Abstract Summary
•
This study evaluates media bias in LLM-generated content and the ability of LLMs to detect subtle ideological bias using datasets PoliGen and EconoLex.
•
Results show a consistent preference for Democratic over Republican positions in political discourse across all seven LLMs evaluated, while biases in economic topics vary among Western models and Chinese models lean towards socialism.
Abstract
While detecting and avoiding bias in LLM-generated text is becoming increasingly important, media bias often remains subtle and subjective, making it particularly difficult to identify and mitigate. In this study, we assess media bias in LLM-generated content and LLMs' ability to detect subtle ideological bias. We conduct this evaluation using two datasets, PoliGen and EconoLex, covering political and economic discourse, respectively. We evaluate seven widely used LLMs by prompting them to generate articles and analyze their ideological preferences via Socratic probing. By using our self-contained Socratic approach, the study aims to directly measure the models' biases rather than relying on external interpretations, thereby minimizing subjective judgments about media bias. Our results reveal a consistent preference of Democratic over Republican positions across all models. Conversely, in economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.