Potential And Limitations Of Media Bias Detection Using ChatGPT

3 main points
✔️ Comprehensive study of methods for detecting media bias and investigation of their limitations
✔️ Validation and comparison of ChatGPT's bias detection capabilities:experiments focused on its ability to identify media bias and show good results on specific tasks (e.g., hate speech detection)
✔️& nbsp; Confirms that fine-tuned models are inferior in areas that require deeper background understanding, such as identifying gender and race bias

ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models
written by Zehao Wen, Rabih Younes
(Submitted on 29 Mar 2024)
Comments: Published on arxiv, published on Applied and Computational Engineering
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

code：

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

With the advent of the digital age, information is instantly disseminated around the world, and it is not uncommon for this to include a variety of biases. Media bias, the selective presentation of a particular point of view, can shape people's perceptions of an incident or issue and have a significant impact on public opinion. Indeed, many people feel that major media outlets have a bias, and there is an urgent need to investigate this issue.

This paper investigates how AI techniques can be used to detect and understand media bias. While there is a wide range of methods for identifying media bias, from manual content analysis by human raters to computational methods using machine learning and natural language processing techniques, these methods have limitations. For example, approaches that focus on specific political biases or fake news, which have been well studied, face challenges in capturing the nuances of language that contribute to bias and in scalability.

Among these, the use of ChatGPT, a large-scale language model developed by OpenAI, is of particular interest, as ChatGPT, based on the GPT-3.5 engine, has shown remarkable capabilities in a variety of natural language processing tasks, including translation, sentiment analysis, inference, and summarization. This paper examines ChatGPT's ability to identify media bias and explores ways to improve its accuracy. The performance of ChatGPT is also compared to fine-tuned language models such as BART.

For future research on media bias, ChatGPT provides valuable insight into the ability to identify multiple biases, including racial bias, gender bias, and cognitive bias.

Experimental setup

This paper presents an experiment aimed at identifying and evaluating media bias. The data used are selected from the Media Bias Identification Benchmark (MBIB), a dataset compiled by Wessel et al. MBIB is a comprehensive dataset specifically designed to evaluate different media bias detection techniques, consisting of 115 It is a comprehensive dataset consisting of 115 datasets specifically designed to evaluate different media bias detection techniques. From this dataset, nine tasks and 22 related datasets were selected to evaluate media bias detection techniques.

These data are also preprocessed appropriately for the different tasks, and labels are converted to binary format. This facilitates the integration of different data sets and simplifies the format of the task. In particular, the continuous label data sets have been binarized using the author's recommended thresholds.

Here we focus specifically on six of the nine tasks included in the MBIB and extensively evaluate ChatGPT's ability to detect media bias in each of them.

The datasets associated with the selected tasks are proportionally split into training and testing subsets depending on their size. For most bias identification tasks, a training-testing split of 80-20 is employed for the dataset, but due to the large amount of data (e.g., 2 million examples) included in the cognitive bias and hate speech tasks, 10% of these datasets are randomly selected, followed by an 80-20 split 80-20 splits. The sizes employed in each task areshown inthe tablebelow.

Threehighly regardedmodels (ConvBERT, BART, and GPT-2) were also selected for comparison to evaluate ChatGPT's performance in detecting media bias. They have shown excellent performance on a variety of natural language processing (NLP) tasks.These models have been finetuned for each bias identification task using the training dataset described aboveThe performance of these models has been evaluated on the test dataset and compared to ChatGPT results.

As for ChatGPT,we use the GPT-3.5-turbo version of ChatGPT, which offers an excellent balance between efficiency and cost. To improve the reproducibility of the results, the model is set to be deterministic in its behavior, i.e., the temperature of the model is set to zero, so that it always produces the same response to the same prompts. The prompt design methodology takes its ideas from prior research and has ChatGPT generate three concise prompts for each of the six bias identification tasks to maximize its ability to do so.

For example, a query on identifying racial bias might look like this"Please provide three brief prompts that will maximize your ability to identify if the given text contains racial bias."

These prompts are tested with a limited number of examples (60) selected from the original data set and randomly selected from different data sets. Each task prompt contains an equal number of positively labeled examples indicating the presence of bias and negatively labeled examples indicating its absence. This allows us to select the most effective prompts. The results are shown in the table below.

In addition, special instructions are attached to the task prompts so that the model responds in a manner that can be processed automatically. With this instruction, the model provides output in JSON format, including a "bias" column to indicate whether bias is present in the text as 1 or 0. This process improves the accuracy of bias identification and contributes to standardization of the method.

Experimental results

Here weprovide a comprehensive evaluation of ChatGPT's performance on six media bias identification tasks by comparing the performance of ChatGPT to other fine-tuned models. This evaluation is essential for understanding the effects of bias recognition and mitigation in different contexts and for contributing to the development of more balanced AI systems. Analyzing model performance using specific metrics provides insight into their strengths, limitations, and potential for improvement.

As recommended by MBIB, two metrics are used: one is the Micro Average F1-score. A single F1-score is calculated based on the predictions made by the model on the entire test set. This method ignores variation in which data set each example came from. This metric is useful for easily capturing the overall performance of the model. The other is the Macro Average F1-score. An F1-score is computed for each individual dataset in the test set and the results are averaged to obtain a Macro Average score. This approach ensures that all data sets contribute equally to the final score, regardless of their size.

The performance of the ChatGPT and the fine-tuned models is shown in the table below.

Overall, fine-tuned models such as BART, ConvBERT, and GPT-2 generally perform better at identifying bias. This is likely because these models are trained to adapt to the patterns of bias recognized by human labelers. On the other hand, ChatGPT's zero-shot approach relies only on a wide range of data patterns and has been found to be less accurate at bias identification.

In particular, with respect to gender and race bias, ChatGPT is markedly inferior to the fine-tuned model, showing false positives in many cases. For example, a statement may be incorrectly interpreted by ChatGPT as having a gender bias, while human raters or other models may view it as neutral. An example is the statement "I can't stand a Yankee voice commentating on football. CRINGE," which ChatGPT describes as "reinforcing gender roles by assuming that football commentary is a male-dominated field" and labels the bias labels it as being. This hypersensitive response may be the result of a stereotype or bias tied to a particular word or phrase during the learning process. In this case, "Yankee voice" is associated with the assumption that football commentary is primarily done by men.

Furthermore, ChatGPT is significantly inferior to models such as BART and ConvBERT in detecting cognitive biases and fake news. This is because these types of biases are deeply dependent on context and fine-grained language nuances and are difficult to address with simple zero-shot learning. In the case of fake news, its ambiguous and often deceptive nature makes it difficult to distinguish from the truth based on linguistic cues alone.

However, ChatGPT also performs relatively well in detecting hate speech. Hate speech is easy to identify due to its blatant and aggressive language patterns, which is why it seems to perform well in the zero-shot model.

On the task of detecting context bias at the text level, the results are comparable to those of fine-tuned methods. This may be because ChatGPT's extensive architecture is particularly well suited to capture the subtle meanings of human communication. Through comprehensive training, this large-scale model has acquired a multifaceted understanding of language. As a result, ChatGPT has the ability to gain insight into and interpret the impact of context on language.

However, the performance of all models in this study is heavily influenced by the quality of the available data sets. For example, the model struggles with data sets that contain fewer data examples, while it performs better with data sets that contain more examples. Due to the limited amount of data, it is possible that these macro-average scores do not fully reflect the true capabilities of the model.

As a result, ChatGPT shows some degree of proficiency, but it is not clear whether it can serve as a definitive detector of media bias in its current form. However, testing with a small number of prompts may improve its performance.This approach needs further validation for consistency in the ChatGPT data set.

Summary

This paper compares ChatGPT's ability to detect media bias with other fine-tuned models (BART, ConvBERT, and GPT-2), although ChatGPT shows notable success in identifying hate speech and text-level context bias, it was found to perform poorly on tasks requiring deeper contextual understanding, such as gender, race, and cognitive biases.

While demonstrating the progress that large-scale language models are achieving in language understanding, it highlights that there are still challenges in more subtle understanding of context and bias. It is noted that the subjectivity of bias and the nature of the data on which ChatGPT was trained may have influenced the performance differences between these models.

Future papers are expected to further improve the capabilities of these models using new approaches, including few-shot prompts and human evaluation. This paper provides insight into the future development of AI and its societal implications.

• 
• 
• 

Categories related to this article

• Large Language Models
• Machine Learning
• Deep Learning
• Natural Language Processing
• ChatGPT
• GPT