Can LLM Recreate A Persona Based On The Big Five!
3 main points
✔️ Investigated whether LLMs could replicate their assigned personality trait personas based on the Big Five
✔️ Conducted Big Five personality test and story writing task and evaluated by LLMs and humans
✔️ In both tasks, traits consistent with their assigned personality trait personas Found.
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
written by Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Jad Kabbara, Deb Roy
(Submitted on 4 Mar 2023)
Comments: IC2S2 2023
Subjects: Computation and Language (cs.CL)
code：
The images used in this article are from the paper, the introductory slides, or were created based on them.
Introduction
In recent years, Large Language Models (LLMs), such as ChatGPT, have focused on the ability to have human-like conversations, and attention has turned to building personalized AI agents to support humans.
Several studies, especially in academic fields such as the social sciences, have suggested that generative agents may be able to simulate human personality traits.
On the other hand, despite these advances, little research has been done to assess how accurately and consistently personalized LLMs can reproduce specific personality traits.
Against this background, this paper describes a paper that investigated whether LLMs can reproduce personality traits by having LLMs simulate personas based on the Big Five and extracting psycholinguistic features, human ratings, and personality predictions from the generated content.
What is the Big Five?
The Big Five (Big Five ), which this paper deals with, was proposed by the American psychologist Lewis Goldberg, who called for a " Big Five " of human individuality.
- Openness
- Conscientiousness
- Extraversion
- Agreeableness
- Neuroticism
The theory is based on the following five factors.
In addition, this paper uses the Big Five Inventory (BFI), a scale designed to measure Big Five personality traits, after having an LLM agent simulate a persona based on one of the above personality traits.
Experiment Summary
The workflow of the experiments in this paper is illustrated in the following figure.
In this experiment, the following procedure is followed as shown in the figure.
- To begin, run prompts to generate LLM personas with distinct personality traits
- Then have the generated LLM personas perform the story writing task
- Use the Linguistic Inquiry and Word Count (LIWC) framework to investigate whether the stories described by LLM personas contain linguistic patterns that indicate assigned personality traits
- Evaluate stories described by LLM personas, both human and LLM
- Have both humans and LLMs perform the task of predicting personality traits of the writer's LLM persona from the story
We will explain one by one.
LLM Persona Simulation
In this experiment, we used two LLM models, GPT-3.5 and GPT-4, to simulate 10 LLM personas for each of the Big Five personality traits, generating a total of 320 personas.
We then conducted an evaluation using the aforementioned BFI to confirm that the generated LLM personas adequately reproduced the Big Five.
Story Writing
The 320 LLM personas generated were then asked to "Please share a personal story in 800 words. Do not explicitly mention your personality traits in the story. Do not explicitly mention your personality traits in the story. Do not explicitly mention your personality traits in the story. The participants were prompted to write a text-based story to be used in the analysis.
LIWC Analysis
Next, psycholinguistic features were extracted from the stories described by the LLM personas using the LIWC (Linguistic Inquiry and Word Count) framework, a method for categorizing attributes by abstracting and categorizing vocabulary from text.
The purpose of this analysis is to identify language patterns that correspond to the character traits of the traits by examining the correlation between the traits derived from the stories and the personality traits assigned to the LLMs.
Story Evaluation
The stories described by the LLM personas were then evaluated by both humans and LLMs using the following criteria
- Readability: Is the story easy to read, well structured, and flows naturally?
- Personalness: Is the story unique and does it clearly express the writer's thoughts and feelings?
- Redundancy: concise story, no unnecessary content
- Cohesiveness: Is the story well written?
- Likeability: Is it interesting to read?
- Believability: Is the story compelling and realistic for the actual situation?
Personality Prediction
Finally, we endorsed each human and LLM to predict the personality traits of the writer's LLM persona from a given story on a scale of 1 to 5.
The purpose of this experiment is to evaluate whether the stories described by LLM personas can effectively demonstrate personality traits to a level that is identifiable by both humans and LLMs.
Experimental results
In this paper, we conducted an experiment using 320 LLM personas generated by two LLM models, GPT-3.5 and GPT4, to confirm the following two research questions.
- Do LLM personas reflect assigned personality traits?
- Can we obtain the linguistic patterns of each personality trait from the stories described by the LLM personas?
- Are the stories described by the LLM personas adequately written?
- Can we predict personality traits of LLM personas from stories?
Each of these will be explained.
Do LLM personas reflect assigned personality traits?
To confirm this research question, this experiment calculated personality scores for 320 LLM personas based on their responses to the BFI and analyzed the distribution of these scores as a function of the assigned personality traits by applying a t-test.
The results are shown below.
The results ofthis experiment showed that LLM personasreflect the assigned personas, with statistically significant differences in all personality traits.
Do LLM personas reflect assigned personality traits?
To confirm this research question, this experiment extracted psycholinguistic traits from stories generated by LLM personas using LIWC and calculated Point Biserial Correlations (PBC) between these traits and assigned personality traits.
The point biserial correlation coefficient is a coefficient suitable for analyzing the relationship between a binary variable and a continuous variable, and is used here to look at the correlation between assigned personality traits (= binary variables) and LIWC characteristics (= continuous variables).
The following table summarizes LIWC characteristics that have statistically significant correlations with personality traits.
The results of this experiment showed that when LLMs were given the persona of Neuroticism, they were more likely to use negative vocabulary such as Anxiety and Negative Tone, and that the personality traits assigned to LLM personas had a significant impact on their linguistic style The results show that the assigned personality traits have a significant impact on the linguistic style of the LLM persona.
In addition, and importantly, these correlations reflect patterns observed in stories described by humans, confirming the consistency of word usage between human and LLM personas. (The GPT-4 results were more consistent with humans than the GPT-3.5.)
Are the stories described by the LLM personas adequately written?
To confirm this research question, this experiment evaluated stories generated by LLM personas, both human and LLM.
The evaluation results are shown in the table below.
Notably, the stories generated by the GPT-4 personasreceived a high rating of 4.0 or higher in the Readability, Cohesiveness, and Believability categories forboth humans and LLMs. The reason for this is that the report received a high rating of 4.0 or higher in the
The results confirm that the stories generated by the LLM personas are not only linguistically fluent and structurally coherent, but also compelling.
Can we predict personality traits of LLM personas from stories?
To confirm this research question, this experiment treated the personality traits of each persona as a binary classification problem and calculated the accuracy with which both humans and LLMs predict personality traits.
The results of the experiment are shown in the graph below.
Experimental results showed that the accuracy of humans predicting personality traits from stories described by GPT-4 personas was as low as 68% for Extraversion and 51% for Agreeableness, confirming that the text-based personality prediction task is difficult for humans The results of the study are as follows.
On the other hand, GPT-4 achieved 97% accuracy in Extraversion,68% in Agreeableness, and 69% in Conscientiousness, indicating that it can predict personality traits with very high accuracy. The results are shown in the following table.
Summary
How was it? In this article, we described a paper that investigated whether LLMs can reproduce personality traits by having LLMs simulate personas based on the Big Five and extracting psycholinguistic traits, human ratings, and personality predictions from the generated content.
Experiments conducted in this paper have shown that LLMs can adequately simulate a given persona, as well as that word usage reflects personality traits and that personality traits can be predicted by LLMs, suggesting the great potential of LLMs.
On the other hand, there are many other points that need to be examined, such as the fact that this experiment did not simulate more natural situations such as interactions or collaboration between LLM personas, or that it focused only on English and did not investigate other languages.
As this research field progresses, it may soon be possible to create AI agents that replicate human personality and behave like humans.
The details of the workflow and experimental results presented here can be found in this paper for those interested.
Categories related to this article
![[JMMLU] Prompt Polit](https://aisholar.s3.ap-northeast-1.amazonaws.com/media/July2024/jmmlu-520x300.png)
