An Approach That Draws Out LLM's Superior Problem-solving Abilities By Assigning Multiple Personas Has Emerged!
3 main points
✔️Proposes Solo Performance Prompting (SPP), an approach that draws on the diverse expertise of LLMs by repeatedly providing feedback through multiple personas
✔️ Finds that assigning multiple detailed personas elicits superior problem solving skills in LLMs
✔️ Demonstrated the effectiveness of SPP by comparing it to Chain-of-Thought Prompting
Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
written by Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, Heng Ji
(Submitted on 11 Jul 2023 (v1), last revised 14 Jul 2023 (this version, v2))
Comments: work in progress
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
code：
The images used in this article are from the paper, the introductory slides, or were created based on them.
Introduction.
While Large Language Models (LLMs) have shown remarkable performance in generating agents to solve common tasks, problems such as hallucination (LLMs output plausible lies that are not true) and factualerror (a phenomenon in which LLM outputs plausible lies that are not true), and factual error (a phenomenon in which LLM misrepresents facts).
Because of these problems, unlike humans who can take advantage of the ability to integrate different cognitive processes and information, called cognitive synergy, today's LLMs are more like "do-it-yourselfers," so to speak, with a vast amount of aggregated knowledge.
In addition, while recent work such as Chain-of-Thought prompting and self-refinement have successfully enhanced LLM reasoning through intermediate step generation and iterative revision, hallucination in internal knowledge acquisition and factual errors in internal knowledge acquisition remain a major challenge.
To solve these problems, the author of this paper has found it effective to create a cognitive synergist, an intelligent agent that links multiple personas and combines their unique abilities and expertise to improve problem solving and overall effectiveness in complex tasks. We considered it effective to create a cognitive synergist, an intelligent agent that combines the unique abilities and expertise of each persona.
This paper proposes Solo Performance Prompting(SPP), an approach to transform a single LLM into a cognitive synergist by repeating feedback with multiple personas, and describes a paper that found that assigning multiple detailed personas to an LLM elicits better problem-solving ability through comparative experiments. The paper will discuss a comparison experiment that found that assigning multiple detailed personas to LLMs elicits better problem-solving abilities.
Solo Performance Prompting
The figure below illustrates the difference between Chain-of-Thought Prompting, a conventional prompting method, and Solo Performance Prompting (SPP), our new method.
The biggest difference is thatSPPtransforms a single agent generated by LLMinto a cognitive synergist that dynamically sets personas and modifies them with feedback toeffectively solve various tasks.
The structure of dynamically identifying and simulating different personas based on task input improves problem solving and overall performance in complex tasks and effectively elicits internal LLM knowledge.
The figure below provides a concrete example of how SPP works in a challenging task that requires integrating information from a variety of domains, such as the game "The Legend of Zelda," the movie "Harry Potter," and the album by singer Jay Chou.
While existing standard prompts do not provide adequate output due to missing important information or factual errors, SPP provides participants with special personas, such as Harry Potter fans or Jay Chou fans, with suggestions on how to approach the task based on their own expertise. SPP is able to obtain appropriate information by providing suggestions on how to approach the task based on the expertise of each participant.
The leader persona, the AI assistant (user), then proposes an initial solution, seeks feedback from other participants, iteratively modifies the solution based on that feedback, and outputs the final solution.
These sequences consist of three elements: Persona Identification, Beginning Remarks, and Multi-Persona Iterative Collaboration.
Persona Identification
Given input, SPP first generates a list of participants with various personas that may be valid for task resolution.
For example, as in the specific example described above, "What is the last song on singer Jay Chou's second album?" to search for the knowledge "What is the last song on singer Jay Chou's second album?", we generate the persona of a Jay Chou fan.
Also, personas here are not defined manually, but automatically by the language model.
Beginning Remarks
After the various personas are generated by the language model, the user, as the AI assistant, will propose the first solution, and then each persona will make statements about how to approach the task from its own perspective.
In the specific example mentioned above, a fan of Joy Chou made a statement pointing out that the last song on Jay Chou's second album is "An Jing".
Repeated revisions through such statements effectively improve the solution initially proposed by the AI assistant.
Multi-Persona Iterative Collaboration
After the AI assistant proposes an initial solution, other participants provide feedback, and the AI assistant repeats the process of modifying the solution over and over until all participants are satisfied with the final solution.
When this process is complete, the words "Finish collaboration! " will be output and a final solution will be generated.
This structure allows SPP to effectively elicit domain knowledge and reduce hallucination based solely on a single large-scale language model.
Experiments (TriviaCreative Writing Task)
To demonstrate the effectiveness of the Solo Performance Prompting (SPP) proposed in this paper, we conducted a comparison experiment with Chain-of-Thought Prompting (CoT) using the following tasks. (GPT-4-32K API was used for all experiments)
The Trivia Creative Writing Task, shown in the figure below, would be a task that asks the model to draw a coherent story about a topic while incorporating answers to N trivia questions.
Because larger N includes more questions, the model must elicit more diverse domain knowledge, and this paper experimented with two evaluation settings, N=5 and N=10.
The results of this experiment (N=5) are shown in the figure below.
From the figure, it can be seen that CoT attempts to generate a reasonable plan to solve the task, but the final answer shows problems such as factual errors and hallucination, while SPP is able to answer appropriately.
In addition, it is worth noting that these improvements are more pronounced at N=10 than at N=5, suggesting that SPPs that generate a variety of personas to elicit different expertise are more effective for tasks that require knowledge to be captured from a large number of different domains.
summary
How was it? In this article, we proposed Solo Performance Prompting (SPP), an approach to transform a single LLM into a cognitive synergist by repeatedly providing feedback through multiple personas, and described a paper that found that assigning multiple detailed personas to an LLM elicited better problem-solving ability through comparative experiments. The paper found that assigning multiple detailed personas to LLMs elicited better problem-solving abilities in comparative experiments.
While the comparative experiments conducted in this paper demonstrate the effectiveness of this method compared to existing methods, there are still some problems , such as the fact that responses may be inaccurate even when detailed personas are assigned.
In addition, it is still unclear to what extent assigning personas can increase knowledge in specific areas, and professional diagnosis will be required to quantify the impact of these personas, so future trends will be closely watched.
The details of Solo Performance Prompting and the experiments presented in this paper can be found in this paper for those who are interested.
Categories related to this article
