Textfocals, A User-driven Text Improvement Tool Using LLM

3 main points
✔️ Proposal for Textfocals, which leverages large-scale language models to allow users to revise their own writing as they review it
✔️ Customizable prompt menus and card-style sidebars designed to make large-scale language model output accessible to users
✔️ User testing demonstrates that large-scale language model views can help users improve their writing while maintaining their uniqueness

Towards Full Authorship with AI: Supporting Revision with AI-Generated Views
written by Jiho Kim, Ray C. Flanagan, Noelle E. Haviland, ZeAi Sun, Souad N. Yakubu, Edom A. Maru, Kenneth C. Arnold
(Submitted on 2 Mar 2024)
Comments: 15 pages, 2 figures; Accepted to 5th Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN) at ACM IUI 2024
Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

code： 

The images used in this article are from the paper, the introductory slides, or were created based on them.

Summary

Large-scale language models can produce text comparable to that written by professional writers. Two main dialogue technologies are currently in use: one is interactive (e.g., OpenAI's ChatGPT and Google's Gemini) and the other is predictive text completion (e.g., GitHub Copilot). These technologies perform well on many tasks. On the other hand, however, in writing, they leave some or all of the creative decision-making to the system.

For example, OpenAI's ChatGPT allows users to specify their desired output based on a purpose such as "I want you to write a short note to introduce yourself," but such a system may undermine the creativity and originality of the user, affecting their unique ideas and expressions It is said that there is a possibility that the user's creativity and originality may be affected.

This paper proposes Textfocals, a human-centered system for leveraging large-scale language models to support writing. textfocals is a UI prototype designed on the principle that users generate content, and large-scale language models modify user writing The large scale language model restricts the user from modifying the writing. Instead, through summaries, questions, and writing advice generated by the large-scale language model (hereafter referred to as "large-scale language model views"), users can be presented with suggestions for improvement and refinement, andencouraged to consider revisions.

With usability in mind, Textfocals offers two UI features: one streamlines the revision process by adapting the view to the point where the writer is currently revising. The otherprovides the ability to select or modify pre-designed prompts, since it is difficult forauthorsto create appropriate prompts, and provides easy access to large language model features that meet current revision needs.

Previous research has investigated UIs that display summaries generated by large-scale language models in an interactive sidebar within a text editor, and UIs that support template-based prompt engineering to improve user-defined feedback. However, Textfocalsis thefirstprototype tostudy UI features to encourage person-initiated revision.

Formative evaluations conducted with four participants qualitatively assessed the effectiveness of the large-scale language model summary, question, and advice views in aiding writing revision. The results show that the large-scale language model views help users refine the ideas under consideration and make writing more efficient.

This study is an important step in showing how large-scale language models can support creativity and originality in writing.

Design and Implementation

In writing, it is very important to review one's own writing and make new discoveries. However, while writing tools with large-scale language models such as OpenAI's ChatGPT and Google's Gemini can generate sentences comparable to professional writers, it can be difficult and burdensome for users to think of appropriate questions to ask (prompt engineering). In addition, when incorporating the generated responses into the text, it is necessary to devise ways to ensure that copyright and uniqueness are secured.

Textfocals provides the following two features to address these issues.

• Menu of customizable pre-programmed prompts:prompts that letlarge languagemodels "observe" rather than generate text
• Card-style sidebar: interface that allows users to interact with responses (views) of a large language model

The figure below shows the overall interaction flow inTextfocals. the Textfocalsprototype is being developed as a task pane add-in for Microsoft Word. it uses React and Microsoft's JavaScript API to check for changes in the cursor position within a document. the cursor position changes within the document. It retrieves the text of a paragraph and sends it to the backend (Python FastAPI server) with a prompt. The backend queries the OpenAI API GPT-3.5 and streams the output to the frontend, which parses it (e.g. Markdown rendering) and displays the generated view.

Also shown below isa UI image of theTextfocalsprototype;Textfocals has a button to select a predefined prompt, which requests the user's observations about the text from a large language model. The prompt buttons provide a brief summary of the functionality and a prompt editor that allows the user to review and edit the content. This approach is intended to aid the writing process by encouraging users to modify prompts.

Examples of useful view categories include

• Summary View
  • Thesis Statement: "Write one sentence that seems to be the argument of the paragraph, and repeat it more briefly."
  • Important Concepts: "List the important concepts in the paragraph, with the three most relevant concepts in Markdown format."
• Question View
  • Questions the Writer Was Attempting to Answer: "List the questions the writer was attempting to answer in this paragraph."
  • Questions a Reader Might Have: "As a reader, you ask the writer questions about definitions and logical connections."
• Advisory View
  • Advice: "Provide bulleted advice on how to improve the paragraph."

It also implements a scrollable sidebar to interact with the generated views. This is an intermediate interface between the user's writing and the view of the large-scale language model. To start the dialogue process, the large language model is populated with the first paragraph and the "Thesis Statement" prompt. Subsequently, when a particular section of text is selected, the paragraph containing it is passed as input to the large-scale language model. The large-scale language model also generates views of the preceding and following paragraphs, allowing the user to browse and interact with the views within the context of nearby paragraphs. In addition, hovering over a card highlights the related paragraph, facilitating navigation through the document.In this way, Textfocals provides users with tools to review and improve their own writing.

Research Results - How Large Language Model Views Help Authors

We are conducting a full-formative evaluation of four university staff and professors to determine how the generative view of alarge-scale languagemodel affects users' writing. Each participant is working on a newsletter, grant application, argumentative essay, or blog post andbrings their own writing draft (approximately 1 page). Participantsare briefed to dictate their thoughts while interacting with Textfocals and the chatbot interface.

Textfocals includes prompts to summarize the Thesis Statement and Important Concepts for each paragraph, which not only helps to reorganize the document, but also helps to identify new ideas. For example,whenone of the participants (P4) requested a list ofImportant Concepts fora short paragraph, the systemidentifiedthe Concept of"tagging creators"and noted it as an idea to be further developed. Thus, we found that the summary view helped the user to delve deeper into a particular area.

The summary view also helps participants compare how the writing is understood by readers and identify differences from the intended message. Participants treat the summary view as written by an external reader and understand how the writing is received by the reader. For example,one of the participants (P4) commented, "This view helps me understand how the reader understands this writing."Thus, it is clear that thesummary view is helpful in modifying the intended message to communicate it clearly.

Textfocals also includes a question view that poses anthropomorphic questions to the participants' writing, which also helps them review their writing from the reader's perspective. For example,one of the participants (P3) stated that he would make revisions if his current writing did not adequately answer the question view.Thus, it can be seen that thequestioning view also helpstheuser to create writing that is appropriate for the reader.

Furthermore, while participants generally found the advice views for both superficial and substantive improvements useful, many requested to see examples of how to specifically implement these improvements. For example, after reading the advice view on "reorganizing paragraphs to flow logically and smoothly,"one participant (P4) said that it got him thinking, "How can I be more logical?" This demand for real-life examples can sometimes trigger unexpected dialogue. Also, for example,one of the participants (P2) asked Textfocals for specific suggestions for the "break up paragraphs to make them more readable" view.

Thus, modifications for clarity and other advice require the inclusion of specific examples in the system response. An interactive interface could be a natural way to provide these concrete examples. For example,one of the participants (P3) tried to obtain specific examples by copying and pasting text from the AI view multiple times into the chatbot interface.

The initial results of this study indicate that the large scale language model generation view canbe useful fordiscovering undeveloped ideas in writing, tailoring writing to the reader, andmakingclear and specificimprovements Textfocals and its UI interface can be an effective tool for authors to maintain their thoughts and ideas while improving their writing improve their writing while maintaining their own thoughts and ideas.

Research Results - Design Issues and Areas for Improvement

Several design challenges and areas for improvement are evident in how the system presents views and allows users to interact.

If the system provides feedback that requires a large amount of space and applies to different parts of the user's document, designers need to consider how to visualize the relationship between the feedback and the document and to which areas the feedback should be displayed. In the study, participants requested views that focused on different parts of the document, but they were not able to fit all feedback on the screen at once.

Although the prototype visualizes each view by highlighting the portion of the document that is relevant to the view,we found that participants often found it difficult to understand because thehighlights are not directly linked to the view, and in other situations the highlights often indicate relevance.

For example, P1 misinterpreted the yellow highlighting when hovering over text as an indication of error and commented, "I don't know if it indicates good or bad." Similarly, P3 misinterpreted the yellow highlighting of cards as the most relevant view and commented, "I think the ones marked in yellow are the ones with the highest relevance." Other participants also found this feature confusing and unintuitive. For this reason, a subtle, colorless outline that does not indicate additional meaning is considered better when indicating which part of the document the sidebar relates to.

We also found that when using the sidebar, some participants felt that the scope, or range, of the text provided as input to the large-scale language model was unclear. For example, P1 commented, "Is the prototype looking at the entire document or just selected portions?" Similarly, P2 comments, "Is the prototype looking at the entire document or just selected paragraphs? This suggests that visual cues that clearly indicate the range of text used by the large-scale language model to generate the view would be useful.

In addition, participants used a variety of predefined prompts for their views, indicating that the variety was useful. Some participants edited the predefined prompts or created new prompts to suit their own needs. For example, one participant edited the "Questions a reader might have" prompt to specify the type of reader, while another asked to reuse the view feature to improve the text in a larger language model. However, creating and managing prompts was difficult for users, and while a fill-in-the-blank approach such as FeedbackBuffet may solve part of this challenge, it also requires assistance for users to create new prompts and recall them later. This is an area where further research is needed.

Summary

This paper proposes Textfocals, a human-centered system for utilizing large-scale language models to support writing.Revision is the process of critically reviewing and evaluating what you have written. This allows us to identify areas for improvement and opportunities for new development, and to make appropriate revisions. By using pre-generated prompts and allowing the large-scale language model to generate output, users can review their writing objectively and make new discoveries. User testing has shown that participants find the output generated in this way useful. They have also been able to identify gaps between the intended message and the reader's understanding and gain insight into how to tailor the text to the reader. These results have been shown to help improve the writing and enrich the content.

It also shows that pre-defined prompts and a prompt editor are effective in encouraging users to generate external perspectives rather than replacing or continuing their own sentences. This UI ingenuity allows usersto use thelarge-scale languagemodelto enhance their thought process,rather than letting thelarge-scale languagemodel replace their own thinking. This has been shown to allow users to revise their writing while maintaining their autonomy as authors.

It has been shown that the "Textfocals" proposed in this paper is a useful tool for users to review their own writing and make new discoveries to enhance the content. It is hoped that further research will further clarify the effectiveness of this tool.

• 
• 
• 

Categories related to this article

• Large Language Models
• Machine Learning
• Deep Learning