Can ChatGPT Increase Human Productivity?

ChatGPT 18/07/2023

3 main points
✔️ A study examining the effects of applying ChatGPT to professional writing tasks
✔️ An experiment employing 444 professionals as subjects found that work time decreased by 0.8 standard deviations and the quality of deliverables increased by 0.4 standard deviations
✔️ ChatGPT was also found to have greater efficacy for less skilled subjects and to increase job satisfaction and self-efficacy

Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
written by Shakked Noy, Whitney Zhang
(Submitted on 6 Mar 2023, last revised 11 Apr 2023)
Comments: Published on SSRN.
Subjects: artificial intelligence, generative AI, labor markets, productivity

The images used in this article are from the paper, the introductory slides, or were created based on them.

introduction

Important questions include: how will generative AI affect worker productivity in existing tasks,
will it replace workers or complement their skills,
will the benefits gained vary depending on the skill level of the worker,
will it affect worker job satisfaction,
.
The authors conducted an experiment with subjects to answer these questions.

experiment

We assembled 444 professional marketers, grant writers, consultants, data analysts, HR professionals, and managers to work on the writing task.

Tasks can be employed for 20~30 minutes, such as press releases, short reports, analysis plans, delicate email writing, etc.

Half of the participants belong to the treatment group and are taught how to use ChatGPT between the first and second tasks before being allowed to use it in the second task. The other half of the participants belong to the control group and are allowed to use the LaTeX editor Overleaf instead of ChatGPT.

The quality of the subject's deliverables is evaluated by experts in terms of quality of writing, quality of content, originality, and overall score. Three evaluators are assigned to each deliverable. In addition, time required, satisfaction, self-efficacy, and attitudes toward automation are also surveyed.

result

In the following, the results of the experiment with subjects are described in nine sections.

1. use of ChatGPT

Eighty-one percent of subjects in the treatment group chose to use ChatGPT for the second task, with a perceived usefulness score of 4.4 out of 5.

2. productivity

As shown in Fig. 1 (a) and (b), compared to the control group, the treatment group took 10 minutes less time for the second task and their performance increased by 0.45 standard deviation.

As shown in Fig. 1 (c) and (d), the distribution as a whole shows a shift in the direction of shorter time required and better performance.

Fig. 2 (a) and (b) show the change in productivity as a function of the subject's skill level.

(a) plots the results of the first task on the horizontal axis and the results of the second task on the vertical axis, showing that the worse the results of the first task were, the better the results improved by using ChatGPT.

(b) plots the time required for the second task on the vertical axis, with the performance of the first task on the horizontal axis, and shows that the time required for the second task is uniformly reduced by using ChatGPT, regardless of the performance of the first task.

When both the treatment and control groups were instructed to fix the duration of the task at 15 minutes, the ChatGPT improved performance by 0.39 standard deviations.

3. elimination of productivity bias

As seen in Fig. 2 (a), subjects with lower performance in the first task benefited more from ChatGPT in the second task. This means that the use of ChatGPT eliminates to some extent the bias by performance.

4. complementarity between human and machine

Of the treatment group, 68% of subjects reported that they submitted the initial output of the ChatGPT without any editing. There was no difference in performance between the raw ChatGPT output and the modified responses. These results mean that there was no evidence that "human modification of ChatGPT output improves quality.

5. changes in time spent on tasks

Using ChatGPT changed the percentage of time spent on writing tasks; as shown in Fig. 3 (a), ChatGPT reduced the amount of time spent brainstorming and drafting.

6. differences in writing skills

We found that the willingness to pay for the use of the ChatGPT and the degree of grade improvement obtained remained largely unchanged across subjects, regardless of their writing skills (self- and peer-rated).

7. job satisfaction and self-efficacy

We found that using the ChatGPT increased job satisfaction by 0.40 standard deviations and self-efficacy by 0.20 standard deviations.

8. approach to automation

Most of the subjects had never used ChatGPT before participating in the experiment, but were asked how their attitudes toward ChatGPT had changed as a result of their participation in the experiment. The three questions were (a) how concerned are you about AI replacing workers in your occupation, (b) how optimistic are you that AI will increase productivity in your occupation, and (c) how optimistic or pessimistic are you about the future development of AI? The results of the questions showed that, compared to the control group, the treatment group increased (a) their worry by 0.26 standard deviations, (b) their expectation of increased productivity by 0.39 standard deviations, and (c) their degree of optimism about the future by 0.20 standard deviations.

9. follow-up on ChatGPT use

When surveyed again two weeks after participating in the experiment, 33% of the subjects in the treatment group indicated that they used the software in their real work. Specifically, they used it for a variety of tasks, including writing letters of recommendation, responding to customer service calls, brainstorming, using search engines, and drafting emails. Conversely, many of those who said they did not use ChatGPT in their actual work said it was because it lacked context-specific knowledge. The problem is that it is not specific to the customer, deals with real-time information, and is not capable of dealing with company-specific information.

summary

In the paper presented here, we investigated how much the use of ChatGPT improves productivity in writing jobs and how it changes the work experience of workers.

For lower skill levels, it was found to contribute to improving the quality of deliverables, while for higher skill levels, it had the effect of significantly reducing the time required to complete the task.

This paper gives us an idea of the impact that generative AI technologies, such as ChatGPT, may have on the labor market in the future.

Categories related to this article

Abe