The Impact of ChatGPT on Worker Productivity

Personally, I believe that the greatest challenges the generative AI revolution will pose in relation to people management in organizations will not so much be about this technology’s ability to automate HR tasks and processes as about the response companies give to the increase in their workers’ productivity because of the adoption of solutions based on these technologies.

In relation to this issue, I find particularly interesting an experiment conducted earlier this year by two MIT doctoral students, Shakked Noy and Whitney Zhang, whose results we can take as an “early warning” of some possible, sometimes counterintuitive, effects of the emergence of generative AI technologies in the world of work.

The authors’ objective was to analyze how ChatGPT impacts productivity, skills, and job satisfaction of workers. To do this, they selected a sample of 444 participants, all of them university-educated, with experience in areas such as marketing, consulting, data analysis, human resources, and business management. Each person had to perform two writing tasks relevant to their occupation, such as drafting press releases or delicate emails. These 20- to 30-minute tasks resembled their real work. Additionally, they were incentivized with financial bonuses to produce high-quality work.

The sample was randomly divided into two groups: a treatment group, which was given access to ChatGPT to perform the second task, and a control group that did not have access to this tool.

(Important: When assessing the results of this experiment, we must bear in mind that the working paper was published on March 10, 2023, so the version of ChatGPT used by the members of the treatment group to perform the assigned tasks must have been at most based on the GPT-3.5 model, significantly less powerful than the latest version available at the time of writing these lines, based on the GPT-4 model, which hit the market on March 14 of this year.)

To measure the research results, various aspects were analyzed, such as productivity (time spent and quality of work), complementarity between humans and machines, changes in the structure of writing tasks, the demand for skills and competences, job satisfaction, and self-efficacy, as well as beliefs about automation. The works were evaluated by experienced professionals from the same areas as the participants, who rated the overall quality, the quality of the writing, the quality of the content, and its originality.

Some results were surprising and could have significant implications for how we understand the role of artificial intelligence in the workplace. This is, from my point of view, the most notable:

1- ChatGPT significantly improves productivity in professional writing tasks, increasing both speed and quality of work.

The experiment demonstrated ChatGPT’s potential to increase productivity in professional writing tasks. Participants using this tool were able to complete their tasks 23% faster, and the quality of their work, assessed by independent experts, increased by 20%. This was reflected in a 0.45 standard deviation increase in the ratings given by evaluators to the group using the tool, with improvements observed in overall quality and specific aspects such as writing, content, and originality.

2- Workers tend to use ChatGPT as a tool to substitute human effort rather than a complement to amplify their capabilities.

The study revealed another interesting result: Workers in the study chose to use ChatGPT primarily to automate parts of their work and reduce their effort rather than as a complement to enhance their skills or produce higher quality work. In this sense, it is noteworthy that 68% of the treatment group did not edit the initial output generated by ChatGPT before submitting it, and on average, participants remained active in the task for only three minutes after pasting large amounts of text (presumably generated by the tool).

3- Using ChatGPT reduces performance inequality among participants (although it does not particularly benefit those with lower writing skills)

The use of ChatGPT also reduced performance inequality among participants, so that those who scored lower in the first round benefited more from access to ChatGPT. This phenomenon may be related to two factors: 1) the quality of the texts produced by ChatGPT is higher than the quality of the texts written by many human professionals in their work; 2) high-performing professionals use ChatGPT to reduce their effort rather than to produce higher quality work.

Regarding this issue, an unexpected finding of the study was that although, a priori, one might assume that ChatGPT would be an especially beneficial tool for those who have greater difficulties expressing their ideas in writing, access to the tool did not disproportionately favor those with lower writing skills. One hypothesis is that ChatGPT contributes to improving users’ arguments, or even the originality of their approaches, more than their written expression. Another hypothesis is that precise writing is essential for formulating effective prompts to ChatGPT, which makes those with poorer written expression unable to capture the benefit that, theoretically, the tool could provide them.

4- Access to ChatGPT increases job satisfaction and self-efficacy of participants (to the point that they are willing to pay for access to this tool)

Noy and Zhang’s experiment revealed that using ChatGPT also increases participants’ job satisfaction. As for the causes of this improvement, although many participants reported in an open-ended question at the end of the questionnaire that they had enjoyed discovering and working with this tool, it is possible that it has much to do with ChatGPT’s ability to automate tedious or annoying components of tasks and reduce the time users need to complete them. Another aspect where ChatGPT had a positive, albeit less obvious, impact was on workers’ self-efficacy, i.e., their perception of their ability to complete tasks and achieve goals. A reflection of these positive assessments by workers is their notable willingness to pay for access to ChatGPT in their jobs. Specifically, participants said they were willing to allocate, on average, around 0.5% of their monthly salary to subscribe to ChatGPT.

5- Experience with ChatGPT can change workers’ perceptions of artificial intelligence

Finally, the study also sheds light on how direct experience with AI can reshape workers’ perceptions of these technologies. Although after trying ChatGPT, workers showed greater concern about the possibility of their roles being replaced by AI, interestingly, their optimism also increased about how AI could in the short term enhance their productivity in their current roles, and their overall optimism about future advances in the field of AI.

DISCUSSION

According to the results of this experiment, workers use the productivity increases provided by generative AI solutions like ChatGPT to save time and reduce their effort rather than to increase the quality with which they perform their tasks (although this also increases) or the volume of output they produce. This positively impacts their job satisfaction, self-efficacy, and the optimism with which they contemplate present and future advances in the field of artificial intelligence…

Let’s see what happens when their bosses find out… ;-)

Jokes aside, it’s important to note that these are not isolated data. The results of Noy and Zhang’s experiment with ChatGPT coincide with those of studies conducted among users of other generative AI solutions, like Copilot, an automatic code development tool developed by GitHub. As we read in the 2023 AI Index Report from Stanford University, the aspects most valued by users of this tool in September 2022 were the increased speed with which it allowed them to do repetitive tasks, the faster completion of their assignments, the lesser mental effort they had to make to carry out these repetitive tasks, or the lesser time they had to spend looking for references.

These data also agree with those of a recent Microsoft study, according to which, although 49% of people are worried about AI replacing their jobs, 70% would be willing to delegate as many tasks as possible to AI to reduce their workload. On the other hand, when participants were asked to imagine their ideal job in 2030, what they primarily valued were things like producing high-quality work in half the time, learning new skills twice as fast, better understanding how to use their time and energy effectively, not having to assimilate unnecessary or irrelevant information, or cutting in half the time spent in meetings and responding to emails and chat messages. In other words, what people value most are those improvements that save them time and mental effort.

Why might this be?

The Microsoft report I referenced earlier provides a clue. According to this report, in Spain, 60% of people say they have serious problems finding enough time and energy to get their work done, and these people are also 4.9 times more likely to say they have difficulties with innovation and strategic thinking.

This may all have to do with the “digital debt” recently discussed by Satya Nadella. 

“We’re all carrying digital debt: the inflow of data, emails, meetings, and notifications has outpaced humans’ ability to process it all. And the pace of work is only intensifying. Everything feels important, so we spend our workdays trying to get out of the red. (…) There are only so many minutes in the day—and every minute we spend managing this digital debt is a minute not spent on the creative work that leads to innovation. In a world where creativity is the new productivity, digital debt is more than an inconvenience—it’s impacting business.”

This might also have to do with the large amount of absurd and zero-added-value work generated in many organizations, sometimes with the sole purpose of justifying equally absurd hierarchical structures. We mentioned earlier the large number of people in Spain who have difficulties with innovation due to the problems they have in finding enough time and energy to get their work done. We could now tell those people that, according to the same report, 60% of Spanish business leaders are concerned about the lack of innovation in their organizations. Let’s see what they think.

Personally, I get the feeling that the arrival of the generative artificial intelligence revolution may cause a serious aggravation of the “productivity paranoia” that we have already discussed on previous occasions. The difference is that this time the “productivity theater” many managers affected by this disease provoked among their collaborators when they made them return to the office after the pandemic of COVID19 or subjected them to online monitoring systems of their work will be even harder to detect.

In addition, we must bear in mind that, from what we are seeing, if we don’t do anything differently these tools are not going to improve the performance of people who already do a good job but, above all, the performance of less productive people. We might think that just with this, the overall performance of the organization will improve, but there’s a new problem. The reality is that the work of lower-performing employees will improve, yes, but probably only to match a standard that everyone will now be able to reach and, therefore, it will not differentiate us from our competitors.

CONCLUSION

Tools like ChatGPT can be powerful aids in enhancing productivity and job satisfaction, but they also pose new challenges and questions. How can we ensure that these technologies are used to augment, not replace, human effort? How can these tools be used to drive creativity and innovation, rather than merely automating tasks? How can leaders and human resources professionals manage the impact of these technologies on the distribution of people’s performance and the perception of work’s value?

Furthermore, we must not overlook the importance of addressing issues like this “digital debt” that threatens to overwhelm us and that on some occasions is a consequence of that “productivity paranoia” that too many bosses still suffer from. At the same time, organizations need to tackle outdated and low-value-added work structures and practices that drain the energy and time of their people. Instead of adding more pressure on workers to be more productive, we need to rethink the meaning we give to the term “productivity” in our organizations and create environments where people can do meaningful and creative work.

In conclusion, generative artificial intelligence has the potential to be a great ally in this process, but it could also exacerbate problems if not handled consciously and strategically. Perhaps we should start by thinking about what we want work to look like in the era of generative AI in our organizations, and what we need to change to make that possible.

REFERENCES

Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Available at SSRN 4375283.

Microsoft. (2023). Will AI Fix Work? 2023 World Trend Index.

Stanford Institute for Human-Centered Artificial Intelligence. (2023). Artificial Intelligence Index.