How AI could already be affecting worker performance

At the beginning of 2023, when I asked attendees—usually “knowledge workers”—at a talk or conference whether they had already used any generative artificial intelligence tool, only a few hands would go up. Today, the landscape is very different. Most people have had some interaction with AI, and a significant number report that their jobs are starting to undergo changes due to the use of these new tools. But what are those changes?

Based on research conducted on this topic to date, it seems that, at least for now, the rise of AI is having more of an augmentation effect than a substitution effect. This means it enhances the performance of workers who integrate it into their tasks, rather than replacing human labor outright. However, this impact is rarely uniform. It tends to disproportionately favor certain groups, either narrowing or widening, depending on the case, the gap between high- and low-performing employees. There is also evidence that, under certain circumstances, the use of AI can have the opposite effect, harming users’ performance and resulting in worse outcomes compared to the level they could achieve without this technology. In summary, it’s a bit of a mess.

Is AI leveling the performance of workers?  

The first studies conducted after the launch of ChatGPT in November 2022 already provided revealing insights on this topic. One example is the experiment conducted by Shakked Noy and Whitney Zhang to examine the effects of the then-recently released ChatGPT 3.5 on writing tasks typical of various professions. 

They found that workers who scored lower without AI not only completed their tasks faster when incorporating it into their work, but they also improved the quality of their writing. On the other hand, individuals who scored high before using ChatGPT also completed their tasks more quickly when using the tool, but the quality of their writing remained the same. This led to a reduction in the quality gap between the work of the two groups.  

The authors attributed this effect to the participants’ tendency to use these tools primarily to save effort rather than to improve the quality of their work.

However, subsequent studies have shown that the performance of more skilled or experienced professionals can also improve with the use of generative AI tools, although the improvements they achieve tend to be smaller than those of their less productive colleagues. As a result, the “equalizing effect” of AI, while moderated, persists.  

In this regard, a set of field experiments conducted by Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz analyzed the impact of GitHub Copilot on the productivity of software developers at Accenture, Microsoft, and a third company. The results showed an average increase of 26% in the number of tasks completed by programmers using this tool. However, the effects varied by level of experience and skill. Specifically, less experienced programmers increased their output by 27% to 39%, while veteran programmers also experienced notable improvements, albeit more modest, ranging from 8% to 13%.  

In the same vein, noteworthy research by Fabrizio Dell’Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani focused on consultants at the Boston Consulting Group (BCG). The authors found that using a generative AI tool—in this case, ChatGPT 4—during a new product launch project improved the performance of less skilled consultants by 43%, while the performance of more skilled consultants rose by an impressive 17%.

Is AI widening the gap between high- and low-performing professionals?

Despite evidence suggesting that AI can help level the performance of professionals who use it, more recent research indicates that, in certain circumstances, the use of generative AI in specific jobs may have the opposite effect, increasing the gap between more and less productive professionals.

For example, a study conducted by Aidan Toner-Rodgers analyzed the impact of artificial intelligence in the field of innovation by examining the introduction of a new material discovery technology to 1,018 scientists in the R&D lab of a large U.S. company. AI-assisted researchers discovered 44% more new materials, resulting in a 39% increase in patent applications and a 17% rise in subsequent product innovations. Moreover, these compounds featured more novel chemical structures and led to more radical inventions.

However, the introduction of this technology had uneven effects depending on the historical productivity of the researchers. While the bottom third of scientists experienced few benefits, the output of the top-performing researchers nearly doubled. Since the AI automated 57% of “idea generation” tasks, researchers shifted their focus to the new task of evaluating the candidate materials proposed by the AI model. Herein lay the difference: the top-performing scientists leveraged their specialized knowledge to prioritize the most promising AI suggestions, while less skilled researchers wasted significant resources pursuing false positives.

The conclusions of Toner-Rodgers align significantly with the findings of a meta-analysis conducted by MIT professors Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone, based on 106 studies published between January 1, 2020, and June 30, 2023. One of the key findings was that systems combining AI and humans performed worse when AI alone outperformed humans. Conversely, when human performance alone exceeded that of AI, systems combining humans and AI achieved better results than humans working without these tools.

Since, in the vast majority of the human-AI systems examined in the meta-analysis, humans made the final decisions after receiving input from an AI solution, the authors attributed these differences to the fact that when humans generally outperform algorithms, they are also better at deciding when to rely more on the AI’s recommendations and when to trust their own judgment. This, in turn, depends both on the capability of the AI solution employed and the skills of its users.

In any case, this is not the only way in which AI can widen the gap between high- and low-performing workers. A study conducted by Christoph Riedl and Eric Bogert on the behavior of users on an online chess platform reveals another mechanism through which the use of AI tools can cause this effect. The researchers found that higher-skilled players learned faster than less-skilled players, which was linked to the former seeking AI feedback more frequently and being more likely to do so after a loss. In contrast, less-skilled players tended to request feedback less often and were more likely to do so after their successes rather than their failures.

On the other hand, many AI tools are designed to adapt to the specific needs of each user. This feature can boost the productivity of those using such solutions but may also amplify the differences between high- and low-performing workers. For example, an AI model that learns from the individual strengths and weaknesses of each worker is likely to be much more effective in the hands of a highly skilled professional than in those of someone with less experience. This could further increase the performance gap between the two individuals.

Is AI negatively affecting the performance of some workers? 

The use of AI does not always guarantee improvements in worker performance. While it is a powerful tool, it can also be misleading and, in certain contexts, negatively impact the outcomes of professionals who use it. One example of this is found in the experiments by Dell’Acqua and colleagues, previously mentioned. When users (BCG consultants) applied AI to tasks where its performance was inferior to that of humans, their results significantly worsened compared to what they achieved without using the tool. Specifically, in one particular task, participants in the control group (without access to AI) achieved 84.5% correct answers, while those using AI—whether or not they had prior training with the tool—achieved only 70% and 60%, respectively.  

This phenomenon can partly be explained by the tendency of professionals to accept AI results uncritically, a behavior the authors linked to a characteristic of generative AI solutions: they can be highly persuasive, influencing how users perceive the quality of their recommendations, even when those recommendations are incorrect.  

In the same vein, three experiments conducted in 2023 by Deusto University professors Lucía Vicente and Helena Matute are noteworthy. In these experiments, participants completed a classification task in a fictional medical context. The task involved determining, with the support of recommendations from a “biased” AI solution, whether a tissue sample was affected by a disease based on the proportion of dark and light cells in the sample. The researchers found that the influence of AI bias can extend beyond the initial interaction with the system and affect humans’ future decisions, even when the AI is no longer present.  

The authors suggested that this might occur because participants reduce their oversight of information and increase their trust in AI recommendations as their experience with the system grows. This, in turn, can lead to less analytical and more superficial processing of the objective information from the tissue samples.

So, what do we do?  

When it comes to the causes of this variety of effects, research on the impact of AI on worker performance suggests that the nature and magnitude of this impact depend on a combination of factors. Key among these are workers’ skills, their attitudes toward AI and work, the capabilities of the technological solutions employed, the types of tasks to which they are applied, and how users engage with them.  

In other words, if employers want to optimize the impact of AI on their employees’ performance, they face the challenge of monitoring and managing a range of variables over which they often have only limited influence.  

One illustrative example is the use of AI tools by workers in their daily tasks. Even in this area, employers often lack awareness or control over the extent to which their employees use these solutions. A Microsoft report published in May 2024 revealed that 78% of professionals using AI in their jobs relied on their own tools (known as BYOAI, or “bring your own AI”), a phenomenon observed across all generations of workers.  

More recently, this trend was confirmed in the second edition of the Future for Work Institute survey on the impact of AI on the work of HR professionals. Nearly one in two respondents who used AI admitted to employing tools in their work that were not provided by their company.  

Furthermore, when discussing the impact of AI on worker performance, we must consider that this is a technology evolving at an incredibly rapid pace, and its impact on user performance is closely tied to the capabilities of the solutions employed.  

In this regard, the findings of research conducted by Qiao, Rui, and Xiong in 2024 are particularly thought-provoking. Their study suggests the existence of a “tipping point” for each occupation regarding the impact of AI on the demand for professionals in that specific type of work. Before reaching this tipping point, human workers benefit from the performance and productivity improvements achieved using AI tools. However, once AI surpasses the tipping point, any further improvement in AI performance harms human workers by triggering a displacement effect, which negatively affects both their workload and their income.

Therefore, companies need to acknowledge that, especially among their “knowledge workers,” it is very likely that many have used AI solutions in their professional activities throughout 2024, and this may have influenced their performance. It would be wise to consider this when managing the work of these individuals, particularly in performance evaluation and management processes. Given the time of year, many organizations are likely in the peak season for these processes.  

The company may conclude that its current system remains valid, but to reach that conclusion, its leaders should at least consider some fundamental questions. For example:  

– Is their method of measuring and comparing employee productivity still valid now that a significant portion of their work is being carried out with the support of AI tools?  

– Should the assessment of merit and effort be adjusted according to the level of assistance these tools provide to each individual worker being evaluated? 

– If so, how can they measure this level of assistance when, in many cases, they don’t even know which AI tools these individuals are using in their work?  

Additionally, the advent of AI will require many organizations to rethink their performance expectations. Since AI is a widely accessible technology, companies seeking to differentiate themselves from their competitors will need individuals capable of achieving objectives that algorithms cannot. But how can this shift be reflected in performance management systems, especially when AI continues to advance and expand its capabilities?  

Let’s think about it…