MIT and Stanford ChatGPT study shows low skilled workers benefit the most

Since OpenAI’s release of ChatGPT last November, the buzz around generative AI has been nothing short of deafening. Some are excited about its potential to transform the way we work, create, and live, while others are wary of the dangers it poses to jobs and the nefarious ways it can be used. We know that programs like Midjourney, DALL-E, and GPT-4 are enabling millions of people to generate images and text, but so far very few studies have dug into the impact these tools are having, be it positive or negative, on people and the workforce.

One such study was released this month. Titled “Generative AI at Work,” the paper, by teams from Stanford University and MIT, is one of the first times researchers take a microscope to the way generative AI is actually affecting peoples’ jobs. The team looked at how employees of a Fortune 500 company were impacted by generative AI when they started using it as part of their day-to-day work.

The Future of Work, by keynote Matthew Griffin

The study followed 5,179 customer service agents at a large software firm, whose name wasn’t disclosed, over the course of a year. The employees, mostly based in the Philippines, were split into two groups: one was given access to an AI whose help they could choose to integrate into their work, while the other continued as usual.

The AI was trained on data from over 5,000 successful customer service interactions, likely in the form of recordings of high-performing employees having conversations with customers and resolving their issues. The AI then monitored customer interactions in real time and gave agents suggestions of what to say. The employees could choose to use the suggestions word for word, dismiss them altogether, or use a tweaked version.

The researchers looked at how long it took for agents to solve customers’ issues and how successfully they did so. The results? Good things all around.

For one, the AI enabled customer service agents to get through calls more quickly, resolve more customer complaints successfully, and even handle multiple customer calls at once. The agents using the AI resolved 13.8 percent more issues per hour than they’d been able to without the AI.

And that’s not all. Since the AI’s suggestions skewed towards helping agents be patient and empathetic with frustrated customers, the customers treated the agents better, losing their tempers and raising their voices less  - it’s not pretty, but let’s be honest, we’ve all been there. As a result, the agents were happier and more satisfied with their work.

Perhaps not surprisingly, the AI was the most helpful for the least-skilled workers and those who had been with the company for the shortest time. Meanwhile, the highest-skilled and most experienced agents didn’t benefit much from using the AI. This makes sense, since the tool was trained on conversations from these workers; they already know what they’re doing.

“High-skilled workers may have less to gain from AI assistance precisely because AI recommendations capture the knowledge embodied in their own behaviors,” said study author Erik Brynjolfsson, director of the Stanford Digital Economy Lab.

The AI enabled employees with only two months of experience to perform as well as those who’d been in their roles for six months. That’s some serious skill acceleration. But is it “cheating”? Are the employees using the AI skipping over valuable first-hand training, missing out on learning by doing? Would their skills grind to a halt if the AI were taken away, since they’ve been repeating its suggestions rather than thinking through responses on their own?

It’s possible that an over-reliance on the tool could be detrimental to employees’ ability to build up and retain skills. But, ideally they are learning by doing, just in a faster way, since they’re skipping over the drudgery of many unpleasant interactions with angry customers.

Where does this leave high-skilled employees, though? If their work is being used to train AIs that then freely give their skills to inexperienced employees, that could create issues around fairness and compensation as companies hire less skilled workers on lower salaries to replace their higher earning staff. If you’ve been honing your soothing one-liners for years then a newbie comes in saying all the same things by month two on the job, you’re not going to be thrilled - especially if you’re not getting paid a lot more than the newbie.

Finally, since the AI was essentially training newer employees, their managers didn’t need to spend as much time training them - and more of their time was thus freed up. That means managers could take on bigger teams, which means the company could ultimately hire more employees - if it’s selling enough of its products - and do more business. It seems this particular “generative AI” generated a lot more than just conversation suggestions: it generated employee satisfaction, skill acquisition, and free time. So Win-Win-Win?

Will the same hold true for other scenarios where these tools are implemented? Could be, but they should be introduced with caution and oversight nonetheless, as there are likely many secondary effects generative AI could have on a workplace that wouldn’t become apparent right away, and may not be wholly positive.

“We need far more research here,” said Brynjolfsson. “The impact of AI on productivity may vary over time, and adding these tools to the office could require complementary organizational investments, skills development, and business process redesign. And AI systems may impact worker and customer satisfaction, attrition, and patterns of behaviour. There’s so much we don’t know.”