Coursera’s AI tells your boss what skills you lack and how to get them

Here’s the conundrum with corporate online learning, there are so many classes available from sites like Coursera, edX, and Udacity that companies don’t know what content to offer their employees. And once companies do choose a learning program, or track, then it’s tough for them to figure out what skills their employees pick up and how well they’ve mastered them. In short, they need an objective metric to evaluate proficiency, and a new Artificial Intelligence (AI) powered tool developed by Coursera aims to be that metric.

The feature, which the Bay Area startup announced a few days ago, lets companies that subscribe to its training programs see which of their employees are earning top scores in Coursera classes, how their employees’ skills measure up to their competitors’; and what courses would help fill any knowledge gaps. Companies will be able to access the tool, which uses machine learning to derive insights, in the online dashboard of their Coursera profiles later this year.

The new feature is just one example of the ways online learning providers are using AI to match learners with courses, assess their ability, and tweak class content in response to feedback. Coursera has a data science team that does everything from “collecting and storing data in a warehouse to interpreting information for making internal decisions to building algorithms that feed back into the site,” according to Emily Glassberg Sands, who leads the group.

Meanwhile, Udacity’s AI research team, which it established in 2017, analyses student sentiment to see how lessons can be improved and computes whether learners like the changes. Udacity has also used AI based chatbots to help students find relevant courses and answer common questions during the enrolment process, and edX says it experiments with AI to increase how well people learn and teach.

By addressing some of the most pernicious challenges in online learning, the hope is that these raft of new AI features could inspire more people, and paying companies, to sign up for the company’s training. Increasing the effectiveness of educational programs and measuring their impact are the top two priorities for companies that pay to train their staff, according to a 2017 survey of US based businesses by Training magazine.

The market also has plenty of room to grow, for example, Coursera’s business program, which targets companies, currently has 1,400 customers, and overall, Coursera has 31 million learners, which makes it the world’s largest online learning provider.

“The Achilles’ heel of the corporate learning industry is that no one knows how to demonstrate the return on investment (ROI),” notes Leah Belsky, who heads the Coursera for Business program. “Companies know their people need to learn new skills to stay competitive, but they have a hard time communicating what the value of that learning is.”

Coursera, too, wanted to be able to quantify the benefits of its classes. So a year and a half ago, its data science team began developing machine-learning algorithms to map the 40,000 skills taught on their platform. First, the team used Natural Language Processing (NLP) to figure out how often instructors mentioned particular concepts during their classes. That information helped it identify which classes imparted which skills. Coursera can get some of that data by simply asking its instructors. But it also uses NLP because educators often think of themselves as teaching theoretical concepts while learners want to know what specific tools and technologies they will master, says Glassberg Sands.

Most recently, the team incorporated a psychometrics methodology called Item Response Theory (IRT) into some of its machine learning models to assess learners’ abilities based on how they performed on Coursera quizzes and assignments. The approach allowed the team to measure proficiency in a given skill area across learners who were answering different questions of varying difficulty, says Glassberg Sands. That’s important because a more advanced learner will, on average, take harder courses on Coursera and attempt harder questions than a beginner. The IRT model accounts for that by figuring out what skill a particular quiz question is assessing and the relative difficulty of that quiz question before evaluating a learner’s proficiency.

“What you get is a percentile for your employees in each skill area, relative to whatever comparison group you choose, whether that’s all the companies on Coursera or just companies in your industry or your country or companies of a similar size,” says Glassberg Sands.

Most online learning providers let their corporate customers see which of their employees are enrolled in classes, how they’re progressing, and the feedback they’re providing on their experiences. Coursera says machine learning enables it to examine how its millions of learners are performing across its platform and provide predictive insights, such as a given learner’s skill level, along with more standard student metrics. In theory, these machine learning models should also get smarter as learners and instructors create more data that can be fed back into the system.

Coursera expects the skills benchmarking information, which will be updated daily, to be useful to learning and development specialists, HR professionals, and hiring managers, who previously would have had to guess what skills other companies were acquiring and which of their employees were most expert in, say, a particular programming language. But it’s also bound to rankle some Coursera learners who don’t want their bosses making employment decisions based on their online learning performance.

Glassberg Sands and Belsky contend that the technology will benefit individuals in other ways, such as bringing attention to skilled people who might otherwise be overlooked. Software giant Adobe, which has been testing the new Coursera feature for the past month, agrees.

“We probably have some experts who are located in remote offices where we may not be able to identify their level of talent easily,” says Justin Mass, who leads Adobe’s digital learning program. “These features give us a true sense of what our employees know and are good at, which should help us assess talent more intelligently.”