OpenAI’s GPT-3 moves us closer to realising computers that write their own code

When OpenAI first published a paper on their new language generation AI called GPT-3, the descendent of what came to be known notoriously as the world’s most dangerous AI GPT-2, the hype was slow to build. The paper indicated GPT-3, the biggest natural language AI model yet, was advanced, but it only had a few written examples of its output. Then OpenAI gave select access to a beta version of GPT-3 to see what developers would do with it, and minds everywhere were blown.

Developers playing with GPT-3 have taken to Twitter with examples of its capabilities: short stories, press releases, articles about itself, a search engine. Perhaps most surprising was the discovery GPT-3 can write simple computer code. When web developer, Sharif Shameem, modified it to spit out HTML instead of natural language, the program generated code for webpage layouts from prompts like “a button that looks like a watermelon.”

Post on X

This is mind blowing.

View on X →

“I used to say that AI research seemed to have an odd blind spot towards automation of programming work, and I suspected a subconscious self-preservation bias,” tweeted John Carmack, legendary computer game developer and consulting CTO at Oculus VR. “The recent, almost accidental, discovery that GPT-3 can sort of write code does generate a slight shiver.”

While the discovery of GPT-3’s coding skills may have been somewhat serendipitous, there is, in fact, a whole field dedicated to the development of machine learning algorithms that can code that I’ve written about before and that feed into a new trend that Microsoft is increasingly referring to as the rise of Citizen Developers – people who have no coding skills but who can use these AI’s and programs to create their own applications nevertheless without having to so much as lift a finger.

Programs including Microsoft’s DeepCoder, an AI that writes its own code, and Google’s Bayou project that’s funded by the US government that can write a couple of hundred of lines of code all by itself. And then there’s DeepTab Nine that can write code in hundreds of programming languages.

Research in this field has been making slow but accelerating progress, and asides from the aforementioned examples a new algorithm just recently took another step.

The algorithm, called Machine Inferred Code Similarity (MISIM), is the brainchild of researchers from Intel, Georgia Institute of Technology, University of Pennsylvania, and MIT. Trained on the huge amount of code already publicly available on the web, MISIM can figure out what a program is supposed to do. Then, after finding other similar programs and comparing it to them, MISIM can offer ways to make the program faster or more efficient.

It isn’t the first machine learning algorithm to make recommendations or compare similarity, but according to the researchers in a new preprint paper on MISIM, it was up to 40 times more accurate at the task when it went head to head with several of its most advanced competitors.

Near term, the AI could be a useful sidekick for today’s programmers. Further out, the field could open programming to anyone who can describe what they want to create in everyday language or bring machines that write and maintain their own code – in short the democratization of coding and the emergence of a world where all you have to do is talk to a computer and describe the program you’d like it to build and it builds it for you.

The pursuit of computers that can code is almost as old as modern computer science itself. While there have been advances in programming automation, the recent explosion in machine learning is accelerating progress in a field called machine programming.

In a 2018 paper on the field, a group of Intel and MIT researchers wrote, “The general goal of machine programming is to remove the burden of writing correct and efficient code from a human programmer and to instead place it on a machine.”

Researchers are pursuing systems that can automate the steps required to transform a person’s intent—that is, what they want a piece of software to do—into a working program. They’re also aiming to automate the maintenance of software over time, like, for instance, finding and fixing bugs, keeping programs compatible, or updating code to keep up with hardware upgrades.

That’s easier said than done, of course. Writing software is as much art as it is science. It takes a lot of experience and creativity to translate human intent into the language of machines.

But as GPT-3 shows, language is actually a skill machine learning is rapidly mastering, and programming languages are not so different from English, Chinese, or Swahili. Which is why GPT-3 picking up a few coding skills as a byproduct of its natural language training is notable.

While algorithmic advances in machine learning, like GPT-3, are key to machine programming’s success, they’d be useless without good training data. Luckily, there’s a huge amount of publicly available code on sites like GitHub - replete with revision histories and notes - and code snippets and comment threads on sites like Stack Overflow. Even the internet at large, with accessible webpages and code, is an abundant source of learning material for AI to gobble up.

In theory, just as GPT-3 ingests millions of example articles to learn how to write, machine programming AI’s could consume millions of programs and learn to code. But how to make this work in practice is an open question. Which is where MISIM comes in.

MISIM advances machine programming a step by being able to accurately identify what a snippet of code is supposed to do. Once it’s classified the code, it compares it to millions of other snippets in its database, surfaces those that are most similar, and suggests improvements to the code snippet based on those other examples.

Because MISIM classifies the code’s purpose at a high level, it can find code snippets that do the same thing but are written differently - there’s more than one way to solve the same problem - and even snippets in other programming languages. Simplistically, this is a bit like someone reading a New Yorker article, identifying its topic, and then finding all the other articles on that topic - whether they’re in Der Spiegel or Xinhua.

Another benefit of working at that higher level of classification is the program doesn’t need the code to be compiled. That is, it doesn’t have to translate it into the machine code that’s executed by the computer. Since MISIM doesn’t require a compiler, it can analyze code snippets as they’re being written and offer similar bits of code that could be faster or more efficient. It’s a little like an email autocomplete feature finishing your sentences.

Intel plans to offer MISIM to internal developers for just this purpose. The hope is it’ll prove a useful sidekick, making the code-writing process faster, easier, and more effective. But there’s potentially more it can do. Translation between computer languages, for example, could also be a valuable application. It could perhaps help coders update government software written in archaic languages to something more modern.

But Justin Gottschlich, director of machine programming at Intel, has an even grander vision: The full democratization of coding, as I’ve talked about many times now.

Combine MISIM (or something like it) with natural language AI, and future programmers could simply write down what they want a piece of software to do, and the computer whips up the code. That would open programming to anyone with a decent command of their native language and a desire to make something cool.

As Gottschlich told MIT Technology Review, “I would like to see 8 billion people create software in whatever way is most natural for them.”