DeepMind, the subsidiary of Google’s parent Alphabet, has created a new AI system called AlphaCode. After building an AI that can teach itself to walk, an AI capable of strategically outthinking humans to an AI that can detect certain diseases better than doctors, DeepMind’s latest AI – AlphaCode – wants to “write computer programs at a competitive level”.
Computer programs are the bedrock of various AI systems but AI itself trying to write a competitive program shows the advancement of the technology. AlphaCode is good but it is not great, just yet. However, like past AI systems from DeepMind, the AlphaCode could also mature to beat humans in no time.
An AI capable of writing competitive computer programs
DeepMind says AlphaCode is built using transformer-based language models to generate code at an “unprecedented scale”. The AI then smartly filters the codes to a small set of promising programs. In order to validate its test, DeepMind tested AlphaCode against coding challenges used in human competitions. The results may not be shocking yet but they do show why DeepMind stays ahead on the AI innovation front.
The program from DeepMind achieved an “estimated rank” that placed it within the top 54 per cent of human coders. The Alphabet subsidiary calls it a significant step forward for autonomous coding. However, it is important to note that these tests need not necessarily represent the programming challenges faced by a competitive computer programmer.
The company further adds that AlphaCode was tested against challenges curated by Codeforces, a competitive coding platform that sees tens of thousands of participants compete from around the world. The coding platform shares weekly problems and issues ranking for coders. For evaluation of its own system, DeepMind selected 10 recent contests on the basis of each being newer than its training data.
“AlphaCode placed at about the level of the median competitor, marking the first time an AI code generation system has reached a competitive level of performance in programming competitions,” DeepMind explains.
AlphaCode: How is it evaluated
For its test, DeepMind selected 10 of the recent challenges from Codeforces and these challenges were fed into AlphaCode in the same form they are given to humans. The AI system generated a large number of possible answers and then narrowed them down by running the code and checking the output as a human programmer would do. The 10 challenges that AlphaCode was tested on have been tackled by 5,000 users on the Codeforces site.
AlphaCode ranked within the top 54.3 per cent of responses and DeepMind estimates a Codeforces Elo score of 1238, placing it within the top 28 per cent of users who competed on the coding platform in the last six months.
“I can safely say the results of AlphaCode exceeded my expectations. I was sceptical because even in simple competitive problems it is often required not only to implement the algorithm, but also (and this is the most difficult part) to invent it. AlphaCode managed to perform at the level of a promising new competitor. I can’t wait to see what lies ahead,” says Mike Mirzayanov, Founder of Codeforces.
Where does it go now
DeepMind is very careful of the fact that AlphaCode’s current skill is limited to the domain of competitive programming. However, the early results show that AlphaCode could be used to create tools that make programming more accessible and could even one day be fully automated.
DeepMind is not alone in experimenting with building an AI program capable of writing competitive computer programs. Microsoft is collaborating with AI lab OpenAI to adapt the latter’s GPT-3 to function as an autocomplete program that finishes strings of code. The idea here is to make autocomplete on coding software work similar to the Smart Compose feature on Gmail.
The progress made with developing AI coding systems in recent years shows that the first steps have been taken. However, these programs are still far from taking over the work from human programmers. There is a possibility that these AI programs will be integrated slowly into low-level coding solutions and might even act as assistants. However, we are probably a few years away, before AI takes over the role of human programmers.