How Music Experts Taught AI To Compose Beethoven

So we all know by now how the world is saturated in artificial intelligence — AI — right? We've got consumer-driven AI that transforms a person into a collection of data points that companies aggregate and sell, as Marketing AI Institute explains of Facebook. AlphaZero, the AI created by Google firm DeepMind, taught itself chess with no human involvement in a mere four hours, as The Guardian reminds us. (It obliterated the best players in the world.) Authors AI boasts that its AI "Marlowe" can critique your fiction, including "narrative arc and plot structure analysis" and "major characters' personality traits." IFL Science reports that lawyer Damien Riehl spearheaded a project to create an AI to compose "every 12-note melody that has ever been written or can ever be written." His catalog, open to the public at All the Music, stands at 68 billion melodies strong.

So what do we need people for, again? Except shuffling around resources and information, and maybe cutting down trees and petting dogs. Well, as UDiscover explains, we need people to create AI to do the work of people that can now be done by AI. Capisce?

More to the point: Beethoven. When he composed his gorgeous "symphony to end all symphonies," his ninth — the one with "Ode to Joy" — he was already working on his 10th. Per The Guardian, he passed away in 1827 and left us with pieces of his unfinished work. And now? An AI has stepped up to finish it.

As Smithsonian Magazine says, musicologist Barry Cooper tried to complete Beethoven's 10th symphony in 1988. He reconstructed, based on notes and theory, what he believed was a faithful rendition of the first movement of the symphony. But Beethoven's notes were too threadbare to go past that point. This time, the challenge was to "teach" an AI to understand Beethoven enough to predict the entire rest of the symphony. Until then, AI hadn't been able to add anything more than a few seconds to an original musical work.

The "head teacher" in this case, so to speak, was project head Ahmed Elgammal, professor at the Department of Computer Science at Rutgers University, and founder and director of Rutger's Art and Artificial Intelligence Laboratory. Elgammal had been in charge of the lab's "AICAN" program, as he says on The Conversation. The AICAN program is an AI that can learn various "existing styles and aesthetics and can generate innovate images of its own." AICAN's artwork has been showcased worldwide and sold at auctions. Its guiding artistic principle is "the least effort," which essentially means that it seeks a balance between too novel and too banal. It was fed 80,000 images from all periods and styles, and can not only create its own work and name it, but evaluate others' work, as well. In the end, it seems that human viewers couldn't tell that the artwork was made by an AI, not another person. 

In 2019, as Elgammel wrote at The Conversation, the Karajan Institute in Salzburg, Austria (Mozart's birthplace) contacted him and asked him to apply his expertise to Beethoven's 10th symphony. He was joined by a team of music experts: Austrian composer Walter Werzowa, "computational music expert" Mark Gotham, who would feed the AI literally everything Beethoven had ever written in order to allow it to make correct and comprehensive choices, and Harvard University musicologist and pianist Robert Levin, who had previously completed unfinished work by Mozart and Bach.

Using Beethoven's entire body of work and sketches of his musical phrases from his 10th symphony, they had to infer Beethoven's creative process. Specifically, they had to figure out how to develop short motifs into entire elongated passages, and whether or not those passages conformed to the musical structure of a scherzo, trio, or fugue, for instance. Then, they had to teach the AI how to harmonize with a melodic line, develop bridges between passages, write a coda to finish the whole thing, and assign parts to various instruments.

One of the team said that the AI "reminded him of an eager music student who practices every day, learns, and becomes better and better." For two years they all worked together, from an initial Turing-test-type "did an AI write this" media conference in 2019 all the way to the final, completed symphony. It aired on October 9, 2021, and is available in full on Magenta Musik 360. (The performance starts at about 16:30.)