AI can judge your piano skills just by watching you play

25 January 2021, 17:10 | Updated: 25 January 2021, 17:21

Could a computer really assess a pianist’s skill level with human accuracy?

A new study claims this artificial intelligence can judge a musician’s pianistic talents just by watching a video of them tickling the ivories.

Researchers say the AI could be used to facilitate online music teaching, looking specifically to our current COVID-19 times.

Brendan Morris, an assistant professor in Electrical and Computer Engineering at the University of Nevada, Las Vegas, led the study which examined 61 piano performance videos on YouTube.

Firstly, Morris asked a trained pianist to manually grade every pianist’s skill level on a 10-point scale. On the scale, based on a syllabus developed by the Music Teachers National Association, levels 1-9 signify a pre-collegiate skill level, with level 10 representing a collegiate or post-collegiate mastery.

The researchers used half these videos and their assigned grades to train a neural network, creating an AI that can assess piano playing from just a video.

Read more: You can now play LEGO’s functioning 3,622-piece grand piano >

Firstly, researchers taught the AI to judge each pianist’s music-making visually, basing its assessment on technical skills, which build from grade to grade, and virtuosic skills, which are unique to high-skill level pianists.

For instance, in a visual examination of Chopin’s Etude Op. 25 No. 11, the four octave A minor scale at high speed would immediately indicate a high-grade piece (start the video below at 0:39).

From this, the researchers determined, “The lack of such skills would not give any indication as to the level of a pianist, but the presence of such skills would immediately indicate a very high level of technical achievement.

“For example, professional pianists who must play at high speeds may play eight note intervals with their first and third fingers, which is incredibly difficult for the average pianist.”

The AI’s accuracy in this test was 74 per cent, set against the expert pianist’s judgement.

Secondly, the AI judged the videos aurally, making a call on the pianists’ level based on the speed of the playing – “a simple yet valuable tool to judge the technical skill required for a piece,” researchers claimed.

But while any competition judge or teacher worth their salt would be able to determine level and skill based on the variety of style, clarity and dynamic in a piece, it was here that this rather unfamiliar AI judge struggled to live up to its human counterpart, achieving just 64 percent accuracy.

Read more: Schubert's 'Unfinished' Symphony completed by artificial intelligence >

Chopin - Etude Op. 25 No. 11 (Winter Wind)

Researchers concluded that both a visual and aural assessment achieved the best results with a 75 percent precision rating, which New Scientist magazine describes as “near-human accuracy”.

Based on the evidence, the researchers said they hoped the automated evaluation could make music lessons more accessible to disadvantaged groups in society.

They also suggested the AI could provide second opinions in the case of any controversial or biased assessments.

“With our work, we hope to inspire future work in the direction of automated piano skills assessment and tutoring,” researchers concluded.

“While our approach yielded good results, we believe there is a significant scope for improvement in this direction.”