Wednesday, 06 December 2023
Researchers at the University of Nottingham have developed a musical robot that can help musicians with disabilities to improvise and access instruments in a way that hasn’t previously been possible.
Jess+ has been developed as part of the DigiScore project, which is the first large-scale collaboration between the university’s Department of Music, Orchestras Live, and Sinfonia Viva. The AI robot helps musicians with disabilities to collaborate as part of a mixed ensemble and to improvise in the creation of live scores where dexterity can be a barrier.
The robot was created by Craig Vear, Professor of Music, and Computer Science at the University of Nottingham, and used in a series of workshops involving musicians Jess Fisher (who has a physical disability) and Clare Bhabra and Deirdre Bencsik from Sinfonia Viva.
During the workshops, the robot moved and drew marks on paper in response to the real-time sound created by a live musician. The musician in turn would interpret these movements and marks as notational material and make a sound in response, much like a graphic score. The central research question with the project was: How to build an embodied-AI system that facilitates co-creation for an improvising ensemble of disabled and non-disabled musicians, so that the expressive qualities of all musicians are amplified and bound together in a co-creative system?
Professor Vear, who is the Principle Investigator of DigiScore, drew on his expertise in the fields of music, digital performance, creative technologies, AI, creativity, gaming mixed reality and robotics, as a focus for the Jess+ project.
It's been a privilege to work with these amazing musicians to develop AI that benefits and transforms lives. In a time when creative AI is perceived as taking something away from musicians, we developed an AI that extended, amplified and transformed musicianship, creativity and togetherness. We feel that this approach can be used in the future to enable those facing barriers to make music, and to express themselves in a supportive, inclusive and enhancing way."
There are significant barriers to music-making for disabled musicians. One such barrier is that while non-disabled people can make music in many ways, this is less true for disabled musicians, who need new accessible instruments, new creative processes, and new hierarchies of ‘success.’
Another issue is how a mixed ensemble of disabled and non-disabled musicians might improvise together in such a way that dexterity is not a limiting factor. And a further barrier is how disabled musicians can be excluded from the creative process in which a music score is created for them. All these obstacles can leave creative potentials untapped, set up hierarchical measurements of creative involvement, and limit the potential input of disabled musicians.
In tacking these issues, the research team, along with the project partners Sinfonia Viva, a UK-based orchestra and educational organisation, and Orchestras Live, a national producer creating inspiring orchestral experiences for communities across England, were able to use AI and Professor Vear’s digital score concept to generate new modes of music-making.
The findings from the project revealed that the musicians formed unexpected and distinct relationships with Jess+, each perceiving they were in-the-loop with the system, as they found it to be a good listener with its own creative ‘voice.’
It was non-judgmental and accepting, which the participants said promoted a new freedom of expression and confidence in taking musical risks when improvising.
Images courtesy of BBC R&D. Musicians L-R: Jess Fisher, Clare Bhabra, Deidre Bencsik
For musician Jess, she felt that the system allowed her to express the emotions that she is sometimes not able to express through her current digital setup. Referring to it on several occasions as a “friend” and as a “storyteller,” she said that being extended through the system meant that she could feel like she was able to express her feeling directing onto a score.
“I wanted to explore that part of me, and I want my emotions that are in here to get expressed outwardly through that,” said Jess.
A recent performance from the Jess+ team was held at the BBC studios in London for a performance as part of the Bridging Responsible AI Divides (BRAID) launch event. BRAID is a three-year national research programme which aims to bring together UK-based researchers from across the arts and humanities to drive responsible AI innovation, in partnership with the BBC and the Ada Lovelace Institute.
The video of the performance at the BBC can be viewed here.
Story credits
More information is available from Professor Craig Vear in the University of Nottingham’s Department of Music, at craig.vear@nottingham.ac.uk
Notes to editors:
About the University of Nottingham
Ranked 32 in Europe and 16th in the UK by the QS World University Rankings: Europe 2024, the University of Nottingham is a founding member of the Russell Group of research-intensive universities. Studying at the University of Nottingham is a life-changing experience, and we pride ourselves on unlocking the potential of our students. We have a pioneering spirit, expressed in the vision of our founder Sir Jesse Boot, which has seen us lead the way in establishing campuses in China and Malaysia - part of a globally connected network of education, research and industrial engagement.
Nottingham was crowned Sports University of the Year by The Times and Sunday Times Good University Guide 2024 – the third time it has been given the honour since 2018 – and by the Daily Mail University Guide 2024.
The university is among the best universities in the UK for the strength of our research, positioned seventh for research power in the UK according to REF 2021. The birthplace of discoveries such as MRI and ibuprofen, our innovations transform lives and tackle global problems such as sustainable food supplies, ending modern slavery, developing greener transport, and reducing reliance on fossil fuels.
The university is a major employer and industry partner - locally and globally - and our graduates are the second most targeted by the UK's top employers, according to The Graduate Market in 2022 report by High Fliers Research.
We lead the Universities for Nottingham initiative, in partnership with Nottingham Trent University, a pioneering collaboration between the city’s two world-class institutions to improve levels of prosperity, opportunity, sustainability, health and wellbeing for residents in the city and region we are proud to call home.
More news…