Musicians across the world are now a step closer to feeling the real-time thrill of performing together\u2014without being in the same room. A new technology is changing how music is made, practiced, and taught by turning video recordings into lifelike virtual performances. <\/p>\n
This breakthrough blends the rhythm of live music with the power of virtual reality<\/a>, delivering a vivid and personal music experience that reaches beyond physical boundaries.<\/p>\n A New Way to Play Music Together<\/p>\n The Joint Active Music Sessions, known as JAMS, is a virtual music platform designed to make remote performances feel real. Developed by a team of researchers, this tool allows musicians to rehearse, perform, and teach in virtual spaces using avatars. These digital characters mimic real human movement and expressiveness, letting artists interact just like they would in a live setting.<\/p>\n he Coull Quartet playing with motion-capture markers on their string instruments. They are surrounded by motion-capture cameras. (CREDIT: Maria Witek) <\/p>\n To join a session, a musician only needs a smartphone<\/a> to record themselves and a virtual reality headset. The software then transforms the video into an avatar that plays in perfect sync with another user. Musicians can see each other\u2019s actions\u2014like the tilt of a violinist\u2019s bow or a key hand gesture\u2014at exactly the right moment.<\/p>\n This kind of precise coordination is crucial. In live music, the slightest delay between players can break rhythm and focus. That delay, called latency, can cause problems even at 10 milliseconds. But JAMS eliminates that issue entirely. <\/p>\n \u201cLatency is the delay between a sound production and when it reaches the listener,\u201d says Dr. Max Di Luca, one of the platform\u2019s creators at the University of Birmingham<\/a>. \u201cPerformers can start to feel the effects of latency as low as 10 milliseconds, throwing them off-beat, breaking their concentration, or distracting them from the technical aspects of playing.\u201d<\/p>\n With JAMS, every beat hits at the right time, every glance and motion lands naturally, and every session feels like an in-person gathering. The experience is smooth, without interruption or lag.<\/p>\n Capturing the Feeling of Playing Live<\/p>\n JAMS goes far beyond simple video sharing. It uses avatars that respond in real-time and give performers visual cues that matter. This includes small but important elements\u2014like eye contact or shared body language\u2014that help musicians stay in sync and communicate while playing. That feeling of shared presence is central to music.<\/p>\n Unlike traditional video calls, the VR headset delivers a highly immersive setting. It places the musicians in a shared virtual space where their avatars interact as if they were on stage or in a rehearsal room. Even the placement of faces at eye level improves the connection between performers.<\/p>\n This visual accuracy doesn\u2019t just add to the realism\u2014it also boosts learning. Watching a skilled musician\u2019s movements closely can help less experienced players improve faster. \u201cYou can adapt the avatar that other people play with, or learn to play better through practice with a maestro,\u201d says Dr. Di Luca.<\/p>\n Computer shows the digital process of creating the life-like avatars in ARME. (CREDIT: Maria Witek) Building a Social Network for Musicians<\/p>\n The vision behind JAMS includes more than just virtual practice sessions. It aims to create a digital space<\/a> where musicians can connect, perform, teach, and grow as a community. It could become something like a music-focused version of Spotify or Myspace, where people meet to make music, share their work, and even perform for larger virtual audiences.<\/p>\n The platform is made with musicians in mind\u2014whether they are seasoned professionals or just starting out. This sense of inclusiveness is one of JAMS’ strengths. It gives beginners a space to improve while also offering advanced players tools for professional performance.<\/p>\n The avatars aren\u2019t just for show\u2014they are built using a special algorithm created during the Augmented Reality Music Ensemble (ARME) project. This project united experts from several areas, including psychology, computer science<\/a>, engineering, music, sport science, and math. <\/p>\n Together, they built a model that could capture the complex timing and movement of real musicians. The result is a responsive avatar that can match the physical actions of its human counterpart with surprising accuracy.<\/p>\n From Practice Room to Big Opportunities<\/p>\n JAMS isn\u2019t only a practice tool\u2014it opens doors to many other uses. It can be adjusted for lipsyncing and dubbing in TV or film. It also gathers performance data that can be used to create digital \u201ctwins\u201d of musicians. These digital versions can be licensed for games, animations<\/a>, or virtual concerts, giving artists more ways to share and profit from their work.<\/p>\n This could also change how music rights are handled. Digital avatars performing real pieces might allow music to be shared across platforms in new ways, offering publishers more control and more options for distribution.<\/p>\n There\u2019s also potential for educational growth. Music teachers can use JAMS to create practice partners for their students, helping them improve timing, technique, and musical expression. Students can interact with avatars modeled after famous performers, making learning more exciting and relatable.<\/p>\n LiDAR (Light Detection and Ranging) captured 3D data of musicians to create lifelike avatars for virtual and augmented reality performances. (CREDIT: Maria Witek) A Step Toward the Future of Music<\/p>\n