3-D movement captured to conduct music

Imagine the sound mixing desk in a concert hall controlled not by a technician manipulating hundreds of knobs and sliders, but by pointing to speakers and changing volume and tone with the movement of an arm. This futuristic orchestra conductor is being made reality by the work of researchers in the school of music at Leeds.

Dr Kia Ng of the Interdisciplinary Centre for Scientific Research in Music is developing ways of capturing human movement in three dimensions and using it to instruct computers to control or create music.

Ultimately the technique could also be used for everyday tasks like scrolling a web page with the movement of a hand, which could be especially useful for people with restricted mobility.

To capture 3-D movement, infra-red light is projected onto tiny reflective balls attached to clothing and monitored with cameras. The changing position of the balls is determined by triangulation and the computer recognises the movement as a gesture which it turns into instructions for music software.

Dr Ng said: “The biggest challenge is to train the system to anticipate movement. To make sense of a gesture it needs to know not only where an object has been and where it is, but also where it will be. Based on what it has seen before it builds up a catalogue of likely gestures and the software anticipates which one it thinks a movement will be.”

The research is a development of Dr Ng’s techniques tracking movement in two dimensions (Reporter 452).