Fantasia: Motion as magic
Designing gesture controls that feel natural, and even magical: Fantasia: Music Evolved with Disney, and an unreleased next-generation Kinect game.
At a glance
- Situation
- Kinect games asked players to control everything with their bodies, and to the camera, exuberant dancing and a deliberate command could look a lot alike.
- Task
- Design gestures that felt natural and easy to learn, stayed comfortable over long sessions, and that the system could reliably tell apart.
- Action
- On Fantasia, embedded research in the scrum teams, planned studies with Disney's research labs, and designed the overworld and song controls. On a next-gen game, built an ergonomic gesture library from player testing, with motion data and telemetry for machine-learning detection.
- Result
- Controls grounded in how people actually move, including a steering scheme players found on their own: lean to steer, twist to turn.
Your whole body is the controller
Fantasia: Music Evolved set out to be a spiritual successor to Disney's 1940 film, which Walt Disney called a voyage of discovery in the realms of colour, sound and motion. Players would shape music with their hands across about a dozen worlds. At the same time, I was designing controls for an unreleased Xbox One game that let players move freely through a 3D world.
Both raised the same problem. To a camera, an enthusiastic flourish and a deliberate command can look very similar, and players get tired.
Natural to learn, comfortable to keep doing
Design gestures that felt natural and even magical, were easy to learn but comfortable over a long session, and that the system could reliably tell apart from everything else a player's body was doing.
Design from how people move
Put research in the stand-up
On Fantasia, I embedded with the scrum teams, joining daily stand-ups and staying close to features that were iterating or due for playtesting, so research and design feedback reached development quickly. I built research into the team's agile workflow, created milestone schedules and test plans with Disney's user research team in Los Angeles, and designed gameplay and controls for the overworld and song levels.
The design questions were unusual ones: how to make an interaction feel magical, how to make audiovisual landscapes double as interface, and how to separate exuberant movement from deliberate input.
Borrow from cycling, surfing and flight
For the next-gen game, players needed to move freely through 3D space. I drew on cycling, surfing and bird flight, and tested gesture ideas with players. Most steered by leaning and turned by twisting their torsos, so the controls followed them.
Teach the system with real movement
I built an ergonomic gesture library for 3D motion controls, motion-captured and graded many variations of each gesture for machine-learning detection, and added telemetry to track how players moved and where detection misfired. I worked with level and systems designers on gameplay that suited the controls, balancing how easy a gesture was to learn against how comfortable it was to repeat, and how to handle a hasty one.
Controls grounded in real bodies
- Embeddedresearch and design feedback in the daily rhythm of the scrum teams.
- Lean and twista steering scheme players found on their own, drawn from cycling, surfing and flight.
- Telemetryto find detection misfires in real play, not just in the lab.