Tangible Media Group

A recent focus of our lab has been making use of Tangible Displays and Body Object Space to develop new assistive technologies. As a test case, we prototyped Nintendo’s Super Mario Land side-scrolling game for visually impaired users, using body movement analogies to control Mario in the game. Mario and 2D side scrollers present a particularly interesting case, as they keep the main character location in the center of the display and move the world around the character. The shape display itself provides spatial audio of enemy positions. We make use of the AUFLIP sensor platform to pick up body movements—walking and jumping, causing Mario to do the same in-game. This enables users to keep their hands engaged to understand the game landscape, while using their body to control Mario at the same time.

CONJURE aims to enable tactile play of Super Mario Land using body motions, spatial auditory cues, and a large-scale tactile shape display.

Super Mario Land is streamed from an emulator and transformed into a pin height map for the tactile TRANSFORM shape display with image processing.

The transformed emulator image is explored on the shape display. Mario stays centered on the display, enabling users to maintain a reference point from which to explore the game world. Adversaries in-game create unique sounds via their movements on the pin-display that are spatially identifiable via the shape display's large scale. These sounds enable users to track and defeat opponents by jumping on them.

To play the game, the user wears a sensor on their ankle that captures walking and jumping in place. These actions control the main player in the game world, enabling them to play while keeping their hands available to explore and understand the game world.