Tangible Media Group

Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials

Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. 2012. Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (January 2012), 38-51. DOI=10.1145/2065327.2065337

DOI: http://doi.acm.org/10.1145/2065327.2065337

Abstract

“Radical Atoms” is our vision of human interaction with future dynamic materials that are computationally reconfigurable. “Radical Atoms” was created to overcome the fundamental limitations of its precursor, the “Tangible Bits” vision. Tangible Bits - the physical embodiment of digital information and computation - was constrained by the rigidity of “atoms” in comparison with the fluidity of bits. This makes it difficult to represent fluid digital information in traditionally rigid physical objects, and inhibits dynamic tangible interfaces from being able to control or represent computational inputs and outputs. In order to augment the vocabulary of Tangible User Interfaces or TUIs, we use dynamic representations such as co-located projections or “digital shadows”. However the physical objects on the tabletop stay static and rigid. To overcome these limitations, we began to experiment with a variety of actuated and kinetic tangibles, which can transform their physical positions or shapes as an additional output modality beyond the traditional manual input mode of TUI’s. Our vision of “Radical Atoms” is based on hypothetical, extremely malleable and reconfigurable materials that can be described by real-time digital models so that dynamic changes in digital information can be reflected by a dynamic change in physical state and vice-versa. Bidirectional synchronization is key to making Radical Atoms a tangible but dynamic representation & control of digital information, and enabling new forms of Human Computer Interaction. In this article, we review the original vision and limitations of Tangible Bits and introduce an array of actuated/kinetic tangibles that emerged in the past 10 years of Tangible Media Group’s research to overcome the issue of atoms’ rigidity. Then we illustrate our vision of interactions with Radical Atoms which do not exist today, but may be invented in next 100 years by atom hackers (material scientists, self-organizing nano-robot engineers, etc.) and speculate new interaction techniques and applications which would be enabled by the Radical Atoms.

PingPong++: Community Customization in Games and Entertainment

Xiao Xiao, Michael S. Bernstein, Lining Yao, David Lakatos, Lauren Gust, Kojo Acquah, and Hiroshi Ishii. 2011. PingPong++: community customization in games and entertainment. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology (ACE ‘11), Teresa Romão, Nuno Correia, Masahiko Inami, Hirokasu Kato, Rui Prada, Tsutomu Terada, Eduardo Dias, and Teresa Chambel (Eds.). ACM, New York, NY, USA, , Article 24 , 6 pages. DOI=10.1145/2071423.2071453

DOI: http://doi.acm.org/10.1145/2071423.2071453

Abstract

In this paper, we introduce PingPong++, an augmented ping pong table that applies Do-It-Yourself (DIY) and community contribution principles to the world of physical sports and play. PingPong++ includes an API for creating new visualizations, easily recreateable hardware, an end-user interface for those without programming experience, and a crowd data API for replaying and remixing past games. We discuss a range of contribution domains for PingPong++ and share the design, usage, feedback, and lessons for each domain. We then reflect on our process and outline a design space for community-contributed sports.

ZeroN: Mid-air Tangible Interaction Enabled by Computer-Controlled Magnetic Levitation

Jinha Lee, Rehmi Post, and Hiroshi Ishii. 2011. ZeroN: mid-air tangible interaction enabled by computer controlled magnetic levitation. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ‘11). ACM, New York, NY, USA, 327-336.

DOI: DOI=10.1145/2047196.2047239 http://doi.acm.org/10.1145/2047196.2047239

Abstract

This paper presents ZeroN, a new tangible interface element that can be levitated and moved freely by computer in a three dimensional space. ZeroN serves as a tangible representation of a 3D coordinate of the virtual world through which users can see, feel, and control computation. To accomplish this we developed a magnetic control system that can levitate and actuate a permanent magnet in a predefined 3D volume. This is combined with an optical track- ing and display system that projects images on the levitating object. We present applications that explore this new interaction modality. Users are invited to place or move the ZeroN object just as they can place objects on surfaces. For example, users can place the sun above physical objects to cast digital shadows, or place a planet that will start revolving based on simulated physical conditions. We describe the technology, interaction scenarios and challenges, dis- cuss initial observations, and outline future development.

Rope Revolution: Tangible and Gestural Rope Interface for Collaborative Play

Lining Yao, Sayamindu Dasgupta, Nadia Cheng, Jason Spingarn-Koff, Ostap Rudakevych, and Hiroshi Ishii. 2011. Rope Revolution: tangible and gestural rope interface for collaborative play. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology (ACE ‘11), Teresa Romão, Nuno Correia, Masahiko Inami, Hirokasu Kato, Rui Prada, Tsutomu Terada, Eduardo Dias, and Teresa Chambel (Eds.). ACM, New York, NY, USA, , Article 11 , 8 pages. DOI=10.1145/2071423.2071437 http://doi.acm.org/10.1145/2071423.2071437

DOI: http://doi.acm.org/10.1145/2071423.2071437

Abstract

In this paper we describe Rope Revolution, a rope-based gaming system for collaborative play. After identifying popular rope games and activities around the world, we developed a generalized tangible rope interface that includes a compact motion-sensing and force-feedback module that can be used for a variety of rope-based games. Rope Revolution is designed to foster both co-located and remote collaborative experiences by using actual rope to connect players in physical activities across virtual spaces. Results from this study suggest that a tangible user interface with rich metaphors and physical feedback help enhance the gaming experience in addition to helping remote players feel connected across distances. We use this design as an example to motivate discussion on how to take advantage of the various physical affordances of common objects to build a generalized tangible interface for remote play.

Sourcemap: eco-design, sustainable supply chains, and radical transparency

Leo Bonanni. 2011. Sourcemap: eco-design, sustainable supply chains, and radical transparency. XRDS 17, 4 (June 2011), 22-26. DOI=10.1145/1961678.1961681

DOI: http://doi.acm.org/10.1145/1961678.1961681

Abstract

Industry and consumers need tools to help make decisions that are good for communities and for the environment

Duet for Solo Piano: MirrorFugue for Single User Playing with Recorded Performances

Xiao Xiao and Hiroshi Ishii. 2011. Duet for solo piano: MirrorFugue for single user playing with recorded performances. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA ‘11). ACM, New York, NY, USA, 1285-1290.

DOI: DOI=10.1145/1979742.1979762 http://doi.acm.org/10.1145/1979742.1979762

Abstract

MirrorFugue is an interface that supports symmetric, real-time collaboration on the piano using spatial metaphors to communicate the hand gesture of collaborators. In this paper, we present an extension of MirrorFugue to support single-user interactions with recorded material and outline usage scenarios focusing on practicing and self-reflection. Based on interviews with expert musicians, we discuss how single-user interactions on MirrorFugue relate to larger themes in music learning and suggest directions for future research.

RopePlus: Bridging Distances with Social and Kinesthetic Rope Games

Lining Yao, Sayamindu Dasgupta, Nadia Cheng, Jason Spingarn-Koff, Ostap Rudakevych, and Hiroshi Ishii. 2011. Multi-jump: jump roping over distances. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA ‘11). ACM, New York, NY, USA, 1729-1734. DOI=10.1145/1979742.1979836

DOI: http://doi.acm.org/10.1145/1979742.1979836

Abstract

Rope-based games such as jump rope, tug-of-war, and kite- flying promote physical activity and social interaction among people of all ages and especially in children during the de- velopment of their coordination skills and physical fitness. Our RopePlus system builds on those traditional games by enabling players to participate remotely through interacting with ropes that connect physical and virtual spaces. The RopePlus platform is centered around the rope as a tangible interface with various hardware extensions to allow for multi- ple playing modes. In this paper, we present two games that have been implemented in detail: a kite-flying game called Multi-Fly and a jump-rope game called Multi-Jump. Our work aims to expand tangible interface gaming to real-time social playing environments.

Multi-Jump: Jump Roping Over Distances

Lining Yao, Sayamindu Dasgupta, Nadia Cheng, Jason Spingarn-Koff, Ostap Rudakevych, and Hiroshi Ishii. 2011. Multi-jump: jump roping over distances. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA ‘11). ACM, New York, NY, USA, 1729-1734. DOI=10.1145/1979742.1979836

DOI: http://doi.acm.org/10.1145/1979742.1979836

Abstract

Jump roping, a game in which one or more people twirl a rope while others jump over the rope, promotes social interaction among children while developing their coordination skills and physical fitness. However, the traditional game requires that players be in the same physical location. Our ‘Multi-Jump’ jump-roping game platform builds on the traditional game by allowing players to participate remotely by employing an augmented rope system. The game involves full-body motion in a shared game space and is enhanced with live video feeds, player rewards and music. Our work aims to expand exertion interface gaming, or games that deliberately require intense physical effort, with genuine tangible interfaces connected to real-time shared social gaming environments.

Direct and Gestural Interaction with Relief: A 2.5D Shape Display

Daniel Leithinger, David Lakatos, Anthony DeVincenzi, Matthew Blackshaw, and Hiroshi Ishii. 2011. Direct and gestural interaction with relief: a 2.5D shape display. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ‘11). ACM, New York, NY, USA, 541-548.

DOI: http://doi.acm.org/10.1145/2047196.2047268

Abstract

Actuated shape output provides novel opportunities for experiencing, creating and manipulating 3D content in the physical world. While various shape displays have been proposed, a common approach utilizes an array of linear actuators to form 2.5D surfaces. Through identifying a set of common interactions for viewing and manipulating content on shape displays, we argue why input modalities beyond direct touch are required. The combination of free hand gestures and direct touch provides additional degrees of freedom and resolves input ambiguities, while keeping the locus of interaction on the shape output. To demonstrate the proposed combination of input modalities and explore applications for 2.5D shape displays, two example scenarios are implemented on a prototype system.

Kinected Conference: Augmenting Video Imaging with Calibrated Depth and Audio

Anthony DeVincenzi, Lining Yao, Hiroshi Ishii, and Ramesh Raskar. 2011. Kinected conference: augmenting video imaging with calibrated depth and audio. In Proceedings of the ACM 2011 conference on Computer supported cooperative work (CSCW ‘11). ACM, New York, NY, USA, 621-624. DOI=10.1145/1958824.1958929

DOI: http://doi.acm.org/10.1145/1958824.1958929

Abstract

The proliferation of broadband and high-speed Internet access has, in general, democratized the ability to commonly engage in videoconference. However, current video systems do not meet their full potential, as they are restricted to a simple display of unintelligent 2D pixels. In this paper we present a system for enhancing distance-based communication by augmenting the traditional video conferencing system with additional attributes beyond two-dimensional video. We explore how expanding a system’s understanding of spatially calibrated depth and audio alongside a live video stream can generate semantically rich three-dimensional pixels containing information regarding their material properties and location. We discuss specific scenarios that explore features such as synthetic refocusing, gesture activated privacy, and spatiotemporal graphic augmentation.

MirrorFugue: Communicating Hand Gesture in Remote Piano Collaboration

Xiao Xiao and Hiroshi Ishii. 2010. MirrorFugue: communicating hand gesture in remote piano collaboration. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction (TEI ‘11). ACM, New York, NY, USA, 13-20. DOI=10.1145/1935701.1935705

DOI: http://doi.acm.org/10.1145/1935701.1935705

Abstract

Playing a musical instrument involves a complex set of continuous gestures, both to play the notes and to convey expression. To learn an instrument, a student must learn not only the music itself but also how to perform these bodily gestures. We present MirrorFugue, a set of three interfaces on a piano keyboard designed to visualize hand gesture of a remote collaborator. Based their spatial configurations, we call our interfaces Shadow, Reflection, and Organ. We describe the configurations and detail studies of our designs on synchronous, remote collaboration, focusing specifically on remote lessons for beginners. Based on our evaluations, we conclude that displaying the to-scale hand gestures of a teacher at the locus of interaction can improve remote piano learning for novices.

Recompose: Direct and Gestural Interaction with an Actuated Surface

Matthew Blackshaw, Anthony DeVincenzi, David Lakatos, Daniel Leithinger, and Hiroshi Ishii. 2011. Recompose: direct and gestural interaction with an actuated surface. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA ‘11). ACM, New York, NY, USA, 1237-1242. DOI=10.1145/1979742.1979754

DOI: http://dx.doi.org/10.1145/1979742.1979754

Abstract

In this paper we present Recompose, a new system for manipulation of an actuated surface. By collectively utilizing the body as a tool for direct manipulation alongside gestural input for functional manipulation, we show how a user is afforded unprecedented control over an actuated surface. We describe a number of interaction techniques exploring the shared space of direct and gestural input, demonstrating how their combined use can greatly enhance creation and manipulation beyond unaided human capability.

deFORM: An Interactive Malleable Surface For Capturing 2.5D Arbitrary Objects, Tools and Touch

Sean Follmer, Micah Johnson, Edward Adelson, and Hiroshi Ishii. 2011. deForm: an interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ‘11). ACM, New York, NY, USA, 527-536.

DOI: DOI=10.1145/2047196.2047265 http://doi.acm.org/10.1145/2047196.2047265

Virtual Guilds: Collective Intelligence and the Future of Craft

BONANNI, L. AND PARKES, A. Virtual Guilds: Collective Intelligence and the Future of Craft. The Journal of Modern Craft, Volume 3, Number 2, July 2010 , pp. 179-190(12)

DOI: http://dx.doi.org/10.2752/174967810X12774789403564

Abstract

In its most basic definition, craft refers to the skilled practice of making things, which is shaped as much by technological advancements as by cultural practices. Richard Sennett discusses the nature of craftsmanship as an enduring, basic human impulse, the desire to do a job well for its own sake.1 This encompasses a much broader context than skilled labor and promotes an objective standard of excellence which incorporates shapers of culture, policy, and technology as craftsmen. The emerging nature of craft is transdisciplinary in its formation and must consider how emerging materials, processes and cultures influence the objects we make and how the processes of design and production can be used to reflect new social values and to change cultural practices. In order to re-think the kind of objects we make, it is necessary to rethink the way we craft our objects. Digital technologies and media are defining a new sort of craft, seamlessly blending technology, design, and production into the post-industrial landscape. As an early pioneer in redefining craftsmanship to include digital processes, Malcolm McCullough explored the computer as a craft medium inviting interpretation and subltleties, with the combined skill sets of the machine and the human (both mind and hands) providing a structured system of transformations resulting in a crafted object.2 The nature of digital technologies also allows craft to evolve into a form which is decentralized and distributed, and can give rise to excellence through a collective desire and a combined multiplicity of knowledge through community Craft is inherently a social activity, shaped by communal resources and motivations. The collective approach of craft communities - or guilds - is characterized by the master-apprentice model, where practitioners devote significant time passing on their skills to the next generation. The open source software movement embodies the communal character and the highly skilled practices of craft guilds. Until recently skilled handicraft relied on hands-on teaching and access to local physical resources. Mass media and the internet make it possible to transmit skills and resources to isolated individuals, making possible entirely new kinds of distributed craft communities. These “Virtual Guilds” form at the margins of established domains, extending the reach of specialized knowledge and technology. Virtual Guilds benefit from the free exchange of expert information to bring about innovation in sometimes neglected domains. The growth of open-source software projects provides the model by which dispersed, collective innovation becomes possible in other domains. Shared resources maintained by a socially motivated community form the backbone of these non-commercial efforts. Digital channels of communication can extend this free exchange of information to the domain of craft, so that specialized designs and processes can be shared among a wide audience. Online distribution provides access to rare materials and tools and provides a market for craft products. Several Virtual Guilds exist today, and they are contributing important inventions and new domains to often neglected markets. These communities of skilled practitioners are characterized by their marginal nature, where the free and open exchange of ideas is carried forward for collective benefit. At the same time, the popularity of Virtual Guilds and the commercial success of their inventions endanger the free exchange of information on which they are built. The survival of collective craft communities is important to under-served groups and for technological innovation, so it is essential that more practitioners engage in collective action. The new generation of digital design and fabrication tools lays the groundwork for more skilled craftspeople to collectively expand on their practice.

Construction by replacement: a new approach to simulation modeling

James Hines, Thomas Malone, Paulo Gonçalves, George Herman, John Quimby, Mary Murphy-Hoye, James Rice, James Patten, Hiroshi Ishii, Syst. Dyn. Rev. July 2010

DOI: http://dx.doi.org/10.1002/sdr.437

Abstract

Simulation modeling can be valuable in many areas of management science, but it is often costly, time consuming, and difficult to do. To reduce these problems, system dynamics researchers have previously developed standard pieces of model structure, called molecules, that can be reused in different models. However, the models assembled from these molecules often lacked feedback loops and generated few, if any, insights. This paper describes a new and more promising approach to using molecules in system dynamics modeling. The heart of the approach is a systematically organized library (or taxonomy) of predefined model components, or molecules, and a set of software tools for replacing one molecule with another. Users start with a simple generic model and progressively replace parts of the model with more specialized molecules from a systematically organized library of predefined components. These substitutions either create a new running model automatically or request further manual changes from the user. The paper describes our exploration using this approach to construct system dynamics models of supply chain processes in a large manufacturing company. The experiment included developing an innovative “tangible user interface” and a comprehensive catalog of system dynamics molecules. The paper concludes with a discussion of the benefits and limitations of this approach. Copyright © 2010 John Wiley & Sons, Ltd. Syst. Dyn. Rev. (2009)

Small Business Applications of Sourcemap: A Web Tool for Sustainable Design and Supply Chain Transparency

Bonanni, L., Hockenberry, M., Zwarg, D., Csikszentmihalyi, C., and Ishii, H. 2010. Small business applications of sourcemap: a web tool for sustainable design and supply chain transparency. In Proceedings of the 28th international Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA, April 10 - 15, 2010). CHI ‘10. ACM, New York, NY, 937-946.

DOI: http://doi.acm.org/10.1145/1753326.1753465

Abstract

This paper introduces sustainable design applications for small businesses through the Life Cycle Assessment and supply chain publishing platform Sourcemap.org. This web-based tool was developed through a year-long participatory design process with five small businesses in Scotland and in New England. Sourcemap was used as a diagnostic tool for carbon accounting, design and supply chain management. It offers a number of ways to market sustainable practices through embedded and printed visualizations. Our experiences confirm the potential of web sustainability tools and social media to expand the discourse and to negotiate the diverse goals inherent in social and environmental sustainability.

Beyond: collapsible tools and gestures for computational design

Jinha Lee and Hiroshi Ishii. 2010. Beyond: collapsible tools and gestures for computational design. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems (CHI EA ‘10). ACM, New York, NY, USA, 3931-3936. DOI=10.1145/1753846.1754081

DOI: http://doi.acm.org/10.1145/1753846.1754081

Abstract

Since the invention of the personal computer, digital media has remained separate from the physical world, blocked by a rigid screen. In this paper, we present Beyond, an interface for 3-D design where users can directly manipulate digital media with physically retractable tools and hand gestures. When pushed onto the screen, these tools physically collapse and project themselves onto the screen, letting users perceive as if they were inserting the tools into the digital space beyond the screen. The aim of Beyond is to make the digital 3-D design process straightforward, and more accessible to general users by extending physical affordances to the digital space beyond the computer screen.

Tangible Interfaces for Art Restoration

BONANNI, L., SERACINI, M., XIAO, X., HOCKENBERRY, M., COSTANZO, B.C., SHUM, A., TEIL, R., SPERANZA, A., AND ISHII, H.2010. Tangible Interfaces for Art Restoration. International Journal of Creative Interfaces and Computer Graphics 1, 54-66. DOI: 10.4018/jcicg.2010010105

DOI: http://www.igi-global.com/bookstore/article.aspx?titleid=41711

Abstract

Few people experience art the way a restorer does: as a tactile, multi-dimensional and ever-changing object. The authors investigate a set of tools for the distributed analysis of artworks in physical and digital realms. Their work is based on observation of professional art restoration practice and rich data available through multi-spectral imaging. The article presents a multidisciplinary approach to develop interfaces usable by restorers, students and amateurs. Several interaction techniques were built using physical metaphors to navigate the layers of information revealed by multi-spectral imaging, prototyped using single- and multi-touch displays. The authors built modular systems to accommodate the technical needs and resources of various institutions and individuals, with the aim to make high-quality art diagnostics possible on different hardware platforms, as well as rich diagnostic and historic information about art available for education and research through a cohesive set of web-based tools instantiated in physical interfaces and public installations.

Relief: A Scalable Actuated Shape Display

Leithinger, D. and Ishii, H. 2010. Relief: a scalable actuated shape display. In Proceedings of the Fourth international Conference on Tangible, Embedded, and Embodied interaction (Cambridge, Massachusetts, USA, January 24 - 27, 2010). TEI ‘10. ACM, New York, NY, 221-222.

DOI: http://doi.acm.org/10.1145/1709886.1709928

Abstract

Relief is an actuated tabletop display, which is able to render and animate three-dimensional shapes with a malleable surface. It allows users to experience and form digital models like geographical terrain in an intuitive manner. The tabletop surface is actuated by an array of 120 motorized pins, which are controlled with a low-cost, scalable platform built upon open-source hardware and software tools. Each pin can be addressed individually and senses user input like pulling and pushing.

g-stalt: a chirocentric, spatiotemporal, and telekinetic gestural interface.

Jamie Zigelbaum, Alan Browning, Daniel Leithinger, Olivier Bau, and Hiroshi Ishii. 2010. g-stalt: a chirocentric, spatiotemporal, and telekinetic gestural interface. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI ‘10). ACM, New York, NY, USA, 261-264. DOI=10.1145/1709886.1709939 http://doi.acm.org/10.1145/1709886.1709939

DOI: http://dl.acm.org/citation.cfm?id=1709939

Abstract

In this paper we present g-stalt, a gestural interface for interacting with video. g-stalt is built upon the g-speak spatial operating environment (SOE) from Oblong Industries. The version of g-stalt presented here is realized as a three-dimensional graphical space filled with over 60 cartoons. These cartoons can be viewed and rearranged along with their metadata using a specialized gesture set. g- stalt is designed to be chirocentric, spatiotemporal, and telekinetic.

Play it by eye, frame it by hand! Gesture Object Interfaces to enable a world of multiple projections.

Cati Vaucelle. Play it by eye, frame it by hand! Gesture Object Interfaces to enable a world of multiple projections.Thesis (Ph. D.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2010.

Abstract

Tangible Media as an area has not explored how the tangible handle is more than a marker or place-holder for digital data. Tangible Media can do more. It has the power to materialize and redefine our conception of space and content during the creative process. It can vary from an abstract token that represents a movie to an anthropomorphic plush that reflects the behavior of a sibling during play. My work begins by extending tangible concepts of representation and token-based interactions into movie editing and play scenarios. Through several design iterations and research studies, I establish tangible technologies to drive visual and oral perspectives along with finalized creative works, all during a child’s play and exploration.

I define the framework, Gesture Object Interfaces, expanding on the fields of Tangible User Interaction and Gesture Recognition. Gesture is a mechanism that can reinforce or create the anthropomorphism of an object. It can give the object life. A Gesture Object is an object in hand while doing anthropomorphized gestures. Gesture Object Interfaces engender new visual and narrative perspectives as part of automatic film assembly during children’s play. I generated a suite of automatic film assembly tools accessible to diverse users. The tools that I designed allow for capture, editing and performing to be completely indistinguishable from one another. Gestures integrated with objects become a coherent interface on top of natural play. I built a distributed, modular camera environment and gesture interaction to control that environment. The goal of these new technologies is to motivate children to take new visual and narrative perspectives.

In this dissertation I present four tangible platforms that I created as alternatives to the usual fragmented and sequential capturing, editing and performing of narratives available to users of current storytelling tools. I developed Play it by Eye, Frame it by hand, a new generation of narrative tools that shift the frame of reference from the eye to the hand, from the viewpoint (where the eye is) to the standpoint (where the hand is). In Play it by Eye, Frame it by Hand environments, children discover atypical perspectives through the lens of everyday objects. When using Picture This!, children imagine how an object would appear relative to the viewpoint of the toy. They iterate between trying and correcting in a world of multiple perspectives. The results are entirely new genres of child-created films, where children finally capture the cherished visual idioms of action and drama. I report my design process over the course of four tangible research projects that I evaluate during qualitative observations with over one hundred 4- to 14-year-old users. Based on these research findings, I propose a class of moviemaking tools that transform the way users interpret the world visually, and through storytelling.

WOW pod

Vaucelle, C., Shada, S., and Jahn, M. 2010. WOW pod. In Proceedings of the 28th of the international Conference Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA, April 10 - 15, 2010). CHI EA ‘10. ACM, New York, NY, 4813-4816. DOI= http://doi.acm.org/10.1145/1753846.1754237

DOI: http://doi.acm.org/10.1145/1753846.1754237

Abstract

WOW Pod is an immersive architectural solution for the advanced massive online role-playing gamer that provides and anticipates all life needs. Inside, the player finds him/herself comfortably seated in front of the computer screen with easy-to-reach water, pre-packaged food, and a toilet conveniently placed underneath a built-in throne.

Trackmate: Large-Scale Accessibility of Tangible User Interfaces

Adam Kumpf. Trackmate: Large-Scale Accessibility of Tangible User Interfaces. Thesis (M.S.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2009.

Abstract

There is a long history of Tangible User Interfaces (TUI) in the community of human- computer interaction, but surprisingly few of these interfaces have made it beyond lab and gallery spaces. This thesis explores how the research community may begin to remedy the disconnect between modern TUIs and the everyday computing experience via the creation and dissemination of Trackmate, an accessible (both ubiquitous and enabling) tabletop tangible user interface that scales to a large number of users with minimal hardware and configuration overhead. Trackmate is entirely open source and designed: to be community-centric; to leverage common objects and infrastructure; to provide a low floor, high ceiling, and wide walls for development; to allow user mod- ifications and improvisation; to be shared easily via the web; and to work alongside a broad range of existing applications and new research interface prototypes.

Ubiquitous Computing in Chaos and Tangible Bits (in Japanese) 混迷するユビキタスの未来とタンジブル・ビット

Hiroshi Ishii, Ubiquitous Computing in Chaos and Tangible Bits (in Japanese), Next Generation Service Forum, Focus Column, June 19, 2009,

Abstract

混迷するユビキタスの未来とタンジブル・ビット ■宣伝色に染まった「ユビキタス」 　「ユビキタス」（Ubiquitous）の文脈は今ひどく混迷している。 　「至る所にある」というユビキタスの辞書的な意味が転じて、日本のメディアでは「いつでも・どこでも」ネットアクセスできる多機能モバイル・コンピューティングという意味で使われているように思える。各人がたくさんのコンピュータを使うユビキタスの時代──という未来像は、小型情報通信機器の販売を促進する旗としては、確かに有効な宣伝コピーではある。しかし、「いつでも・どこでも」（any time, any place）は、かつて80年代の「高度情報化社会」や90年代の「マルチメディア社会」が華々しく論じられたときに、何度も出てきた使い古されたスローガンだった。では、「ユビキタス」も結局はこれらのスローガンと同様、一時的な流行歌に過ぎず、時が経てばすぐ忘れ去られてしまうのか。ユビキタスに未来はないのか。それは、ユビキタスの原点の理解にかかっていると思う。 ■ユビキタス・コンピューティングの原点 　今は亡きマーク・ワイザー（Mark Weiser）がユビキタス・コンピューティングの概念を「The Computer for the 21st Century」（21世紀のコンピュータ）と題した論文として「サイエンティフィック・アメリカン」誌に発表したのは1991年だった。「最も深い技術とは、見えなくなるものである。日々の生活環境と区別がつかないほど、その中に溶け込む。」※1この冒頭の文章が、ユビキタス・コンピューティングの精神と哲学を最も明快に示している。コンピュータが「環境にすっかり溶け込み消えてしまう」ことがそのビジョンであり、それはまさにテクノロジーの浸透した世界を人間がどのように認知するのかという観点から、インターフェースの理想を描いたものである。しかし残念ながら、この論文の中で紹介されていた具体的なプロトタイプ（大小さまざまなスクリーン付きの情報通信端末）は、彼の理念である「見えないコンピュータ」を十分に説明するレベルには達していなかった。彼がこの論文の中で使った「ユビキタス」という言葉は、彼が本当に実現したいビジョンを明快に表すものではなかった。 ■マーク・ワイザーからのメール 　実際、本人自身もユビキタス・コンピューティングについての誤解をかなり気にかけていたようだ。’97年1月26日に「Tangible Bits」（タンジブル・ビット）の論文※2を読んだワイザーから、私が受け取ったメールの一部※3をここに紹介する。 　ひとつお願いがあります。テニュアを得た元教授として、それまでのすべての研究から自身の研究を際立たせる必要があることを、よく理解しています。また、私に対する献辞にも、とても感謝しています。ありがとう！ 　私のお願いは、ユビキタス・コンピューティングについて、その名称から生じた誤解が広まるのを防ぐこと、これをあなたに手伝ってほしいのです。ユビキタス・コンピューティングは、「コンピュータ」をユビキタスにすることだけでは決してありません。これは、あなたの研究のように、コンピュータをメディアとして環境に溶け込ませるということだったのです。 （中略） 　誤解を招きやすいため、私はユビキタス・コンピューティングという言い方をやめようとしてきました。しかし、その後も幾度にわたってこの言葉が重くのしかかってきます。そこで、Things That Thinkなどを含み、私がかかわっているさまざまな仕事すべての総称として、これを使うようにしていました。拡張現実という言葉をしばらく使ったこともありますが、これもまた異なった意味合いが生み出されてしまいました。私はテーマとしてカーム・テクノロジーという言葉を用い始めましたが、これは研究プロジェクトというよりも、その目的にふさわしいものです。「タンジブル・ビット」はとても素晴らしく、総称にふさわしいかもしれませんが、そうするとあなたの研究プロジェクトを示す名前ではなくなってしまいます！ 　われわれが何かひとつの共通な事柄を掲げ、その下でそれぞれの差異を規定していけば、みんなにとって得になるのではないでしょうか。もっとも、それを何と呼ぶかに苦心するでしょうが。 　とにかく、これはすばらしい仕事です。近いうちにMITを訪問して話をするのを楽しみにしています。 ─ マーク ■ネーミングの重要性 　ワイザーが私の「タンジブル・ビット」のビジョンに共感を覚え、さらにそれが彼のユビキタス・コンピューティングの思想と深い地下水脈を通じてつながっていることを明快に指摘してくれたことに、私は強い知的興奮を覚えた。彼がメールの中で述べたように、「ユビキタス」という言葉は、彼の思想を表現するには不適切なラベルだったのだと思う。本来の意味を十分理解されず、マーケティングの文脈で濫用されているラベル「ユビキタス」は、残念ながら、かつてはやった「ニューメディア」や「マルチメディア」のように、一過性の流行歌のような運命にあると思われる。もしワイザーが生きていたなら、おそらく彼は「ユビキタス・コンピューティング」というラベルを捨て、「見えないコンピュータ」をコア・コンセプトとした新しいビジョンを作り上げていたに違いない。 　コンピュータが1人あたり何台あるか、それが分散しているか、集中しているか、携帯型か、環境埋め込み型か──それらは、彼の究極のインターフェース理念とは本来無関係のはずであった。後にワイザーは、「Calm Technology」（穏やかな技術）という言葉を使って環境的（アンビエント）なインターフェースを強調しようとしたが、「遍在するコンピュータ」と誤解されたユビキタス・コンピューティングは、すでにひとり歩きを始めていた。 ■タンジブル・ビットとは 　私が提唱している「タンジブル・ユーザー・インターフェース」(Tangible User Interface [TUI]) の目的は、物理世界における人とモノとのインタラクションをベースにして、コンピュータの内部にあるデジタル情報とのインタラクションをシームレスに融合することにある。TUIの基本定義と特徴は、直接触れて操作できるインターフェースのデザインであり、’96年から多様なTUIプロトタイプを発表してきた。最も純粋なタンジブルと呼べる入出力一体型TUIとは、入力と出力とが完全に一体化しており、人に力覚情報を提示するフォース・ディスプレー技術によって、実際の動きを表現出力に用いている。直接触れることのできない「インタンジブル」な表現はなく、すべてがタンジブルな物理的媒体のみで成り立っている点が「純粋なTUI」たるゆえんだ。それによって物理世界そのものをデジタル世界とのインターフェースにする道が開け、現在の複雑かつ不透明なデジタル世界への窓口を、認知的に「透明」にすることが可能になるのだ。 　これらのTUIは、概念的にはそろばんに近い。そろばんは、10進数という情報を珠の位置関係で物理的に表現する。10本の指でその珠に直接触れ、情報を操作・計算する。そこには、入力と出力の境界は存在しない。情報表現と操作手段が密に結合した物理的インターフェースの明快さ、直接性がそろばんの特徴である。TUIはこのそろばんの特徴に、デジタル情報の表示手段としてフォース・ディスプレーを加味し、ビット（デジタル情報）をアトム（物理的世界）の着物でつつみ込んだことに特徴がある。 ■透明なインターフェースの追求 　私が’99年に発表した「musicBottles」（ミュージックボトル）は、ワイザーの哲学のコアである透明なインターフェースの概念を自分なりに解釈して形にした TUI の代表例である。これは、私からマーク・ワイザーにささげる贈り物だった。 　ミュージックボトルは、この透明なインターフェースのコンセプトを可視化・可触化し、モノの持つメタファーに加えて情緒的・審美的な価値にも注目しつつ、「ミニマル・デザイン」を意図的に追求した作品である。 　人類が数千年に渡って使ってきたガラス製のボトル。そのメタファーとアフォーダンスをデジタル世界に拡張することにより、ミュージックボトル・プロジェクトはインターフェースの透明性（transparency）を追求する。ガラスボトルをデジタル情報のコンテナおよびコントローラーとして使い、フタの開け閉めという単純な操作だけでデジタルコンテンツへのアクセスを実現するシンプルなインターフェース。 　このプロジェクトの原点は、私が母親への贈り物として温めていた「天気予報の小瓶」のアイデアだった。台所で料理をしている最中、しょうゆ瓶のフタを開けるとしょうゆの香りが漂ってくる──彼女が慣れ親しんだ物理世界のモデルをベースに、天気予報というデジタル情報へアクセスするための青い小瓶をデザインしようと考えていたのだ。朝起きて枕元にある青い小瓶のフタを開ける。小鳥のさえずりが聞こえれば天気は晴れ、雨の音が聞こえてくれば、雨天というアイデアだ。 　しかし、’98年の夏の終わりに長い闘病生活を続けてきた母が亡くなり、天気予報の小瓶をプレゼントする機会は永遠に失われてしまった。その年の暮れ、私と当時博士課程に在学していたリッチ・フレッチャーとのディスカッションからミュージックボトルのアイデアが生まれ、母への追悼の意味も込めてこのプロジェクトを開始することにした。翌’99年4月27日には敬愛するマーク・ワイザーが急逝し、彼が私に残した言葉がこれにさらなる個人的な意味合いを添える。 　パソコンや携帯電話が登場するはるか昔から人類の日常生活に遍在していたガラス瓶に、デジタルコンテンツを詰めることで、ミニマルかつユニバーサルな情報へのインターフェースを実現する。この可能性は、音楽コンテンツに限定されるものではない。例えば天気予報の小瓶はもちろん、詩の入った香水の瓶、物語の入ったワインボトルなど、多様な応用が考えられるだろう。実用性を追求するなら、薬瓶がたくさん置かれた棚を想像してほしい。薬の服用パターンと照らし合わせて患者に服用を促す、その情報を病院へ送るなど、ガラス瓶を使ったサービスはいくらでも考えられる。私たちの生活の奥深くに浸透しているが故に、ガラス瓶のインターフェースには数多くの用途が広がっているのだ。 　デザインされたテーブル上のボトル、それを開けるときのガラスの感触、流れ出る音楽に同期してボトルの中で乱反射するLEDの光──それらは、独特の情緒的・審美的な体験を作り上げる。審美的な喜びは、単純なスイッチやマウスのクリックからは決して得ることができない。そして、この体験はあらゆるガラス瓶の中に入り得るコンテンツを想像するという喜びももたらしてくれる。インタラクティブ・アートとインターフェース・デザインとの境界線をも、あいまいにするのだ。 　人々の日常生活に溶け込む「透明なインターフェース」に加え、機能や性能が中心の従来型インターフェース・デザインとは異なる美的価値の追求が、ミュージックボトル・プロジェクトの大切なメッセージなのである。 石井 裕（MIT Media Lab 副所長） ※1. 【原文】 「The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.」（Weiser, M. The Computer for the 21st Century. Scientific American, 1991, 265(3), pp. 94-104. ※2. Ishii, H. and Ullmer, B., Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms, in Proceedings of Conference on Human Factors in Computing Systems (CHI ‘97), (Atlanta, March 1997), ACM Press, pp. 234-241.

MIT: Creation, Collaboration and Competition (in Japanese) 米国MITの独創・協創・競創の風土

Ishii, H. 2009. MIT: Creation, Collaboration and Competition (in Japanese). In the Journal of the IEICE (Institute of Electronics, Information, and Communication Engineers). Vol. 92. No. 5, pp.327-331

Abstract

1994年にMITにヘッドハントされ、1995年にNTTの研究所からMITメディアラボに転身。「これまでの研究を捨て、新しいテーマで再出発すること」を条件にメディラボ準教授に採用されてから、生き残りをかけた「競創」の戦いが始まった。 オリジナリティを徹底的に追求しながら、同時に大きなインパクトを作り出すための戦略、テニュア取得という大きなプレッシャーに耐えながらの全力疾走、テニュア取得後のラボ全体のための研究資金調達のプレッシャーなど、MITでの過去１４年間の体験は、米国の競創社会の縮図だと言える。本稿では、独創的研究で世界を目指す読者に向けて、私の体験をまとめてみた。

Wetpaint: Scraping Through Multi-Layered Images

Bonanni, L., Xiao, X., Hockenberry, M., Subramani, P., Ishii, H., Seracini, M., and Schulze, J. 2009. Wetpaint: scraping through multi-layered images. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI ‘09. ACM, New York, NY, 571-574.

DOI: http://doi.acm.org/10.1145/1518701.1518789

Abstract

A work of art rarely reveals the history of creation and interpretation that has given it meaning and value. Wetpaint is a gallery interface based on a large touch screen that allows curators and museumgoers to investigate the hidden layers of a painting, and in the process contribute to the pluralistic interpretation of the piece, both locally and online. Inspired by traditional restoration and curatorial methods, we have designed a touch-based user interface for exhibition spaces that allows “virtual restoration” by scraping through the multi-spectral scans of a painting, and “collaborative curation” by leaving voice annotations within the artwork. The system functions through an online social image network for flexibility and to support rich and collaborative commentary for local and remote visitors

Burn Your Memory Away: One-time Use Video Capture and Storage Device to Encourage Memory Appreciation

Chi, P., Xiao, X., Chung, K., and Chiu, C. 2009. Burn your memory away: one-time use video capture and storage device to encourage memory appreciation. In Proceedings of the 27th international Conference Extended Abstracts on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI EA ‘09. ACM, New York, NY, 2397-2406.

DOI: http://doi.acm.org/10.1145/1520340.1520342

Abstract

Although modern ease of access to technology enables many of us to obsessively document our lives, much of the captured digital content is often disregarded and forgotten on storage devices, with no concerns of cost or decay. Can we design technology that helps people better appreciate captured memories? What would people do if they only had one more chance to relive past memories? In this paper, we present a prototype design, PY-ROM, a matchstick-like video recording and storage device that burns itself away after being used. This encourages designers to consider lifecycles and human-computer relationships by integrating physical properties into digitally augmenting everyday objects.

Stress OutSourced: A Haptic Social Network via Crowdsourcing

Chung, K., Chiu, C., Xiao, X., and Chi, P. 2009. Stress outsourced: a haptic social network via crowdsourcing. In Proceedings of the 27th international Conference Extended Abstracts on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI EA ‘09. ACM, New York, NY, 2439-2448.

DOI: http://doi.acm.org/10.1145/1520340.1520346

Abstract

Stress OutSourced (SOS) is a peer-to-peer network that allows anonymous users to send each other therapeutic massages to relieve stress. By applying the emerging concept of crowdsourcing to haptic therapy, SOS brings physical and affective dimensions to our already networked lifestyle while preserving the privacy of its members. This paper first describes the system, its three unique design choices regarding privacy model, combining mobility and scalability, and affective communication for an impersonal crowd, and contrasts them with other efforts in their respective areas. Finally, this paper describes future work and opportunities in the area of haptic social networks.

Some Challenges for Designers of Shape Changing Interfaces

Zigelbaum, J., Labrune, J.B. Some Challenges for Designers of Shape Changing Interfaces. CHI 2009 Workshop on Transitive Materials (2009).

Abstract

In this paper we describe some challenges we find in the design of shape changing user interfaces through our own work and thoughts on the current state of the art in HCI. Due to the large set of possibilities for shape changing materials we are faced with a too-large constraint system. Without a good understanding and the beginning of a standardization or physical language for shape change it will be hard to design interactions that make sense beyond those in very limited, one-off applications. We are excited by the challenge that this poses to researchers and look forward to understanding how to use programmable and shape changing materials in the future.

Fusing Computation into Mega-Affordance Objects

Chung, K., Ishii, H., Fusing computation into mega-affordance objects. CHI 2009 Workshop on Transitive Materials (2009).

Abstract

In this paper, I present the concept of “Mega- Affordance Objects” (MAOs). An MAO is a common object with a primitive form factor that exhibits multiple affordances and can perform numerous improvised functions in addition to its original one. In order to broaden the reach of Tangible User Interfaces (TUIs) and create compelling everyday applications, I propose applying computational power to Mega-Affordance Objects that are highly adaptable and frequently used. This approach will leverage the capabilities of smart materials and contribute to the principles of Organic User Interface (OUI) design.

Spime Builder: A Tangible Interface for Designing Hyperlinked Objects

Bonanni, L., Vargas, G., Chao, N., Pueblo, S., and Ishii, H. 2009. Spime builder: a tangible interface for designing hyperlinked objects. In Proceedings of the 3rd international Conference on Tangible and Embedded interaction (Cambridge, United Kingdom, February 16 - 18, 2009). TEI ‘09. ACM, New York, NY, 263-266. DOI= http://doi.acm.org/10.1145/1517664.1517719

DOI: http://doi.acm.org/10.1145/1517664.1517719

Abstract

Ubiquitous computing is fostering an explosion of physical artifacts that are coupled to digital information – so-called Spimes. We introduce a tangible workbench that allows for the placement of hyperlinks within physical models to couple physical artifacts with located interactive digital media. A computer vision system allows users to model three-dimensional objects and environments in real-time using physical materials and to place hyperlinks in specific areas using laser pointer gestures. We present a working system for real-time physical/digital exhibit design, and propose the means for expanding the system to assist Design for the Environment strategies in product design.

Proverbial Wallet: Tangible Interface for Financial Awareness

Kestner, J., Leithinger, D., Jung, J., and Petersen, M. 2009. Proverbial wallet: tangible interface for financial awareness. In Proceedings of the 3rd international Conference on Tangible and Embedded interaction (Cambridge, United Kingdom, February 16 - 18, 2009). TEI ‘09. ACM, New York, NY, 55-56.

DOI: http://doi.acm.org/10.1145/1517664.1517683

Abstract

We propose a tangible interface concept for communicating personal financial information in an ambient and relevant manner. The concept is embodied in a set of wallets that provide the user with haptic feedback about personal financial metrics. We describe how such feedback can inform purchasing decisions and improve general financial awareness.

Stop-Motion Prototyping for Tangible Interfaces

Bonanni, L. and Ishii, H. 2009. Stop-motion prototyping for tangible interfaces. In Proceedings of the 3rd international Conference on Tangible and Embedded interaction (Cambridge, United Kingdom, February 16 - 18, 2009). TEI ‘09. ACM, New York, NY, 315-316.

DOI: http://doi.acm.org/10.1145/1517664.1517729

Abstract

Stop-motion animation brings the constraints of the body, space and materials into video production. Building on the tradition of video prototyping for interaction design, stop motion is an effective technique for concept development in the design of Tangible User Interfaces. This paper presents a framework for stop-motion prototyping and the results of two workshops based on stop-motion techniques including pixillation, claymation and time-lapse photography. The process of stop-motion prototyping fosters collaboration, legibility and rapid iterative design in a physical context that can be useful to the early stages of tangible interaction design.

Design of Haptic Interfaces for Psychotherapy

Vaucelle, C., Bonanni, L., and Ishii, H. 2009. Design of haptic interfaces for therapy. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI ‘09. ACM, New York, NY, 467-470.

DOI: http://doi.acm.org/10.1145/1518701.1518776

Abstract

Touch is fundamental to our emotional well-being. Medical science is starting to understand and develop touch-based therapies for autism spectrum, mood, anxiety and borderline disorders. Based on the most promising touch therapy protocols, we are presenting the first devices that simulate touch through haptic devices to bring relief and assist clinical therapy for mental health. We present several haptic systems that enable medical professionals to facilitate the collaboration between patients and doctors and potentially pave the way for a new form of non-invasive treatment that could be adapted from use in care-giving facilities to public use. We developed these prototypes working closely with a team of mental health professionals.

Play-it-by-eye! Collect Movies and Improvise Perspectives with Tangible Video Objects.

Vaucelle, C. and Ishii, H. 2009. Play-it-by-eye! Collect movies and improvise perspectives with tangible video objects. In Artificial Intelligence for Engineering Design, Analysis and Manufacturing (2009), Special Issue: Tangible Interaction, 23, 305–316. Cambridge University Press.

DOI: http://portal.acm.org/citation.cfm?id=1560677.1560685&coll=ACM&dl=GUIDE&CFID=48339682&CFTOKEN=69593101

Abstract

We present an alternative video-making framework for children with tools that integrate video capture with movie production. We propose different forms of interaction with physical artifacts to capture storytelling. Play interactions as input to video editing systems assuage the interface complexities of film construction in commercial software. We aim to motivate young users in telling their stories, extracting meaning from their experiences by capturing supporting video to accompany their stories, and driving reflection on the outcomes of their movies. We report on our design process over the course of four research projects that span from a graphical user interface to a physical instantiation of video. We interface the digital and physical realms using tangible metaphors for digital data, providing a spontaneous and collaborative approach to video composition. We evaluate our systems during observations with 4- to 14-year-old users and analyze their different approaches to capturing, collecting, editing, and performing visual and sound clips.

Cost-effective Wearable Sensor to Detect EMF

Vaucelle, C., Ishii, H., and Paradiso, J. A. 2009. Cost-effective wearable sensor to detect EMF. In Proceedings of the 27th international Conference Extended Abstracts on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI’09. ACM, New York, NY, 4309-4314.

DOI: http://doi.acm.org/10.1145/1520340.1520658

Abstract

In this paper we present the design of a cost-effective wearable sensor to detect and indicate the strength and other characteristics of the electric field emanating from a laptop display. Our Electromagnetic Field Detector Bracelet can provide an immediate awareness of electric fields radiated from an object used frequently. Our technology thus supports awareness of ambient background emanation beyond human perception. We discuss how detection of such radiation might help to “fingerprint” devices and aid in applications that require determination of indoor location.

Sculpting Behavior A Tangible Language for Hands-On Play and Learning

Hayes Raffle. Sculpting Behavior A Tangible Language for Hands-On Play and Learning. Thesis (Ph. D.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2008.

DOI: http://hdl.handle.net/1721.1/44912

Abstract

For over a century, educators and constructivist theorists have argued that children learn by actively forming and testing – constructing – theories about how the world works. Recent efforts in the design of “tangible user interfaces” (TUIs) for learning have sought to bring together interaction models like direct manipulation and pedagogical frameworks like constructivism to make new, often complex, ideas salient for young children. Tangible interfaces attempt to eliminate the distance between the computational and physical world by making behavior directly manipulable with one’s hands. In the past, systems for children to model behavior have been either intuitive-but-simple (e.g. curlybot) or complex-but-abstract, (e.g. LEGO Mindstorms). In order to develop a system that supports a user’s transition from intuitive-but-simple constructions to constructions that are complex-but-abstract, I draw upon constructivist educational theories, particularly Bruner’s theories of how learning progresses through enactive then iconic and then symbolic representations. This thesis presents an example system and set of design guidelines to create a class of tools that helps people transition from simple-but-intuitive exploration to abstract-and-flexible exploration. The Topobo system is designed to facilitate mental transitions between different representations of ideas, and between different tools. A modular approach, with an inherent grammar, helps people make such transitions. With Topobo, children use enactive knowledge, e.g. knowing how to walk, as the intellectual basis to understand a scientific domain, e.g. engineering and robot locomotion. Queens, backpacks, Remix and Robo add various abstractions to the system, and extend the tangible interface. Children use Topobo to transition from hands-on knowledge to theories that can be tested and reformulated, employing a combination of enactive, iconic and symbolic representations of ideas.

SpeakCup: Simplicity, BABL, and Shape Change

Zigelbaum, J., Chang, A., Gouldstone, J., Monzen, J. J., and Ishii, H. 2008. SpeakCup: simplicity, BABL, and shape change. In Proceedings of the 2nd international Conference on Tangible and Embedded interaction (Bonn, Germany, February 18 - 20, 2008). TEI ‘08. ACM, New York, NY, 145-146.

DOI: http://doi.acm.org/10.1145/1347390.1347422

Abstract

In this paper we present SpeakCup, a simple tangible interface that uses shape change to convey meaning in its interaction design. SpeakCup is a voice recorder in the form of a soft silicone disk with embedded sensors and actuators. Advances in sensor technology and material science have provided new ways for users to interact with computational devices. Rather than issuing commands to a system via abstract and multi-purpose buttons the door is open for more nuanced and application-specific approaches. Here we explore the coupling of shape and action in an interface designed for simplicity while discussing some questions that we have encountered along the way.

Picture This! Film assembly using toy gestures

Vaucelle, C. and Ishii, H. 2008. Picture this!: film assembly using toy gestures. In Proceedings of the 10th international Conference on Ubiquitous Computing (Seoul, Korea, September 21 - 24, 2008). UbiComp ‘08, vol. 344. ACM, New York, NY, 350-359.

DOI: http://doi.acm.org/10.1145/1409635.1409683

Abstract

We present Picture This! a new input device embedded in children’s toys for video composition. It consists of a new form of interaction for children’s capturing of storytelling with physical artifacts. It functions as a video and storytelling performance system in that children craft videos with and about character toys as the system analyzes their gestures and play patterns. Children’s favorite props alternate between characters and cameramen in a film. As they play with the toys to act out a story, they conduct film assembly. We position our work as ubiquitous computing that supports children’s tangible interaction with digital materials. During user testing, we observed children ages 4 to 10 playing with Picture This!. We assess to what extent gesture interaction with objects for video editing allows children to explore visual perspectives in storytelling. A new genre of Gesture Object Interfaces as exemplified by Picture This relies on the analysis of gestures coupled with objects to represent bits.

From Touch Sensitive to Aerial Jewelry

Cati Vaucelle. From Touch Sensitive to Aerial Jewelry (Book Chapter). In Fashionable Technology, The intersection of Design, Fashion, Science, and Technology. Editor Seymour, S., Springer-Verlag Wien New York, 2008

Abstract

Now that we constantly travel by plane, use GIS, google map, satellite imagery, our vision is expanded. Our everyday objects have a language that adapts itself to our influences. On the other end, as much as the car has influenced painting and the representation of space and movement, we wanted to show how the use of new technologies can change our way to design personal objects as exemplified by Aerial Jewelry

Handsaw: Tangible Exploration of Volumetric Data by Direct Cut-Plane Projection

Bonanni, L., Alonso, J., Chao, N., Vargas, G., and Ishii, H. 2008. Handsaw: tangible exploration of volumetric data by direct cut-plane projection. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 251-254.

DOI: http://doi.acm.org/10.1145/1357054.1357098

Abstract

Tangible User Interfaces are well-suited to handling three-dimensional data sets by direct manipulation of real objects in space, but current interfaces can make it difficult to look inside dense volumes of information. This paper presents the SoftSaw, a system that detects a virtual cut-plane projected by an outstretched hand or laser-line directly on an object or space and reveals sectional data on an adjacent display. By leaving the hands free and using a remote display, these techniques can be shared between multiple users and integrated into everyday practice. The SoftSaw has been prototyped for scientific visualizations in medicine, engineering and urban design. User evaluations suggest that using a hand is more intuitive while projected light is more precise than keyboard and mouse control, and the SoftSaw system has the potential to be used more effectively by novices and in groups.

Renaissance Panel: The Roles of Creative Synthesis in Innovation

Hockenberry, M. and Bonanni, L. 2008. Renaissance panel: the roles of creative synthesis in innovation. In CHI ‘08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 2237-2240.

DOI: http://doi.acm.org/10.1145/1358628.1358658

Abstract

The Renaissance ideal can be expressed as a creative synthesis between cultural disciplines, standing in stark contrast to our traditional focus on scientific specialization. This panel presents a number of experts who approach the synthesis of art and science as the modus operandi for their work, using it as a tool for creativity, research, and practice. Understanding these approaches allows us to identify the roles of synthesis in successful innovation and improve the implementation of interdisciplinary synthesis in research and practice.

Future Craft: How Digital Media is Transforming Product Design

Bonanni, L., Parkes, A., and Ishii, H. 2008. Future craft: how digital media is transforming product design. In CHI ‘08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 2553-2564.

DOI: http://doi.acm.org/10.1145/1358628.1358712

Abstract

The open and collective traditions of the interaction community have created new opportunities for product designers to engage in the social issues around industrial production. This paper introduces Future Craft, a design methodology which applies emerging digital tools and processes to product design toward new objects that are socially and environmentally sustainable. We present the results of teaching the Future Craft curriculum at the MIT Media Lab including principal themes of public, local and personal design, resources, assignments and student work. Novel ethnographic methods are discussed with relevance to informing the design of physical products. We aim to create a dialogue around these themes for the product design and HCI communities.

Slurp: Tangibility, Spatiality, and an Eyedropper

Zigelbaum, J., Kumpf, A., Vazquez, A., and Ishii, H. 2008. Slurp: tangibility spatiality and an eyedropper. In CHI ‘08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 2565-2574.

DOI: http://doi.acm.org/10.1145/1358628.1358713

Abstract

The value of tangibility for ubiquitous computing is in its simplicity-when faced with the question of how to grasp a digital object, why not just pick it up? But this is problematic; digital media is powerful due to its extreme mutability and is therefore resistant to the constraints of static physical form. We present Slurp, a tangible interface for locative media interactions in a ubiquitous computing environment. Based on the affordances of an eyedropper, Slurp provides haptic and visual feedback while extracting and injecting pointers to digital media between physical objects and displays.

Reality-Based Interaction: A Framework for Post-WIMP Interfaces

Jacob, R. J., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., and Zigelbaum, J. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 201-210.

DOI: http://doi.acm.org/10.1145/1357054.1357089

Abstract

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

AR-Jig: A Handheld Tangible User Interface for 3D Digital Modeling (in Japanese)

Anabuki, M., Ishii, H. 2008. AR-Jig: A Handheld Tangible User Interface for 3D Digital Modeling. Transactions of the Virtual Reality Society of Japan, Special Issue on Mixed Reality 4 (Japanese Edition), Vol.13, No.2, 2008

Abstract

AR-Jig: A Handheld Tangible User Interface for 3D Digital Modeling Mahoro Anabuki*1 and Hiroshi Ishii*2 Abstract —- We introduce AR-Jig, a new handheld tangible user interface for 3D digital modeling in Augmented Reality space. AR-Jig has a pin array that displays a 2D physical curve coincident with a contour of a digitally-displayed 3D form. It supports physical interaction with a portion of a 3D digital representation, allowing 3D forms to be directly touched and modified. This project leaves the majority of the data in the digital domain but gives physicality to any portion of the larger digital dataset via a handheld tool. Through informal evaluations, we demonstrate AR-Jig would be useful for a design domain where manual modeling skills are critical. Keywords: actuated interface, augmented reality, handheld tool, pin array display

Tangible Bits: Beyond Pixels

Ishii, H. 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international Conference on Tangible and Embedded interaction (Bonn, Germany, February 18 - 20, 2008). TEI ‘08. ACM, New York, NY, xv-xxv.

DOI: http://doi.acm.org/10.1145/1347390.1347392

Abstract

Tangible user interfaces (TUIs) provide physical form to digital information and computation, facilitating the direct manipulation of bits. Our goal in TUI development is to empower collaboration, learning, and design by using digital technology and at the same time taking advantage of human abilities to grasp and manipulate physical objects and materials. This paper discusses a model of TUI, key properties, genres, applications, and summarizes the contributions made by the Tangible Media Group and other researchers since the publication of the first Tangible Bits paper at CHI 1997. http://tangible.media.mit.edu/

Topobo in the Wild: Longitudinal Evaluations of Educators Appropriating a Tangible Interface

Parkes, A., Raffle, H., and Ishii, H. 2008. Topobo in the wild: longitudinal evaluations of educators appropriating a tangible interface. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI ‘08. ACM, New York, NY, 1129-1138.

DOI: http://doi.acm.org/10.1145/1357054.1357232

Abstract

What issues arise when designing and deploying tangibles for learning in long term evaluations? This paper reports on a series of studies in which the Topobo system, a 3D tangible construction kit with the ability to record and playback motion, was provided to educators and designers to use over extended periods of time in the context of their day-to-day work. Tangibles for learning - like all educational materials - must be evaluated in relation both to the student and the teacher, but most studies of tangibles for learning focus on the student as user. Here, we focus on the conception of the educator, and their use of the tangible interface in the absence of an inventor or HCI researcher. The results of this study identify design and pedagogical issues that arise in response to distribution of a tangible for learning in different educational environments.

The Everyday Collector

Vaucelle, C. The Everyday Collector. In Extended Abstracts of the 10th international Conference on Ubiquitous Computing (Seoul, Korea, September 21 - 24, 2008). UbiComp ‘08, vol. 344. ACM, New York, NY.

Abstract

This paper presents the conceptualization of the Everyday Collector as a bridge between the traditional physical collection and the growing digital one. This work supports a reflection on the collection impulse and the impact that digital technologies have on the physical act of collection.

Electromagnetic Field Detector Bracelet

Vaucelle, C., Ishii, H. and Paradiso,.J. Electromagnetic Field Detector Bracelet. In Extended Abstracts of the 10th international Conference on Ubiquitous Computing (Seoul, Korea, September 21 - 24, 2008). UbiComp ‘08, vol. 344. ACM, New York, NY.

Abstract

We present the design of a cost-effective wearable sensor to detect and indicate the strength and other characteristics of the electric field emanating from a laptop display. Our bracelet can provide an immediate awareness of electric fields radiated from an object used frequently. Our technology thus supports awareness of ambient background emanation beyond human perception. We discuss how detection of such radiation might help to “fingerprint” devices and aid in applications that require determination of indoor location.

TILTle: exploring dynamic balance

Modlitba, P., Offenhuber, D., Ting, M., Tsigaridi, D., and Ishii, H. 2007. TILTle: exploring dynamic balance. In Proceedings of the 2007 Conference on Designing Pleasurable Products and interfaces (Helsinki, Finland, August 22 - 25, 2007). DPPI ‘07. ACM, New York, NY, 466-472. DOI= http://doi.acm.org/10.1145/1314161.1314207

DOI: http://doi.acm.org/10.1145/1314161.1314207

Abstract

In this paper we introduce a novel interface for exploring dynamic equilibria using the metaphor of a traditional balance scale. Rather than comparing and identifying physical weight, our scale can be used for contrasting digital data in different domains. We do this by assigning virtual weight to objects, which physically affects the scale. Our goal is to make complex comparison mechanisms more visible and graspable.

Simplicity in Interaction Design

Chang, A., Gouldstone, J., Zigelbaum, J., and Ishii, H. 2007. Simplicity in interaction design. In Proceedings of the 1st international Conference on Tangible and Embedded interaction (Baton Rouge, Louisiana, February 15 - 17, 2007). TEI ‘07. ACM, New York, NY, 135-138.

DOI: http://doi.acm.org/10.1145/1226969.1226997

Abstract

Attaining simplicy is a key challenge in interaction design. Our approach relies on a minimalist design exercise to explore the communication capacity for interaction components. This approach results in expressive design solutions, useful perspectives of interaction design and new interaction techniques.

Zstretch: A Stretchy Fabric Music Controller

Chang, A. and Ishii, H. 2007. Zstretch: a stretchy fabric music controller. In Proceedings of the 7th international Conference on New interfaces For Musical Expression (New York, New York, June 06 - 10, 2007). NIME ‘07. ACM, New York, NY, 46-49.

DOI: http://doi.acm.org/10.1145/1279740.1279746

Abstract

We present Zstretch, a textile music controller that supports expressive haptic interactions. The musical controller takes advantage of the fabric’s topological constraints to enable proportional control of musical parameters. This novel interface explores ways in which one might treat music as a sheet of cloth. This paper proposes an approach to engage simple technologies for supporting ordinary hand interactions. We show that this combination of basic technology with general tactile movements can result in an expressive musical interface.

AR-Jig: A Handheld Tangible User Interface for Modification of 3D Digital Form via 2D Physical Curve

Anabuki, M.; Ishi, H., “AR-Jig: A Handheld Tangible User Interface for Modification of 3D Digital Form via 2D Physical Curve,” Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on , vol., no., pp.55-66, 13-16 Nov. 2007

DOI: http://doi.ieeecomputersociety.org/10.1109/ISMAR.2007.4538826

Abstract

We introduce AR-Jig, a new handheld tangible user interface for 3D digital modeling in augmented reality (AR) space. AR-Jig has a pin array that displays a 2D physical curve coincident with a contour of a digitally displayed 3D form. It supports physical interaction with a portion of a 3D digital representation, allowing 3D forms to be directly touched and modified. Traditional tangible user interfaces physically embody all the data; in contrast, this project leaves the majority of the data in the digital domain but gives physicality to any portion of the larger digital dataset via a handheld tool. This tangible intersection enables the flexible manipulation of digital artifacts, both physically and virtually. Through an informal test by end-users and interviews with professionals, we confirmed the potential of the AR-Jig concept while identifying the improvements necessary to make AR-Jig a practical tool for 3D digital design.

Interfacing Video Capture, Editing and Publication in a Tangible Environment

Vaucelle, C. and Ishii, H. 2007. C. Interfacing Video Capture, Editing and Publication in a Tangible Environment. In. Baranauskas et al. (Eds.): INTERACT 2007, LNCS 4663, Lecture Notes in Computer Science, Part II, pp. 1 – 14, 2007. Springer Berlin / Heidelberg publisher.

DOI: http://dx.doi.org/10.1007/978-3-540-74800-7_1

Abstract

The paper presents a novel approach to collecting, editing and performing visual and sound clips in real time. The cumbersome process of capturing and editing becomes fluid in the improvisation of a story, and accessible as a way to create a final movie. It is shown how a graphical interface created for video production informs the design of a tangible environment that provides a spontaneous and collaborative approach to video creation, selection and sequencing. Iterative design process, participatory design sessions and workshop observations with 10-12 year old users from Sweden and Ireland are discussed. The limitations of interfacing video capture, editing and publication in a self-contained platform are addressed.

Keywords: Tangible User Interface - Video - Authorship - Mobile Technology - Digital Media - Video Jockey - Learning - Children - Collaboration

Senspectra: A Computationally Augmented Physical Modeling Toolkit for Sensing and Visualization of Structural Strain

LeClerc, V., Parkes, A., and Ishii, H. 2007. Senspectra: a computationally augmented physical modeling toolkit for sensing and visualization of structural strain. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI ‘07. ACM, New York, NY, 801-804.

DOI: http://doi.acm.org/10.1145/1240624.1240744

Abstract

We present Senspectra, a computationally augmented physical modeling toolkit designed for sensing and visualization of structural strain. Senspectra seeks to explore a new direction in computational materiality, incorporating the material quality of malleable elements of an interface into its digital control structure. The system functions as a decentralized sensor network consisting of nodes, embedded with computational capabilities and a full spectrum LED, and fl exible joints. Each joint functions as an omnidirectional bend sensing mechanism to sense and communicate mechanical strain between neighboring nodes.

Tug n Talk: A Belt Buckle for Tangible Tugging Communication

Adcock, M., Harry, D., Boch, M., Poblano, R.-D. and Harden, V. Tug n’ Talk: A Belt Buckle for Tangible Tugging Communication. Presented at alt.chi 2007

Abstract

Tug and Talk is prototype communication system with which you can send a “tug” to another person. The Tug and Talk device sits on your belt and connects to your shirt. Another Tug and Talk user can tug on the chain coming out of their matching belt and their tugging pattern is replicated as a tug on your own shirt. Tugs can express lots of different ideas, depending on context. A tug could be brief and small to see if someone is interruptable, or large, fast, and long to get someone’s attention in an urgent situation. We think this sort of tangible social channel between people is a powerful idea, and we implemented two prototype devices to explore its potential.

Touch . Sensitive Apparel

Vaucelle, C. and Abbas, Y. 2007. Touch: sensitive apparel. In CHI ‘07 Extended Abstracts on Human Factors in Computing Systems (San Jose, CA, USA, April 28 - May 03, 2007). CHI ‘07. ACM, New York, NY, 2723-2728.

DOI: http://doi.acm.org/10.1145/1240866.1241069

Abstract

Touch·Sensitive is a haptic apparel that allows massage therapy to be diffused, customized and controlled by people while on the move. It provides individuals with a sensory cocoon. Made of modular garments, Touch·Sensitive applies personalized stimuli. We present the design process and a series of low fidelity prototypes that lead us to the Touch·Sensitive Apparel.

Jabberstamp: Embedding sound and voice in traditional drawings

Raffle, H., Vaucelle, C., Wang, R., and Ishii, H. 2007. Jabberstamp: embedding sound and voice in traditional drawings. In Proceedings of the 6th international Conference on interaction Design and Children (Aalborg, Denmark, June 06 - 08, 2007). IDC ‘07. ACM, New York, NY, 137-144.

DOI: http://doi.acm.org/10.1145/1297277.1297306

Abstract

We introduce Jabberstamp, the first tool that allows children to synthesize their drawings and voices. To use Jabberstamp, children create drawings, collages or paintings on normal paper. They press a special rubber stamp onto the page to record sounds into their drawings. When children touch the marks of the stamp with a small trumpet, they can hear the sounds playback, retelling the stories they created. We describe our design process and analyze the mechanism between the act of drawing and the one of telling, defining interdependencies between the two activities. In a series of studies, children ages 4—8 use Jabberstamp to convey meaning in their drawings. The system allows collaboration among peers at different developmental levels. Jabberstamp compositions reveal children’s narrative styles and their planning strategies. In guided activities, children develop stories by situating sound recording in their drawing, which suggests future opportunities for hybrid voice-visual tools to support children’s emergent literacy.

Remix and Robo: Improvisational performance and competition with modular robotic building toys

Raffle, H., Yip, L., and Ishii, H. 2007. Remix and robo: sampling, sequencing and real-time control of a tangible robotic construction system. In ACM SIGGRAPH 2007 Educators Program (San Diego, California, August 05 - 09, 2007). SIGGRAPH ‘07. ACM, New York, NY, 35.

DOI: http://doi.acm.org/10.1145/1282040.1282077

Abstract

We present Remix and Robo, new composition and performance based tools for robotics control. Remix is a tangible interface used to sample, organize and manipulate gesturally-recorded robotic motions. Robo is a modified game controller used to capture robotic motions, adjust global motion parameters and execute motion recordings in real-time. Children use Remix and Robo to engage in (1) character design and (2) competitive endeavors with Topobo, a constructive assembly system with kinetic memory. Our objective is to provide new entry paths into robotics learning. This paper overviews our design process and reports how users age 7-adult use Remix and Robo to engage in different kinds of performative activities. Whereas robotic design is typically rooted in engineering paradigms, with Remix and Robo users pursue cooperative and competitive social performances. Activities like character design and robot competitions introduce a social context that motivates learners to focus and reflect upon their understanding of the robotic manipulative itself.

Mechanical Constraints as Computational Constraints in Tabletop Tangible Interfaces

Patten, J. and Ishii, H. 2007. Mechanical constraints as computational constraints in tabletop tangible interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI ‘07. ACM, New York, NY, 809-818.

DOI: http://doi.acm.org/10.1145/1240624.1240746

Abstract

This paper presents a new type of human-computer interface called Pico (Physical Intervention in Computational Optimization) based on mechanical constraints that combines some of the tactile feedback and affordances of mechanical systems with the abstract computational power of modern computers. The interface is based on a tabletop interaction surface that can sense and move small objects on top of it. The positions of these physical objects represent and control parameters inside a software application, such as a system for optimizing the configuration of radio towers in a cellular telephone network. The computer autonomously attempts to optimize the network, moving the objects on the table as it changes their corresponding parameters in software. As these objects move, the user can constrain their motion with his or her hands, or many other kinds of physical objects. The interface provides ample opportunities for improvisation by allowing the user to employ a rich variety of everyday physical objects as mechanical constraints. This approach leverages the user’s mechanical intuition for how objects respond to physical forces. As well, it allows the user to balance the numerical optimization performed by the computer with other goals that are difficult to quantify. Subjects in an evaluation were more effective at solving a complex spatial layout problem using this system than with either of two alternative interfaces that did not feature actuation.

Senspectra: A Computationally Augmented Physical Modeling Toolkit for Sensing and Visualization of Structural Strain

LeClerc, V., Parkes, A., and Ishii, H. 2007. Senspectra: a computationally augmented physical modeling toolkit for sensing and visualization of structural strain. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI ‘07. ACM, New York, NY, 801-804.

DOI: http://doi.acm.org/10.1145/1240624.1240744

Abstract

We present Senspectra, a computationally augmented physical modeling toolkit designed for sensing and visualization of structural strain. Senspectra seeks to explore a new direction in computational materiality, incorporating the material quality of malleable elements of an interface into its digital control structure. The system functions as a decentralized sensor network consisting of nodes, embedded with computational capabilities and a full spectrum LED, and flexible joints. Each joint functions as an omnidirectional bend sensing mechanism to sense and communicate mechanical strain between neighboring nodes. Using Senspectra, a user incrementally assembles and refines a physical 3D model of discrete elements with a real-time visualization of structural strain. While the Senspectra infrastructure provides a flexible modular sensor network platform, its primary application derives from the need to couple physical modeling techniques utilized in architecture and design disciplines with systems for structural engineering analysis. This offers direct manipulation augmented with visual feedback for an intuitive approach to physical real-time finite element analysis, particularly for organic forms.

Reflecting on Tangible User Interfaces: Three Issues Concerning Domestic Technology

Zigelbaum, J., and Csikszentmihályi, C. Reflecting on Tangible User Interfaces: Three Issues Concerning Domestic Technology. CHI 2007 Workshop on Tangible User Interfaces in Context and Theory (2007).

Abstract

As tangible interface design continues to gain currency within the mainstream HCI community and further manifests within the space of consumer electronics how will its impact be realized and how as designers of new technologies can we shape that impact? In this paper we examine the question of choice in technology design from the perspective of the social sciences and then reflect on ways that TUI designers could use these insights within their own practices. Of particular interest to this work is the repurposing and transplantation of current technologies into the domestic environment. The home has been a focus for much of the new work in HCI and in the near future we will see a continuation and increase in the development of domestic technologies. Much of the current work developing connected homes and ubiquitous systems for domestic use is compelling, though it seems to run directly counter to insights gained from the social sciences and philosophy of technology. In particular computer scientists, designers, anthropologists, and historians all offer very different points of departure concerning commercialization of domestic space and privacy versus data sharing. These differences may indicate a fertile area for research. We’ve identified three issues for domestic technology design: 1) context and the differentiation of constraints, 2) the privitization of space, and 3) the perception of control. These issues are not original to this work, nor are they exhaustive. Our work here is to discuss them within the context of tangible interface and domestic technology design as a means for critical reflection.

Jabberstamp: embedding sound and voice in traditional drawings

Raffle, H., Vaucelle, C., Wang, R., and Ishii, H. 2007. Jabberstamp: embedding sound and voice in traditional drawings. In ACM SIGGRAPH 2007 Educators Program (San Diego, California, August 05 - 09, 2007). SIGGRAPH ‘07. ACM, New York, NY, 32.

DOI: http://doi.acm.org/10.1145/1282040.1282074

Abstract

Children in our culture are accustomed to creating people and things and places - with implied context - in their drawings. Since the first days they draw, parents will ask “who is that? Where are they? What are they doing?” From early on, children have learned through drawing to provide the information necessary for an audience to understand the story that is going on in their drawing. Conversely, learning how to contextualize an oral or written story in the absence of images is a much slower learning process for children, and children’s ability to use language to communicate when and where their story takes place is considered a milestone in literacy development. Jabberstamp is the first tool that allows children to synthesize their drawings and voices. To use Jabberstamp, children create drawings, collages or paintings on normal paper. They press a special rubber stamp onto the page to record sounds into their drawings. When children touch the marks of the stamp with a small trumpet, they can hear the sounds playback, retelling the stories they created. In a series of studies, children ages 4-8 use Jabberstamp to convey meaning in their drawings. The system allows collaboration among peers at different developmental levels. Jabberstamp compositions reveal children’s narrative styles and their planning strategies. In guided activities, children develop stories by situating sound recording in their drawing, which suggests future opportunities for hybrid voice-visual tools to support children’s emergent literacy.

Interaction Techniques for Musical Performance with Tabletop Tangible Interfaces

Patten, J., Recht, B., and Ishii, H. 2006. Interaction techniques for musical performance with tabletop tangible interfaces. In Proceedings of the 2006 ACM SIGCHI international Conference on Advances in Computer Entertainment Technology (Hollywood, California, June 14 - 16, 2006). ACE ‘06, vol. 266. ACM, New York, NY, 27.

DOI: http://doi.acm.org/10.1145/1178823.1178856

Abstract

We present a set of interaction techniques for electronic musical performance using a tabletop tangible interface. Our system, the Audiopad, tracks the positions of objects on a tabletop surface and translates their motions into commands for a musical synthesizer. We developed and refi ned these interaction techniques through an iterative design process, in which new interaction techniques were periodically evaluated through performances and gallery installations. Based on our experience refi ning the design of this system, we conclude that tabletop interfaces intended for collaborative use should use interaction techniques designed to be legible to onlookers. We also conclude that these interfaces should allow users to spatially reconfi gure the objects in the interface in ways that are personally meaningful.

Glume: Exploring Materiality in a Soft Augmented Modular Modeling

Parkes, A., LeClerc, V., and Ishii, H. 2006. Glume: exploring materiality in a soft augmented modular modeling system. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 1211-1216.

DOI: http://doi.acm.org/10.1145/1125451.1125678

Abstract

This paper presents Glume, a system modular primitives – six silicone bulbs, embedded with sculptable gel and a full spectrum LED- attached to a central processing “nucleus.” The nodes communicate capacitively to their neighbors to determine a network topology taking advantage of the novel conductive characteristics of hairgel. As a modular, scalable platform, Glume provides a system with discrete internal structure coupled with a soft organic form, like the skeleton defi nes the structure of a body, to provide a means for expression and investigation of structures and processes not possible with existing systems.

BodyBeats: Whole-Body, Musical Interfaces for Children

Zigelbaum, J., Millner, A., Desai, B., and Ishii, H. 2006. BodyBeats: whole-body, musical interfaces for children. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 1595-1600.

DOI: http://doi.acm.org/10.1145/1125451.1125742

Abstract

This work in progress presents the BodyBeats Suite— three prototypes built to explore the interaction between children and computational musical instruments by using sound and music patterns. Our goals in developing the BodyBeats prototypes are (1) to help children engage their whole bodies while interacting with computers, (2) foster collaboration and pattern learning, and (3) provide a playful interaction for creating sound and music. We posit that electronic instruments for children that incorporate whole-body movement can provide active ways for children to play and learn with technology (while challenging a growing rate of childhood obesity). We describe how we implemented our current BodyBeats prototypes and discuss how users interact with them. We then highlight our plans for future work in the fields of whole-body interaction design, education, and music.

3D and Sequential Representations of Spatial Relationships among Photos

Anabuki, M. and Ishii, H. 2006. 3D and sequential representations of spatial relationships among photos. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 472-477.

DOI: http://doi.acm.org/10.1145/1125451.1125555

Abstract

This paper proposes automatic representations of spatial relationships among photos for structure analysis and review of a photographic subject. Based on camera tracking, photos are shown in a 3D virtual reality space to represent global spatial relationships. At the same time, the spatial relationships between two of the photos are represented in slide show sequences. This proposal allows people to organize photos quickly in spatial representations with qualitative meaning.

Mechanical Constraints as Common Ground between People and Computers

Patten, J. M. 2006 Mechanical Constraints as Common Ground between People and Computers. Doctoral Thesis. UMI Order Number: AAI0808956., Massachusetts Institute of Technology.

Abstract

This thesis presents a new type of human-computer interface based on mechanical constraints that combines some of the tactile feedback and affordances of mechanical systems with the abstract computational power of modern computers. The interface is based on a tabletop interaction surface that can sense and move small objects on top of it. Computation is merged with dynamic physical processes on the tabletop that are exposed to and modified by the user in order to accomplish his or her task. The system places mechanical constraints and mathematical constraints on the same level, allowing users to guide simulations and optimization processes by constraining the motion of physical objects on the interaction surface. The interface provides ample opportunities for improvisation by allowing the user to employ a rich variety of everyday physical objects as interface elements. Subjects in an evaluation were more effective at solving a complex spatial layout problem using this system than with either of two alternative interfaces that did not feature actuation.

SENSPECTRA: An Elastic, Strain-Aware Physical Modeling Interface

Vincent Leclerc. SENSPECTRA: An Elastic, Strain-Aware Physical Modeling Interface. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2006.

Abstract

Senspectra is a computationally augmented physical modeling toolkit designed for sensing and visualization of structural strain. The system functions as a distributed sensor network consisting of nodes, embedded with computational capabilities and a full spectrum LED, which communicate to neighbor nodes to determine a network topology through a system of flexible joints. Each joint, while serving as a data and power bus between nodes, also integrates an omnidirectional bend sensing mechanism, which uses a simple optical occlusion technique to sense and communicate mechanical strain between neighboring nodes. Using Senspectra, a user incrementally assembles and refines a physical 3D model of discrete elements with a real-time visualization of structural strain. While the Senspectra infrastructure provides a flexible modular sensor network platform, its primary application derives from the need to couple physical modeling techniques utilized in the architecture and industrial design disciplines with systems for structural engineering analysis, offering an intuitive approach for physical real-time finite element analysis. Utilizing direct manipulation augmented with visual feedback, the system gives users valuable insights on the global behavior of a constructed system defined as a network of discrete elements.

The Texture of Light

Vaucelle, C. 2006. The texture of light. In ACM SIGGRAPH 2006 Research Posters (Boston, Massachusetts, July 30 - August 03, 2006). SIGGRAPH ‘06. ACM, New York, NY, 27.

DOI: http://doi.acm.org/10.1145/1179622.1179651

Abstract

The Texture of Light is research on lighting principles and the exploration of life feed video metamorphosis in the public space using reflection of light on transparent materials. The Texture of Light is an attempt to fight the boredom of everyday life. This project employs the simple use of chemistry, Plexiglas, and plastic patterns to form a reconstruction of reality, giving it a texture and an expressive form. The transformation of life feed video comes from physical, plastic circles that act as different masks of reality. These masks can be moved around and swapped by the public, enabling collective expression. This metamorphosis of the public space is presented in real time as a moving painting and is projected on city walls. The public can record video clips of their ‘moving painting’ and project them back onto different city locations.

Affective TouchCasting

Bonanni, L. and Vaucelle, C. 2006. Affective TouchCasting. In ACM SIGGRAPH 2006 Sketches (Boston, Massachusetts, July 30 - August 03, 2006). SIGGRAPH ‘06. ACM, New York, NY, 35.

DOI: http://doi.acm.org/10.1145/1179849.1179893

Abstract

The sense of touch is not only informative: certain kinds of touch are directly related to emotions. Haptics can enrich the experience of broadcast media through tactile stimulus that is mapped to emotional response and distributed over the body. This sketch applies affective touch research to haptic broadcast in a wearable device that can record, distribute and play back touch information. TouchCasting augments broadcast media with affective haptics that can be experienced in public as a new form of art.

Collaborative Simulation Interface for Planning Disaster Measures

Kobayashi, K., Narita, A., Hirano, M., Kase, I., Tsuchida, S., Omi, T., Kakizaki, T., and Hosokawa, T. 2006. Collaborative simulation interface for planning disaster measures. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 977-982.

DOI: http://doi.acm.org/10.1145/1125451.1125639

PlayPals: Tangible Interfaces for Remote Communication and Play

Bonanni, L., Vaucelle, C., Lieberman, J., and Zuckerman, O. 2006. PlayPals: tangible interfaces for remote communication and play. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 574-579.

DOI: http://doi.acm.org/10.1145/1125451.1125572

Abstract

PlayPals are a set of wireless figurines with their electronic accessories that provide children with a playful way to communicate between remote locations. PlayPals is designed for children aged 5-8 to share multimedia experiences and virtual co-presence. We learned from our pilot study that embedding digital communication into existing play pattern enhances both remote play and communication.

TapTap: A Haptic Wearable for Asynchronous Distributed Touch Therapy

Bonanni, L., Vaucelle, C., Lieberman, J., and Zuckerman, O. 2006. TapTap: a haptic wearable for asynchronous distributed touch therapy. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 580-585.

DOI: http://doi.acm.org/10.1145/1125451.1125573

Abstract

TapTap is a wearable haptic system that allows nurturing human touch to be recorded, broadcast and played back for emotional therapy. Haptic input/output modules in a convenient modular scarf provide affectionate touch that can be personalized. We present a working prototype informed by a pilot study.

3D and Sequential Representations of Spatial Relationships among Photos

Anabuki, M. and Ishii, H. 2006. 3D and sequential representations of spatial relationships among photos. In CHI ‘06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI ‘06. ACM, New York, NY, 472-477.

DOI: http://doi.acm.org/10.1145/1125451.1125555

Abstract

This paper proposes automatic representations of spatial relationships among photos for structure analysis and review of a photographic subject. Based on camera tracking, photos are shown in a 3D virtual reality space to represent global spatial relationships. At the same time, the spatial relationships between two of the photos are represented in slide show sequences. This proposal allows people to organize photos quickly in spatial representations with qualitative meaning.

Beyond Record and Play: Backpacks: Tangible Modulators for Kinetic Behavior

Raffle, H., Parkes, A., Ishii, H., and Lifton, J. 2006. Beyond record and play: backpacks: tangible modulators for kinetic behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI ‘06. ACM, New York, NY, 681-690.

DOI: http://doi.acm.org/10.1145/1124772.1124874

Abstract

Digital Manipulatives embed computation in familiar children’s toys and provide means for children to design behavior. Some systems use “record and play” as a form of programming by demonstration that is intuitive and easy to learn. With others, children write symbolic programs with a GUI and download them into a toy, an approach that is conceptually extensible, but is inconsistent with the physicality of educational manipulatives. The challenge we address is to create a tangible interface that can retain the immediacy and emotional engagement of “record and play” and incorporate a mechanism for real time and direct modulation of behavior during program execution.We introduce the Backpacks, modular physical components that children can incorporate into robotic creations to modulate frequency, amplitude, phase and orientation of motion recordings. Using Backpacks, children can investigate basic kinematic principles that underly why their specific creations exhibit the specific behaviors they observe. We demonstrate that Backpacks make tangible some of the benefits of symbolic abstraction, and introduce sensors, feedback and behavior modulation to the record and play paradigm. Through our review of user studies with children ages 6-15, we argue that Backpacks extend the conceptual limits of record and play with an interface that is consistent with both the physicality of educational manipulatives and the local-global systems dynamics that are characteristic of complex robots.

Interaction Techniques for Musical Performance with Tabletop Tangible Interfaces

Patten, J., Recht, B., and Ishii, H. 2006. Interaction techniques for musical performance with tabletop tangible interfaces. In Proceedings of the 2006 ACM SIGCHI international Conference on Advances in Computer Entertainment Technology (Hollywood, California, June 14 - 16, 2006). ACE ‘06, vol. 266. ACM, New York, NY, 27.

DOI: http://doi.acm.org/10.1145/1178823.1178856

Abstract

We present a set of interaction techniques for electronic musical performance using a tabletop tangible interface. Our system, the Audiopad, tracks the positions of objects on a tabletop surface and translates their motions into commands for a musical synthesizer. We developed and refined these interaction techniques through an iterative design process, in which new interaction techniques were periodically evaluated through performances and gallery installations. Based on our experience refining the design of this system, we conclude that tabletop interfaces intended for collaborative use should use interaction techniques designed to be legible to onlookers. We also conclude that these interfaces should allow users to spatially reconfigure the objects in the interface in ways that are personally meaningful.

Designing the "World as your Palette"

Ryokai, K., Marti, S., and Ishii, H. 2005. Designing the world as your palette. In CHI ‘05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA, April 02 - 07, 2005). CHI ‘05. ACM, New York, NY, 1037-1049.

DOI: http://doi.acm.org/10.1145/1056808.1056816

Abstract

“The World as your Palette” is our ongoing effort to design and develop tools to allow artists to create visual art projects with elements (specifically, the color, texture, and moving patterns) extracted directly from their personal objects and their immediate environment. Our tool called “I/O Brush” looks like a regular physical paintbrush, but contains a video camera, lights, and touch sensors. Outside of the drawing canvas, the brush can pick up colors, textures, and movements of a brushed surface. On the canvas, artists can draw with the special “ink” they just picked up from their immediate environment. We describe the evolution and development of our system, from kindergarten classrooms to an art museum, as well as the reactions of our users to the growing expressive capabilities of our brush, as an iterative design process.

The world as a palette : painting with attributes of the environment

Kimiko Ryokai. Thesis (Ph. D.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2005.

Bottles: A Transparent Interface as a Tribute to Mark Weiser

Hiroshi Ishii, “Bottles: A Transparent Interface as a Tribute to Mark Weiser.” IEICE TRANSACTIONS on Information and Systems Vol.E87-D No.6 pp.1299-1311

Abstract

This paper first discusses the misinterpretation of the concept of “ubiquitous computing” that Mark Weiser originally proposed in 1991. Weiser’s main message was not the ubiquity of computers, but thetransparency of interface that determines users’ perception of digital technologies embedded in our physical environment seamlessly. To explore Weiser’s philosophy of transparency in interfaces, this paper presents the design of an interface that uses glass bottles as “containers” and “controls” for digital information. The metaphor is a perfume bottle: Instead of scent, the bottles have been filled with music — classical, jazz, and techno music. Opening each bottle releases the sound of a specific instrument accompanied by dynamic colored light. Physical manipulation of the bottles — opening and closing — is the primary mode of interaction for controlling their musical contents. The bottlesillustrates Mark Weiser’s vision of the transparent (or invisible) interface that weaves itself into the fabric of everyday life. The bottles also exploits the emotional aspects of glass bottles that are tangible and visual, and evoke the smell of perfume and the taste of exotic beverages. This paper describes the design goals of the bottle interface, the arrangement of musical content, the implementation of the wireless electromagnetic tag technology, and the feedback from users who have played with the system.

Topobo: A 3-D Constructive Assembly System with Kinetic Memory

Hayes Solos Raffle. Topobo: A 3-D Constructive Assembly System with Kinetic Memory. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2004.

DOI: http://hdl.handle.net/1721.1/26920

Tangible User Interfaces (TUIs): A Novel Paradigm for GIS

Ratti, C., Wang, Y., Ishii, H., Piper, B. Frenchman, D., “Tangible User Interfaces (TUIs): A Novel Paradigm for GIS,” Trans. GIS, vol. 8, no. 4, 2004, pp. 407–421.

DOI: http://dx.doi.org/10.1111/j.1467-9671.2004.00193.x

Abstract

In recent years, an increasing amount of effort has gone into the design of GIS user interfaces. On the one hand, Graphical User Interfaces (GUIs) with a high degree of sophistication have replaced line-driven commands of first-generation GIS. On the other hand, a number of alternative approaches have been suggested, most notably those based on Virtual Environments (VEs). In this paper we discuss a novel interface for GIS, which springs from recent work carried out in the field of Tangible User Interfaces (TUIs). The philosophy behind TUIs is to allow people to interact with computers via familiar tangible objects, therefore taking advantage of the richness of the tactile world combined with the power of numerical simulations. Two experimental systems, named Illuminating Clay and SandScape, are described here and their applications to GIS are examined. Conclusions suggest that these interfaces might streamline the landscape design process and result in a more effective use of GIS, especially when distributed decision-making and discussion with non-experts are involved.

egaku: Enhancing the Sketching Process

Yoon, J., Ryokai, K., Dyner, C., Alonso, J., and Ishii, H. 2004. egaku: enhancing the sketching process. In ACM SIGGRAPH 2004 Posters (Los Angeles, California, August 08 - 12, 2004). R. Barzel, Ed. SIGGRAPH ‘04. ACM, New York, NY, 42.

DOI: http://doi.acm.org/10.1145/1186415.1186464

Abstract

egaku is a tabletop user interface designed to enhance the ideation process with seamless image management tools. Designers sketch ideas as the system captures high-resolution images of the sketches and organizes them in a transparent image management structure. The system’s ability to determine and recognize layer associations allows users to quickly and intuitively visualize, retrieve, navigate through, and switch between layers of information without the hassle of traversing through multiple sheets of paper. With its strong emphasis on maintaining and enhancing the natural affordances of physical tracing paper, egaku allows users to overlay multiple digital translucent images to compose and compare different designs.

Phoxel-Space: an Interface for Exploring Volumetric Data with Physical Voxels

Ratti, C., Wang, Y., Piper, B., Ishii, H., and Biderman, A. 2004. PHOXEL-SPACE: an interface for exploring volumetric data with physical voxels. In Proceedings of the 5th Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (Cambridge, MA, USA, August 01 - 04, 2004). DIS ‘04. ACM, New York, NY, 289-296.

DOI: http://doi.acm.org/10.1145/1013115.1013156

Abstract

Phoxel-Space is an interface to enable the exploration of voxel data through the use of physical models and materials. Our goal is to improve the means to intuitively navigate and understand complex 3-dimensional datasets. The system works by allowing the user to define a free form geometry that can be utilized as a cutting surface with which to intersect a voxel dataset. The intersected voxel values are projected back onto the surface of the physical material. The paper describes how the interface approach builds on previous graphical, virtual and tangible interface approaches and how Phoxel-Space can be used as a representational aid in the example application domains of biomedicine, geophysics and fluid dynamics simulation

I/O Brush: Drawing with Everyday Objects as Ink

Ryokai, K., Marti, S., and Ishii, H. 2004. I/O brush: drawing with everyday objects as ink. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI ‘04. ACM, New York, NY, 303-310.

DOI: http://doi.acm.org/10.1145/985692.985731

Abstract

We introduce I/O Brush, a new drawing tool aimed at young children, ages four and up, to explore colors, textures, and movements found in everyday materials by “picking up” and drawing with them. I/O Brush looks like a regular physical paintbrush but has a small video camera with lights and touch sensors embedded inside. Outside of the drawing canvas, the brush can pick up color, texture, and movement of a brushed surface. On the canvas, children can draw with the special “ink” they just picked up from their immediate environment. In our study with kindergarteners, we found that children not only produced complex works of art using I/O Brush, but they also engaged in explicit talk about patterns and features available in their environment. I/O Brush invites children to explore the transformation from concrete and familiar raw material into abstract concepts about patterns of colors, textures and movements.

Topobo: A Constructive Assembly System with Kinetic Memory

Raffle, H. S., Parkes, A. J., and Ishii, H. 2004. Topobo: a constructive assembly system with kinetic memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI ‘04. ACM, New York, NY, 647-654.

DOI: http://doi.acm.org/10.1145/985692.985774

Abstract

We introduce Topobo, a 3D constructive assembly system embedded with kinetic memory, the ability to record and playback physical motion. Unique among modeling systems is Topobo’s coincident physical input and output behaviors. By snapping together a combination of Passive (static) and Active (motorized) components, people can quickly assemble dynamic biomorphic forms like animals and skeletons with Topobo, animate those forms by pushing, pulling, and twisting them, and observe the system repeatedly play back those motions. For example, a dog can be constructed and then taught to gesture and walk by twisting its body and legs. The dog will then repeat those movements and walk repeatedly.

Bringing clay and sand into digital design — continuous tangible user interfaces

Ishii, H., Ratti, C., Piper, B., Wang, Y., Biderman, A., and Ben-Joseph, E. 2004. Bringing Clay and Sand into Digital Design — Continuous Tangible user Interfaces. BT Technology Journal 22, 4 (Oct. 2004), 287-299.

DOI: http://dx.doi.org/10.1023/B:BTTJ.0000047607.16164.16

Abstract

Tangible user interfaces (TUIs) provide physical form to digital information and computation, facilitating the direct manipulation of bits. Our goal in TUI development is to empower collaboration, learning, and decision-making by using digital technology and at the same time taking advantage of human abilities to grasp and manipulate physical objects and materials. This paper presents a new generation of TUIs that enable dynamic sculpting and computational analysis using digitally augmented continuous physical materials. These new types of TUI, which we have termed ‘Continuous TUIs’, offer rapid form giving in combination with computational feedback. Two experimental systems and their applications in the domain of landscape architecture are discussed here, namely ‘Illuminating Clay’ and ‘SandScape’. Our results suggest that by exploiting the physical properties of continuous soft materials such as clay and sand, it is possible to bridge the division between physical and digital forms and potentially to revolutionise the current design process.

Super Cilia Skin: A Textural Interface

Raffe, H., Tichenor, J., Ishii, H. 2004. Super Cilia Skin: A Textural Interface. Textile, Volume 2, Issue 3, pp. 1–19

Abstract

Super Cilia Skin is a literal membrane separating a computer from its environment. Like our skin, it is haptic I/O membrane that can sense and simulate movement and wind flow. Our intention is to have it be universally applied to sheath any surface. As a display, it can mimic another person’s gesture over a distance via a form of tangible telepresence. A hand-sized interface covered with Super Cilia Skin would produce subtle changes in surface texture that feel much like a telepresent “butterfly kiss.”

Topobo: A Gestural Design Tool with Kinetic Memory

Amanda Parkes. Topobo: A Gestural Design Tool with Kinetic Memory. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2004.

DOI: http://hdl.handle.net/1721.1/28768

Abstract

The modeling of kinetic systems, both in physical materials and virtual simulations, provides a methodology to better understand and explore the forces and dynamics of our physical environment. The need to experiment, prototype and model with programmable kinetic forms is becoming increasingly important as digital technology becomes more readily embedded in physical structures and provides real-time variable data the capacity to transform the structures themselves. This thesis introduces Topobo, a gestural design tool embedded with kinetic memory—the ability to record, playback, and transform physical motion in three dimensional space. As a set of kinetic building blocks, Topobo records and repeats the body’s gesture while the system’s peer-to-peer networking scheme provides the capability to pass and transform q gesture. This creates a means to represent and understand algorithmic simulations in a physical material, providing a physical demonstration of how a simple set of rules can lead to complex form and behavior. Topobo takes advantage of the editability of computer data combined with the physical immediacy of a tangible model to provide a means for expression and investigation of kinetic patterns and processes not possible with existing materials.

PINS : a haptic computer interface system.

Bradley Carter Kaanta. PINS : a haptic computer interface system. Thesis (M. Eng. and S.B.)—Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2004.

DOI: http://hdl.handle.net/1721.1/28419

Tangible Query Interfaces: Physically Constrained Tokens for Manipulating Database Queries

Ullmer B, Ishii H, Jacob R.J.K (2003) Tangible query interfaces: physically constrained tokens for manipulating database queries. In: Proceedings of the 9th IFIP international conference on human-computer interaction (INTERACT 2003), Zurich, Switzerland, September 2003.

Abstract

We present a new approach for using physically constrained tokens to express, manipulate, and visualize parameterized database queries. This method extends tangible interfaces to enable interaction with large aggregates of information. We describe two interface prototypes that use physical tokens to represent database parameters. These tokens are manipulated upon physical constraints, which map compositions of tokens onto interpretations including database queries, views, and Boolean operations. We propose a framework for “token + constraint” interfaces, and compare one of our prototypes with a comparable graphical interface in a preliminary user study.

Super Cilia Skin: An Interactive Membrane

Raffle, H., Joachim, M. W., and Tichenor, J. 2003. Super cilia skin: an interactive membrane. In CHI ‘03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA, April 05 - 10, 2003). CHI ‘03. ACM, New York, NY, 808-809.

DOI: http://doi.acm.org/10.1145/765891.766004

Abstract

Super Cilia Skin is a literal membrane separating a computer from its environment. Like our skin, it is haptic I/O membrane that can sense and simulate movement and wind flow. Our intention is to have it be universally applied to sheath any surface. As a display, it can mimic another person’s gesture over a distance via a form of tangible telepresence. A hand-sized interface covered with Super Cilia Skin would produce subtle changes in surface texture that feel much like a telepresent “butterfly kiss.”

Applications of Computer-Controlled Actuation in Workbench Tangible User Interfaces

Daniel Maynes-Aminzade. Applications of Computer-Controlled Actuation in Workbench Tangible User Interfaces. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2003.

The actuated workbench : 2D actuation in tabletop tangible interfaces

Gian Antonio Pangaro. The actuated workbench : 2D actuation in tabletop tangible interfaces. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2003.

DOI: http://hdl.handle.net/1721.1/17620

Abstract

The Actuated Workbench is a new actuation mechanism that uses magnetic forces to control the two-dimensional movement of physical objects on flat surfaces. This mechanism is intended for use with existing tabletop Tangible User Interfaces, providing computer-controlled movement of the physical objects on the table, and creating an additional feedback layer for Human Computer Interaction (HCI). Use of this actuation technique makes possible new kinds of physical interactions with tabletop interfaces, and allows the computer to maintain consistency between the physical and digital states of data objects in the interface. This thesis focuses on the design and implementation of the actuation mechanism as an enabling technology, introduces new techniques for motion control, and discusses practical and theoretical implications of computer-controlled movement of physical objects in tabletop tangible interfaces.

The Actuated Workbench: Computer-Controlled Actuation in Tabletop Tangible Interfaces

Pangaro, G., Maynes-Aminzade, D., Ishii, H. 2003. The Actuated Workbench: Computer-Controlled Actuation in Tabletop Interfaces. ACM Trans. Graph. 22, 3 (Jul. 2003), 699-699.

DOI: http://doi.acm.org/10.1145/882262.882330

Abstract

The Actuated Workbench is a device that uses magnetic forces to move objects on a table in two dimensions. It is intended for use with existing tabletop tangible interfaces, providing an additional feedback loop for computer output, and helping to resolve inconsistencies that otherwise arise from the computer’s inability to move objects on the table.

ComTouch: A Vibrotactile Communication Device

Angela Chang. ComTouch: A Vibrotactile Mobile Communication Device. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

Tangible Interfaces for Manipulating Aggregates of Digital Information

Brygg Ullmer. Tangible Interfaces for Manipulating Aggregates of Digital Information. Thesis (Ph. D.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

The Illuminated Design Environment: a 3D Tangible Interface for Landscape Analysis

Ben Piper. The Illuminated Design Environment: a 3D Tangible Interface for Landscape Analysis. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

Hover: Conveying Remote Presence

Maynes-Aminzade, D., Tan, B., Goulding, K., and Vaucelle, C. 2002. Hover: conveying remote presence. In ACM SIGGRAPH 2002 Conference Abstracts and Applications (San Antonio, Texas, July 21 - 26, 2002). SIGGRAPH ‘02. ACM, New York, NY, 194-194.

DOI: http://doi.acm.org/10.1145/1242073.1242207

Abstract

This sketch presents Hover, a device that enhances remote telecommunication by providing a sense of the activity and presence of remote users. The motion of a remote persona is manifested as the playful movements of a ball floating in midair. Hover is both a communication medium and an aesthetic object.

Audiopad: A Tag Based Interface for Musical Performance

Patten, J., Recht, B., and Ishii, H. 2002. Audiopad: a tag-based interface for musical performance. In Proceedings of the 2002 Conference on New interfaces For Musical Expression (Dublin, Ireland, May 24 - 26, 2002). E. Brazil, Ed. New Interfaces For Musical Expression. National University of Singapore, Singapore, 1-6.

Abstract

We present Audiopad, an interface for musical performance that aims to combine the modularity of knob based controllers with the expressive character of multidimensional tracking interfaces. The performer’s manipulations of physical pucks on a tabletop control a real-time synthesis process. The pucks are embedded with LC tags that the system tracks in two dimensions with a series of specially shaped antennae. The system projects graphical information on and around the pucks to give the performer sophisticated control over the synthesis process

Illuminating Clay: A Tangible Interface with potential GRASS applications

Piper B., Ratti C., Ishii H., 2002, Illuminating Clay: a tangible interface with potential GRASS applications. Proceedings of the open-source GIS - GRASS users conference, Trento, Italy, September 2002.

Abstract

This paper introduces Illuminating Clay, an alternative interface for manipulating and navigating landscape representations that has been designed according to the specific needs of the landscape analyst

ComTouch: A Vibrotactile Communication Device

Chang, A., O’Modhrain, S., Jacob, R., Gunther, E., and Ishii, H. 2002. ComTouch: design of a vibrotactile communication device. In Proceedings of the 4th Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (London, England, June 25 - 28, 2002). DIS ‘02. ACM, New York, NY, 312-320.

DOI: http://doi.acm.org/10.1145/778712.778755

Abstract

We describe the design of ComTouch, a device that augments remote voice communication with touch, by converting hand pressure into vibrational intensity between users in real-time. The goal of this work is to enrich inter-personal communication by complementing voice with a tactile channel. We present preliminary user studies performed on 24 people to observe possible uses of the tactile channel when used in conjunction with audio. By recording and examining both audio and tactile data, we found strong relationships between the two communication channels. Our studies show that users developed an encoding system similar to that of Morse code, as well as three original uses: emphasis, mimicry, and turn-taking. We demonstrate the potential of the tactile channel to enhance the existing voice communication channel.

PegBlocks: a Learning Aid for the Elementary Classroom

Illuminating Clay: A 3-D Tangible Interface for Landscape Analysis

Piper, B., Ratti, C., and Ishii, H. 2002. Illuminating clay: a 3-D tangible interface for landscape analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI ‘02. ACM, New York, NY, 355-362.

DOI: http://doi.acm.org/10.1145/503376.503439

Abstract

This paper describes a novel system for the real-time computational analysis of landscape models. Users of the system - called Illuminating Clay - alter the topography of a clay landscape model while the changing geometry is captured in real-time by a ceiling-mounted laser scanner. A depth image of the model serves as an input to a library of landscape analysis functions. The results of this analysis are projected back into the workspace and registered with the surfaces of the model.We describe a scenario for which this kind of tool has been developed and we review past work that has taken a similar approach. We describe our system architecture and highlight specific technical issues in its implementation.We conclude with a discussion of the benefits of the system in combining the tangible immediacy of physical models with the dynamic capabilities of computational simulations.

A Tangible Interface for Organizing Information Using a Grid

Jacob, R. J., Ishii, H., Pangaro, G., and Patten, J. 2002. A tangible interface for organizing information using a grid. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI ‘02. ACM, New York, NY, 339-346.

DOI: http://doi.acm.org/10.1145/503376.503437

Abstract

The task of organizing information is typically performed either by physically manipulating note cards or sticky notes or by arranging icons on a computer with a graphical user interface. We present a new tangible interface platform for manipulating discrete pieces of abstract information, which attempts to combine the benefits of each of these two alternatives into a single system. We developed interaction techniques and an example application for organizing conference papers. We assessed the effectiveness of our system by experimentally comparing it to both graphical and paper interfaces. The results suggest that our tangible interface can provide a more effective means of organizing, grouping, and manipulating data than either physical operations or graphical computer interaction alone

Dolltalk: a computational toy to enhance children's creativity

Vaucelle, C. and Jehan, T. 2002. Dolltalk: a computational toy to enhance children’s creativity. In CHI ‘02 Extended Abstracts on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI ‘02. ACM, New York, NY, 776-777.

DOI: http://doi.acm.org/10.1145/506443.506592

Abstract

This paper presents a novel approach and interface for encouraging children to tell and act out original stories. Dolltalk is a toy that simulates speech recognition by capturing the gestures and speech of a child. The toy then plays back a child’s pretend-play speech in altered voices representing the characters of the child’s story. Dolltalk’s tangible interface and ability to retell a child’s story may enhance a child’s creativity in narrative elaboration.

Dolltalk: A computational toy to enhance narrative perspective-taking

Cati Vaucelle. Dolltalk: A computational toy to enhance narrative perspective-taking. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

Pinwheels: Visualizing Information Flow in an Architectural Space

Hiroshi Ishii, Sandia Ren, and Phil Frei. 2001. Pinwheels: visualizing information flow in an architectural space. In CHI ‘01 extended abstracts on Human factors in computing systems (CHI EA ‘01). ACM, New York, NY, USA, 111-112. DOI=10.1145/634067.634135 http://doi.acm.org/10.1145/634067.634135

DOI: http://doi.acm.org/10.1145/634067.634135

Abstract

We envision that the architectural spaces we inhabit will become an interface between humans and online digital information. We have been designing ambient information displays to explore the use of kinetic physical objects to present information at the periphery of human perception. This paper reports the design of a large-scale Pinwheels installation made of 40 computer-controlled pinwheel units in a museum context. The Pinwheels spin in a “wind of bits” that blows from cyberspace. The array of spinning pinwheels presents information within an architectural space through subtle changes in movement and sound. We describe the iterative design and implementation of the Pinwheels, and discuss design issues.

Bottles as a minimal interface to access digital information

Hiroshi Ishii, Ali Mazalek, and Jay Lee. 2001. Bottles as a minimal interface to access digital information. In CHI ‘01 extended abstracts on Human factors in computing systems (CHI EA ‘01). ACM, New York, NY, USA, 187-188. DOI=10.1145/634067.634180 http://doi.acm.org/10.1145/634067.634180

DOI: http://dx.doi.org/10.1145/634067.634180

Abstract

We present the design of a minimal interface to access digital information using glass bottles as “containers” and “controls”. The project illustrates our attempt to explore the transparency of an interface that weaves itself into the fabric of everyday life, and exploits the emotional aspects of glass bottles that are both tangible and visual. This paper describes the design of the bottle interface, and the implementation of the musicBottles installation, in which the opening of each bottle releases the sound of a specific instrument.

Sensetable: A Wireless Object tracking platform for tangible user interfaces.

James Patten. Sensetable: A Wireless Object tracking platform for tangible user interfaces. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

Tangible Interfaces for Interactive Point-of-View Narratives

Alexandra Mazalek. Tangible Interfaces for Interactive Point-of-View Narratives. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

Telling Tales: A new way to encourage written literacy through oral language

Mike Ananny. Telling Tales: A new way to encourage written literacy through oral language. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2002.

genieBottles: An Interactive Narrative in Bottles

A. Mazalek, A. Wood, and H. Ishii. Geniebottles: An interactive narrative in bottles. In Conference Abstracts and Applications SIGGRAPH 2001, page 189, Los Angeles, California USA, August 2001.

LumiTouch: An Emotional Communication Device

Chang, A., Resner, B., Koerner, B., Wang, X., and Ishii, H. 2001. LumiTouch: an emotional communication device. In CHI ‘01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, March 31 - April 05, 2001). CHI ‘01. ACM, New York, NY, 313-314.

DOI: http://doi.acm.org/10.1145/634067.634252

Abstract

We present the Lumitouch system consisting of a pair of interactive picture frames. When one user touches her picture frame, the other picture frame lights up. This touch is translated to light over an Internet connection. We introduce a semi-ambient display that can transition seamlessly from periphery to foreground in addition to communicating emotional content. In addition to enhancing the communication between loved ones, people can use LumiTouch to develop a personal emotional language.Based upon prior work on telepresence and tangible interfaces, LumiTouch explores emotional communication in tangible form. This paper describes the components, interactions, implementation and design approach of the LumiTouch system.

The HomeBox: A Web Content Creation Tool for The Developing World

Piper, B. and Hwang, R. E. 2001. The HomeBox: a web content creation tool for the developing world. In CHI ‘01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, March 31 - April 05, 2001). CHI ‘01. ACM, New York, NY, 145-146.

DOI: http://doi.acm.org/10.1145/634067.634156

Abstract

This paper describes the implementation and testing of the HomeBox, a prototype that seeks to provide a cost effective and scalable means for allowing users in the developing world to publish on the Web. It identifies the key requirements for such a design by drawing lessons from a variety of sources including two studies of networked community projects in Africa and South America. It. It ends with a discussion of possible design developments and plans for field trails in the Dominican Republic

Strata/ICC: Physical Models as Computational Interfaces

Ullmer, B., Kim, E., Kilian, A., Gray, S., and Ishii, H. 2001. Strata/ICC: physical models as computational interfaces. In CHI ‘01 Extended Abstracts on Human Factors in Computing Systems (Seattle, Washington, March 31 - April 05, 2001). CHI ‘01. ACM, New York, NY, 373-374.

DOI: http://doi.acm.org/10.1145/634067.634287

Abstract

We present Strata/ICC: a computationally-augmented physical model of a 54-story skyscraper that serves as an interactive display of electricity consumption, water consumption, network utilization, and other kinds of infrastructure. Our approach pushes information visualizations into the physical world, with a vision of transforming large-scale physical models into new kinds of interaction workspaces.

Designing Touch-based Communication Devices

Chang, A., Kanji, Z., Ishii, H. 2001. Designing Touch-based Communication Devices. CHI 2001 Workshop: Universal design: Towards universal access in the Information Society

DataTiles: A Modular Platform for Mixed Physical and Graphical Interactions

Rekimoto, J., Ullmer, B., and Oba, H. 2001. DataTiles: a modular platform for mixed physical and graphical interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, United States). CHI ‘01. ACM, New York, NY, 269-276.

DOI: http://doi.acm.org/10.1145/365024.365115

Abstract

The DataTiles system integrates the benefits of two major interaction paradigms: graphical and physical user interfaces. Tagged transparent tiles are used as modular construction units. These tiles are augmented by dynamic graphical information when they are placed on a sensor-enhanced flat panel display. They can be used independently or can be combined into more complex configurations, similar to the way language can express complex concepts through a sequence of simple words. In this paper, we discuss our design principles for mixing physical and graphical interface techniques, and describe the system architecture and example applications of the DataTiles system.

Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces

Patten, J., Ishii, H., Hines, J., and Pangaro, G. 2001. Sensetable: a wireless object tracking platform for tangible user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, United States). CHI ‘01. ACM, New York, NY, 253-260.

DOI: http://doi.acm.org/10.1145/365024.365112

Abstract

In this paper we present a system that electromagnetically tracks the positions and orientations of multiple wireless objects on a tabletop display surface. The system offers two types of improvements over existing tracking approaches such as computer vision. First, the system tracks objects quickly and accurately without susceptibility to occlusion or changes in lighting conditions. Second, the tracked objects have state that can be modified by attaching physical dials and modifiers. The system can detect these changes in real-time.We present several new interaction techniques developed in the context of this system. Finally, we present two applications of the system: chemistry and system dynamics simulation

Emerging Frameworks for Tangible User Interfaces

Ullmer, B. and Ishii, H. 2000. Emerging frameworks for tangible user interfaces. IBM Syst. J. 39, 3-4 (Jul. 2000), 915-931.

Abstract

We present steps toward a conceptual framework for tangible user interfaces. We introduce the MCRpd interaction model for tangible interfaces, which relates the role of physical and digital representations, physical control, and underlying digital models. This model serves as a foundation for identifying and discussing several key characteristics of tangible user interfaces. We identify a number of systems exhibiting these characteristics, and situate these within 12 application domains. Finally, we discuss tangible interfaces in the context of related research themes, both within and outside of the human-computer interaction domain.

A Comparison of Spatial Organization Strategies in Graphical and Tangible User Interfaces

Patten, J. and Ishii, H. 2000. A comparison of spatial organization strategies in graphical and tangible user interfaces. In Proceedings of DARE 2000 on Designing Augmented Reality Environments (Elsinore, Denmark). DARE ‘00. ACM, New York, NY, 41-50.

DOI: http://doi.acm.org/10.1145/354666.354671

Abstract

We present a study comparing how people use space in a Tangible User Interface (TUI) and in a Graphical User Interface (GUI). We asked subjects to read ten summaries of recent news articles and to think about the relationships between them. In our TUI condition, we bound each of the summaries to one of ten visually identical wooden blocks. In our GUI condition, each summary was represented by an icon on the screen. We asked subjects to indicate the location of each summary by pointing to the corresponding icon or wooden block. Afterward, we interviewed them about the strategies they used to position the blocks or icons during the task. We observed that TUI subjects performed better at the location recall task than GUI subjects. In addition, some TUI subjects used the spatial relationship between specific blocks and parts of the environment to help them remember the content of those blocks, while GUI subjects did not do this. Those TUI subjects who reported encoding information using this strategy tended to perform better at the recall task than those who did not.

curlybot: Designing a New Class of Computational Toys

Frei, P., Su, V., Mikhak, B., and Ishii, H. 2000. curlybot: designing a new class of computational toys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, April 01 - 06, 2000). CHI ‘00. ACM, New York, NY, 129-136. DOI= http://doi.acm.org/10.1145/332040.332416

DOI: http://doi.acm.org/10.1145/332040.332416

Abstract

We introduce an educational toy, called curlybot, as the basis for a new class of toys aimed at children in their early stages of development – ages four and up. curlybot is an autonomous two-wheeled vehicle with embedded electronics that can record how it has been moved on any flat surface and then play back that motion accurately and repeatedly. Children can use curlybot to develop intuitions for advanced mathematical and computational concepts, like differential geometry, through play away from a traditional computer.

musicBottles

Abstract

musicBottles introduces a tangible interface that deploys bottles as containers and controls for digital information.The system consists of a specially designed table and three corked bottles that “contain” the sounds of the violin, the cello, and the piano in Édouard Lalo’s Piano Trio in C Minor, Op. 7. Custom-designed electromagnetic tags embedded in the bottles enable each one to be wirelessly identified. When a bottle is placed onto the stage area of the table and the cork is removed, the corresponding instrument becomes audible. A pattern of colored light is rear-projected onto the table’s translucent surface to reflect changes in pitch and volume.The interface allows users to structure the experience of the musical composition by physically manipulating the different sound tracks.

The Design of Personal Ambient Displays

Craig Wisneski. The Design of Personal Ambient Displays. Thesis (S.M.)—Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1999.

The Design and Implementation of inTouch: A Distributed, Haptic Communication System

Victor Su. The Design and Implementation of inTouch: A Distributed, Haptic Communication System. Thesis (M. Eng. and S.B.)—Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1999.

The I/O Bulb and the Luminous Room

John Underkoffler, The I/O Bulb and the Luminous Room, Thesis (Ph.D.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts & Sciences, 1999.

Emancipated Pixels: Real-World Graphics in the Luminous Room

Underkoffler, J., Ullmer, B., and Ishii, H. 1999. Emancipated pixels: real-world graphics in the luminous room. In Proceedings of the 26th Annual Conference on Computer Graphics and interactive Techniques International Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., New York, NY, 385-392.

DOI: http://doi.acm.org/10.1145/311535.311593

Abstract

We describe a conceptual infrastructure the Luminous Room for providing graphical display and interaction at each of an interior architectural space’s various surfaces, arguing that pervasive environmental output and input is one natural heir to today’s rather more limited notion of spatially-confined, output-only display (the CRT). We discuss the requirements of such real-world graphics, including computational & networking demands; schemes for spatially omnipresent capture and display; and issues of design and interaction that emerge under these new circumstances. These discussions are both illustrated and motivated by five particular applications that have been built for a real, experimental Luminous Room space, and by details of the current technical approach to its construction (involving a two-way optical transducer called an I/O Bulb that projects and captures pixels).

Urp: A Luminous-Tangible Workbench for Urban Planning and Design

Underkoffler, J. and Ishii, H. 1999. Urp: a luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania, United States, May 15 - 20, 1999). CHI ‘99. ACM, New York, NY, 386-393.

DOI: http://doi.acm.org/10.1145/302979.303114

Abstract

We introduce a system for urban planning - called Urp -that integrates functions addressing a broad range of the fields concerns into a single, physically based workbench setting. The I/O Bulb infrastructure on which the application is based allows physical architectural models placed on an ordinary table surface to cast shadows accurate for arbitrary times of day; to throw reflections off glass facade surfaces; to affect a real-time and visually coincident simulation of pedestrian-level windflow; and so on. We then use comparisons among Urp and several earlier I/O Bulb applications as the basis for an understanding of luminous-tangible interactions, which result whenever an interface distributes meaning and functionality between physical objects and visual information projectively coupled to those objects. Finally, we briefly discuss two issues common to all such systems, offering them as informal thought-tools for the design and analysis of luminous-tangible interfaces.

Towards the Distributed Visualization of Usage History

Paul Yarin. Towards the Distributed Visualization of Usage History. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 1999.

Curlybot

Phil Frei, Victor Su, and Hiroshi Ishii. 1999. Curlybot. In ACM SIGGRAPH 99 Conference abstracts and applications (SIGGRAPH ‘99). ACM, New York, NY, USA, 173-. DOI=10.1145/311625.311972 http://doi.acm.org/10.1145/311625.311972

DOI: http://dx.doi.org/10.1145/311625.311972

PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play

Hiroshi Ishii, Craig Wisneski, Julian Orbanes, Ben Chun, and Joe Paradiso. 1999. PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play. In Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit (CHI ‘99). ACM, New York, NY, USA, 394-401. DOI=10.1145/302979.303115

DOI: http://doi.acm.org/10.1145/302979.303115

The Last Farewell: Traces of Physical Presence

Ishii, H. 1998. Reflections: “The last farewell”: traces of physical presence. interactions 5, 4 (Jul. 1998), 56-ff.

DOI: http://doi.acm.org/10.1145/278465.278474

Abstract

In the Spring of 1995, I was finally able to realize a dream that I’d held for quite a number of years; I was able to visit Hanamaki village, the home of the famous author Miazawa Kenji. Before leaving Japan, I had wanted to see Kenji’s “World of Efertobe” once with my own eyes.

Designing Kinetic Objects for Digital Information Display

Andy Dahley. Designing Kinetic Objects for Digital Information Display. Thesis (S.M.)—Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 1998.

Abstract

We have access to more and more information from computer networks. However, the means of monitoring this changing information is limited by its access through the narrow window of a computer screen. The interactions between people and digital information are now almost entirely confined to the conventional GUI (Graphical User Interface) comprised of a keyboard, monitor, and mouse, largely ignoring the richness of the physical world. As a critical step in moving beyond current interface limitations, this research is attempts to use many parts of our environment to convey information in a variety of ways. Rather than adding more video terminals into an environment, this thesis examines how to move information off the screen into our physical environment, where it is manifested in a more physical and kinetic manner. The thesis explores how these kinetic objects can be used to display information on a more visceral cognitive level than afforded by the interfaces of generalized information appliances like the computer. The approach in this thesis is through several exploratory design studies. A geography of the design space of kinetic objects as digital information displays was developed through this series of design studies so that it can be used in the development of future kinetic displays.

Beyond Input Devices: A New Conceptual Framework for the Design of Physical-Digital Objects

Matthew Gorbet. Beyond Input Devices: A New Conceptual Framework for the Design of Physical-Digital Objects. Thesis (M.S.)—Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1998.

DOI: http://hdl.handle.net/1721.1/29138

Abstract

This work introduces the concept of physical-digital objects: physical objects which allow people to interact with digital information as though it were tangible. I treat the design of physical-digital objects as a new field, and establish a conceptual framework within which to approach this design task. My design strategy treats objects as having both a physical and a digital identity, related to one another by three design principles: coupling, transparency, and mapping. With these principles as a guide, designers can take advantage of emerging digital technologies to create entirely new physical-digital objects. This new design perspective encourages a conceptual shift away from discrete input and output devices as gateways to a digital world, and towards a more seamless interaction with information, enabled by our knowledge and understanding of the physical world. I illustrate this by introducing and discussing seven actual physical-digital object systems, including two which I developed: Bottles and Triangles.

Tangible Interfaces for Remote Communication and Collaboration

Scott Brave. Tangible Interfaces for Remote Communication and Collaboration. Thesis (M.S.)—Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1998.

Abstract

This thesis presents inTouch, a new device enabling long distance communication through touch. inTouch is based on a concept called Synchronized Distributed Physical Objects, which employs telemanipulation technology to create the illusion that distant users are interacting with a shared physical object. I discuss the design and prototype implementation of inTouch, along with control strategies for extending the physical link over an arbitrary distance. User reactions to the prototype system suggest many similarities to direct touch interactions while, at the same time, point to new possibilities for object-mediated touch communication. I also present two initial experiments that begin to explore more objective properties of the haptic communication channel provided by inTouch and develop analysis techniques for future investigations.

mediaBlocks: Physical Containers,Transports, and Controls for Online Media

Ullmer, B., Ishii, H., and Glas, D. 1998. mediaBlocks: physical containers, transports, and controls for online media. In Proceedings of the 25th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH ‘98. ACM, New York, NY, 379-386.

DOI: http://doi.acm.org/10.1145/280814.280940

Abstract

We present a tangible user interface based upon media-Blocks: small, electronically tagged wooden blocks that serve as physical icons (cicons”) for the containment, transport, and manipulation of online media. MediaBlocks interface with media input and output devices such as video cameras and projectors, allowing digital media to be rapidly cpied” from a media source and pasted into a media display. Media-Blocks are also compatible with traditional GUIs, providing seamless gateways between tangible and graphical interfaces. Finally, mediaBlocks act as physical cntrols” in tangible interfaces for tasks such as se-quencing collections of media elements.

Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information

Wisneski, C., Ishii, H., Dahley, A., Gorbet, M., Brave, S., Ullmer, B., Yarin, P. Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information. CoBuild 1998.

Abstract

We envision that the physical architectural space we inhabit will be a new form of interface between humans and digital information. This paper and video present the design of the ambientROOM, an interface to information for processing in the background of awareness. This information is displayed through various subtle displays of light, sound, and movement. Physical objects are also employed as controls for these “ambient media”.

Tangible Interfaces for Remote Collaboration and Communication

Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbet, M., Ullmer, B., and Yarin, P. 1998. ambientROOM: integrating ambient media with architectural space. In CHI 98 Conference Summary on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). CHI ‘98. ACM, New York, NY, 173-174.

DOI: http://doi.acm.org/10.1145/289444.289491

Abstract

Current systems for real-time distributed CSCW are largely rooted in traditional GUI-based groupware and voice/video conferencing methodologies. In these approaches, interactions are limited to visual and auditory media, and shared environments are confined to the digital world. This paper presents a new approach to enhance remote collaboration and communication, based on the idea of Tangible Interfaces, which places a greater emphasis on touch and physicality. The approach is grounded in a concept called Synchronized Distributed Physical Objects, which employs telemanipulation technology to create the illusion that distant users are interacting with shared physical objects. We describe two applications of this approach: PSyBench, a physical shared workspace, and inTouch, a device for haptic interpersonal communication.

ambientROOM: Integrating Ambient Media with Architectural Space

Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbet, M., Ullmer, B., and Yarin, P. 1998. ambientROOM: integrating ambient media with architectural space. In CHI 98 Conference Summary on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). CHI ‘98. ACM, New York, NY, 173-174.

DOI: http://doi.acm.org/10.1145/286498.286652

Abstract

We envision that the physical architectural space we inhabit will be a new form of interface between humans and digital information. This paper and video present the design of the ambientROOM, an interface to information for processing in the background of awareness. This information is displayed through various subtle displays of light, sound, and movement. Physical objects are also employed as controls for these “ambient media”.

Illuminating Light: An Optical Design Tool with a Luminous-Tangible Interface

Underkoffler, J. and Ishii, H. 1998. Illuminating light: an optical design tool with a luminous-tangible interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). C. Karat, A. Lund, J. Coutaz, and J. Karat, Eds. Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 542-549.

DOI: http://doi.acm.org/10.1145/274644.274717

Abstract

We describe a novel system for rapid prototyping of laser- based optical and holographic layouts. Users of this optical prototyping tool - called the Illuminating Light system - move physical representations of various optical elements about a workspace, while the system tracks these compo- nents and projects back onto the workspace surface the simulated propagation of laser light through the evolving layout. This application is built atop the Luminous Room infrastrncture, an aggregate of interlinked, computer-con- trolled projector-camera units called Z/O Bulbs. Philosophi- cally, the work embodies the emerging ideas of the Luminous Room and builds on the notions of ‘graspable media’. We briefly introduce the VO Bulb and Luminous Room con- cepts and discuss their current implementations. After an overview of the optical domain that the Illuminating Light system is designed to address, we present the overall sys- tem design and implementation, including that of an inter- mediary toolkit called voodoo which provides a general facility for object identification and tracking. beam that continues through the beamsplitter.

Triangles: Tangible Interface for Manipulation and Exploration of Digital Information Topography

Gorbet, M. G., Orth, M., and Ishii, H. 1998. Triangles: tangible interface for manipulation and exploration of digital information topography. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Los Angeles, California, United States, April 18 - 23, 1998). C. Karat, A. Lund, J. Coutaz, and J. Karat, Eds. Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 49-56.

DOI: http://doi.acm.org/10.1145/274644.274652

Abstract

This paper presents a system for interacting with digital information, called Triangles. The Triangles system is a physictidigital construction kit, which allows users to use two hands to grasp and manipulate complex digital information. The kit consists of a set of identical flat, plastic triangles, each with a microprocessor inside and magnetic edge connectors. The connectors enable the Triangles to be physically connected to each other and provide tactile feedback of these connections. The connectors also pass electricity, allowing the Triangles to communicate digital information to each other and to a ,desktop computer. When the pieces contact one another, specific connection information is sent back to a computer that keeps track of the configuration of the system.

The metaDESK: Models and Prototypes for Tangible User Interfaces

Ullmer, B. and Ishii, H. 1997. The metaDESK: models and prototypes for tangible user interfaces. In Proceedings of the 10th Annual ACM Symposium on User interface Software and Technology (Banff, Alberta, Canada, October 14 - 17, 1997). UIST ‘97. ACM, New York, NY, 223-232.

DOI: http://doi.acm.org/10.1145/263407.263551

Abstract

The metaDESK is our first platform for exploring the design of tangible user interfaces. The metaDESK integrates multiple 2D and 3D graphic displays with an assortment of physical objects and instruments, sensed by an array of optical, mechanical, and electromagnetic field sensors. The metaDESK “brings to life” these physical objects and instruments as tangible interfaces to a range of graphically-intensive applications. Using the metaDESK platform, we are studying issues such as a) the physical embodiment of GUI (graphical user interface) widgets such as icons, handles, and windows; b) the coupling of everyday physical objects with the digital information that pertains to them.

Models and Mechanisms for Tangible User Interfaces

Brygg Ullmer. Models and Mechanisms for Tangible User Interfaces. Thesis (M.S.)—Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1997.

Abstract

Current human-computer interface design is dominated by the graphical user interface approach, where users interact with graphical abstractions of virtual interface devices through a few general-purpose input “peripherals.” The thesis develops models and mechanisms for “tangible user interfaces” – user interfaces which use physical objects, instruments, surfaces, and spaces as physical interfaces to digital information. Prototype applications on three platforms – the metaDESK, transBOARD, and ambientROOM – are introduced as examples of this approach. These instances are used to generalize the “GUI widgetry,” “optical,” and “containers and conduits” interface metaphors. The thesis also develops engineering mechanisms called proxy-distributed or “proxdist” computation, which provide a layered approach for integrating physical objects with diverse sensing, display, communication, and computation capabilities into coherent interface implementations. The combined research provides a vehicle for moving beyond the keyboard, monitor, and pointer of current computer interfaces towards use of the physical world itself as a kind of computationally-augmented interface.

Triangles: Design of a Physical/Digital Construction Kit

Gorbet, M. G. and Orth, M. 1997. Triangles: design of a physical/digital construction kit. In Proceedings of the 2nd Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (Amsterdam, The Netherlands, August 18 - 20, 1997). S. Coles, Ed. DIS ‘97. ACM, New York, NY, 125-128.

DOI: http://doi.acm.org/10.1145/263552.263592

Abstract

This paper describes the design process and philosophy behind Triangles, a new physical computer interface in the form of a construction kit of identical, flat, plastic triangles. The triangles connect together both mechanically and electrically with magnetic, conducting connectors. When the pieces contact one another, information about the specific connection is passed through the conducting connectors to the computer. In this way, users can create both two and three-dimensional objects whose exact configuration is known by the computer. The physical connection of any two Triangles can also trigger specific events in the computer, creating a simple but powerful means for physically interacting with digital information. This paper will describe the Triangles system, its advantages and applications. It will also highlight the importance of collaborative and multidisciplinarian design teams in the creation of new digital objects that bridge electrical engineering, industrial design, and software design—such as the Triangles

inTouch: A Medium for Haptic Interpersonal Communication

Brave, S. and Dahley, A. 1997. inTouch: a medium for haptic interpersonal communication. In CHI ‘97 Extended Abstracts on Human Factors in Computing Systems: Looking To the Future (Atlanta, Georgia, March 22 - 27, 1997). CHI ‘97. ACM, New York, NY, 363-364.

DOI: http://doi.acm.org/10.1145/1120212.1120435

Abstract

In this paper, we introduce a new approach for applying haptic feedback technology to interpersonal communication. We present the design of our prototype inTouch system which provides a physical link between users separated by distance.

Tangible Bits: Towards Seamless Interfaces between People, Bits, and Atoms

Ishii, H. and Ullmer, B. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI ‘97. ACM, New York, NY, 234-241.

DOI: http://doi.acm.org/10.1145/258549.258715

Abstract

This paper presents our vision of Human Computer Interaction (HCI): “Tangible Bits.” Tangible Bits allows users to “grasp & manipulate” bits in the center of users’ attention by coupling the bits with everyday physical objects and architectural surfaces. Tangible Bits also enables users to be aware of background bits at the periphery of human perception using ambient display media such as light, sound, airflow, and water movement in an augmented space. The goal of Tangible Bits is to bridge the gaps between both cyberspace and the physical environment, as well as the foreground and background of human activities. This paper describes three key concepts of Tangible Bits: interactive surfaces; the coupling of bits with graspable physical objects; and ambient media for background awareness. We illustrate these concepts with three prototype systems – the metaDESK, transBOARD and ambientROOM – to identify underlying research issues.

Bricks: Laying the Foundations for Graspable User Interfaces

Fitzmaurice, G. W., Ishii, H., and Buxton, W. A. 1995. Bricks: laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, May 07 - 11, 1995). I. R. Katz, R. Mack, L. Marks, M. B. Rosson, and J. Nielsen, Eds. Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 442-449

DOI: http://doi.acm.org/10.1145/223904.223964

Abstract

We introduce the concept of Graspable User Interfaces that allow direct control of electronic or virtual objects through physical handles for control. These physical artifacts, which we call “bricks,” are essentially new input devices that can be tightly coupled or “attached” to virtual objects for manipulation or for expressing action (e.g., to set parameters or for initiating processes). Our bricks operate on top of a large horizontal display surface known as the “ActiveDesk.” We present four stages in the development of Graspable UIs: (1) a series of exploratory studies on hand gestures and grasping; (2) interaction simulations using mock-ups and rapid prototyping tools; (3) a working prototype and sample application called GraspDraw; and (4) the initial integrating of the Graspable UI concepts into a commercial application. Finally, we conclude by presenting a design space for Bricks which lay the foundation for further exploring and developing Graspable User Interfaces

Integration of interpersonal space and shared workspace: ClearBoard design and experiments

Ishii, H., Kobayashi, M., and Grudin, J. 1993. Integration of interpersonal space and shared workspace: ClearBoard design and experiments. ACM Trans. Inf. Syst. 11, 4 (Oct. 1993), 349-375. DOI= http://doi.acm.org/10.1145/159764.159762

ClearBoard: a seamless medium for shared drawing and conversation with eye contact

Ishii, H. and Kobayashi, M. 1992. ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, United States, May 03 - 07, 1992). P. Bauersfeld, J. Bennett, and G. Lynch, Eds. CHI ‘92. ACM, New York, NY, 525-532. DOI= http://doi.acm.org/10.1145/142750.142977

DOI: http://doi.acm.org/10.1145/142750.142977

Abstract

This paper introduces a novel shared drawing medium called ClearBoard. It realizes (1) a seamless shared drawing space and (2) eye contact to support realtime and remote collaboration by two users. We devised the key metaphor: “talking through and drawing on a transparent glass window” to design ClearBoard. A prototype of ClearBoard is implemented based on the “Drafter-Mirror” architecture. This paper first reviews previous work on shared drawing support to clarify the design goals. We then examine three methaphors that fulfill these goals. The design requirements and the two possible system architectures of ClearBoard are described. Finally, some findings gained through the experimental use of the prototype, including the feature of “gaze awareness”, are discussed.

Integration of inter-personal space and shared workspace: ClearBoard design and experiments

Ishii, H., Kobayashi, M., and Grudin, J. 1992. Integration of inter-personal space and shared workspace: ClearBoard design and experiments. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work (Toronto, Ontario, Canada, November 01 - 04, 1992). CSCW ‘92. ACM, New York, NY, 33-42. DOI= http://doi.acm.org/10.1145/143457.143459

DOI: http://doi.acm.org/10.1145/143457.143459

Abstract

This paper describes the evolution of a novel shared drawing medium that permits co-workers in two different locations to draw with color markers or with electronic pens and software tools while maintaining direct eye contact and the ability to employ natural gestures. We describe the evolution from ClearBoard-1 (based on a video drawing technique) to ClearBoard-2 (which incorporates TeamPaint, a multi-user paint editor). Initial observations based on use and experimentation are reported. Further experiments are conducted with ClearBoard-0 (a simple mockup), with ClearBoard-1, and with an actual desktop as a control. These experiments verify the increase of eye contact and awareness of collaborator’s gaze direction in ClearBoard environments where workspace and co-worker images compete for attention.