The Sixth Sense by Pranav Mistry

Pranav Mistry’s famous Sixth Sense interface that made a huge splash in the tech media after Pattie Maes’ TED Talk in 2009 (video here) was what I used as a frame of reference for TUIs in order to evaluate the taxonomy proposed by Fishkin. Why I regard the Sixth Sense as a strong example of TUI because it breaks conventions of desk-constrained computing and made it so that the interactions are intuitive and self-evident.

In terms of embodiment, the input in Mistry’s prototype is through color coded finger tips that are used to encode gestures which are then captured by a camera. The output is then projected onto surfaces in front of the user which are also interactive completing the augmented reality cycle. This kind of I/O touches various types of embodiments – full when the user is touching a projected button hinting at direct manipulation, nearby when the gesture indicates to scroll information on a projected page and environmental when the gesture takes an image (but doesn’t display the same image in front of you after it’s being captured hinting at the ambient nature of the process).

Metaphorically, Mistry’s prototype is heavily gesture-driven and the inputs to the device are based on how other devices act – hence, Sixth Sense falls into the ‘metaphor as verb’ category. Here’s where I think the taxonomy’s intention to pull all TUIs towards the full embodied and metaphor angle is not very convincing. While the prototype does feature a camera and a projector, a finished unified product does not necessarily have to appear like a camera and a projector – there are plenty of creative angles the designer could take on how it could look like and making it look like a camera and a projector does not seem very innovative. So Fishkin’s taxonomy is not very useful to guide design changes in existing innovations. However, it is a very robust concept as it helps place existing examples of TUI in various parts of the spectrum and in my opinion, by seeing innovations like the Sixth Sense appear in the non-extreme portions of the spectrum, it can also help guide what-if questions.

Virtual Reality

I found Fishkin’s conceptualization and analysis of TUIs along the two dimensions of embodiment and metaphor to be useful frameworks to think about various TUI. However, I also felt that there were some fuzzy/grey areas wherein it wasn’t clear if a TUI is one or the other or overlaps/spreads across categories. I found the taxonomy to be interesting tool for analysis and organizing but not as useful for innovation.

The example that came to my mind was that of virtual reality. It’s hard to perfectly fit the interaction in the virtual world along the dimension of embodiment wherein the output to the user’s action is happening in the virtual world. Also, along the Metaphor dimension, I felt like VR technology doesn’t fit perfectly in any category (except for none). Similarly, as innovation makes way for new technology, these boundaries of fuzziness could increase.

So, while Fishkin’s taxonomy is a very useful tool for analysing and conceptualizing, I would consider keeping it in my toolset for that purpose and use it within the larger context of user needs and user interactions to help define the solution space of the particular problem that I am working on. 

 

1292809145749560752

Virtual Reality

I found Fishkin’s conceptualization and analysis of TUIs along the two dimensions of embodiment and metaphor to be useful frameworks to think about various TUI. However, I also felt that there were some fuzzy/grey areas wherein it wasn’t clear if a TUI is one or the other or overlaps/spreads across categories. I found the taxonomy to be interesting tool for analysis and organizing but not as useful for innovation.

The example that came to my mind was that of virtual reality. It’s hard to perfectly fit the interaction in the virtual world along the dimension of embodiment wherein the output to the user’s action is happening in the virtual world. Also, along the Metaphor dimension, I felt like VR technology doesn’t fit perfectly in any category (except for none). Similarly, as innovation makes way for new technology, these boundaries of fuzziness could increase.

So, while Fishkin’s taxonomy is a very useful tool for analysing and conceptualizing, I would consider keeping it in my toolset for that purpose and keep my mind open to user needs and user interactions to help define the solution space of the particular problem that I am working on. 

 

1292809145749560752

Inputs without Outputs: What to do with them?

Fishkin’s taxonomy elevates one’s awareness of a question that’s rarely asked, but worth asking: what exactly comprises a TUI, and how can we systematically compare them to each other? That is, we know a TUI when we see one, and we can easily tell them apart, but Fishkin’s text shows us one way of doing so–and I confess that, while reading, I found myself jotting down new ideas provoked by the rubric. So I did find it productive and helpful, in short.

That said, I wonder about a corner not explicitly covered in the text. Fishkin remarks in the section “Calm Computing” on TUIs stripped of their human inputs: a flower’s petals respond to ambient temperature; or a self-shortening string that responds to bus wait times. These examples are “tangible,” I suppose, insofar as they have components that could conceivably be touched, but they don’t demand user input. Can we imagine, however, a device that would have user input and not produce an output? Can we properly categorize such a device as a TUI?

Take, for example, a button that says “press,” but which has no effect. This button would seem to fall into an unclear categorization in terms of embodiment: would it comprise “full” embodiment insofar as the pushing of the button is itself the user experience? Or would it engage the user as “nearby” or “distant” embodiment (depending on the user’s expectations) that never comes to fruition? What about metaphor: is the metaphorical relationship of the experience “none” because the pushing of the button lacks any kind of relationship with the outcome (where pushing buttons classically results in effects), or would the interaction be better described as “full”? Clearly the example is problematic, and we might think of any number of other examples: a light switch that a user flips while simultaneously knowing it connects to nothing; a television screen that a child touches as if it were touch-sensitive; a synthesizer that one plays while it is off. How would Fishkin’s taxonomy account for such examples—and if it cannot, what categories might we add to help it succeed in this way?

panic-button

Lights, Metaphors, Action!

Reading about embodiment and metaphors helped me organize the TUIs I have encountered in my life into neat categories. The Philips Living Colors light at my home makes for a great discussion after reading this paper.

It uses an array of LEDs to generate 16 million different types of colors which it projects at an angle. It comes with a wireless remote control which can be used to change the color of the light. Since it’s operated using a remote, the output device isn’t the same as an input device, the Living Colors light isn’t an example of full embodiment. It’s not an example of nearby embodiment either since the output isn’t taking place in close proximity of the input device. However, it is an example of environmental embodiment since the output takes place around the user. There is a link between the remote and the LED controller, but the output combination of light is viewed exclusive of these.

As for metaphors, the metaphor of noun is really weak in this case since it doesn’t directly draw an analogy between any other light outputting device that the user may have experienced. The transparent material looks fancy and attractive but speaking strictly in metaphorical terms, it’s hard to say if the affordance immediately dovetails with the user’s mental model. The remote control on the other hand, exhibits metaphor of verb. It uses the concept of a touch-sensitive circular shaped dial, which any person who has ever used the iPod classic will immediately associate with. Sliding your finger on any particular color on the dial results in a corresponding color change in the LED controller, which is output immediately for instant feedback to the user. Due to the strong presence of metaphors used as guiding principles in the design process, it is easy and quick to learn how to operate this light. And once you get the hang of it, it’s a deeply satisfying experience to watch the lights change softly in response to your touch.

Living Colors

Living Colors

Mixed Reality ?!?!?

In Kenneth P. Fishkin’s, definition of taxonomy for Tangible User Interfaces, enough stress hasn’t been laid on an important aspect of interaction or the input i.e. the feedback aspect. We need to think about adding another dimension to the taxonomy. This could be based on how much humans perceive the interaction feedback(we could have partially perceived, fully perceived and not perceived categories).
Secondly I am not satisfied with the taxonomy as it limits the ways of interaction. Nowadays devices have started supporting other forms of interaction such as using eyes and brain waves. This dimension could be based on the different ways humans can sense or interact with an interface such as eyes, brain, speech.
The perfect example that comes to my mind to challenge the taxonomy would be Microsoft Hololens. When people wear the device, they see a sort of mixed-reality as there are objects seen that exist as well as objects that do not exist. So people can interact with virtual objects in the virtual world but gives a perception to the user as if the interaction is happening in the real world. Here, the feedback aspect is interesting as we want to know where should we actually categorize this experience because it would belong to all the embodiment types and almost all the metaphor types due to the mixed reality aspect. Hence we need to categorize this into perceivability. We would call this partially perceived as although we are able to see or hear things in the mixed world, we are still not able to touch it as we do not feel it.
In conclusion I would say that, although Fishkin’s article doesn’t cover new forms of interaction, it was a great piece of work for the time when it was written in and helped me clearly understand the basic taxonomy of tangible user interfaces.

8a71046e-7be5-4bb5-b4e2-0f6866fcaa21

Dance Dance Revolution

The UI I was thinking about while reading Fishkin, is the Dance Dance Revolution mat. DDR is a dance video game set up in arcades and available on the Playstation, which uses a mat that the player stands on and interacts with. The mat has arrows that are used to go up, down, left, and right when stepped on. This relates directly to the ‘distant’ embodiment, in that players stand away from the screen. Their input on the mat directly affects the game screen in front of them.

The metaphor being implied in this case is body movement directly representing similar body movement of the character in the game. A step forward on the ‘up’ arrow results in the character performing that action and scoring points if done at exactly the right time. Here, the TUI has a direct noun and verb analogy with its virtual counterpart. In this way, the taxonomy was helpful in understanding how we perceive and are intuitively able to use the DDR mat. It’s clear that the designers of the game wanted to provide an easy to learn experience.

ddr

The Rattling Chair

I’m interested in an interface that does not involve a user interacting with a physical object to induce an output. What if the user’s movements in their proximity within a given parameter of space were the input? The input of information by the user could be detected by sensors that don’t make contact with a physical object — such as sonar sensors, infrared, or motion sensors. I am interested in the ability of a human body to retain its memory of its position within an environment, and how it navigates itself through said environment. What does the body remember through its senses?

In a way this scenario might also be defined as a tangible user interface because it does take place in a physical space.

A blind person navigates through touch and sound. A deaf person navigates through sight and proportion. Each is an example of a person lacking one sense utilizes available senses to make sense of feedback. A blind person may move toward sensations that evoke a pleasing emotional response, such as a sound, a warmth, or a smell. This would be somewhat of an example of an environmental embodiment, where the output (like “Toon Town”) is ambient, but the input is not an avatar, but the user. Chris Downey, a blind architect who lost his sight in the aftermath of successful surgery to remove his brain tumor, speaks of discovering a world of information garnered from different sense that defines his relationship to his environment.

Here is a story from his Ted Talk that illuminates the range of emotion that comes with subsequent discovery of his surroundings:

“So, stepping down out of the bus, I headed back to the corner to head west en route to a braille training session. It was the winter of 2009, and I had been blind for about a year.Things were going pretty well. Safely reaching the other side, I turned to the left, pushed the auto-button for the audible pedestrian signal, and waited my turn. As it went off, I took off and safely got to the other side. Stepping onto the sidewalk, I then heard the sound of a steel chair slide across the concrete sidewalk in front of me. I know there’s a cafe on the corner, and they have chairs out in front, so I just adjusted to the left to get closer to the street. As I did, so slid the chair. I just figured I’d made a mistake, and went back to the right, and so slid the chair in perfect synchronicity. Now I was getting a little anxious. I went back to the left, and so slid the chair, blocking my path of travel. Now, I was officially freaking out. So I yelled, “Who the hell’s out there? What’s going on?” Just then, over my shout, I heard something else, a familiar rattle. It sounded familiar, and I quickly considered another possibility, and I reached out with my left hand, as my fingers brushed against something fuzzy, and I came across an ear, the ear of a dog, perhaps a golden retriever. Its leash had been tied to the chair as her master went in for coffee, and she was just persistent in her efforts to greet me, perhaps get a scratch behind the ear. Who knows, maybe she was volunteering for service. ”

In a designed scenario, a seeing person may remember where they are if they are able to see the objects that anchor his or her position within a space. Suppose they are in a dark, blank room with no objects or markers. They could be disoriented and emotionally unmoored. If they were to move toward one corner and find that that gentle lights illuminate that corner. Suppose in tandem to that light source, perhaps a warmth can also emanate (presence of heater) – the combination of these two sense could suggest a sense of being outdoors, of being in the presence of sunlight. If this person moves away from this area, the light and warmth diminishes. That particular corner would take on an emotional value. Conversely, another area in that blank room could trigger displeasing effects; the person would avoid that area. In short, what ultimately happens is that the user develops a spatial mapping of the environment based on their physical position within that space. It is also about a discovery of the relationship of their body’s physicality and how it relates to an environment, such as an urban community.

iPods; leading the way to modern day TUIs

A definition typically requires a narrow focus and a precise description. However, Fishkin takes the opposite approach in his attempt to define tangible user interfaces. Perhaps part of the challenge in narrowing the scope of TUIs begins with the first word of the acronym. He calls the word “tangible” “a multi-valued attribute” (Fishkin, 348) and, in an attempt to improve the TUI’s broad definition, he developed a taxonomy with two axes.

Throughout Fishkin’s article, I thought about various TUIs and realized that most of what existed when this article was published were either art forms or items of practical use; nothing stuck out to me as an everyday, useful, and enjoyable product. While a TV remote is incredibly useful, it’s only relevant when you’re watching TV. Most of his other examples attempted to bridge the gap between motion and engaging some aspect of our senses (i.e. movement and increasing a sound or a situation where rotation changes what one sees on a display like in the Great Dome example).

When I think of a TUI, I think of something that’s useful and practical in our everyday lives. While this use case may not have always been true, I believe the start of these practical and enjoyable TUIs began with products like MP3 players which first debuted in 1997. Although they may now seem outdated, MP3 players and iPods were revolutionary. They allowed a person to scroll through a digital library of their music using just their finger, creating what I believe to be a pure and useful TUI.

I think Fishkin’s taxonomy was helpful as it described the wide landscape of TUIs and the various environments they fit in. However, I felt that each category within embodiment could be adjusted to include just about any useful tangible object. Because the category is so large, it seems as though almost any item from our current environment has a home on the list. In order to improve this, I think narrowing the scope of what can be included within the embodiment axes of a TUIs taxonomy would be helpful.

With regards to metaphors, I agree with Fishkin’s argument that “once parts of an interface are made physically tangible, a whole realm of physically afforded metaphors becomes available (Fishkin, 349). He also goes on a define a metaphor, in a TUI sense, with the precision required of a definition. There’s no ambiguity about what could be classified as a metaphor (when referring to TUIs); there’s a clear “yes” or “no” response to whether “a system effect of a user action [is] analogous to the real-world effect…”(Fishkin 349). This axes and definition helps to narrow the scope of the wide-ranging world of tangible user interfaces.

 

ipod

source: https://goo.gl/images/C2OUlI

TUI Taxonomy

I wanted to see where Hiroshi Ishii’s pin-based shape display used for telepresence and other projects would fall on Fishkin’s taxonomy. Is it a full metaphor, where it is just an object to be interacted with, or is it only a verb metaphor when moving hands in one location moves them in another location? Is that distant? The effects are both local and far away. It seems to cross boundaries and be hard to define in the taxonomy.

For most other tangible user interfaces, Fishkin’s taxonomy was a great way to consider a spectrum of different interfaces that are called TUI’s. I appreciated the discussion about the proper amount of metaphor, and it helped me to consider how metaphor is helpful for learning, but strong metaphor can limit how an interface will be used.

With an interest in calm computing, I would like to see a similar taxonomy made that considers how much information is obtained without conscious effort. Many TUI’s seek to let the hands feel and guide an interface without the user explicitly thinking about how they feel, which is similar to the background attention given to calm computing.