RR6 – VR and Mixed Reality

1) I particularly liked the feedback through the hand paddles. It was unexpected and really enriched the experience (opening drawers, pulling handles, etc.). As mentioned in the Magic Leap article, some other VR experiences give feedback through vests or clothes – maximizing the number of senses involved (while maintaining continuity between them) could be really interesting. I also felt like certain architectural features in the environments took advantage of being able to place the user in them (overhead beams and ledges, for example) that wouldn’t work as well for traditional displays.

2) At one point, I ended up on the menu screen (pressing a wrong button?) and didn’t know how to navigate it. This issue seems hardly specific to VR, but it felt like a stronger clashing of worlds in VR, maybe because I have the expectation that is should be seamless (have as few distractions from the VR space as possible). I found it similarly challenging to figure how to use the hand paddles – do the different buttons have consistent functions across environments? Finally, I wear glasses, and while I could see without them, the blurriness somewhat detracted from the experience. I imagine this is a widespread issue, and I’m curious what’s been done for it.

Prototyping Mindfully

In the article “Prototyping: Generating ideas or Cargo Cult Designs,” the author discussed the dilemma and limitation of prototyping. In the authors view, prototyping has been a successful and useful tool for interaction designers to foster creativity and to facilitate their design process and discussion with other team members. However, one should be careful in term of over promising the feasibility of these prototypes. In order to prevent having a feature that is not actually feasible in reality, the author recommend the designers to conduct researches about the technologies and their limitations.

In his article, the author argues that while prototyping provides a common focal and discussion point among different team members, over emphasis on the hypothetical features on the prototype could cause illusion to the team members regarding the reality of making these features into an actual product. This false belief of the pseudo feature of a prototype is what the author refers to as a “Cargo Cult Design.” Furthermore, in order to fix this problem, the author proposed that interaction designers should conduct prior researches regarding the background of the potential technologies and their limitations before designing the hypothetical features of a prototype. In this way, while having the discussion with the team, the designers will also have an idea about what’s actually feasible in reality rather than falling into the illusions that everything is feasible. In addition, the author also suggest that in order to prevent a prototype falls into a cargo cult experience, designers should realize that prototyping a helping the team to explore rather than an end product. 

Revisiting Threats and Peripheral Noise

The most plausible explanation for peripheral vision seems to trace itself to evolution. During their hunting and gathering phase, humans needed to be aware of threats that do not dominate their main line of vision; this kind of critical information also gives the perspective of direction of the threat. While Pousman et al’s paper focuses on ambient media that display non-critical information, the fidelity in today’s ambient media and our modern day interpretation of what a threat is can allow us to investigate innovative approaches in ambient media by revisiting of how we can feed critical information to our peripheral senses more responsibly.

Distractions are an unfortunate outcome of the current proliferation of information and some irresponsibly designed stimuli which can exist at the periphery of our senses start to demand our attention. Because of this, we might need innovations that diffuse the noise from our periphery and help focus our attention. Of course, this is not in the way the isolator helmet does it (courtesy Noura: http://laughingsquid.com/the-isolator-a-bizarre-helmet-invented-in-1925-used-to-help-increase-focus-and-concentration/), but ambient media may be able to distinguish peripheral noise from peripheral signal in real-time to help us focus. Interpreting threats – both from an evolutionary and a modern day perspective – would then become an important asset to ambient media despite the various definitions of ambient media relating to non-critical information. Directionally, threats can also extend to time and space – for example, a threat to a student can be missing an important deadline or making the student aware that her workload is about to multiply in the next few weeks.

Ambient media may also help augment our peripheral vision by helping us identify threats which lie beyond our peripheral vision too. For example, on detecting a potential threat pursuing the user at night, ambient media could trigger street lights or a nearby car alarm to attract attention to detract the threat.

Encalming Technology

In their article, Weiser and Brown give the Ambient Information Systems a name called Calm Technology. According to Weiser and Brown, “enhanced peripheral reach increases our knowledge and so our ability to act without increasing information overload” (P.2, Weiser and Brown 1995). This type of technology provide the users information they need to know for their actions in the background and only brings the user’s attention to these information when needed. The best example, which illuminate their point, in my view, is the inner office windows. According to Weiser and Brown, adding inner office windows in a office “connects people inside to the nearby world” by  providing information such as “motion of other people down the hall (it is time for lunch; the big meeting is starting) or noticing the same person peeking in for the third time while you are on the phone (they really want to see me; I forgot an appointment)” ,etc.

These kind of technology is very exciting because it utilize the senses that are normally peripheral to us to provide information which enhances the users’ actions. For example, in the “Dangling String” example, The users will be able to notice something wrong with the bit transmission by some irregular sounds from the strings. In this way the users can focus on other actions and shift their attention to the data transmission part of the system only when the irregular sounds occur.  As Weiser and Brown mentioned, in this way, “more information could be more encalming.” (P5, Weiser & Brown, 1995).

Sensory information in calm computing

First off, I found it very interesting that the authors of Calm Technology chose to discuss glass office windows as an example—I found it very fitting, but surprising! However, it helped me bring the concept of calm computing more into my everyday life, since I don’t regularly encounter items like the Live Wire piece.

One aspect I think is missing from the examples is the concept of progress. One of the reasons, for instance, that users might have a dashboard is to know what their schedule is, what is upcoming, or what time it is. Is there a way to use ambient media to signal progress along some sort of continuum? Currently there are many variations tackling that problem using GUIs and traditional patterns, but it didn’t seem that any of the examples covered it. I take it as an assumption that one would need “progress” to be in the center of their attention in order to grasp it, but why? After all, we can judge progress in terms of distance out of our peripheral vision, or when there are tangible items within our view. But I’m having trouble calling to mind an example of calm computing that incorporates notions of progress when the TUI is in the periphery rather than central.

I’d also be curious to discuss how to bring in our other senses to calm computing. For instance, vision and hearing play a big role in the examples: one is looking at a dashboard, seeing or hearing a Live Wire, or looking through or hearing people through a glass window. But what about taste? What about touch or skin conductance? (These are, after all, classified as TUIs—but there wasn’t any discussion of their tangible properties.) What about temperature variations? And what about smell? Those senses always alert us to potential danger—original ambient media, as another student pointed out in reference to smoke detectors—but how are artists and creators incorporating them into calm computing as a way to communicate meaning and information?

The Sixth Sense by Pranav Mistry

Pranav Mistry’s famous Sixth Sense interface that made a huge splash in the tech media after Pattie Maes’ TED Talk in 2009 (video here) was what I used as a frame of reference for TUIs in order to evaluate the taxonomy proposed by Fishkin. Why I regard the Sixth Sense as a strong example of TUI because it breaks conventions of desk-constrained computing and made it so that the interactions are intuitive and self-evident.

In terms of embodiment, the input in Mistry’s prototype is through color coded finger tips that are used to encode gestures which are then captured by a camera. The output is then projected onto surfaces in front of the user which are also interactive completing the augmented reality cycle. This kind of I/O touches various types of embodiments – full when the user is touching a projected button hinting at direct manipulation, nearby when the gesture indicates to scroll information on a projected page and environmental when the gesture takes an image (but doesn’t display the same image in front of you after it’s being captured hinting at the ambient nature of the process).

Metaphorically, Mistry’s prototype is heavily gesture-driven and the inputs to the device are based on how other devices act – hence, Sixth Sense falls into the ‘metaphor as verb’ category. Here’s where I think the taxonomy’s intention to pull all TUIs towards the full embodied and metaphor angle is not very convincing. While the prototype does feature a camera and a projector, a finished unified product does not necessarily have to appear like a camera and a projector – there are plenty of creative angles the designer could take on how it could look like and making it look like a camera and a projector does not seem very innovative. So Fishkin’s taxonomy is not very useful to guide design changes in existing innovations. However, it is a very robust concept as it helps place existing examples of TUI in various parts of the spectrum and in my opinion, by seeing innovations like the Sixth Sense appear in the non-extreme portions of the spectrum, it can also help guide what-if questions.

Fishkin’s Taxonomy and the Rain Room

One UI that I wondered about (and struggled with how to characterize), is that of the Rain Room. The Rain Room is an embodied, immersive art exhibit that allows a visitor to enter a room and be surrounded by falling water, without getting wet. The room itself responds to the presence of humans and adjusts the rainfall accordingly, so that each human has a small envelope within which they can stand and not be caught “in the rain.”

RainRoom

When reading Fishkin, the Rain Room seemed initially like an example of calm computing. My main rationale for classifying it this way was because the human participants don’t use their hands for controlling the rain—something Fishkin emphasizes as one of the requirements of a TUI rather than calm computing. However, there are many TUIs that do not rely on the hands for the input (someone else’s example of Dance, Dance Revolution comes to mind, where input comes from participants’ feet, mainly). Thus, classification of the Rain Room seemed somewhat problematic: the human participants do control the rain, in the sense that their physical motion is the input that the computer system detects, to then alter output accordingly. However, they do not control the rain with their hands, which seems to be a requirement. Yet, all of Fishkin’s examples of calm computing seem to revolve around systems that do not take any kind of direct input from humans, and the Rain Room most definitely does. If it were an example of calm computing, I would posit that it fits into the “nearby” category of embodiment (which Fishkin couldn’t find offer an example of), and perhaps the verb level of metaphor (after all, moving to avoid the rain in the rain room is like avoiding the rain in real life). Yet, in some ways it seems like “full” metaphor to me, because there is none. To avoid the rain in the Rain Room is to avoid the rain. You are physically being acted upon, experiencing the computers output, in the same manner as in real life.

However, in the end this doesn’t seem to me to fit clearly into any of the taxonomical categories—I would suggest modifying the taxonomy to include a broader definition of input than just hand manipulation, so that the Rain Room could be classified as a TUI rather than calm computing. Much like some of the other students mentioned, it might be helpful for us to discuss more deeply where to draw the line between “what is a TUI” and “what isn’t a TUI,” especially as our computing devices rely more and more on physical manipulation to accomplish our goals.

The handoff

The first thing that came to my mind when I read McCullough’s assertion that the computer is inherently a tool for the mind and not for the hands was a comment made by Steve Jobs during the early years of Apple – he said that the computer is the equivalent of a bicycle for our minds. While both statements reiterate that using computers help us accomplish tasks more efficiently, even these personalities, with a constant finger on technology’s pulse, couldn’t have predicted the rate at which the boundaries between tactility and cognition would begin to blur. Our hands continue to move in accord with the mind, but there’s

The idea of our mental and tangible facilities converging has existed in theory for a while, but only recently have we begun to experience it with virtual reality apps like TiltBrush. It is easy to get excited about such technology because it clearly represents a radical leap forward. However, I feel that studying what’s on the horizon is always helps put the current landscape in perspective. For instance, 3D TVs were all they rage when they came out, but it wasn’t until Oculus Rift that the user really began feeling the sense of ‘presence’ that content developers were aspiring for. Drawing an analogy, I would like to put VR in perspective by bringing in Project Soli.

Project Soli by Google utilizes an advanced radar sensor which is capable of sensing complex gestures and adept at translating them into computer input. For instance, twisting a virtual knob in mid-air can actually increase or decrease the volume of a pair of Soli-enabled speakers. Similarly, rubbing the index finger against the thumb can trigger the grab and scroll interaction on a Soli-connected screen.

Sensors like these open up an unprecedented world of interactions and allow for richer, more authentic user experiences. A technology like Soli can actually untether us from bulky VR headsets and enrich our surroundings with more natural interactions. Going back to McCullough’s assertion, I reckon that the computer is slowly becoming an appendage for the hand. Going ahead, we may not have to consciously retrieve steps from memory to accomplish day-to-day tasks. With a seemingly natural gesture of our hands, the computer shall do our bidding.

The Shift Towards Natural Body Movement

McCullough’s argument that computers are made for the mind and not for the hands is accurate enough to apply to most of computing devices today. However, increasing awareness and improvement have also happened in the past 10 years to make computers more adaptable and friendly to different parts of human organs. Let’s take iPhone as an example, Steve Jobs’s insistence that iPhone should be used easily with our fingers is a huge step forward to force software designers to develop their applications according to how users physically interact with them. Old palmtop computers are considered as failures because human are not used to holding a pen when they interact with computers. So the claim that computers are made just for the mind should be considered as rather outdated since physical interaction with computer is quite different now thanks to touchscreen technology.
On the other hand, there has also been movement of shifting the focus of making computers just for the mind in the past decade as well. Take google glass, virtual reality, and self-driving cars as example. These technologies are created so our hands can do less labor. The interaction is shifted away from the hands to the voice or to other bodily movement. Even though these movements have yet to be widely adopted, the focus of computers right now is leaning towards being a tool for human’s most natural movement. For example, if a task can be completed the most easily with our mouth, then the designer should strive forward to design the interaction with the mouth. Hands are definitely very subtle and sensitive, but the affordance of hands should also be considered because not all tasks should rely on hands. It’s interesting to note the creation of iPad pro has actually bring back the pen just because our hands aren’t precise enough for certain art creation process. So McCullough’s point of view on the relationship of hands are art is also being constantly challenged as technology develops. 

Multimodal Interactions and Passwords

To me, passwords are no longer just an ordered list of characters. My increasing proficiency with a standard US-keyboard has tied in the motion of my hands with the position and combination of the keys on the keyboard to generate the password. The muscle memory in reference here held me in good stead when remembering passwords across various websites and typing them out effortlessly. However, when I enter a need to type in a password to log into a service using my smartphone, the position and the combination of the keys on the touch keyboard are now altered, breaking my flow. What would follow is my visualizing the password as a string of keys and how that often works is reimagining my the hand movements on the standard physical keyboard.

Alluding to McCullough’s point about computers being a tool for the mind and not for the hands, this resonates strongly with me in almost an ironic way as while the keyboard-based computers are accused of restricting what our hands could do and touch-based devices seem to have freed the range of motion for our hands, touch devices in this example force a cognitive responsibility back on the mind and the hands play a subservient role again. This however, as a trend may not last long as technologies like replacing your password with your brainwaves can bypass the hands altogether and workarounds like Slack sending a ‘magic link’ to your inbox to log in are emerging and extant respectively. This is also ideal as typing a password isn’t exactly pleasurable, but it illustrates the importance that hands have in letting the mind not be burdened by mundane tasks.

In so far as an experience goes in terms of reutilizing the notion of touch and movements of hands, Virtual Reality apps like BowSlinger in The Lab (played on an HTC Vive) and Google’s TiltBrush are on the right track to revive the usage of hands in a more generative way. The former uses a multimodal interaction coupling the pull of the arrow on the bow with the sound of the string stretching to give you an illusion of tension in the string; and it certainly engages the hands more actively than writing out passwords. While the technology may be far too advanced to do away with the Vive hand controllers which hold the hand in a semi-closed position and include non-intuitive motions like having a trackpad near the thumb, we might be able to witness devices which provide feedback to various positions on our palms without it having to be in one type of position and also create resistance if needed. For example, would you be able to feel the weight of a heavy object if you tried to lift it in virtual reality? Or would the sensors on your palms recreate the feel of petting a dog?