Experience 46 – Progress Sketches

Experience 46
Andrew Chong, Vivian Liu & Owen Hsiao

Brief Concept:

Our midterm project proposal is an immersive installation piece exploring the following characteristics:

  • Altering specific musical traits to intensify their capacity to induce chills/elicit an emotional response
  • Eliciting agency and involvement of the “audience”, who becomes an active participant in the experience
  • Minimal “cool” media that is fully engaging but open to perceptual interpretation

After feedback and discussion, we modified our base case to maximize the chances of an engaging interaction, by more actively enrolling the visual element.

Unexpected Concord:

Across the previous research cited in our earlier post, the ability of music to evoke chills or a strong emotional response typically relied on unexpected concord. The time frame, dimensions or network of elements brought into play can all differ. As an analogy, harmony relates to the “vertical” aspect of music, as opposed to melody, which is over time. Unexpected concord can both occur on both dimensions, with composers often enrolling more than one at the same time.

For instance, in Andrew Bird’s piece, the interplay of ambient, “un-music-like” birdsong, heavy strings and a high-pitch whale-like instrument employed what we called “spectral complexity” – utilizing different parts of the spectrum, including both pitch and texture, at each time.


That spectral blending is visible from the spectrogram, with the bird song at the start of the piece as a scattered haze, and a “spectral gap” opening up between the heavy strings and high-pitch melody.

In Samuel Barber’s Adagio for Strings, the unexpected concord reflects itself as the resolution of drawn-out tension, where the time dimension comes much more heavily into play. Dissonance through the first five minutes of the piece builds, with a sense of coming resolution, continually drawn out, until a pause and resolution coming after a last phase of continuous rising dissonance, between 5.30 and 5.54. This gap is also clearly visible on the diagram.



Enlisting the visual dimension:

We decided to enlist the visual element as a way to maximize the chances of an engaging interaction for the participant. In our previous examples, unexpected concord occurred only in the audio dimension, though by utilizing both harmonic and melodic elements, and combining the pitch, texture and other elements of the music, composers were able to shape and intensify the experience of unexpected concord. In our installation, we wanted to enlist the participant and his actions as a way to create unexpected concord in both the visual and audio dimensions.

Specifically, where the participant moves to specific locations in the space, the visualization projected on each side of the wall, at first only peripherally related to the music, undergo a marked increase in resonance with the music. The result should both be suggested but somewhat unexpected, in keeping with our interest in using hidden affordances to engender exploration and curiosity for the participant.


We explain the interaction in more detail below. We decided on our base case as a canvas of the key elements we wanted to implement and put in place, that we could then iterate, experiment and improve on by swapping out algorithms, changing light effects, moving actuators, and modifying the participant experience in other ways.

Interaction Walkthrough

The participant enters a dark, enclosed room alone. Faint lights signal hidden affordances. After a lull, Andrew Bird’s Yawny at the Apocalypse begins to play.


Movement towards each general direction induces at first subtle changes in the visualization/music. For instance, the participant’s position would subtly alter the relative loudness of different voices in the piece, as well as the transformations of the music into visualizations on each screen.


This interaction is meant to “coax” each participant towards three specific sites in the room, each of which will induce a different kind of “resonant” visualization with the music. At these specific sites, this “resonant” interaction is activated, with a marked increase in the stimulation and intensity of the visualization.

The entire experience might last several minutes, as the participant moves around the space to explore the specific affordances. While we designed the experience as primarily a solitary experience, it is interesting to explore how participants might interact with more than one person in the room, since actions are “shared” within the installation, with collaborative action required to engender specific effects. Exploration is likewise shared, but where discoveries may not be isolated and remain somewhat mysterious.

In terms of bodily experience – the participant is limited to the constraints of his own dimensions and position – in a smaller installation, he might be able to activate multiple effects alone, without the need to transition or “lose” any aspect of the experience.

As a solitary experience, a sense of tension/dissonance and discovery is accentuated by the “loss” of interactions as they move within the room. As a shared experience, other individuals might then become an avenue for each participant to reclaim that lost experience, with each participant, of course, also playing that role for other individuals in the room.

Such an interaction would be especially interesting to construct in a public space.





Other thoughts:

Yawny’s musical structure is more time-dependent than other pieces, which may detract from the visual interaction we’ve incorporated in the experience. Participants may feel like they’ve missed some aspect of the experience if their exploration of the room does not line up with the specific points in the piece where the composer has set up different forms of resolution.

One way to get around this is to choose pieces with still evocative, “unexpected concord”, but with a far more regular structure. Arvo Part’s Spiegel im Spiegel might be a good choice for this, due to the regularity of its musical structure, which can also be seen in the spectrogram.



Brushy – the brushing companion

The child steps on the stool to brush teeth. There is a pressure sensor on the stool that activates the interface housed in a two way mirror.

The interface has a character (in this case Elmo, but can be changed as the child wishes). The character has his mouth open with teeth visible.

As the child grabs the brush and starts brushing her teeth, her hand’s motion is replicated by a brush in Elmo’s mouth.

The child can see Elmo’s teeth getting cleaner as she brushes.

Once all teeth are white, brushing concludes.

After brushing is done, Elmo says thankyou and goodnight (cuing the child to sleep).




Fun with Drawing Tools – Elena, Ganesh, Leah

Elena Lopez, Ganesh Iyer, Leah Rosenbaum

To create a collaborative puzzle for children which is fun physically and can be digitally enhanced for new ideas.

Fun with Drawing Tools

We envision this project as closely embodied, full-metaphor tangible UI where our main targeted audience – children aged 5 and above – can use drawing tools like a compass, a ruler and maybe a few other daily objects to contribute to a narrative in a game environment on a table top interface. The intention of the team is to encourage children to think of geometric tools in a more intuitive and intimate way. This project can also be thought of something adults can engage in to communicate with each other as conversation starters or as a game to play with over a get-together.

The puzzle is tied into a narrative where the players will use their tools to help a protagonist navigate her way to a reward. This environment would be offset by the tangible constraints that the drawing tools themselves pose and digital constraints through virtual traps in the puzzle which help make the game more difficult and engaging with progress and learned skill. The narrative that we’re looking to tie in this also involves the use of materials – straw, wood and stone – and this led us to exploring possible narratives like Three Little Pigs and the Big Bad Wolf or getting a colony of hardworking ants to secure food amid obstacles.

As a starting point, the most basic game environment could be a fluid path of dots indicating a reward-filled path to the goal and the task of the users is to connect these dots using the drawing tools that they have. To make the learning process more coherent and make gravity more intuitive, we chose to go with a birds’ eye view of the environment, so we are looking at the protagonist from above. The dots can involve materials that stack up across levels and the users may use up materials to build their path to the goal, so they might be nudged to draw optimal paths.

Fun with Drawing Tools 2

The game dynamics physically would be the users drawing paths using soft tools (since the age bar is low, we also want to make sure that the compass and rulers are not sharp/pointed objects). Straight-line paths can be drawn using a ruler and circular-directed paths can be made using a compass for the protagonist to travel along the circumference. An additional advantage is that users can indicate direction of the movement from the way they draw these constituent paths.

Fun with Drawing Tools 3

Next steps:
The next step for us as designers in this particular project would be to scope the project so that our vision is achieved. This would involve detailing out the game narrative – creating a story, reward system – and the virtual game dynamics in addition to understanding intuitive ways in which children use geometric tools so that the physical interactions that we have proposed can be refined.

Looking forward to hearing your feedback! 🙂

Calendar for grandma (Reema, Olivia, Yifei)

We aim to build an interactive calendar for elder people to schedule their activities and events easily with a selected group of people (families, friends, or social worker etc). These people can send voice messages to the elderly through an app and it will light up a corresponding magnet with the person’s image on the grandma’s (Lucy) wall (blinking lights mean action needs to be taken. Slow blinking: outgoing message and waiting for response from recipient. Fast blinking: action/confirmation pending from Lucy for incoming messages)


This is what the system looks like in resting state. The pictures of people are physical magnets which can be moved around and put on the calendar to indicate when an activity is scheduled with them.



Situation 1: The son sends Lucy a message through his app (e.g.  “Mom, I’m visiting on Tuesday”) and his magnet lights up. The light will turn off only after Lucy presses the magnet(and hears his recording) and then moves it to Tuesday. If she wants to reject this visit, she can double click the magnet and the light will go off, and the son will get a message on the app informing him of the rejection.



Situation 2: Lucy initiates interaction (“come visit me”) by moving the son’s magnet to Sunday. The light on the magnet will blink slowly till the son confirms (because he will get a notification on his app prompting him to act).


If he accepts the invitation, the light on Lucy’s wall turns off.


If he rejects it, the lights blink faster because Lucy is required to now move this magnet to its resting position or to another day.

Midterm Project Sketches – Daniel, Safei, Michelle

Project: Memento

Group: Daniel, Michelle, Safei

For our project, we are going to be making a tangible memento to track shared social experiences in a relationship. Below is a diagram for the information architecture.


As an example use case:
Two people in a relationship (parent-child, best friends, siblings, partners, etc.) who care about each other want to capture their shared experiences together so they can remember them later.
One person will get a pair of Mementos and give one to a person they care about, and keep the other.
These Mementos will be with the person at all times, and activate when they are near each other.
When these Mementos are activated, they capture information about the scene around them: if there is music, movement, light, geolocation.
These mementos then display a visual representation of the experience.
As experiences are captured and added onto previous experiences, creating a beautiful visual of the relationship.
When the two are apart, they always have these beautiful visuals of their time together, and they have incentive to hang out again soon to add more visuals to their relationship display.


This can be compared to the Pousman and Stasko ambient design principles (see image below).
The Information Capacity of the Memento is High: it is capturing multiple input from the user and the environment: geolocation, photosensitivity, movement, sound, and if the device is near others.
The Notification Level is Medium: the device exists primarily to showcase the relationship between the users, but can notify the users if they are not spending enough time together or opportunities to spend time together.
The Representational Fidelity is Somewhat High: the Memento does not display exact geo coordinates of the location or time of day spent together, but rather may change color or shape based on these features to represent them in a different way.
The Aesthetic Emphasis is Somewhat High: the device captures aspects of the relationship and displays them beautifully so users can reflect on their relationship and be reminded of the time they spent together from these abstract forms.


Kitchen Choreography & the Kitchen Workspace

Our main goal is to help those who want to learn how to cook and prepare beautiful meals, improve their skills. We want to help choreograph meals so that all components associated with cooking culminate simultaneously. For example, we will have several timers so you can track and time individual items separately. There’s also a digital display of the recipe on the board and a progress indicator which rewards the cook as they achieve milestones. Finally, another important component of presenting a meal is its plating. We also intend to provide suggestions to the cook so that they can prepare for and present their guests with a well-choreographed meal.

Product goals:

  • To educate someone who wants to learn how to cook how to do it well and to enjoy the process
  • Provide a guide to cooks so that they can plate their meals well

img_20161002_161313 img_20161002_163017 slack-for-ios-uploadkitchen-surface

Midterm Sketches – The Patch (Molly + Sasha)

We are proposing a patch that you wear on your skin. The patch can generate warmth, light (possibly in shapes), and pressure. The receiver can then send these sensations back to the original sender. See sketches below for examples.


Small patch, worn on the arm (similar to a temporary tattoo). Allows for sharing gestures, like a stroke of the arm, or sharing a drawing that you feel and that appears on your patch.


Pre-determined shapes (ex: happy face) could be shared when experiencing certain emotions. Can also share a feeling or warmth of a ‘hug’ via a pressurized warmth.