Experience 46
Andrew Chong, Vivian Liu & Owen Hsiao

Brief Concept:

Our midterm project proposal is an immersive installation piece exploring the following characteristics:

• Altering specific musical traits to intensify their capacity to induce chills/elicit an emotional response
• Eliciting agency and involvement of the “audience”, who becomes an active participant in the experience
• Minimal “cool” media that is fully engaging but open to perceptual interpretation

After feedback and discussion, we modified our base case to maximize the chances of an engaging interaction, by more actively enrolling the visual element.

Unexpected Concord:

Across the previous research cited in our earlier post, the ability of music to evoke chills or a strong emotional response typically relied on unexpected concord. The time frame, dimensions or network of elements brought into play can all differ. As an analogy, harmony relates to the “vertical” aspect of music, as opposed to melody, which is over time. Unexpected concord can both occur on both dimensions, with composers often enrolling more than one at the same time.

For instance, in Andrew Bird’s piece, the interplay of ambient, “un-music-like” birdsong, heavy strings and a high-pitch whale-like instrument employed what we called “spectral complexity” – utilizing different parts of the spectrum, including both pitch and texture, at each time.

https://soundcloud.com/andrew-chong-129716715/yawny

That spectral blending is visible from the spectrogram, with the bird song at the start of the piece as a scattered haze, and a “spectral gap” opening up between the heavy strings and high-pitch melody.

In Samuel Barber’s Adagio for Strings, the unexpected concord reflects itself as the resolution of drawn-out tension, where the time dimension comes much more heavily into play. Dissonance through the first five minutes of the piece builds, with a sense of coming resolution, continually drawn out, until a pause and resolution coming after a last phase of continuous rising dissonance, between 5.30 and 5.54. This gap is also clearly visible on the diagram.

https://www.youtube.com/watch?v=KylMqxLzNGo

Enlisting the visual dimension:

We decided to enlist the visual element as a way to maximize the chances of an engaging interaction for the participant. In our previous examples, unexpected concord occurred only in the audio dimension, though by utilizing both harmonic and melodic elements, and combining the pitch, texture and other elements of the music, composers were able to shape and intensify the experience of unexpected concord. In our installation, we wanted to enlist the participant and his actions as a way to create unexpected concord in both the visual and audio dimensions.

Specifically, where the participant moves to specific locations in the space, the visualization projected on each side of the wall, at first only peripherally related to the music, undergo a marked increase in resonance with the music. The result should both be suggested but somewhat unexpected, in keeping with our interest in using hidden affordances to engender exploration and curiosity for the participant.

We explain the interaction in more detail below. We decided on our base case as a canvas of the key elements we wanted to implement and put in place, that we could then iterate, experiment and improve on by swapping out algorithms, changing light effects, moving actuators, and modifying the participant experience in other ways.

Interaction Walkthrough

The participant enters a dark, enclosed room alone. Faint lights signal hidden affordances. After a lull, Andrew Bird’s Yawny at the Apocalypse begins to play.

https://soundcloud.com/andrew-chong-129716715/yawny

Movement towards each general direction induces at first subtle changes in the visualization/music. For instance, the participant’s position would subtly alter the relative loudness of different voices in the piece, as well as the transformations of the music into visualizations on each screen.

This interaction is meant to “coax” each participant towards three specific sites in the room, each of which will induce a different kind of “resonant” visualization with the music. At these specific sites, this “resonant” interaction is activated, with a marked increase in the stimulation and intensity of the visualization.

The entire experience might last several minutes, as the participant moves around the space to explore the specific affordances. While we designed the experience as primarily a solitary experience, it is interesting to explore how participants might interact with more than one person in the room, since actions are “shared” within the installation, with collaborative action required to engender specific effects. Exploration is likewise shared, but where discoveries may not be isolated and remain somewhat mysterious.

In terms of bodily experience – the participant is limited to the constraints of his own dimensions and position – in a smaller installation, he might be able to activate multiple effects alone, without the need to transition or “lose” any aspect of the experience.

As a solitary experience, a sense of tension/dissonance and discovery is accentuated by the “loss” of interactions as they move within the room. As a shared experience, other individuals might then become an avenue for each participant to reclaim that lost experience, with each participant, of course, also playing that role for other individuals in the room.

Such an interaction would be especially interesting to construct in a public space.

Other thoughts:

Yawny’s musical structure is more time-dependent than other pieces, which may detract from the visual interaction we’ve incorporated in the experience. Participants may feel like they’ve missed some aspect of the experience if their exploration of the room does not line up with the specific points in the piece where the composer has set up different forms of resolution.

One way to get around this is to choose pieces with still evocative, “unexpected concord”, but with a far more regular structure. Arvo Part’s Spiegel im Spiegel might be a good choice for this, due to the regularity of its musical structure, which can also be seen in the spectrogram.

The child steps on the stool to brush teeth. There is a pressure sensor on the stool that activates the interface housed in a two way mirror.

The interface has a character (in this case Elmo, but can be changed as the child wishes). The character has his mouth open with teeth visible.

As the child grabs the brush and starts brushing her teeth, her hand’s motion is replicated by a brush in Elmo’s mouth.

The child can see Elmo’s teeth getting cleaner as she brushes.

Once all teeth are white, brushing concludes.

After brushing is done, Elmo says thankyou and goodnight (cuing the child to sleep).

We aim to build an interactive calendar for elder people to schedule their activities and events easily with a selected group of people (families, friends, or social worker etc). These people can send voice messages to the elderly through an app and it will light up a corresponding magnet with the person’s image on the grandma’s (Lucy) wall (blinking lights mean action needs to be taken. Slow blinking: outgoing message and waiting for response from recipient. Fast blinking: action/confirmation pending from Lucy for incoming messages)

This is what the system looks like in resting state. The pictures of people are physical magnets which can be moved around and put on the calendar to indicate when an activity is scheduled with them.

Situation 1: The son sends Lucy a message through his app (e.g.  “Mom, I’m visiting on Tuesday”) and his magnet lights up. The light will turn off only after Lucy presses the magnet(and hears his recording) and then moves it to Tuesday. If she wants to reject this visit, she can double click the magnet and the light will go off, and the son will get a message on the app informing him of the rejection.

Situation 2: Lucy initiates interaction (“come visit me”) by moving the son’s magnet to Sunday. The light on the magnet will blink slowly till the son confirms (because he will get a notification on his app prompting him to act).

If he accepts the invitation, the light on Lucy’s wall turns off.

If he rejects it, the lights blink faster because Lucy is required to now move this magnet to its resting position or to another day.

Project: Memento

Group: Daniel, Michelle, Safei

For our project, we are going to be making a tangible memento to track shared social experiences in a relationship. Below is a diagram for the information architecture.

As an example use case:
Two people in a relationship (parent-child, best friends, siblings, partners, etc.) who care about each other want to capture their shared experiences together so they can remember them later.
One person will get a pair of Mementos and give one to a person they care about, and keep the other.
These Mementos will be with the person at all times, and activate when they are near each other.
When these Mementos are activated, they capture information about the scene around them: if there is music, movement, light, geolocation.
These mementos then display a visual representation of the experience.
As experiences are captured and added onto previous experiences, creating a beautiful visual of the relationship.
When the two are apart, they always have these beautiful visuals of their time together, and they have incentive to hang out again soon to add more visuals to their relationship display.

This can be compared to the Pousman and Stasko ambient design principles (see image below).
The Information Capacity of the Memento is High: it is capturing multiple input from the user and the environment: geolocation, photosensitivity, movement, sound, and if the device is near others.
The Notification Level is Medium: the device exists primarily to showcase the relationship between the users, but can notify the users if they are not spending enough time together or opportunities to spend time together.
The Representational Fidelity is Somewhat High: the Memento does not display exact geo coordinates of the location or time of day spent together, but rather may change color or shape based on these features to represent them in a different way.
The Aesthetic Emphasis is Somewhat High: the device captures aspects of the relationship and displays them beautifully so users can reflect on their relationship and be reminded of the time they spent together from these abstract forms.

Our main goal is to help those who want to learn how to cook and prepare beautiful meals, improve their skills. We want to help choreograph meals so that all components associated with cooking culminate simultaneously. For example, we will have several timers so you can track and time individual items separately. There’s also a digital display of the recipe on the board and a progress indicator which rewards the cook as they achieve milestones. Finally, another important component of presenting a meal is its plating. We also intend to provide suggestions to the cook so that they can prepare for and present their guests with a well-choreographed meal.

Product goals:

• To educate someone who wants to learn how to cook how to do it well and to enjoy the process
• Provide a guide to cooks so that they can plate their meals well