Experience 46
Andrew Chong, Vivian Liu & Owen Hsiao

Brief Concept:

Our midterm project proposal is an immersive installation piece exploring the following characteristics:

• Altering specific musical traits to intensify their capacity to induce chills/elicit an emotional response
• Eliciting agency and involvement of the “audience”, who becomes an active participant in the experience
• Minimal “cool” media that is fully engaging but open to perceptual interpretation

After feedback and discussion, we modified our base case to maximize the chances of an engaging interaction, by more actively enrolling the visual element.

Unexpected Concord:

Across the previous research cited in our earlier post, the ability of music to evoke chills or a strong emotional response typically relied on unexpected concord. The time frame, dimensions or network of elements brought into play can all differ. As an analogy, harmony relates to the “vertical” aspect of music, as opposed to melody, which is over time. Unexpected concord can both occur on both dimensions, with composers often enrolling more than one at the same time.

For instance, in Andrew Bird’s piece, the interplay of ambient, “un-music-like” birdsong, heavy strings and a high-pitch whale-like instrument employed what we called “spectral complexity” – utilizing different parts of the spectrum, including both pitch and texture, at each time.

https://soundcloud.com/andrew-chong-129716715/yawny

That spectral blending is visible from the spectrogram, with the bird song at the start of the piece as a scattered haze, and a “spectral gap” opening up between the heavy strings and high-pitch melody.

In Samuel Barber’s Adagio for Strings, the unexpected concord reflects itself as the resolution of drawn-out tension, where the time dimension comes much more heavily into play. Dissonance through the first five minutes of the piece builds, with a sense of coming resolution, continually drawn out, until a pause and resolution coming after a last phase of continuous rising dissonance, between 5.30 and 5.54. This gap is also clearly visible on the diagram.

https://www.youtube.com/watch?v=KylMqxLzNGo

Enlisting the visual dimension:

We decided to enlist the visual element as a way to maximize the chances of an engaging interaction for the participant. In our previous examples, unexpected concord occurred only in the audio dimension, though by utilizing both harmonic and melodic elements, and combining the pitch, texture and other elements of the music, composers were able to shape and intensify the experience of unexpected concord. In our installation, we wanted to enlist the participant and his actions as a way to create unexpected concord in both the visual and audio dimensions.

Specifically, where the participant moves to specific locations in the space, the visualization projected on each side of the wall, at first only peripherally related to the music, undergo a marked increase in resonance with the music. The result should both be suggested but somewhat unexpected, in keeping with our interest in using hidden affordances to engender exploration and curiosity for the participant.

We explain the interaction in more detail below. We decided on our base case as a canvas of the key elements we wanted to implement and put in place, that we could then iterate, experiment and improve on by swapping out algorithms, changing light effects, moving actuators, and modifying the participant experience in other ways.

Interaction Walkthrough

The participant enters a dark, enclosed room alone. Faint lights signal hidden affordances. After a lull, Andrew Bird’s Yawny at the Apocalypse begins to play.

https://soundcloud.com/andrew-chong-129716715/yawny

Movement towards each general direction induces at first subtle changes in the visualization/music. For instance, the participant’s position would subtly alter the relative loudness of different voices in the piece, as well as the transformations of the music into visualizations on each screen.

This interaction is meant to “coax” each participant towards three specific sites in the room, each of which will induce a different kind of “resonant” visualization with the music. At these specific sites, this “resonant” interaction is activated, with a marked increase in the stimulation and intensity of the visualization.

The entire experience might last several minutes, as the participant moves around the space to explore the specific affordances. While we designed the experience as primarily a solitary experience, it is interesting to explore how participants might interact with more than one person in the room, since actions are “shared” within the installation, with collaborative action required to engender specific effects. Exploration is likewise shared, but where discoveries may not be isolated and remain somewhat mysterious.

In terms of bodily experience – the participant is limited to the constraints of his own dimensions and position – in a smaller installation, he might be able to activate multiple effects alone, without the need to transition or “lose” any aspect of the experience.

As a solitary experience, a sense of tension/dissonance and discovery is accentuated by the “loss” of interactions as they move within the room. As a shared experience, other individuals might then become an avenue for each participant to reclaim that lost experience, with each participant, of course, also playing that role for other individuals in the room.

Such an interaction would be especially interesting to construct in a public space.

Other thoughts:

Yawny’s musical structure is more time-dependent than other pieces, which may detract from the visual interaction we’ve incorporated in the experience. Participants may feel like they’ve missed some aspect of the experience if their exploration of the room does not line up with the specific points in the piece where the composer has set up different forms of resolution.

One way to get around this is to choose pieces with still evocative, “unexpected concord”, but with a far more regular structure. Arvo Part’s Spiegel im Spiegel might be a good choice for this, due to the regularity of its musical structure, which can also be seen in the spectrogram.