Keyboard, Mouse, and “Jaedong’s Hands”

“Besides manual dexterity,” McCullough writes of becoming skilled on a computer, “you may feel some intellectual agility. You will learn to build mental models, and to switch frames of reference when necessary. You alertly monitor feedback from a variety of sources, and recognize and recover from errors before they compound themselves” (McCullough 26). In “Abstracting Craft,” McCullough identifies a problem of definition: to what extent can one consider “working on a computer,” or, better, “making work on a computer,” a kind of craft? The chapter concludes by suggesting that even though computation began as a discipline (seemingly) divorced from the hands, much of modern computer labor demands and fosters skills related to both physical dexterity and mental alacrity. The keyboard-mouse interface, often touted as less “natural” than other potential user interfaces like touchscreens (forgetting that there is no “natural” HCI, but only “naturalized” HCIs), can be mastered in such a way as to elevate the computer user to extraordinary levels of prestige and admiration.

Among the most extraordinary examples of wage-earning computer laborers who depend on mouse-keyboard dexterity are competitive gamers. “Jaedong,” a professional South Korean gamer who makes his living playing the game Starcraft (which conveniently has “craft” in its name), is featured in a YouTube video entitled: “Jaedong’s Hands,” which boasts around half a million views and a 96% “like” ratio (demonstrating the affinity its viewers feel for its subject). The camera hovers on the gamer’s hands as he plays the game, never drifting up to show either the game or the gamer’s face, reveling in his speed and coordination. The meaning of the video is clear: it is an homage to something happening; a culmination of years of rigorous training that manifests itself in a series of clicks and taps that register as entirely illegible when separated from the context of the game. Most of his clicks are the result of a kind of distal memory, where the fingers seem to act of their own accord.

But what is being created when Jaedong plays Starcraft? Is it a performance piece? An athletic event? (Are athletic events craft?) Is it merely media content? Do two players play to yield a culminating image of victory and defeat, like a picture of a Go board after a difficult game?

Finally, there seems to be a medium-specific relationship in gaming between “skill-based games” and “casual games,” where each successive move from keyboard-mouse to game controller, from game controller to touch screen, from touch screen to VR, yield lower and lower skill ceilings and potentials for at least one kind of mastery—the kind demonstrated by professional gamers. How will the most “skilled” VR pioneer behave? What will their movements look like? Is “richness” lost in the move to more “natural” methods of interacting with computers?

Frightened Computing (pt 2)

Oops–hit “post” before adding my second jpeg! The curled fingers in the following image should be read as the visual accompaniment to a user yelling at their arduino:

Frightened Computing

Chapter two of Acting With Technology begins by offering that “…recent trends in interaction design include emotion in design, extending usability to include the “pleasurability” of interactive products (Norman 2004) . . . affective computing (Picard 1997), affective design (Aboulafia and Bannon 2004), [and] autonomous characters (Tomlinson 2005) (Dourish 25). Using two LEDs instead of one introduces affective possibility into an otherwise inanimate series of wires and boards by alluding to a pair of eyes; and by coding the lights to temporarily turn off when the circuit is yelled at one creates perhaps the most basic relationship possible—but a relationship nonetheless, and one that a user should probably find emotionally moving. Relationships between humans and computers are often filled with hostility—a computer appears to not be behaving, and so we express our frustration at it. But what if computers responded to yelling with exhibits of fear, meekness, and regret?


• 1 Arduino Uno
• 1 Breadboard
• 2 LEDs
• Jumper Wires
• 1 Battery pack (4 AA)
• 1 SparkFun Electret Microphone Breakout


const int sampleWindow = 250; //250 MS sample width
unsigned int loudNoise;

void setup()
pinMode(12, OUTPUT); //these control the LED eyes
pinMode(2, OUTPUT);

void loop()

digitalWrite(12, HIGH); //these make the eyes “on” by default
digitalWrite(2, HIGH);

unsigned long start= millis(); // start of sample window
unsigned int peakToPeak = 0; // peak-to-peak level

unsigned int signalMax = 0;
unsigned int signalMin = 1024;

while (millis() – start < sampleWindow) // collecting data for 250 MS (can be changed)
loudNoise = analogRead(0);
if (loudNoise < 1024) // { if (loudNoise > signalMax) // if there’s a loud noise, save the loud value
signalMax = loudNoise;
else if (loudNoise < signalMin) // if there’s not a loud noise, don’t save the loud value { signalMin = loudNoise; } } } peakToPeak = signalMax – signalMin; // max – min = peak-peak amplitude double volts = (peakToPeak * 3.3) / 1024; // convert to volts Serial.println(volts); // to know if the mic is working if (volts >=1.0)
digitalWrite(2, LOW);
digitalWrite(12, LOW); //eyes shut off
delay(2000); // for a certain amount of time (though this could be modified for more expressive behavior)
Serial.println(“YELLING AT ME”);
digitalWrite(12, LOW);
digitalWrite(2, LOW); //not yetlling at me

Activity Theory: fundamentally humanist or anti-humanist?

When my father worked for Philly Carpenter’s Union he wore a hip-holstered flip phone by his right hand at all times. Because he was often using his left hand to hold, clamp, or carry projects, he had an elaborate one-handed method of detaching his phone from his belt, opening it, and answering it. It was an action that was itself a response to the technological availability of small devices, workforce exigencies of rapid communication, his own perceptions of masculinity, and in a way it stood as an act of social rebellion (“a phone holster—really dad?”). But the very act of picking up the phone was usually executed gracefully and with not a small amount of flourish.

To describe the same relationship with vocabulary offered by Kaptelinin and Nardi in Acting with Technology: Activity Theory and Interaction Design: Vern’s relation towards the object of his cell phone had “cross[ed] the border between conscious and automatic processes” (Kaptelinin 68)—he would answer the phone consciously, but his hand motion, which he mastered over a number of years, was performed automatically. Through his engagement with the phone he was able to conduct business, earn a living, gain status (or lose status, depending on whom you ask), and eat regular meals; his work, which was object-oriented by any measure of the phrase (he was a cabinetmaker), acceded in a special sense to the demands of his phone, and the particular human movements and responses on which that phone insisted.

While reading I’ve identified one of the central themes of the Kaptelinin and Nardi text (and to some extent the Dourish text, too) as “development.” The first three chapters of Acting with Technology, for example, based on my rudimentary and unscientific count of the text, include variations on the word “develop” around 200 times, which puts the average per-page count to just shy of three. This is to say: it is a text that is obsessed with the importance of development, whether the term refers to the development of a theory of mind, the development of a child, or the development of a subject’s relationship with the objects that define and orbit them. Even if one disregards the big-picture repetition of the word throughout the first three chapters of Acting with Technology, the text very explicitly identifies “the importance of human development” as the third tenet of activity theory (10). On the following page the authors write that “it would be desirable to establish a practice of design in which the development of users—their ability to grow and change with technology—is of paramount importance” (11). The beginning of Chapter three too hinges on Vygotsky’s increased attention to “developmental potential” of young people.

But what if a user—a subject—loses the ability to develop in relation to objects? While reading I’ve found myself turning my thoughts to a few specific communities: individuals diagnosed with dementia or Alzheimer’s, for example, or simply older individuals who have difficulty incorporating new information into older operational schematics.

To return finally to my first example: Vern, just before he was diagnosed with Parkinson’s and dementia in 2014, bought a new iPhone. The iPhone’s interface, even with Apple’s reputation for intuitive design, has been entirely intractable to him: where once he had a physical button that clicked when he accepted calls, now he has a slider that displays on the phone’s surface. And while one understands that digital literacy is a broader conversation than that which concerns specific populations with specific ailments here, I nevertheless wonder out loud: does a theory of relational ontology that privileges learning and development in its core necessarily prioritize dynamic human-object relationships over static human-object relations? Put another way: what is the minimal form of “action” required by the phrase “interaction”?