Cook���s collage: Two exploratory designs

Cook’s Collage: Two Exploratory Designs Quan T. Tran, Elizabeth D. Mynatt College of Computing Georgia Institute of Technology Atlanta, GA 30332 USA +1 404 385 1102 {quantt, mynatt}@cc.gatech.edu ABSTRACT

Gathered from observations and studies of how people busily manage tasks amidst distractions, forgetfulness, and multitasking in their homes, we identify two common cooking predicaments. We introduce Cook’s Collage, a context-aware system that passively displays recent activities. We posit that a cook, by glancing at these visual cues, can easily resume his task following an interruption when he otherwise might not be able to recover. Keywords

Passive capture, serendipitous information, retrospective memory, smart home. INTRODUCTION

Throughout this work we investigate how people busily manage tasks amidst distractions, forgetfulness, and multitasking in their homes. Performing a task requires maintaining a sequence of actions in some form of shortterm memory. However, short-term memory, with its typical limitations, cannot hold all the needed information to complete a complex task; therefore, steps may be missed or forgotten. To mitigate these shortcomings, people use a number of strategies. One powerful strategy is to place the information “in the world” so that it no longer has to be remembered “in the head” [2]. But when the information “in the world” is transient, this strategy breaks down. To overcome this difficulty, and to complement common remembering strategies, we introduce Cook’s Collage, a context-aware system that captures the transient information of recent activities and passively displays them as visual cues. At home, the maintenance of daily chores and the ebb and flow of relationships are un-ending and vie for attention. For example, the task of cooking consists of multi-faceted instruction sequences, demanding time and attention to detail. However, the act of cooking produces a busy, noisy, and cluttered surrounding that also must be looked after. Additionally, the kitchen with its open space and central location in the home is prone to high traffic and extraneous interruptions. For these reasons, cooking can be readily taxing from interruptions and multi-tasking, and susceptible

to memory lapses. As illustration, we present two common cooking predicaments. Guiding Scenarios

For this scenario, labeled “white mix,” the cook is preparing a bread or cookie recipe. He adds flour, then baking soda, baking powder, salt, and so on. The telephone rings and interrupts him, so he turns his attention to resolve the distraction. After which, he returns to his cooking task only to realize that he has forgotten which ingredients he has already added, and which he has yet to add. Upon inspecting the mixing bowl, he discovers, much to his chagrin, that the uniformnity of the dry ingredients makes it difficult to discern the bowl’s content. A similar dilemma plagues clock devices that time cooking segments. The cook might forget to set the timer. Alternately, she might forget for what she had set the timer upon hearing its nondescript announcement of expired time. This ambiguity worsens when cooking multiple dishes simulatenously. Which timer corresponds to which dish? We label this scenario “missed timers.” SYSTEM PROTOTYPE

Motivated by these scenarios, Cook’s Collage explores how to support retrospective memory (i.e., keeping a record of transpired events). We do not address prospective memory (i.e., the system is not a cooking assistant or instructor) because daily cooking is often flexible, filled with improvisations. The system is composed of a smart environment and a collage display as shown in Figure 1. We employ simple sensing (e.g., contact, tilt, and photodiode sensors) to tag and track objects. We use low-end PC cameras to capture snapshots of the cook’s actions. The cameras are mounted just beneath the kitchen cabinets overlooking the countertops, rendering them virtually invisible and unobtrusive. The cameras capture close-up, hand shots on the countertops, yielding sufficient detailed images with minimal occlusion, while alleviating privacy concerns of “big brother” surveillance. We align the always-on collage along the kitchen wall as to set it inconspicuously within the surroundings. We embellish the collage with a decorative theme and recipe. Despite the recipe display, the system does not rely on understanding the recipe constraints and so remains free form to flexibly support any type of cooking activity.

Comic Strip Design

ACKNOWLEDGMENTS

In the “white mix” scenario, the dilemma consists of not knowing which actions had been performed and which had not. Cook’s Collage addresses this predicament by presenting a temporal ordering of the cook’s last six actions. The display utilizes the sequential layout customarily found in comic strips with the bottom right panel as the most recent. Glancing from the top-left image to the bottom-right image of Figure 2, the cook quickly reviews that he added flour, more cups of flour, then sugar, baking powder, baking soda, and finally salt.

We thank Khai Truong and Gregory Abowd for their collaboration with the initial prototypes. FIGURES

Figure 1. (a) containers (b) stovetop (c) camera position

Color Timer Design

Resolving ambiguities from the second scenario, Cook’s Collage correlates the top-bottom physical stove (Figure 1b) to the top-bottom display representation (Figure 3). Attached to each stove image, a clock notes in real-time the start and duration of different heat settings (e.g., yellowlow, orange-medium, red-high, transparent-off). The stove image updates when its corresponding heat setting changes. Leveraging spatial memory, the cook easily stays abreast of the progress of each stove burner. RELATED WORK

There has been little work in short-term capture and access applications, especially for everyday home life [3]. In contrast, video editing and comic books have extensively produced visual summaries. Converging these fields, Video Manga [1] uses the Japanese Manga style for layout and key framing for image selection.

Figure 2. Comic strip sequence of added ingredients

DISCUSSION & FUTURE WORK

We demonstrated both scenario prototypes to over 1,000 visitors at the ACM1 Exposition.1 Ordinary families immediately identified with the motivating scenarios, appreciated the added value of the passive reminding system, and deemed the close-range cameras unobtrusive and harmless. The smart environment adds no further learning or interaction obstacles for one to use the system. A cook needs not explicitly step through a recipe, actively alerting the system to record each action. The ambient display adds no further distraction. The collage provides information serendipitously, helpful when needed but otherwise easily ignored. Presently, we are designing user studies to evaluate the system by having a cook manage common household interruptions while he is preparing cookie dough in our enhanced kitchen. A human operator assembles the collage in real-time. In doing so, we will gain insight into how to compose effective visual cues and how to select which images to display. More generally, we will investigate how to more effectively support transient information about an activity and environment. We will examine questions such as how long into the past should the display preserve, and how quickly should the passive display transition? 1

ACM1: Beyond Cyberspace – A Journey of Many Directions. San Jose, CA, March 10-14, 2001. www.acm.org/acm1

Figure 3. Color timers for top-bottom stove burners REFERENCES

1. Boreczky, J., Girgensohn, A., Golovchinsky, G., and Uchihashi, S. An Interactive Comic Book Presentation for Exploring Video. CHI 2000 Conference Proceedings, ACM Press, pp. 185-192, 2000. 2. Hutchins, E. Cognition in the Wild. MIT Press, 1995. 3. Truong, K., Abowd, G., Brotherton, J. Who, What, When, Where, How: Design Issues of Capture & Access Applications. Ubicomp 2001 Conference Proceedings, ACM Press, pp. 209-224, 2001.