Hackable urban experiences

Hackable urban experiences Marinela Telo Konstantinos-Alketas Oungrinis Technical University of Crete, School of Architectural Engineering, Transformable Intelligent Environments Laboratory(TUC TIE Lab) TIE Lab, TU Crete ,K4 Building, University Campus, Kounoupidiana, Chania, Greece, 73133 [email protected]

ABSTRACT The proposed paper presents an ongoing research project that focuses on re-defining the relationship between people and public space by bridging the physical environment with the digital realm. The project negotiates the parameters of this relation through an Augmented Reality tool that employs the technological features provided by enhanced reality glasses combined with a wearable computer and a wearable gesture-control device. The user has the ability to create virtual manifestations of a personal version of the surrounding urban environment and intervene digitally by painting, writing, posting and even designing within public space. The proposed AR tool provides users with the ability to creatively produce spatial 'tags' and comments upon a building's facade, a street's urban equipment, even within open spaces, helping them to release their imagination, express their own view about public space and shape a new type of spatial experience. Public space transforms into a canvas for personalized expressions of creativity and appropriations of the usual uncanny aspects of a city. The results of this interaction are artistic, architectural and/or social, but they primarily retain the personal view of the users' spatial perception. Aspects that are usually connected with private space are externalized, facilitating familiarity, initiative and fun. The aforementioned digital interventions can either remain personal or shared within a public cloud accessible by other users that forms a second layer of understanding the urban reality.

BRIEF BIOGRAPHY Marinela Telo is an undergraduate student in the School of Architectural Engineering at the Technical University of Crete and a researcher in Transformable Intelligent Environments Laboratory (TUC TIE LAB). Her research interests are in the area of customizing space based on personal needs, through actually and virtually transforming environments. She believes that individuality and taste have the potential to "mutate" space into something new, an unexplored communal but also personal perception of living. Furthermore, she is participating in research projects focused on the way software open-source tools provide designers with the ability to interact with models and design in the most immediate way. Konstantinos-Alketas Oungrinis is an Assistant Professor in Architectural Design and Innovative Engineering at the Technical University of Crete (TUC) in Greece. He is also the Director of the Transformable Intelligent Environments Laboratory (TIE Lab) at the same university. During 20042005 he was a Visiting Research Associate at the Harvard University Graduate School of Design. He holds a Professional Diploma in Architectural Engineering (1994) and a PhD degree in Building Morphology and Kinetic Structures in Transformable Spaces (2009) from the Aristotle University of Thessaloniki. He is the author of two books: 1)Transformations: Paradigms for Designing Transformable Spaces (2006) Harvard GSD Design and Technologies Report Series, Cambridge, MA and 2) Transformable Architecture: Movement, Adaptation, Flexibility (2011) ION Publishers, Athens. In 2008 he received the Europe 40 under 40 Architecture Award and in 2014 he received the Silver Award at the International Design Awards.

Hackable urban experiences Marinela Telo Konstantinos-Alketas Oungrinis Technical University of Crete, School of Architectural Engineering, Transformable Intelligent Environments Laboratory(TUC TIE Lab)

keywords Urban digital media; augmented reality; spatial experience; people-space communication.

INTRODUCTION The paper presents an ongoing research project that focuses on re-defining the relationship between citizens and urban space. The project negotiates the parameters of this relation through a concept of an Augmented Reality tool which provides users with the ability to create virtual manifestations of a personal version of the surrounding urban environment. The concept of this project stems from different fields of everyday life which include citizens' interventions with built environment, personal technological devices and the growth of social media usage. INTERVENTIONS IN URBAN BUILT ENVIRONMENT The Urban space, as a social and political space, is influenced by the activities of citizens. Since the activities of citizens in public space have various manifestations, this project is focused especially in citizens' interventions in built environment. People react and intervene in their urban environments with various ways and techniques (Photo 1). They use graffiti, art, even a simple way in order to express themselves with interventions to walls, traffic signs, trees, ads, and to any other element of modern city [1]. These kind of gestures in urban space interplay with fine art, architecture, performance, installation, activism and urbanism while they turn public spaces into individual experiences [1].

Photo 1. On the left examples of existing playful urban interventions, on the right examples of hacks on the MIT's Great Dome. Many of these interventions in public space have a critical or/and satirical character. At MIT a series of pranks and interventions have been described by the term of 'hack'. The word “hack” at MIT usually refers to a clever, benign, and “ethical” prank or practical joke, which is both challenging for the perpetrators and amusing to the MIT community (and sometimes even the rest of the world) [2]. Students come together, usually at night, and prepare their hack. For some of them, hacking is part and parcel of the MIT education. They work productively in teams, to solve engineering problems, and communicate them to the wider world. The process of hacking includes brainstorming, strategy sessions, preparation, test runs, and implementation. The success of a hack is almost proportional to the strength of its finer points. They make large objects appear in inaccessible places, such as rewire lecture hall blackboards, or place a police car or a replica of the Apollo lunar module at the top of the Great Dome (Photo 1). In this case, the term 'hack' has nothing to do with computer (or phone) hacking, which in MIT community is named “cracking”. André DeHon discusses how hacking reflects the Institute’s own value system. As he mentions “Hacks provide an opportunity to demonstrate creativity and know-how in mastering the physical world” [2]. The term hacking has expanded its meaning not only at the MIT community. Another definition of the meaning rises from the term "civic hacking". As Levitas defined it on the Code for America blog, civic hacking is "people working together quickly and creatively to make their cities better for everyone" [3]. In June 2013 took place the first National Day of Civic Hacking, where 91 "civic hackers" all around the United States of America brought together developers, designers, artists, urban planners and interested citizens. These folks used open data sets, made available by their governments, to design useful web or mobile applications for citizens, businesses and local government agencies, with the major purpose of improving communication, transparency and accessibility along these channels. Civic hacking expanded to a social movement taking place in communities across the USA. These examples reveal the tendency of citizens or members of a community to express their opinion, make their comments, collaborate with others to improve their surrounding environment and finally to interact with it. PERSONAL DEVICES IN URBAN ENVIROMENTS In the past years, citizens' behavior in public and personal space has gained extra characteristics through the use of technology and internet connectivity. Personal devices, such as mobile phones and tablets are constituent part of everyday life. The use of personal devices in public places often distracts people from making eye contact and face to face communication, but also (sometimes) has an impact on their spatial orientation ( Photo 2).

Photo 2. On the left first mobile phone sidewalks in China, on the right examples of different use of personal devices for interactions. Researchers started to explore the capabilities of interactions in urban spaces through different technologies, including personal devices. Media facades, which are described by Haeusler as the idea of designing or modifying the architecture of buildings by using their surfaces as giant public screens [4], emerged as basic infrastructure for interactions in public space. Interactive media facade provides a direct or indirect interaction mechanism that lets users access and manipulate its content [5]. Böhmer et al. used basic games to create playful engagement with a media facade, utilizing mobile devices as input devices [6]. Boring et al. present a way to interact directly with Media facades through live video on mobile devices [7]. They extend the Touch Projector interface, an augmented reality (AR) approach, allowing multiple users to interact simultaneously with a distant digital screen shown in the screen of a mobile device and using touch input in real-time, in order to accommodate multiple users by showing individual contents on the mobile display that would otherwise clutter the facade’s canvas or distract other users [7]. They built two collaborative multi-user applications: (1) mobile spray-paint application that enables users to change the facade’s color using a touch-screen device, and (2) a jig saw puzzle that requires users to rearrange colored tiles to a predefined order on the media facade [8]. As exemplified by Wiethoff and Gehring, allowing simultaneous interaction with a shared canvas can lead to frustration among the users [5]. However, the interactions with media facade are limited to a specific area where the infrastructure is installed. Scheible et al. used MobiSpray art tool for imposing large-scale ephemeral digital artistic projections on the environment [9]. MobiSpray combines a personal mobile phone, a PC and a video projector into a novel art tool for the creation of ubiquitous digital art. As an equipment that can be carried in a rucksack, it allows the creation and projection of ephemeral digital art anytime, anywhere, and on anything [9]. The users of MobiSpray have the freedom to change and personalize their surrounding environment, however, due to the ephemeral projection, the interaction between users can only be accomplished if they are on the same place at the same time. Another aspect of using personal devices in Urban Environment is the wearable computer system into augmented reality (AR) tool. The AR tool integrates 3D virtual objects into a 3D real environment in real time [10]. As Azuma describes "Augmented reality (AR) is a variation of virtual environments (VE), or virtual reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world with virtual objects superimposed upon or composited with the real world " [10]. Feiner et al. (in 1997) explored at first a way to combine augmented reality and mobile user interfaces, in order to investigate how these two technologies might together make possible wearable computer systems that can support users in their everyday interactions with the world. They created a prototype which assists users who are interested in Columbia university’s campus, overlaying information about items of interest in their vicinity [11]. In the past years, this field has grown rapidly, and as a result many such devices have been built.

SOCIAL MEDIA AND PHYSICAL ENVIROMENT Social Media, as defined by Kaplan and Haenlein, is a group of Internet-based applications built on the ideological and technological foundations of Web 2.0, that allow the creation and exchange of User Generated Content [12]. Despite this general definition, there are various types of Social Media that need to be distinguished further. Brian Solis developed in 2008 The Conversation Prism which is a visual map of the social media landscape (Photo 3), an ongoing study in digital ethnography that tracks dominant and promising social networks and organizes them by how they’re used in everyday life[13].

Photo 3. Conversation Prism The large family of Social Media includes Social networking sites which are applications that enable users to connect by creating personal information profiles, inviting friends and colleagues to have access to those profiles, and sending e-mails and instant messages between each other [12]. One of the largest social networking sites is Facebook with 1,415 million active user (according to Statista for March 2015 [14]). Facebook users have their personal profile which includes any type of information, and at the same time they have access to a home page, which reveals the news of their contacts, to groups, pages, applications and events. The activities of users in digital cloud of Facebook have references of physical world. For example, a Facebook user has three possible options about an event: "join", "maybe" and "invite". These choices have actually an impact on the event that will take place in physical world. Moreover, a Facebook activity that is immediately connected with physical world is the ability to check into a nearby location. This allows the users to add their current location, integrated with Google Maps, to their posts and information. PROPOSED METHODOLOGY The project proposes a personal device as a tool for a series of interactions and appropriations of public space through digital interventions that lead to a personalized urban experience as well as to create a new Social Networking where personal interventions are shared and applied from other users. The project negotiates the parameters of these interventions through an AR tool, an interface that employs the technological features provided by enhanced reality glasses combined with a wearable computer system and a wearable gesture-control device. The users through their personal wearable device have the ability to create virtual manifestations toward a personal version of the surrounding urban environment and intervene digitally by painting, writing, posting, even (re)designing within public space, using user-friendly software as paint, photoshop, google image search, word, even a 3d model software. For example, users can paint on a facade with different colors chosen from the provided color pallet, or they can draw a new image and tag it on a facade. They can write a message that expresses themselves and post it to a specific place that is important for them. They can 'tag' an existed image in open space or they can re-design a building facade according to their preferences (Photo 4).

Photo 4. Examples of same possible digital hacks. On the left a color hack, in the middle an expressive hack and on the right a personal note. The proposed AR tool provides users with the freedom of creating virtual interventions of their preferences anytime, anywhere, and on anything. The ability to creatively produce spatial 'tags' and 'comments' upon a building's facade, a street's urban equipment, even within open spaces, releases users' imagination and expression of their perception of urban environment. Furthermore, this ability shapes a new type of spatial experience. Christopher Alexander mentions " Those of us who are concerned with buildings tend to forget too easily that all the life and soul of a place, all of our experiences there, depend not simply on the physical environment, but on the pattern of events which we experience there” [15]. The proposed AR tool transforms urban environment into a virtual canvas for personalized expressions of creativity and appropriations of the usual uncanny aspects of a city. Harrison and Dourish highlight the critical distinction between “space” and “place” and show how it is actually a notion of "place", rather than "space", which frame interactive behaviour [16]. As they commented "We make a house into a home by arranging it to suit our lives, and putting things there which reflect ourselves. People make places in media spaces with just the same ideas of adaptation and appropriation."[16]. With the proposed tool, aspects which are usually connected with our private space as personal interventions and customization, are externalized in public space, in order to make place in public space, facilitating familiarity, fun and interactivity. The results of these interactions could be either artistic, architectural and/or social, but they primarily retain the personal view of the users' spatial perception, no matter their aesthetic. Every single person, through the experiences of his/her life, has developed a personal point of view and a personal way of understanding his/her surrounding environment. A recent example of citizens' individual expression into urban environment through digital tools is MobisSpray. Users commented: -“First I didn’t dare to do it because you have some kind of illegal feeling, but then when you do it, it

feels liberating.” -“This is fascinating, now I understand why kids do graffiti.” [9].

However, the users' personal expression is subjective, and as so, it might not be acceptable from other users. The proposed tool avoids this kind of issues through a personal device which allows each

user to see his own perspective. On the other hand, if every user had only his own perspective, social issues would rise upon interaction between users. In order to deal with social issues the proposed tool can evolve social networking application, where digital interventions can be shared in a public cloud and be accessible by other users. This Social network provides user the option to live his/her own personalized version of urban experience or explore a city through the “digital eyes” of other users. The potential of a social networking with digital interventions in surrounding environment could form a second layer of understanding urban reality. Information that such a social network can provide, might be crucial for groups of people that involve in urban environment. Such groups are architects, urban designers and sociologists. SYSTEM COMPONENTS The AR tool is proposed to be comprised of an augmented reality medium, with the most preferable to be glasses, such as Google’s or Microsoft’s, that features all the computational support from integrated CPU, GPU and HPU, in the case of HoloLens, required for the operation of the app. An additional crucial component is wearable gesture control device that will facilitate the users’ expressionism. The project proposal to use augmented reality glasses as the main platform is based on the notion that they can provide the user with a unique personal perspective of the surrounding environment. The computational power embedded into AR glasses is already significant and it is growing rapidly nowadays as a result of the possible applications, the projected impact on the market and the competition between the main developers. The vision, location, orientation, processing systems operated allow an increasing ability to manage complex data real time. The existing developed software provides a series of features to capture information, to process information, to align it to specific sites, to link the position and orientation of a user with a specific viewing angle, to assign virtual content on an actual location/object. Based on these abilities, a new software can be specified that can perform the required tasks for the proposed project:  Create and/or edit virtual interventions. This operation includes software that can create and/or edit images, messages, videos and 3d models by using an interface relating for example to programs such as paint and photoshop for images, HWR programs for messages, and Rhino, Maya and 3dsMax for 3d model design. This operation is crucial for the proposed application, and it is essentially the main scope of the project as it the main expressive feature that allow people to ‘hack’ their urban surroundings  Tag into place. This operation includes a software that is able to ‘lock’ the designed/processed information into a specific location on the environment. The exact position and the line of view will be also fixed so the ‘intervention’ will be ‘available’ for viewing  Share digital interventions. This operation describes the compatibility with existing social networks in order for a user’s personal view of the city can be shared. This feature allows for wider impact of virtual urban interventions, and also will improve the quality, as more people get involved in an AR civic hacking and techniques and tools will be improved and disseminated. In the end, taking into consideration the complexity of data that the proposed tool could handle and especially the fidelity with which the data is depicted, the issue of how the user will be able to control his/her device is raised. As the tool aims to excite and incite the users’ expression and creativity, a gesture control device is proposed. These type of HCI systems are met also with an increasing development trend as they surpass the disembodied use of a medium, such as the mouse, and recognizes/interprets directly the movements of the human body and its appendices. In this way the interaction with a computer system and the ability to control it is occurring without physical handling of some device but as an immediate expression of the hands. A characteristic example of such a device is presented by Intel in the Real Sense technology [J]. Real Sense is a gesture control system

miniaturized enough to be used with mobile systems but provide the abilities of the Kinect sensor, enabling a wide range of gestures to be utilized, enhancing expressionism. CONCLUSIONS & FUTURE WORK This paper presents the concept of an application that provides the user with the ability to create virtual urban interventions, personalize his/her experience in urban environment and share this experience with other users, through the use of AR mediums and specifically glasses. Furthermore, this paper describes the technological components that could support such a tool, as well as it argues that the current social operations is indicative that a tipping point for acceptance of such a tool has come. The goal of this project is the development of the proposed tool, through the development of the constituting parts, and especially the software allowing the expression and creativity of users. The project is in a conceptual stage, at the moment, with the scheduled first developing phase an initial operational prototype of the proposed tool, exhibiting only basic features as to apply only pre-edited images in surrounding environment at university campus of Technical University of Crete, Chania, Greece. The framework, in that stage, is to investigate qualitative elements of the relationship between digital and physical elements and how these two can be superimposed in a way that will be comfortable for the human cognitive mechanisms. The results of this phase will occur form the input provided through questionnaires and experiments, as well as the use of EEG devices, while the users will operate the prototype. When the operability of the app is evaluated, both technically and physiologically, additional capabilities of the expressive tools will be developed. For the hardware, standard tablets will be used at first, and at Microsoft’s HoloLens later, and the Intel Real Sense for the gestural control. For the augmented reality tools, an existing toolkit will be used such as the ARmedia 3D SDK, that can provide the platform to develop the first phases. The aim for the coming year is to complete a case study on a specific area in the old town of Chania, allowing interventions in the context of the expressions that different citizens would perform, projecting their different backgrounds, their sensibilities and their vision of the city, providing a reliable operating proof of concept.

ACKNOWLEDGMENTS This research project is being developed at the Transformable Intelligent Environments Laboratory (TIE Lab) in the School of Architectural Engineering at the Technical University of Crete, Greece, with the insights and contribution of Research Associate Marianthi Liapi. REFERENCES [1] [2] [3] [4] [5] [6]

Klanten, R. and Huebner, M. (2010), Urban Interventions: Personal Projects in Public Spaces,Berlin: Gestalten. Peterson, T. F. (2011), Nightwork, A History of Hacks and Pranks at MIT , updated edition, Cambridge, Massachusetts: The MIT press. Published in association with the MIT Museum. Levitas, J.(2013), 'Defining Civic Hacking', Code Of America, accessed 15 May 2015, http://www.codeforamerica.org/blog/2013/06/07/defining-civic-hacking/ Haeusler, M.H.(2009), Media Facades: History, Technology,Content. Ludwigsburg: Avedition. Wiethoff, A., Bauer, T. and Gehring, S. 2014, 'Investigating Multi-User Interactions on Interactive Media Façades', Proceedings of the 2nd Media Architecture Biennale Conference: World Cities,pp. 92-100. Available from: ACM Portal: ACM Digital Library.[10 April 2015]. Böhmer, M., Gehring, S., Löchtefeld, M., Ostkamp, M. andBauer, G. 2011, 'The mighty untouchables: creating playful engagement on media façades', in: Proceedings of the 13th

International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 605-610.

[7]

[8] [9]

[10] [11] [12] [13] [14] [15] [16]

Boring, S., Gehring, S., Wiethoff, A., Blöckner, M., Schöning, J. and Butz, A. 2011, 'Multi-user interaction on media façades through live video on mobile devices', in: CHI’11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2721-2724. Available from: ACM Portal: ACM Digital Library.[10 April 2015]. Wiethoff, A and Gehring, S. 2012, 'Designing interaction with media façades: A case study' in: DIS’12 Proceedings of the Designing Interactive Systems Conference, pp. 308-317. Available from: ACM Portal: ACM Digital Library.[10 April 2015]. Scheible, J and Ojala, T. 2009, 'MobiSpray: mobile phone as virtual spray can for painting BIG anytime anywhere on anything', in: SIGGRAPH ’09 Proceedings of the International Conference on Computer Graphics and Interactive Techniques, pp. 332-341. Available from: ACM Portal: ACM Digital Library.[10 April 2015]. Azuma, R. T. 1997, A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4 , pp. 355 - 385. Feiner,S., MacIntyre,B., Höllerer,T. and Webster,A. 1997,'A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment', in : Wearable Computers, 1997. Digest of Papers., First International Symposium on, pp. 74 - 81. Kaplan, A.M., Haenlein, M. 2010, 'Users of the world, unite! The challenges and opportunities of Social Media', in: Business Horizons Volume 53, Issue 1,pp .59–68. Brian Solis, The Conversation Prism, acceded 17 May 2015, http://www.conversationprism.com// Statista, The Statistics Portal, accessed 27 May 2015, http://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-ofusers/ Christopher, A. 1979, The Timeless Way of Building, New York: Oxford University Press Harrison, S. and Dourish,P. 1996, Re-Place-ing Space: The Roles of Place in Collaborative Systems, in: CSCW '96 Proceedings of the 1996 ACM conference on Computer supported cooperative work, pp. 67-76. Available from: ACM Portal: ACM Digital Library.[21March 2015].