A Comparative Evaluation of User Preferences for Extra-User Interfaces

A Comparative Evaluation of User Preferences for Extra-User Interfaces Jérémie Melchior1, Jean Vanderdonckt1, and Peter Van Roy2 1

Louvain School of Management, Louvain Interaction Lab, Place des Doyens, 1 2Computer Science Department, Place Sainte-Barbe, 2 Université catholique de Louvain, B-1348 Louvain-la-Neuve, Belgium {jeremie.melchior, jean.vanderdonckt, peter.vanroy}@uclouvain.be

Abstract. This paper aims to investigate user preferences for extra-user interfaces, formerly known as meta-user interfaces, which are used to browse and/or control molding and distribution of a user interface in various applications, such as context-sensitive systems, across several displays, devices, or platforms. For this purpose, we defined a catalogue of fourteen distribution primitives that are typically provided by an extra-UI and classified into four categories, i.e., simple primitives, basic primitives, advanced primitives, and management operations. Based on this catalogue, a comparative analysis of the state of the art was conducted in order to identify which interaction styles have been properly used and in order to discuss the rationale behind these usages. From this analysis, we set up and conducted a comparative evaluation of user preferences by fourteen participants testing six selected distribution primitives in four different interaction styles. The research outcomes exemplified that there were significant differences on user preferences between interaction styles with regards to gender, experience level, and primitive type. Additionally, the comparative evaluation results have highlighted the potential use of new interaction styles (i.e., gesture, simple touch, and multiple touch) for supporting distribution primitives that have significantly gained user preferences and were well perceived as usable interfaces among the participants. The results of this comparative evaluation motivate the development of DISTRICT, a toolkit for providing distribution primitives according to different interaction styles depending on the end user. Keywords: Distributed user interfaces, Distribution primitive, Extra-user interface, Interaction style, Mega-user interface, Meta-user interface.

1

Introduction

An Extra-User Interface (Extra-UI) [23], formerly known as Meta-User Interface (Meta-UI) [9], is defined as the User Interface (UI) used for controlling the molding and distribution of a particular UI based on a library of self-explanative transformations [26]. An Extra-UI may cover a wide spectrum of services ranging from lowlevel functions such as changing parameters [15] until high-level services such as coupling/uncoupling interaction resources [4]. Designing and developing an Extra-UI todays remains an open and challenging research and development problem that is motivated by the following observations that augment existing findings [23,31]:  Extra-UIs have been successfully applied in several domains of computer science, such as ambient intelligence [3,21,22,26] (e.g., for controlling devices in a house), context-aware computing, especially for plastic UIs (e.g., for expressing how the UI should adapt to the context of use [10]), distributed systems [12,15] (e.g., for

2

Melchior, J., Vanderdonckt, J., Van Roy, P.

managing distributed user interfaces across displays [7,8]), Human-Robot Interaction (HRI) (e.g., in order to provide access to robot services), multimodal interaction [21,22] (e.g., for selecting input/output modalities [2]), pervasive [30] and ubiquitous computing [18,19], especially for Multi-Device Environments (MDEs) [13,29] (e.g., for controlling how parts or whole of UIs are migrated to different interaction devices), tabletop interaction [28] and in workflow management systems (e.g., for allocating UIs to end users depending on the work progress [8]).  Extra-UIs have been developed for different domains [31] of human activities (e.g., geography, traffic control) and objects (e.g., pictures [28], maps, services [31]) thus suggesting that they cover a wide range of interactive applications.  These Extra-UIs have been developed most of the time without any particular goal in mind other than supporting a particular function, even without knowing it, thus resulting into an ontological and methodological confusion on the terms and methods used for its development. Several Extra-UIs have been researched and developed with different goals and means that make them hard to compare.  The Extra-UI adheres to the principle “The end user should be kept in the loop” [10,30] relevant to the area of End-User Development (EUD), in particular when a mixed-initiative between the end user, the system itself, and perhaps any external third party occurs for controlling the UI. Developing an Extra-UI not only should result from user-centered design (an Extra-UI is after all a particular UI), but also from knowledge coming from other domains, such as distributed systems. These observations lead us to conclude that there is a common ground between pieces of related work in this area from which a set of generic functions could emerge that would benefit from both a theoretical and empirical analysis that could then be applied and retrofitted. Dearman & Pierce [11] report in their US study how 27 people from academic and industrial research were using their platforms: they revealed that on average they employ more than five computing platforms in four different configurations. These results suggest that functions should be provided for the end user on how to share UI portions across these platforms and contexts, but also how to divide them [13]. The association between a user’s activities and a particular device is problematic for multiple device users. Activities span over multiple devices, each being different depending on the users and the circumstances. Users assign different roles to devices by choice or constraint. Each user has personal techniques for accessing information across devices. Participants reported managing information across their devices as the most challenging aspect of multiple devices [11], thus encouraging improvements for an Extra-UI that effectively and efficiently supports these capabilities. In order to address the aforementioned shortcomings and the problem independently of the application domain as possible, this paper provides a comparative evaluation of user preferences for Extra-UIs. The remainder of this paper is structured as follows: Section 2 delivers a comparative analysis of the state of the art, Section 3 reports on a comparative evaluation of user preferences for interaction styles found for extra-UIs. Section 4 re-examines the notion of extra-UI under the light of new interaction techniques and Section 5 discusses how these lessons learnt have motivated the development of DISTRICT, a toolkit with an Extra-UI with multiple interaction styles. Section 6 concludes the paper by summarizing the main findings of the paper and discussing future avenues of this work.

A Comparative Evaluation of User Preferences for Extra-User Interfaces

2

3

Related Work

An interesting comparison could be established between the concepts introduced for UI control [26] of an interactive system and their counterparts in Model-Driven Engineering (MDE) (Fig. 1): the Final User Interface (FUI) consists of the running UI belonging to any interactive system; in order to control this FUI in case of remolding or distribution [9], the Extra-UI is located on top of the FUI from a software architecture point of view since it manipulates the FUI elements. For this purpose, any model pertaining to UI could be manipulated: UI model, task model, domain & concepts model, abstract UI, concrete UI, and context of use to name the most representative ones. In MDE, these are located at the M1 level of models. The Meta-UI is therefore the UI that is responsible for controlling the concepts of the models for UI, i.e. the UI metamodels. Similarly, the Meta-Meta-UI is the UI that is responsible for controlling the concepts of the UI meta-models, i.e. the UI meta-meta-models. Until now, we only observed Extra- UIs and Meta-UIs. In MDE, Salay et al. [25] defined the notion of macromodel in order to encompass all potential systems for controlling the interactive system at any level of abstraction, while maintaining consistency between them. Similarly, Sottet et al. [26] defined the notation of megamodel in order to encompass all potential systems for controlling the UI of the interactive system at any level of abstraction (ranging from M0 to M3). The remainder of this paper is related only to the M1 level in HCI, where the extra-UI is located (highlighted portion in Fig. 1). Roudaut & Coutaz [23] conducted the first state of the art ever made in comparing 25 extra-UIs in ambient intelligence against criteria representing: (i) the objects manipulated by the Extra-UI (i.e., their nature and manipulation), (ii) the extra-UI external presentation (i.e., embedded or not in the FUI, offering observability and/or predictability), (iii) capability to ensure services (i.e., resource discovery, assembly, distribution, remolding, and parameterizing), and (iv) their expandability. This survey revealed that many extra-UIs exist that provide similar or dissimilar capabilities, but with very different metaphors (e.g., the separation of jigsaw pieces will disconnect the objects or devices [20]) and interaction styles, thus suggesting a further study. Their survey compared high-level services, which suggests that also low-level services could be used as new comparison criteria to complement the survey.

Fig. 1. Structure of concepts in HCI vs. MDE.

4

Melchior, J., Vanderdonckt, J., Van Roy, P.

Vanderhulst et al. [31] pioneered the field by conceptualizing the first reference framework for extra-UIs for pervasive systems. In this framework, an extra-UI runs on devices that offer services depending on tasks executed by users. Then environment, the interaction resources, and the domains resources are then characterized and mapped. Based on this framework, we would like to further examine the services offered and the way they are offered to determine whether there are some aspects affected by the users carrying out these corresponding tasks. Based on these results, we defined and applied a procedure for conducting a comparative analysis of the state of the art in order to identify which basic services are supported, which interaction styles are used and for these services in order to discuss the rationale behind these usages. This procedure consists of the following steps: A. An exploratory study of related work based on the extra-UI definition and the 25 extra-UI analyzed in [23] resulted into identifying 43 pieces of work where an extra-UI was considered explicitly or implicitly. These pieces of work were discovered by literature review, systematic analysis, and browsing digital libraries. B. In these 43 pieces of work, we listed all basic extra-UIs services found and classified them into four categories of a catalog of distribution primitives ((hereby defined as an atomic function for ensuring the remolding and/or the distribution of a particular FUI to be offered by an extra-UI): 1. Simple primitives affect one or several parameters of a UI element (e.g., a simple control, a container, a widget) at a time: set parameter, get parameter value, display a UI element (with or without behavior), undisplay a UI element, and expose a UI element (only shows a UI element without any control over it). 2. Basic primitives reshuffle one or many UI elements at a time without affecting them: copy a UI element to another location, move a UI element from a source to a target location, switch between two UI elements, permute a list of UI elements, and add any UI element to a stack or removing it. 3. Advanced primitives reconfigure one or several UI elements at once while affecting them: merge a list of UI elements into a container, split the UI elements of a container or a list into separate UI elements, replace a UI element by another one (simple or compound), distribute UI elements across containers depending one a criteria, reset a previously executed distribution, append a UI element to a container or a list, compose or decompose interaction resources (as in [2,3]) and as analyzed in [21]. 4. Management primitives are responsible for maintaining the internal representation of the UI subject to change by the extra-UI without touching any UI element: save a configuration, restore a configuration, undo, redo, import an external UI model into an internal representation, export an internal representation into an external UI model or in a UI Description Language (UIDL). C. A final comparative analysis was obtained (Table 1) by: (i) keeping the most frequently found distribution primitives, thus reducing the list to only 17 distribution primitives, (ii) removing duplicate and more frequently found configurations based on the selected primitives, thus refining the initial list of 43 entries to 12 entries, which were all different with respect to at least one criteria. In the rest of this section, we only briefly discuss some representative examples issued from this comparative analysis. This discussion will help us to defining the requirements of the comparative evaluation to be conducted in the next section.

Permute

Append

5

Expose

Export

Import

Reset

Distribute

Separate

Switch

Merge

Transform

Replace

Move

Copy

Undisplay

Display

Set

Interaction style

Name, reference, and year

A Comparative Evaluation of User Preferences for Extra-User Interfaces

FormsVBT [1] DD, DM, T T T T P P P (1989) MW, PL CPN2000 [4] PM P P P P (2000) Proximal UI DD, DM, T T T T P P [18] (2003) MW Aris [6] (2004) MW, II T P P P JigSaw [20] DD, II P P P P P P P P (2004) AttachMe [11] II, MS T T T T T T T (2005) Lightweight MS P P P P P P T services [27] (2005) CESAM [21] DD, II P P P P P P P (2006) MigriXML [15] VR, DD, P P T P T (2006) DM Impromptu [7] MS P P P P P P T (2008) MASP [19,20] P P P P P (2009) Ext. Tcl/Tk PM P T T T T P P [14] (2009) District (2011 – CM, MS, T T T T T T P T T P P T T T T T P this work) DM Table 1. Comparative analysis of extra-UIs: their interaction styles (DD = drag & drop, DM = direct manipulation, MS = menu selection, MW = multi-windowing, II = iconic interaction, VR = virtual reality) and supported distribution primitives (P=partial support, T=total support).

Table 1 reproduces the analytical comparison for the finally selected 12 pieces of work against the possible interaction styles and 17 finally selected distribution primitives. Each distribution primitive is either partially totally supported when all UI element types are supported by the operation. When not all types are covered, the support is said to be partial. The Multi-Access Service Platform (MASP) [21,22] controls Distributed User Interfaces (DUIs) in smart environments through an extra-UI (Fig. 2a) based on a graphical representation of the different tasks that a user can display on a screen. Supported distribution primitives are involved in four services: migration for transferring a service from one interaction resource to another, adaptation to an interaction resource, distribution of UI elements across interaction resources, and multimodality. It allows the distribution of a task on a screen but does not give any choice of the UI that will be displayed. Nevertheless, it only focuses on one single task and has a limited multi-user interaction. There is no support for collaboration and it is not possible to control the distribution in finer granularity.

6

Melchior, J., Vanderdonckt, J., Van Roy, P.

Fig. 2. Some extra-UIs: (a) MASP [21,22], (b) Cameleon-RT [3].

Web sites and applications have been particularly investigated through the angle of distribution primitives. The Migration project [17] implemented an extra-UI for partial or total migration of web pages from one platform to another that is discovered at run-time, thus enabling flexibility in the migration. Cameleon-RT [3] provides a simple extra-UI (Fig. 2b) for controlling the distribution of parts of a website on several platforms. The user checks where to display parts of the website and which parts: the title, the contents, and the navigation bar can be displayed on mobile and stationary devices depending on the needs of the user. Lightweight services [29] includes an extra-UI enabling end user to access the UI of any service (e.g., zoom in/out, browse) on any device of any user (e.g., a mobile device for a standing person and a laptop for a sitting person). The only limitation is that the service could be requested only once and that the entire service UI is moved. Jigsaw’s [20] metaphor comes from a jigsaw: when pieces are connected together, they become connected. The separation of jigsaw pieces will disconnect the objects or devices. Fig. 3a graphically depicts a sentence saying the condition “if someone rings a bell, take a picture and send it to my PDA”. Fig. 3b reproduces a series of window services from one display to another, like copy, move, reshuffle, re-arrange, here based on tabs and metaphors borrowed from paper [6]. Most of the time, the extra-UI is separated [23] from the UI subject to remolding or distribution. For instance, AttachMe [12] adds icons to the desktop to successively detach toolbars from an application, migrate them to another platform, re-attach them on this platform, “plastify” them, and restore the initial configuration. Some metaphor of extra-UI may be weaved [23] into the UI itself: a scissor metaphor is used for cutting the window into two regions [23] (Fig. 4a). The representation of the world of discourse subject to remolding and distribution is also another property [23]. For instance, ARIS’s extra-UI [6] mimics the physical setup of a meeting room (Fig. 4b) with four walls having zero, one or two wall displays, a door, and a table on which a PDA and two devices are located.

Fig. 3. Some other extra-UIs: (a) Jigsaw [20], (b) window services [6].

A Comparative Evaluation of User Preferences for Extra-User Interfaces

7

Fig. 4. Advanced extra-UIs: (a) a scissor for cutting, (b) ARIS [7].

This extra-UI is triggered by clicking on a dedicated icon in the title bar (Fig. 4b) and allows the user to move a virtual representation of a window from where he started the extra-UI. Dragging a window from one device/display and dropping it onto another device/display results in effectively moving the window to its new position. IMPROMPTU’s [7] extra-UI (Fig. 5) supports collaboration between several users with a view on the windows shared by any user (Fig. 5a) and view on the windows shared by a user from the device where it is displayed (Fig. 5b). Multi-user collaboration is also supported in the Lightweight Services [29] thanks to a HTTP-based daemon allowing the distribution of web applications. Instead of creating a DUI, distribution of web applications through services offering is provided. They allow the full control of the distribution through user-driven distribution and support automatic distribution through system-driven distribution. This mixed approach is particularly interesting because it is almost the only one to offer this mixed-distribution mechanism. They are even going further with automatic redistribution in case of changes in the interaction space [30,31]. The work and UI of a disconnected device will not be lost, thanks to the redistribution of this part. MigriXML [16] exploits a virtual reality scene obtained from a physical environment model in order to render windows and dialog boxes of graphical UIs (Fig. 6). By pressing a toggle button for migration, the end user moves any window to any display in direct manipulation, which finally results into transferring a UI description from one platform to another. From this comparative analysis of the state of the art, we could draw the following conclusion: distribution primitives are effectively implemented in various pieces of work, but in many different ways, with different interaction styles and techniques. We do not know today which one is prevalent depending on the distribution primitive for an extra-UI to become widely acceptable. Little or no work was detected for any meta-UI (according to the definition of Section 2) since nothing was found for supporting the manipulation of the abstract UI, the context model, the task model or the domain model with direct implication on the lower levels. One notable exception concerns COMETS [26], a toolkit that supports manipulating the models required for running the final UI, either at design-time or at run-time.

8

Melchior, J., Vanderdonckt, J., Van Roy, P.

Fig. 5. IMPROMPTU’s extra-UI for windows sharing (a) and displaying (b).

Fig. 6. MigriXML’s extra-UI for moving windows across platforms.

3

Experimental study

Due to the heterogeneity of distribution primitives and their implementation demonstrated in the previous section, this section aims to conduct an experimental study in order to determine the user preference for particular interaction styles for each major distribution operation identified in Section 2. The results of this study will motivate the development of DISTRICT (last line of Table 1), described in Section 4. 3.1

Method and protocol

Participants and apparatus. We conducted a user trial of 14 participants (8 female, 6 male) who were recruited from a database of volunteers coming from different disciplines (e.g., marketing, finance, medicine, management) and having different ages (22% between 19 and 25-year old participants, 57% between 26 and 40-year old participants, and 21% of more than 40-year old participants). No participant has any prior knowledge of an extra-UI. The physical setup was similar for all participants. The computer used for the experiment was equipped with an Intel Pentium M 1.6GHz CPU, 2Gb DDR of RAM and a 15 inches LCD screen with a resolution of 1400 x 1050 pixels. This apparatus was selected as it was considered representative. Task and procedure. Each participant received a detailed explanation of the experimental study that was uniformly conveyed through an interactive presentation (implemented with the animation language of Microsoft PowerPoint 2010): to express their preference for interaction styles for distribution primitives. Following the short introduction to the test procedure and test purpose, they performed some training with the tool. Following the training session, the 14 participants were presented 6 distribution primitives (i.e. Set, Copy, Move, Replace, Merge, and Split), each with four dif-

A Comparative Evaluation of User Preferences for Extra-User Interfaces

9

ferent interaction styles for each operator (i.e., A=form filling with iconic interaction, B=direct manipulation with drag & drop, C=command language, and D=menu selection). Each interaction style was implemented as animated examples thanks to Microsoft PowerPoint 2010 macro-command language. Each participant received a different sequence of interaction styles in order to avoid the order influencing the results. There are 14 sequences and each distribution primitive has a different sequence that was generated by a pseudo-random generator. Table 2 (left) shows the order of interaction styles for each sequence and Table 2 (right) shows which sequence was selected for each distribution primitive for each participant. Each participant therefore provided an answer to the following questions:  Which devices have you already used or owned? (desktop, mobile phone, tablet PC, interactive kiosk)  What is your experience level with a computer/laptop, mobile phone/smartphone, tablet PC? Interactive kiosk? (preference on a 5-point Likert scale)  What is your favorite style for each distribution primitive (A, B, C, or D)?  What are the aspects that you found the most interesting, if any? (open question)  What are the aspects that you found the less interesting, if any? (open question)  What do you think of using pen-based gestures for some distribution primitives? (open question)  What do you think of using simple/multi-touch gestures for some distribution primitives? (open question) The dependent variables used to assess the participant task performances were twofold: the preferences for an interaction style for each distribution primitive (i.e., A, B, C, or D) and the comments gathered for each interaction style (i.e., number of positive and negative comments for each interaction style). All the relevant data captured from the presentation were captured in a log file to be imported in a statistical software package. We now justify these main choices of this setup. ID 1 2 3 4 5 6 7 8 9 10 11 12

Random Sequence C A A C B D B B A B D D

Par- SET COPY MOV REMERG ticip. E PLACE E B A D 1 1 2 3 4 5 B C D 2 2 3 4 5 6 B D C 3 3 4 5 6 7 A D B 4 4 5 6 7 8 D A C 5 5 6 7 8 9 A C B 6 6 7 8 9 10 C A D 7 7 8 9 10 11 A C D 8 8 9 10 11 12 C D B 9 9 10 11 12 1 C D A 10 10 11 12 1 2 A C B 11 11 12 1 2 3 C B A 12 12 1 2 3 4 13 1 12 11 10 9 14 9 8 7 6 5 Table 2. Random sequences generated and their repartition by participant.

SPLIT 6 7 8 9 10 11 12 1 2 3 4 5 8 4

10

Melchior, J., Vanderdonckt, J., Van Roy, P.

Justifications. When it comes to evaluate alternative UI designs, Tullis & Albert [27] report that a couple of self-reported metrics are particularly relevant. One is asking each participant to choose which alternative interaction style they would most like to use in the future for a distribution primitive as a forced choice comparison. Another one is asking each participant to provide comments on each alternative interaction style dived into two classes: what were the most positive aspects that you appreciated (if any), what were the most negative aspects that you regretted (if any). This is what we did. In order to consistently analyze all verbal comments, Tullis & Albert’s protocol for verbal analysis was used that classifies any comment into three classes: positive (when a clearly positive tone is expressed), negative (when a clearly negative tone is expressed), or neutral (when no clear tone is expressed or in any other case). The six distribution primitives selected were the most frequently found ones in the literature that were considered fundamental for an extra-UI and because of their associated spectrum of interaction styles. Indeed, we did not find all interaction styles possible for each primitive. SET was the first primitive selected for its simplicity and its largest scope of possible interaction styles. COPY and MOVE were also selected for the same reasons in the set of basic primitives. SWITCH and PERMUTE are less interesting because they could be obtained as a composition of other primitives such as MOVE. The three last operations were REPLACE, MERGE, and SPLIT because of their large coverage and representativeness of the aims and goals of an extra-UI. The DISTRIBUTE and RESET primitives require too much implementation effort, APPEND primitive is a sub-primitive of MERGE, they were not selected. The four interaction styles were selected based on their intrinsic properties (Table 3) and because some interaction styles are typically combined with others. For instance, the command language is assumed to be appropriate when the task prerequisites are moderate, the task productivity should be high, and so forth. Interaction

Prereq-

Produc-

Objective

Task struc-

Task

Task

style

uisites

tivity

environment

turing

im-

complexity

portance Command

moderate

high

non existent

Low

high

minimal

moderate

non existent

moderate to

low

moderate

language Menu selec-

low to moderate

tion

high

Form filling

moderate

moderate

Existent

High

high

moderate

Multi-

moderate

moderate

Existent

low to mod-

high

high

windowing

erate

Direct

minimal

manipulation

to maxi-

moderate

Existent

Low

high

high

Moderate

high

Existent

moderate

moderate

low

Minimal

low

Existent

low to mod-

low

low to mod-

mal Iconic interaction Virtual reality

erate

erate

Table 3. Characterization of interaction styles in terms of task parameters.

A Comparative Evaluation of User Preferences for Extra-User Interfaces

3.2

11

Results and discussion

The survey was based on a 14x6x4 factorial design: 14 participants were involved, 6 distribution operators were selected and 4 different interaction styles for each operator. All the fourteen participants completed the 24 trials, thus giving a total sampling of 336 trials. No outlier was removed since all tasks have been completed without any problem and interruption. Fig. 7 summarizes the participants’ experiences regarding their usage of various computing platforms, ranging from no experience to expert. 7 6 5 4 3 2 1 0

None Little Regular user Experimented Expert

Fig. 7. Platform experience of participants.

14 12 10 8 6 4 2 0

d c b a

Fig. 8. Distribution of preferred interaction styles for each distribution primitive.

Fig. 8 graphically reproduces the results obtained for the user preferences: -

-

Form filling with iconic interaction (A) was considered globally as the most preferred interaction style for advanced primitives taken together (i.e., Replace, Merge, and Split) in terms of occurrences and a percentile analysis revealed the the 50%-percentile (x50%) is in favor of this alternative, followed by menu selection, and direct manipulation. Direct manipulation (e.g., with drag and drop - B) is the most preferred interaction

12

-

Melchior, J., Vanderdonckt, J., Van Roy, P.

style (B) for the Set primitive belonging to the set of simple primitives, probably because of the visual counterpart of the UI element property. This was assessed in terms of occurrences and of the 90%-percentile (x90%) which is in favor. Direct manipulation is also the preferred interaction style for Copy and Move, belonging to the set of advanced primitives. This was assessed in terms of occurrences and of the 55%-percentile (x55%) is in favor of this alternative Command language (C) remains the least preferred interaction style. However, a one-way ANOVA procedure (F=1.6933, p=0.1588) suggests that only highly experienced participants appreciated this interaction style in general. No other statistically significant correlation was found. Command language was only appreciated for the Split primitive since it involves several complex parameters as opposed to more obvious parameters for other primitives. 14 12 10 8 6 4 2

Neutral Negative Positive

0

14 12 10 8 6 4 2 0

Neutral Negative Positive

Fig. 9. Distribution of participants’ comments for all distribution primitives (a) and all interaction styles, explicit or implicit (b).

A Comparative Evaluation of User Preferences for Extra-User Interfaces

13

Figure 9 graphically reproduces the distribution of comments gathered from the participants regarding the overall usage of the 6 distribution primitives (Fig. 9a) and regarding the overall usage of interaction styles for all primitives (Fig. 9b). All the tested primitives are considered vital by participants (in terms of occurrences and a 50%-percentile (x50%) which is in favor globally speaking). Participants’ feedbacks are mostly positive for the primitives. Each primitive has always at least one interaction style that makes it easy to use and natural to understand, particularly Drag & Drop. The styles can vary depending on the context of use, such as the SET operation. There is a question about how precise a primitive like SET should be if we use direct manipulation or Drag & Drop. The diversity of styles proposed for each operation is well appreciated. If the SET has to be precise, they will use direct values by command language, modifying code or changing values of a property. The results for each style taken individually are less mitigated. While Drag & Drop is widely the most interesting and natural one for the participants, command language and code modification are again the less interesting and less natural. Participants have approved the other styles but seem less convinced by them. The menu selection has been criticized because less intuitive than direct manipulation and Drag & Drop. Virtual reality is very interesting for the MOVE operation. Participants say that it is nice to have Drag & Drop features in virtual reality. It allows the user to see what she is going to do and to specify exactly where she moves the UI elements. The reactions for pen-touch-multitouch interactions are separated. About a half of the participants likes these interactions because it is more natural and because we have fingers that can be used for that. There more positive feedbacks for single-touch than multi-touch because participants found multitouch a little less precise and easy to use. The negative feedbacks got for both interactions are the complexity and the feeling of using fingers as interaction mechanisms.

4

Implementation

Motivated by the results of the experimental study, we implemented DISTRICT, a toolkit that provide distribution primitives to the developers so that they can be manipulated through three interaction styles: by command language for expert users who are willing to be efficient, by menu selection for intermittent users who can complete their command by selecting items (e.g., commands, parameters) from the corresponding menu, and by direct manipulation based on the physical properties of UI elements, as suggested by the experimental study. It creates application with UI separated in two-parts: the proxy and the rendering. The proxy is represented as a separate part of the application than the rendering. The first keeps the state of the application and ensures the core functionalities, while the second displays the user interface. Application supporting DUI allows the rendering to be distributed on other platforms while the proxy stays where the application has been created. The toolkit runs in MS Windows 7, Apple Mac OS X, Linux, and Android. The Apple iOS is under development. Each UI element is described as a record containing several properties and values. It ensures compatibility with XML because the keys/values become the name/value pairs of the XML markup. The DUI can be controlled by a command line interface, a meta-UI or even by the applications themselves. The toolkit is developed

14

Melchior, J., Vanderdonckt, J., Van Roy, P.

upon the multiplatform environment Tcl/Tk. In order to formally define the language expressing distribution primitives, an Extended Backus Naur Form (EBNF) grammar has been defined. The distribution primitives supported are:  SET  TO {value, percentage} [ON ]: assigns a value to a CUI widget property or a percentage of the actual value on a platform identified in a cluster. For instance, SET “pushButton_1.height” TO 10 will size the push button to a height of 10 units while SET “pushButton_1.height” TO +10 increases its height by 10%. Note that the platform reference is optional: when it is not provided, we assume that the default platform is used.   DISPLAY  [AT x,y] [ON ]: displays a CUI widget at a x,y location on a platform identified in a cluster, where x and y are integer positions (e.g., in characters or pixels). For instance, DISPLAY  “pushButton_1”  AT  1,1  ON  “Lap‐ top” will display an identified push button at coordinates 1,1 on the laptop. UN‐ DISPLAY    [AT  x,y] [ON  ]  is the inverse operation. DISPLAY    [AT  x,y] [ON  ]  displays a given message on a designated platform in the cluster (mainly for user feedback in an optional console).  COPY    [ON  ]  TO  []  [ON  ]: copies a CUI widget from a source platform identified in a cluster to a clone on a target platform, thus creating a new identifier. This identifier can be provided as a parameter to the primitive or created automatically by the primitive to handle it.  MOVE  TO x,y [ON ] [IN n steps]: moves a CUI widget to a new location indicated by its coordinates x and y, possibly in a fixed amount of steps, on a target platform in the cluster.  REPLACE    BY  : replaces a CUI widget Widget1 by another one Widget2. Sometimes the replacement widget could be determined after a (re)distribution algorithm, thus giving the following definition: REPLACE   BY  . This mechanism could be applied to contents and image transformations: images are usually transformed by local or remote algorithms (e.g., for resizing, converting, cropping, clipping, repurposing), thus giving the following definition: TRANSFORM  BY .  MERGE    [ON  ]  TO  []  [ON  ]: merges a collection of CUI widgets from a source platform identified in a cluster into a container widget on a target platform, thus creating a new identifier. Again, when source and target platforms are not provided, we assume that the default platform is used. SEPARATE is the inverse primitive. SEPARATES   [ON ] TO [] [ON ]: splits a collection of CUI widgets (typically, a container) from a source platform identified in a cluster into CUI widgets on one or many target platforms.  SWITCH    [ON  ]  TO  []  [ON  ]: switches two CUI widgets between two platforms. When the source and target platforms are equal, the two widgets are simply substituted.  DISTRIBUTE    INTO    [BY  : computes a distribution of a series of UI Elements into a series of UI Containers, possibly by calling an external algorithm, local or remote.

A Comparative Evaluation of User Preferences for Extra-User Interfaces

5

15

Towards New Interaction Styles for Extra-UIs

The extra-UIs allow the user to trigger several operations. Nevertheless, when the number of operations become high, it is too difficult for the user to find the right button or action to trigger the operation. A solution for this problem would be to use natural interaction between the human and the computer. The most natural interactions are with pen, tactile and multi-touch. In the survey there were two questions about these alternative interactions and metaphors. There are positive feedbacks about using touch interaction but there are also negative feedbacks because of the complexity and precisions of the metaphor. Some interactions are easier with a mouse and a keyboard, while others are more natural with our fingers. In Table 4, some gestures have been proposed to realize the primitives. The first result of this table is that some operations do not correspond to a natural gesture. What kind of gesture could we do for copy and paste with a pen or with only one finger? When no pertinent gesture was found, they have been placed in a contextual menu. This menu can be opened with one finger maintain more than one second, or by tapping with two fingers. The content of the menu depends on the action point of the interaction. If the user hits an object, she gets specific operations for this object. If she hits a border of an object, she gets inputoutput operations.

Table 4. Revisiting distribution primitives for gesture, single and multi-touch interaction.

16

Melchior, J., Vanderdonckt, J., Van Roy, P.

An interaction outside objects will pop up a menu independent of the object. Several gestures applied on more than one object (e.g. switch and replace). These operations need at least to object to exist. It is harder for the user to guess the result of these operations if the feedback is not clear enough. The gestures are different for singletouch and multi-touch devices. This difference is really important because it means that it is not easy for user to be accustomed to these interactions. Some operations have to stay the same (e.g. move, undo and redo) while some have to be different (e.g. copy, switch and merge). A combination of the metaphors introduced in the paper and touch or multi-touch gestures would be interesting. It would allow supporting many primitives while being not too complex for the end user to use. Adding gestures to the extra-UI is really possible. In Table 4, there are propositions of gestures for each primitive by single-touch (i.e., with one finger) and multitouch (i.e., with more than one finger). The hand for touch section is either one finger or a pen. For pen-based gestures, we relied on the ISO standard on pen-based interaction [14]. For the touch-based interaction, we relied on the sliding widgets [17] and the catalogue of gestures from [24].The purple cycle is the action point which means where the finger hit the display. The label of each cell is describing how the action should take place. For example, the MOVE in single touch is a pressure on the object, and a movement while maintaining the pressure. The red arrow shows the movement of the finger on the display. A red circle means that at this point, the user has to wait while maintaining pressure. For example, the MERGE operation in single touch is by first touching the display where the object is. Maintaining the touch while moving on another objet; waiting upon the other object until a menu pops up. The last labels are Maintain+Shake and Tap. The shake means that when the object has been moved to its destination, the object has to be shaken in order to trigger the switch primitive between the current object and the one behind. The tap label means that the user touches for less than one second the display, which acts as a press-release touch interaction. For the RESET primitives in multi-touch, the user taps with two fingers to pop up a menu. For the APPEND primitive, the red line means that one of the borders of the objects is beside a border of another object which will merge/append them. The results of the primitive are not necessarily shown in the square.

6

Conclusion

In this paper, we have investigated user preferences for extra-user interfaces. The contributions are a catalog of distribution primitives classified into four categories, a comparative analysis of the related work consistent to the catalog and the first survey on metaphors for Extra-UI. The catalog is the first classification of the primitives that can be used for distribution. The outcomes have shown that users have different preferences with regards to the primitives and their experience. We also noticed that new interactions techniques like pen, touch and multi-touch interactions can improve the control over DUIs. The results of the survey highlight the needs of natural metaphors, the importance of feedback and the difference of preferences. It raises the question of combining the metaphors to let user choose the one he/she prefers depending on the context of use. In addition, gestures could lower the complexity of interacting with

A Comparative Evaluation of User Preferences for Extra-User Interfaces

17

DUI thanks to the natural metaphor of direct manipulation. All the metaphors have been proven to be useful but they should be adapted to the context, i.e. user experience, task experience. Basically, there is still work to do before integrating DUI in a day-to-day environment, and user preferences should not be overlooked. Acknowlegments. We gratefully acknowledge the support of the ITEA2 Call 3 UsiXML project under reference 20080026 funded by Région Wallonne –DGO6 and FP7 Serenoa supported by the European Commission.

References 1. Avrahami, G., Brooks, K.P., Brown, M.H.: A Two-view Approach to Constructing User Interfaces. In: Proc. of SIGGRAPH’89 (Boston, July 31-August 4, 1989). Computer Graphics, vol. 23, no. 3, pp. 137–146 (July 1989) 2. Benoît, A., Bonnaud, L., Caplier, A., Jourde, F., Nigay, L., Serrano, M., Damousis, I., Tzovaras, D., Lawson, L.: Multimodal Signal Processing and Interaction for a Driving Simulator: Component-based Architecture. J. of Multi. UIs, vol. 1, no. 1, pp. 49–58 (2007) 3. Balme, L., Demeure, A., Barralon, N., Coutaz, J., Calvary, G., Cameleon-RT: A software architecture reference model for distributed, migratable, and plastic user interfaces. In: Proc. of 2nd European Symposium on Ambient Intelligence EUSAI’2004. LNCS, vol. 3295, pp. 291–302. Springer, Heidelberg (2004) 4. Barralon, N.: Meta UI: vers un Desktop++. In: Proc. of Rencontres Jeunes Chercheurs en Interaction Homme Machine RJHCI’2004. 5. Beaudouin-Lafon, M., Lassen, H.M.: The architecture and implementation of CPN2000, a post-WIMP graphical application. In: Proc. of the 13th ACM Symposium on User interface Software and Technology UIST’2000, pp. 181–190. ACM Press, New York (2000) 6. Beaudouin-Lafon, M.: Novel Interaction Techniques for Overlapping Windows. In: Proc. of 14th ACM Symposium on User Interface Software and Technology UIST’2001, pp. 153– 154. ACM Press, New York (2001) 7. Biehl, J.T., Bailey, B.P.: ARIS: An Interface for Application Relocation in an Interactive Space. In: Proc. of Graphics Interface GI’2004, pp. 107–116. Canadian Human-Computer Communications Society, Waterloo (2004) 8. Biehl, J.T., Baker, W.T., Bailey, B.P., Tan, D.S., Inkpen, K.M., Czerwinski, M.: IMPROMPTU: A New Interaction Framework for Supporting Collaboration in Multiple Display Environments and its Field Evaluation for Co-located Software Development. In: Proc. of the 26th ACM Conf. on Human Factors in Computing Systems CHI’2008, pp. 939– 948. ACM Press, New York (2008) 9. Coutaz, J.: Meta-User Interfaces for Ambient Spaces. In: Proc. of the 5th Int. Workshop on Task Models and Diagrams for User Interface Design TAMODIA’2006. LNCS, vol. 4385, pp. 1–15. Springer, Heidelberg (2006) 10. Coutaz, J.: User Interface Plasticity: Model Driven Engineering to the Limit! In: Proc. of 2nd Int. Conf. on Engineering Interactive Computing Systems EICS’2010, pp. 1–8. ACM Press, New York (2010) 11. Dearman, D., Pierce, J.S.: It's on my other computer!: computing with multiple devices. In: Proc. of the 26th ACM Conference on Human Factors in Computing systems CHI’2008, pp. 767–776. ACM Press, New York (2008) 12. Grolaux, D., Vanderdonckt, J., Van Roy, P.: Attach me, Detach me, Assemble me like You Work. In: Proc. of 10th IFIP TC 13 Int. Conf. on Human-Computer Interaction INTERACT’2005. LNCS, vol. 3585, pp. 198-212. Springer, Heidelberg (2005)

18

Melchior, J., Vanderdonckt, J., Van Roy, P.

13. Hutchings, H.M., Pierce, J. S.: Understanding the whethers, hows, and whys of divisible interfaces. In: Proc. of the Working Conf. on Advanced Visual Interfaces AVI’2006, pp. 274–277. ACM Press, New York (2006) 14. ISO/IEC 14754: Pen-based interfaces - Common Gestures for text editing with pen-based system. International Standard Organization, Geneva (1999) 15. Melchior, J., Grolaux, D., Vanderdonckt, J., Van Roy, P.: A toolkit for peer-to-peer distributed user interfaces: concepts, implementation, and applications. In: Proc. of 1st Symp. on Engineering Interactive Computing Systems EICS’2009, pp. 69–78. ACM Press (2009) 16. Molina, J.P., Vanderdonckt, J., González, P.: Direct Manipulation of User Interfaces for Migration. In: Proc. of 10th ACM Int. Conf. on Intelligent User Interfaces IUI’2006, pp. 140–147. ACM Press, New York (2006) 17. Moscovich, T.: Contact area interaction with sliding widgets. In: Proc. of the 22nd ACM Symposium on User interface software and technology UIST’2009, pp. 13–22. ACM Press. 18. Paternò, F., Santoro, C., Scorcia, A., Bandelloni, R., Mori, G.: Web user interface migration through different modalities with dynamic device discovery. In: Proc. of 2nd Int. Workshop on Adaptation and Evolution in Web Systems Engineering AEWSE’2007. vol. 267, CEUR Workshop Proceedings (2007). http://sunsite.informatik.rwth-aachen.de/Publicati ons/CEUR-WS/Vol-267/paper4.pdf 19. Rekimoto, Yuji Ayatsuka, Michimune Kohno, Hauro Oba, Proximal Interactions: A Direct Manipulation Technique for Wireless Networking. In: Proc. of 9th IFIP TC 13 Int. Conf. on Human-Computer Interaction INTERACT’2003, pp. 511-518. IOS Press, Amsterdam (2003) 20. Rodden, T., Crabtree, A., Hemmings, T., Koleva, B., Humble, J., Akesson, K.P., Hansson, P.: Configuring the Ubiquitous Home. In: Proc. of Int. Workshop on Cooperative Systems Design, Scenario-Based Design of Collaborative Systems COOP’2004, pp. 227–242. IOS Press, Amsterdam (2004) 21. Roscher, D., Blumendorf, M., Albayrak, S.: A Meta User Interface to Control Multimodal Interaction in Smart Environments. In: Proc. of the 13th Int. Conf. on Intelligent User Interface IUI’2009, pp. 481–482. ACM Press, New York (2009) 22. Roscher, D., Blumendorf, M., Albayrak, S.: Using Meta User Interfaces to Control Multimodal Interaction in Smart Environments. In: Proc. of 4th Int. Workshop on Model Driven Development of Advanced User Interfaces MDDAUI’2009. vol. 439, CEUR Workshop Proceedings (2009). http://ceur-ws.org/Vol-439/paper4.pdf 23. Roudaut, A., Coutaz, J.: Meta-IHM, ou comment contrôler son espace interactif ambiant. In: Proc. of 3èmes Journées Francophones Mobilité et Ubiquité UbiMob’2006. 24. Roudaut, A., Lecolinet, E. Guiard, Y.: MicroRolls: Expanding Touch-Screen Input Vocabulary by Distinguishing Rolls vs. Slides of the Thumb. In: Proc. of ACM Conf. on Human Aspects in Computing Systems CHI’2009, pp. 927–936. ACM Press, NY (2009) 25. Salay, R., Mylopoulos, J., Easterbrook, S.M.: Using Macromodels to Manage Collections of Related Models. In: Proc. of CAiSE’2009. LNCS, vol. 5565, pp. 141–155. Springer, Heidelberg (2009) 26. Sottet, J.-S., Calvary, G., Favre, J.-M., Coutaz, J.: Megamodeling and Metamodel-driven Engineering for Plastic User Interface: Mega-UI. In: Seffah, A., Vanderdonckt, J., Desmarais, M. (eds.), Human-Centered Software Engineering, pp. 173–200. Springer (2007) 27. Tullis, T., Albert, B.: Measuring The User Experience – Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann Publishers, San Francisco (2008) 28. Vanacken, D., Demeure, A., Luyten, K., Coninx, K.: Ghosts in the Interface: Meta-user Interface Visualizations as Guides for Multi-touch Interaction. In: Proc. of IEEE Int. Workshop on Horizontal Interactive Human Computer System TABLETOP’2008, pp. 87-90. IEEE Computer Society Press, Los Alamitos (2008)

A Comparative Evaluation of User Preferences for Extra-User Interfaces

19

29. Vandervelpen, Ch., Vanderhulst, G., Luyten, K, Coninx, K.: Light-Weight Distributed Web Interfaces: Preparing the Web for Heterogeneous Environments. In: Proc. of the 5th Int. Conf. on Web Engineering ICWE’2005, pp. 197–202. Springer, Heidelberg (2005) 30. Vanderhulst, G., Luyten, K., Coninx, K.. Put the User in Control: Ontology-Driven Metalevel Interaction for Pervasive Environments. In: Proc. of the 1st Int. Workshop on Ontologies in Interactive Systems ONTORACT’2008, pp. 51–56. IEEE Computer Society (2008) 31. Vanderhulst, G., Schreiber, D., Luyten, K., Muhlhauser, M., Coninx, K.: Edit, Inspect and Connect your Surroundings: A Reference Framework for Meta-UIs. In: Proc. of 1st Symposium on Engineering Interactive Computing Systems EICS’2009, pp. 167–175. ACM Press, new York (2009)