Audio-Visual Analogy: A Synesthetic Experiment on Birdsongs

Audio-Visual Anology: A Synesthetic Experiment on Birdsongs Keywords: Computation, Synesthesia, Gesture Tracking, Pattern Mapping, Audio-Visual Performance. Seçkin Maden Department of Architecture, Architectural Design Computing Graduate Program Istanbul Technical University

Abstract This study explores possible audio-visual analogies in interactive design. Aural and visual codes that brain generates, and their simultaneous grouping principles in perceptual system is the main influence for this new experimental design schema. Computational methods, which have an important role in structure-based thinking, may contribute to both analytic and heuristic processes of this process. Human-machine interaction in design, has an increasing importance by challenging conventional methods of the era. Through newly developed tools, the leading role of hand-eye coordination in design is also augmented with their interactive and real-time features. Study’s aim is to reveal to those features that may constitute a basis for synesthetic recognition processes in design. Through notions of ‘hearing with shapes/colors’, ‘seeing with sound’ and newly developed computation tools with real-time implementation capabilities, an audio-visual experiment is proposed. Here, selected birdsongs of local species of Istanbul, regarding their variance of rhythm and tone in a whole, are analyzed in order to reveal their inherent proportional qualities of tone and rhythm. User defined visual patterns in real-time are matched with audial data so as to provide a user to control both visuals and sonic data in a unified, synesthetic sense. Producing an instant unification code for both audial and visual input, which has a meaningful, memorable content for our perceptual system is the main focus of the study.

Introduction Problem Definition: When linking image, sound and time; the issue is raised of according to which process or code structural features should be translated from one sensory level to the other. What are the main features of an interactive system that responds to a designer’s instant groupings of perceptual cross-modalities? problem // subjectivity // every synesthete has his or her individual constant couplings of activating and activated perceptions, or his or her own synesthetic reality, which clearly complicates the study of the synesthetic phenomenon. 1. Selecting special cases of audio-visual cross-modality based on natural sounds and hand silhouettes. 2. Observation of real–time digital analysis of data produced (decode). Produced data may involve a birdsong, recorded human voice or an instant hand sketch.

3. Generation of sounds and forms in a reverse order, through the structural properties of decoding process (encode). This stage can be seen as a testing process of an analogy, which was set between audial and visual parameters. Research Questions: # How does designer of an audio-visual interface defines a synesthesia in a certain case, which responds directly to our perceptual world? ( pitch-area, frequency-color, tone color, edge definition?) (encode) # If one form of perception activates the other, can digital mediums be used to record and manipulate these constant couplings through desired needs? # On a basis, which is set by coupling relations of sound and form, how can a user produce variations without having any formal education on both fields? # What is the contribution of intuitive processes like sketching and hand gestures to both digital encoding and decoding processes?

1.Background 1.A. Synesthesia and Audio-Visual Commensurability

Synesthesia: syn, "together" / aisthēsis, "sensation" Synesthesia is a neurological phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. Synesthetic responses commonly arise automatically, without requiring effort and being under relatively little control (e.g., Mattingley et al., 2001). The study proposed here, focuses on this notion of instantaneous cognitive groupings of our perceptual system. Audial and visual acts, which are triggered and thus memorized by similar events, are grouped by nervous system and brain. A synesthesia-like phenomenon was first mentioned in 1690 by John Locke, in his text An Essay Concerning Human Understanding. He described a blind man who associated the color scarlet red with the sound of a trumpet .[1] In 1720, the ophthalmologist T. W oolhouse also reported the case of another blind man who perceived colors in response to sounds. [2] In 1812, the physician G. T. L. Sachs published a report on two albinos who perceived sounds and numbers in color (Ione and Tyler, 2004). The term synesthesia was probably first used in 1866 by the French physiologist and neurologist Alfred Vulpian, though with a different meaning than is common today.[3] Eugen Bleuler, himself a synesthete, published a first quantitative study of synesthesia in which he classified 12% of the 600 participants as synesthetes (Bleuler and Lehmann 1881). In his studies,

high notes, for example, tend to activate bright colors. Bleuler also ruled out that these perceptions could be learned, and he identified the brain as the original source. Related with learning issues,there is a good deal of evidence that the occurrence of synesthesia is substantially influenced by early childhood experiences. W itthoft and colleagues discovered, for example, that in the case of one female synesthete there was a 100% agreement between the colors she associated with individual letters, and the colors of magnetic letters her parents had used to decorate the refrigerator in her childhood (W itthoft et al. 2006).

1.B. Cross-Modal Correspondences in Audio-Visual Analogy As had been the case with Eugen Bleuler, research interest focused on identifying shared characteristics between color perceptions, since despite obvious differences there were evidently certain regularities: a connection between pitch and brightness was proposed frequently. There were also scattered reports of a connection between volume and size, volume and brightness, and pitch and size (Emrich, Neufeld, Sinke, 2010). According to Marks, even non-synesthetes perceive high-pitched vs. low-pitched sounds to resemble bright vs. dark colors—the resemblances evident in various tasks of cross-modal comparison (Marks, 1975; Ward et al., 2006). Where music-color synesthetes see brighter colors in high- pitched notes (e.g., “gold, yellow and white moving . . . like a rippling stream” (Mulvenna and Walsh, 2005; p. 399). People lacking the induced qualia of synesthesia nevertheless recognize cross-modal similarities. Similarly Ramachandran and Hubbard (2001:19) cite the consistency of an (albeit simple) mapping between shape and sound to support the same point: asked to link two shapes, one round, one spiky, with two names, bouba and kiki, subjects overwhelmingly associate kiki with the angular form and bouba with the rounded one (Whitelaw,2008). In the work of Messiaen, Scriabin and perhaps Schoenberg (via Kandinsky), synesthetic experience formed the basis for a systematised set of pitch-color correspondences, though even these are not straightforward. The correspondences are different for each composer, as we would expect based on recent science (Whitelaw,2008). Moreover, each is conditioned by, what Cook (1998:46) argues is a mixture of subjective and cultural factors.

2. Audio-Visual Experiment: Proposed research and the audio-visual experiment below, focuses on the possibilities of systematized audio-visual correspondences. Due to the extensive variety of synesthetic groupings observed on many subjects, that is affected by subjective and cultural factors as Cook mentioned, study aims to reveal user defined groupings of audial and visual stimulants. User definitions of form and color of a particular sound sample, and generation of an audial data to test its corresponding form through users’ instant, subjective interpretation, are observed through this aim. If one can recall a data through its corresponding audial or visual form defined by the digital interface, the analogy of audial and visual data can be considered as proper.

The analogies that are used in proposed experiment can be grouped as, higher values of pitch and tone >> brighter colors, illumination lower values of pitch and tone >> darker colors, edgy, sharp tones >> angular forms _smooth, textural tones >> rounded corners.

Likewise conventional design processes, first an analysis process of selected natural sounds is proposed. The interface maps user defined colors and forms to definite audial parts and makes them more recognizable and memorable. This process can be grouped as decoding process. Decoding of the parameters in these possible groupings may allow for an augmented cross-modal learning and experimenting process based on intuition. Proposing a color code for a frequency or an angular edge condition for a pitch level, may extend the limits of linear recognition methods for both fields (Figure-1). Here, in decoding process, birdsongs of specific local species in Istanbul are selected for analysis pattern recognition. Due to their perceivable pitch fluctuations and higher complexity among other natural sounds, birdsongs are the subject of the experiment. Through experiment, proposed interface analyzes the species, which has larger birdsong repertoire size, for generative, variant pattern explorations. Field recordings of selected birdsongs are gathered from the online archive of The Cornell Lab of Ornitology / Macaulay Library. The species to analyzed, were selected in accordance with their larger population in Istanbul for providing a better recognition on local testers of the interface and their larger and more complex birdsong repertoire than any other local species.

GH

Figure-1. Basic Schema of Proposed Birdsong Analysis Process (Decoding)

First, the field recording of the White Wagtail birdsong is played by interactive programming environment, MAXMSP. Distinctive pitches and time intervals of birdcalls (distinctive parts of a birdsongs) are extracted and parametricised. Through UDP signals in local network, numerical data is sent to Rhinoceros/Grasshopper environment. Here numerical data of pitch values are collected and visualised in Y-Coordinate (Figure-2). Time intervals between neighboring birdcalls are visualized in X-Coordinate. The system positions points on those coordinates and groups them in accordance with a specific distance threshold between two neighboring points. This threshold value varies and depends on the density of the birdcalls of each species. Here in White Wagtail example, the value is determined as 450 units for a proper analysis and visualization. According

to this neighboring distance relation, the points, whose distances to another are smaller than this threshold, are grouped. After the interface sets the coordinates of points and their groupings, which corresponds distinctive birdcalls, it draws polylines between those points and assigns them different colors due to the their features of curve discontinuity. A specific colors is related to the count of kinks that curve has. Kink (discontinuity) count is interpreted as the main visualization symbol of a birdcall that distinguishes from another. Here a birdcall, that is visualized as a polyline, that has no discontinuity, colored with red. 1 with purple, 2 with cyan, 3 with yellow and 4 with blue (Figure-2,3).

Figure-2. Visual Coding of Generated Sounds

Figure-3. Grasshopper Definition of Birdsong Analysis and Visualization Process (Decoding)

In encoding phase, a user imitates sound samples of the birdsong with hand gestures (leap motion device will be used for gesture tracking). In real-time interface will assign a distinctive color in accordance with its gesture’s geometric features. This will be used for augmenting the recognition of a pattern. A user will be able to re-generate a form and its specific color with hand gestures in real-time. So a user will be able to manipulate, initial sound or form, through a structural understanding of evolving patterns. This can be regard as a vice-versa process of decoding phase of sound to form analysis (Figure-4,2).

Figure-4. Basic Schema of Hand Gesture Driven Pattern and Sound Generation Process (Encoding)

Figure-5. MAXMSP and Grasshopper Definitions of Hand Gesture Driven Pattern and Sound Generation Process

The interface that is used for the re-generation of sonic and visual patterns through audio-visual groupings, which are established in decoding process, can be seen in Figure-5. Recent memory of first phase’s synesthetic process of analysis influences generation of user-defined audial and visual patterns. In order to achieve this, first, parametric values of hand gestures, that imitates previously heard birdcalls and seen patterns are converted to 2D point coordinates and interpolation curves (Figure 5,2). Then, kinks on curves are determined and their count on each curve visualized with the same coding as decoding process. When a kink determined in a hand gesture, a numerical value of Y-coordinate of that move is sent to MAXMSP environment for sound generation. A user instantly recognizes whether a pattern that is generated fits with the analyzed ones or not. This comparison occurs in both visually and aurally.

Video-1. The Audio-Visual Experiment

3. Conclusion An audio-visual experiment is proposed in order to discuss possible analogies between different perceptual qualities of design and analysis processes. Here, a White Wagtail birdsong is analyzed through digital visualization techniques. MAXMSP and Rhinoceros / Grasshopper interfaces are used. For establishing a better recognition of audial patterns, a visual pattern language is proposed by means of geometric-sonic analogies. Smallest groupings of birdsongs, -birdcalls-, are visualized with curvilinear geometries and their parts that define discontinuities with kinks (points). A color code is proposed for each curve related with its kink count. Discontinuities or shifts in a sonic / geometric form is interpreted as the notions that makes them recognizable. In this sense, the feature of color in visuals and discontinuities in sound forms are matched. After the analysis stage, a gesture tracking driven sound and pattern generation process is proposed. It is aimed to observe simultaneous comparisons of generated forms of both audial and visual. The common pattern language that is used in both processes is aimed to enforce recognition process in similar interactive design processes.

Footnotes [1] John Locke, An Essay Concerning Human Understanding, Book 3, Chapter 4, 1690 available online at http://oregonstate.edu/instruct/phl302/texts/locke/locke1 /Essay_contents.html, [2] Marks, L.E., "On colored-hearing synesthesia: Cross-modal translations of sensory dimensions". Psychological Bulletin 82 (3): 303–331 [3] Cf. Alfred Vulpian, Leçons sur la physiologie générale et comparée du système nerveux faites au Museum d’Histoire Naturelle: Rédigés par Ernest Brémond (Paris, 1866), 464f. Vulpian referred to the photic sneeze reflex as synesthésie. Jules Millet referred to Vulpian in Audition Colorée (Paris, 1892), 14. Cf. also Emilie Noulet, Le Premier Visage de Rimbaud (Brussels, 1953), 122, and Dann 1998, p.188. References

Ione, Amy and Tyler, Christopher, Neuroscience, History and the Arts. Synesthesia: is Fsharp Colored Violet, 2004 Bleuler, Eugen and Lehmann, Zwangsmässige Lichtempfindungen durch Schall und verwandte Erscheinungen auf dem Gebiete der andern Sinnesempfindungen, Bleuler,1881 Hinderk M. Emrich, Janina Neufeld, Christopher Sinke , Synesthesia, 2010 Mattingley, J. B., Rich, A. N., Yelland, G., and Bradshaw, J. L., Unconscious priming eliminates automatic binding of colour and alphanu- meric form in synaesthesia, 2001, Nature 410, 580–582

Witthoft, Nathan and Winawer, Jonathan, Synesthetic colors determined by having colored refrigerator magnets in childhood, 2006 Marks, L. E., On colored-hearing synesthesia: cross-modal translations of sensory dimensions, 1975, 82, 303–331, 1975 Ward, J., Huckstep, B., and Tsakanikos, E., Sound-colour synaesthesia: to what extent does it use cross-modal mechanisms common to us all. Cognition 42, 264–280, 2006 Mulvenna, C. M., and Walsh, V., Synaesthesia, 2005, 15, 399–400 Karwoski, T. F., Odbert, H. S., and Osgood, C. E., Studies in synesthetic think- ing. II. The role of form in visual responses to music, 1942. J. Gen. Psychol. 26, 199–222 Whitelaw, M., Synesthesia and Cross-Modality in Contemporary Audiovisuals, 2008, 4 Cook, Nicholas, Analysing Musical Multimedia. Oxford: Oxford University Press, 1998 Ramachandran, V. S. ; Hubbard, E. M., Synaesthesia: A window into perception, thought and language, Journal of Consciousness Studies 8, 2001, (12): 3–34