Years of Freedom — a Report on my Stay as a Post-Doctoral Researcher at the Media Lab

With this post, I want to give an overview of the last three years of my work-life that I spent at the Media Lab of Aalto University. I also take it as a possibility to look back in reflection, trying to identify my development as a researcher over the time, outline turning points and give credit to all the people that supported me to survive and celebrate both the beautiful Finnish winters and summers.

I will start with thanking the officials who made possible this intense time by selecting me as a post-doctoral researcher and supporting my stay. I am deeply grateful to Lily Diaz (the head of research of the Department of Media), Philip Dean (the Head of the Department of Media) and Helena Hyvönen (the Dean of the School of Arts, Design and Architecture).

As with every report, I will start with a brief description of my background. This is followed by a text on the first big challenge I put myself into; the challenge of open research. I will then continue with a description of the research strand of Sonic Research Culture that I followed during my stay, namely, research into Raw Materials, Grids, Otherness, and Instrument Building for and with people with disabilities.

Background Link to heading

My professional background is in Computer Science in the Natural Science, which I studied from 1999 to 2006 at Bielefeld University in Germany. My focus within this area was robotics.

In 2010, I received a PhD for my work on Tangible Auditory Interfaces, a combination of scientific sonification with tangible interaction. Between 2006 and 2010, I worked as a research assistant at various institutes of Bielefeld University, prominently in the Ambient Intelligence Group (lead by Thomas Hermann) and the Neuroinformatics Group (lead by Helge Ritter) of the CITEC Cognitive Interaction Technology, Center of Excellence.

With this rather scientific background, I applied for the post-doctoral researcher position at Aalto University and was appointed for the following three years.

I remember sitting in Philip Dean’s office when signing the contract. I asked him about what the department will expect from me. His answer was simple: the only thing they want me to do is get my research career started. A request, I was obeying with joy over the course of the next three years.

The self-assigned challenge of open research Link to heading

A post-doctoral position gives a lot of freedom to try out something new, go an experimental route. I decided to take that opportunity and experiment on variations of research methods.

In the science communities I worked in previously, a widespread habit was to keep research topics and ideas as secret as possible to prevent it from being “stolen” by others. I experienced a lot of anxiety around me about opening up unfinished (i.e. unpublished) research to the public (e.g. by publishing preliminary results and observations on weblogs). On the other hand, I considered it essential to discuss with fellows about the challenges and potential of my work; something that makes it necessary to give away unpolished information and, even more difficult, to admit to have a misconception, or not being the expert in the field yet.

So, in the very beginning of my research stay, I decided to ignore the broadly accepted policy to retain all information until the research is carried out properly at least for one project. Even more, I tried to present all relevant information on both the conceptual and the practical process on my weblog while it was happening.

design sketch for the whispertable project

This first investigation was Storyboard and Whispertable, a collaboration with Juha Kronqvist. We set our task to design and build a table or surface that emits structure-borne sounds, which can be perceived and manipulated by custom-build stethoscope-like artefacts. As an application, we intended to enrich deserted places in our working environment by transferring sound (and therefore content) of more popular places. Our hope was that this will attract people, making the deserted areas more lively. We wrote several blog posts on our observations.

Although this project eventually did not take off (or just because of it), I still took away a lot of new insights; already the intent to publish as soon as possible had several effects on my research practice. First and foremost, it forced me to write down and revise all the work and observations we did. This in turn had the effect I hoped for; colleagues from near and far gave constructive comments and suggestions. A serious issue with this approach was, though, that, in order to meet an acceptable level of quality, a significant amount of time had to be reserved to just write those blogpost. Time that would otherwise have been invested into project development. On the long run, though, I understood that this artificial reduction of pace and the serialisation of thought are a good base for qualitatively high work; decisions were undertaken on a much more informed level than before.

To conclude, it is my strong believe that making research public while it happens is highly advantageous and easily levels out the risks of idea stealing that it brings. In fact, I applied it to most of my other projects as well and I never had any issue with anyone properly publishing my ideas before I did.

Looking back at this first project, I also see the lack of knowledge on design thinking I had in those days; specifically knowledge on interaction design, its methods and its approach to embrace research. I am very happy that I was able to gain a significant extent of this knowledge over the past three years by the help of the Media Lab researcher community. Thank you for your support.

Sonic Research Culture Link to heading

Sometimes, especially in the beginning of my stay in Finland, it felt like being dropped into a parallel universe where things seem to be the same. However, after a closer look, everything turns out to be different. I was eager to explore and I grasped for every opportunity to gather thoughts and ideas around this new environment.

Luckily there was Pixelache 2011, and a friend of mine, Marije Baalman held a workshop on distributed realtime sensing technology. I attended, and got to know Tapio Mäkelä, the head of M.A.R.I.N., a “Media Art Research Interdisciplinary Network”. He invited me to join his residency program and spend 10 days on an island in the Saaristomeri.

Anemos Sonore Link to heading

During this beautiful and inspiring stay, I decided to continue working on the idea I had at the MARIN workshop; turn the natural phenomenon of wind sound. I already got very nice feedback in the workshop presentations at pixelache, however the days on the island turned the idea into something bigger. Anemos Sonore is about listening to wind as a holistic phenomenon. The tools I prototyped (I called them tools as I see them as sensing equipment for sonic investigations) augment the calm soundscape of the archipelago; they invite for intense listening sessions, in which the sense of wind (e.g. as a tactile impression on the skin or visually observed in the movements of trees and waves) is complemented by drone-like sounds captured from the tools’s ribbon movements. Still in the time bubble of the island, I made a series of videos which are, admittedly, too long for a quick overview but may give you a hint of an insight on how it was there in Naantali.

Back in civilisation, I presented the project at several events and also reworked the system to show it at the SuperCollider Symposium in London and at the CARTES Flux Festival in Espoo.

The bigger picture Link to heading

Being in Finland meant for me that nature grabbed my attention immediately. Maybe this was because nature and especially the weather are more radical than in central-europe. Before moving here, I was not aware that my mood is so depending on amount of daylight or outside temperature. This dependency (and me recognising it) initiated Anemos Sonore, which is directly connected to weather and nature.

And this affection to natural phenomena and their communication by means of sound and interaction continued; One of the biggest supporters was the finnish society of bioart and its director Erich Berger, who not only gave me the opportunity to spend time at their arctic residency program but also invited me to join their Field_Notes laboratory in 2013, a bi-annual event where international artists meet and work in the harsh but beautiful environment of northern Lapland. This event opened up my horison quite a bit since it connected me with so many great people form all over the world who came together because of the same interests: curiosity, intervention and observation within the arctic circle. It was in relation to this event that I again found my connection to natural science, now, however from a more questioning point of view. Namely I am curious to understand how the tools used in the process of measuring and “data generation” influence the experience and therefore the way we look at natural phenomena.

And, since every scientific method adds its self-inherent subjective element to an observation (this also includes methods as fundamental as inductivity in gaining knowledge), why not choose methods that create positive effects, that draw a holistic world view? In any case, I continued what I started with Anemos Sonore and build an instrument to “bump” nature.

Hive Five Link to heading

Related to this is Hive Five, an artwork and music event featuring bees and their sounds. It emerged from the Melliferopolis project of Christina Stadlbauer, supported by Aalto Biophilia) and its lovely Ulla Taipale.

I hope to continue this research strand also in the future. Lots of things to discover still.

Raw Materials Link to heading

In continuation (and abstraction) of my research in Tangible Auditory Interfaces, I examined the nature of raw materials by means of sonic investigations. Central questions in this project were

• What is the nature of materiality?
• What is are inherent gestalt of digital and physical materials?
• How can methods incorporating hearing add to the quality of knowledge gained in the process?

I intended to investigate how digital-borne raw material can be handled, i.e. formed and carved, just as done every day with physical material. As induction of feedback is seen as a valid method to learn about an element and it, by nature, does not require prior knowledge about the material under examination, I chose it to investigate two examples of material, both ranging in different extremes of the materiality continuum: Paper and Code.

Vellum [2011 – 2012] Link to heading

Vellum is the paper-part of the investigation, carried out as a performance. It is a collaborative work carried out together with the media artist Erich Berger to whom I am grateful not only for insightful discussions on the theme but also for his friendship and support within the last three years. Vellum was presented as part of the 2012 Cumulus Conference held at Aalto University.

In Vellum the characteristics of paper as raw material is sonically exposed by means of both artistic and scientific approaches. The fundamental principle of audio feedback is applied, initiating a dynamic process in which the material articulates itself. In the performance, the artists bring intense spatial soundscapes into existence, consisting of a multitude of different feedbacks all originating in the paper itself. They break up, fade away and abruptly change character, revealing a complex sonic vocabulary.

The central element of the Vellum setup are three vellum loops mounted on the ceiling. Each of these probes has a contact microphone attached and is closely positioned to a loudspeaker. It therefore serves as an acoustic coupler, closing the loop between microphone and speaker. Due to its nature in feedback, the piece is highly sensible to the environment in which it is presented; it reacts to even tiny air movements.

Chip interpretations / BetaBlocker [2010-2013] Link to heading

Chip Interpretations is the code part of the material investigation. It is a collaborative work carried out together with the code artist and musician Dave Griffiths. The project was carried out as artistic research practice, combining artistic research with several performances. The project was finalised in an article for the Computer Music Journal where it will be published early next year. I would like to thank my collaborator and friend Dave for sharing his uplifting way of being and his unusual way of thinking about the world that surrounds us all.

Chip interpretations is an attempt to sonically examine digital raw material and is the code part of the investigation of raw materials. Within three years of artistic research and performance practice, we investigated the sonic gestalt of (a specific form of) digital material, utilising a fictional CPU that treats arbitrary input data (encoded as a series of 8bit values) as its program on which it generates continuous sonic output. We gained understanding of the CPU’s behaviour by applying data sets consisting e.g. of random values or parameterised waveforms and listened to its output. In another attempt, we implemented small programs in the machine’s assembler language aiming to trigger specific behaviours (e.g. output an impulse or sawtooth waveform).

Oulipop rehearsal at live.code.festival

Further, the project undertook a practice turn; it was used in a number of international performances. We invited artists to accompany us in both the conceptualisation and the performances themselves, which resulted in a broad variety of creative applications. This step towards artistic practice turned out to be crucial in order to get familiar with the CPU’s sonic specificity. I honestly enjoyed this part of the project the most. Working with such great artists such as Sara Hildebrand Marques Lopes, Dave Griffiths and Tom Hall made the difference for me; We managed to really break up the research part and turn it into artistic practice on various levels. Among others, this was reaching the audience and connect them with such an abstract theme as CPUs and their inner workings. Below this article, I gathered some photos of these performances.

Further, the project undertook a practice turn; it was used in a number of international performances. We invited artists to accompany us in both the conceptualisation and the performances themselves, which resulted in a broad variety of creative applications. This step towards artistic practice turned out to be crucial in order to get familiar with the CPU’s sonic specificity.

Identifying my interest in materiality and working on it for an extended period of time was a highly rewarding task. It allowed me to gather interesting people around me and try out new methods of research and investigation. I personally enjoyed the most interacting with the guest artists. Working with such great people as Sara Hildebrand Marques Lopes, Dave Griffiths and Tom Hall made all the difference for me; we managed to really break up the research part and turn it into artistic practice on various levels and I had to explain the theme of the project to people not familiar with the concept of computation. It was, at times, a frustrating endeavour. But the reward was not only that they learned about computers and their inner workings but also that I gained a better understanding of the subject on my side. This lead to performances that reportedly managed to reach the audience.

Chip Interpretations with Tom Hall

SuperCollider, Grids, Modality and a Passion for Controllers Link to heading

My involvement with the SuperCollider language started with programming the code for my diploma thesis. I soon got hooked and, as SuperCollider is an open source language, found myself developing and contributing to the language. Being part of the developer community, I was involved in writing various chapters for The SuperCollider Book (together with such great people as Alberto de Campo, Julian Rohrhuber, Marije Baalman, Stefan Kersten and Christopher Frauenberger) which was published by MIT press in 2011.

Modality Link to heading

But computer science was not the only thing I was interested in. Being raised by parents with such divers interests as music, graphic design and visual art but also mathematics and computer science, I was keen to explore creation and research for the sake of art; I wanted to (and still want to) find out, how I can possibly integrate artistic practice into my work with tangible and auditory interaction.

Luckily, almost at the same time I started working at the Media Lab, I was invited to join the Modality group, formed by an international group of people that see themselves as both software developers and artists, mainly using the SuperCollider programming language. With the help of the media lab, I had the pleasure to be part of two one-week-long meetings, one at STEIM in Amsterdam and one at BEK in Bergen. These meetings were filled with intense hacking sessions to implement the base of what will eventually be a unifying service to integrate hardware controllers into SuperCollider. At the same time the meetings allowed us to perform our musical pieces at these internationally knowns places. I had the pleasure to perform a live version of Verber and Greyball in Bergen.

I am very happy that funding for several Modality meetings in northern countries is secured. This will help me to keep the connection to the modalityTeam and help setting up the ambitious software environment we envision. It is simply great to be with such inspired people and come up with things like controller type semantics.

Controllers Link to heading

For a long time, I was obsessed with hardware controllers (for musical purposes) of any kind. I collected GamePads, Drum controller, MPC-like devices, keyboards and strips. It was kind of frustrating that I never got around to actually use them; I have to admit that got caught up in the programming part, trying to make them work on every level. Being involved in the Modality effort (and also learning about design methods), however, I realised that I was tackling the problem from the wrong side; unlike for professional mass-produced products, a performance system that I design and build for myself is a highly subjective system that does not need to be self-explanatory. The only person who needs to understand what it does is me.

Rather than coding every possibility into a controller, I needed to look at it as a device with possibilities; features I can assign to certain functionalities of my performance software. For this, I found quick sketching and outlining of mapping strategies to be essential. I created the controller booklet a set of PDFs that graphically outline the physiognomy of each of my controllers. They are quickly printed out and can then be used to sketch out mapping ideas.

Grids Link to heading

In my affection to controllers, one device truly sticked out: the Manta by Jeff Snyder. It features a beautiful hexagonal grid surface-sensitive pads and made me look at grids as structuring elements. Over the last years, I learned a lot about grid layouts in music and graphic design, how to describe them mathematically and what specific benefits each of them features.

It is valuable knowledge that influences my artistic work, however, was never explicitly featured in any of my works. Maybe I will change that at some point in the future. At least, I made the variety of regular grids a second element in my Controller Booklet.

Design that matters Link to heading

In June 2012, I attended the Designing Interactive Systems conference in Newcastle with an interactive demonstration on Auditory Augmentation Part of the conference was a keynote by Chris Csikszentmihályi, Professor of Media Design Matters at the Art Center College of Design Pasadena, CA. His talk (similar to the one he held at SHARE 2012 for which there is a video) was mainly about his work as the founder of MITs Computing Culture group, which worked to create unique media technologies for cultural and political applications. All the works (of his students) he showed were focused on design interventions that either made a political statement or that actually helped other people in doing so. It was an inspiring talk and I recommend to see Chris talking if you have the chance.

Coffee Link to heading

Schematic overview of the Coffee Dialogue study

In the same week, I attended the “web of things hackathon,” a workshop-like hackathon that accompanied the conference.¹ After a short introduction to the concept of the web of things and an initial brainstorm on possible projects for the day, we split up into two groups and we had to come up with a theme for working on the “web of things”. Inspired by Chris’ talk in which he encouraged (interaction) designers to do things that matter rather than abstract design studies, we came up with the idea to increase transparency of the coffee supply-chain. But, instead of concentrating on the consumer, we thought it might be valuable to think of how to provide information about the coffee to the actual harvesters working at the coffee farms.

After this short inspirational interlude, I came back home and started to think on how I could use my skills to “Think of citizens rather than users”, as Chris put it in his keynote. The thoughts eventually resulted in the DEIND project, a design investigation to facilitate electronic digital music practice for neurodiverse people.

DEIND Link to heading

Participant S1 listening to the 3rd generation prototype of the DEIND project

This project was a collaboration with Julian Parker, Ramyah Gowrishankar, Mila Moisio, Jussi Mikkonen, Vesa Välimäki and Pirjo Kääriäinen. The project was partially funded by Aalto media factory.

The DEIND project was an attempt to connect neurodiverse people with the field of contemporary electronic and digital music practice. In pursuit of this, people with autistic spectrum disorders were invited to take part in the design process of electronic instruments. In difference to music therapy approaches, we focused on the design of tools and methods for artistic expression rather than therapeutic outcome. The close integration of target group members into the research process encouraged a bilateral learning process: on the one hand, we created an intense and fruitful experience for the participants; on the other hand, the involved researchers were able to identify design challenges specific to the target group and reveal new perspectives on their respective area of research.

Within the funding period, we realised three design iterations including field work sessions of one week each. The reception of our interventions was throughout positive; participants as well as carers reacted positive, sometimes even enthusiastic. It left a good feeling in me and supports my efforts to find further funding possibilities for this project. Some of the performances of the participants were published online.

As for dissemination, we published a paper at SMC 2013 on ComplexRes , a DSP filter and released DEIND UGens, an open source audio filter suite for the SuperCollider programming language that contains implementations of five sound filters that were developed by Julian and me specifically for the DEIND project. The project was presented at various institutes such as PITLab, CIID, the Medialogy department of AAU and the Sound is Information symposium in Stockholm. An overview paper on the project is planned for NIME 2014. Furthermore, we showed the third prototype generation at the opening of the TAUKO studio shop where it was quite popular.

Design that matters Link to heading

After 10 years of mainly dealing with basic research in tangible interfaces and sonification, I eventually discovered that I can apply my skills to make a difference to people’s everyday life and (at least minimally) change society. It is a good feeling.  The immense idealist support of people like Andrea Botero and Teemu Leinonen helped me and I am deeply grateful for their efforts. I plan to continue in this direction, so you can expect more musical interventions in the future.