On symbolic representations in the upstairs scenario

25 July 2011

This is a post that reflects my ambiguous thoughts on direct and symbolic information representation. Conceptualizing upstairs and discussing its possible types of representation with both my cooperation partners and various interaction design researchers reminded me of the insights I had in relation to JugglingSounds (see Chapter 9.6 of my Dissertation, see also this page for details and demo videos). JugglingSounds is a system for real-time auditory monitoring of artistic juggling and swinging. Existing approaches to real-time monitoring of motions may be found between the extremes of (a) strict full analysis, then displaying the results (referred to as qualitative display) and (b) displaying raw data in simple forms (referred to as quantitative display). In my opinion, this holds true as well for monitoring systems, especially for shared presence:

An explicit data capturing of e.g. states (by switches) provides an appropriate view on already known features (in the case of switches, the designer needs to know where to put those).

Substitution of the switch with, e.g., sound or vision capturing devices followed by a detailed analysis works the same way. The system developer has to integrate her knowledge about the system’s intended environment, making such a system extremely efficient and the produced data easy to algorithmically process.

Systems that incorporate only those features often use, in terms of the pure data representation algorithms, relatively simple displays with predefined sets of qualities: “If this happens in the measuring, turn that on, and determine its quality depending on that other feature”. In Sonification this often leads to auditory icons, mapping arbitrary sounds (in the sense that their sounds are not directly data-driven) to events triggered by the analysis system. Ths by no means implies that the systems or their sounds are simple; moreover, in order to work properly, the used sonic features should be chosen wisely, resulting in complex mappings. Those mappings can be read as both an internal model and a code for the user who has to understand which state of the system represents which state in the observed measurements. The creation of such a mapping is clearly a design task.

In contrast to this approach, it is possible to directly transfer captured vibrations or or video from one place to the other as we intended it. This direct mapping of time-dynamic measurements of general, non-trivial, pseudo-analogue signals to equivalent displays provides what I call a more direct interconnection since its analysis is shifted from machine-powered algorithms to the massive pattern-recognition abilities of the receiver who may or may not find structural information like the ones described in the full analysis approach. He might, however, unveil structures the designer haven’t though of. Even more important, his ability to make sense of the presented data is not depending of his knowledge about the system but on his knowledge of the expected behaviours on the other side of the system. Two people observing the same vibration patterns of someone walking around in the other space might get differing interpretations and conclusions on what happened. While one knows the person and his walking behaviour quite well and is therefore able to identify him through the system, the other possibly only recognises that it is a pattern caused by a human. The system design is limited to deciding which existing cues to mediate and how to create an appropriate representation.

In JugglingSounds, we noticed that a simple mapping of the low-level streams to sound results almost always in an uninteresting and sonically overloaded soundscape where important parts are difficult to separate from unimportant parts. Our approach where to combine the low-level mapping with some basic high-level features in order to pre-filter the data stream for interesting parts. Utilizing this technique for sonic presence mediation would add several valuable features to the system: A user settable available flag could serve, well, as an available flag indicating whether or not the space in which the user resides is switched on or not. This behavior could e..g be represented by a red light on both sides. This also serves as an explicite privacy controller.

A symbolic history of activity can be added to the system, by processing basic features such as average amplitude of the signal over the last minutes.