3.3: Using sonification to augment cognition

Perception, attention, and situational awareness

One way in which sonification could be used to enhance or augment cognition, is by improving situational awareness. Here I refer to situational awareness broadly, as maintaining conscious knowledge of the immediate environment and all the events happening within it. Our ability to maintain situational awareness, while obviously important for many tasks, is limited not only by what information is available in the environment, but also by our ability to process it (e.g., capacity limitations in working memory, attention, perception, etc.). Work in this area has demonstrated some success in improving situational awareness and task performance by using sonification to facilitate attention and perception processes.

Attention is perhaps the most obvious way sonification could be used to improve situational awareness, and the most easily demonstrated. Maintaining situational awareness in complex environments requires that we constantly monitor multiple streams of information. One obvious way to facilitate situational awareness is to offload the monitoring task using auditory alerts or alarms (Hermann et al., 2011; Nees & Walker, 2009). The ubiquity of auditory alarms, from phone alerts to emergency vehicle sirens, makes it easy to over-look. However, they provide an easy way to offload what would be cognitively demanding task (i.e., vigilance or prospective memory), allowing the listener to engage in other tasks. The use of complex auditory alarms has proven useful in a range of settings and tasks including medical or patient monitoring (Cabrera, Ferguson, & Laing, 2005), air-traffic controllers (Cabrera et al., 2005), and piloting aircraft (Edworthy et al., 2004).

Situational awareness in complex environments can be difficult because of the overwhelming amount of information and our limitations in dividing attention. Another way that sonification can aid situational awareness is by transforming multiple streams of information into a more useful, easier-to-manage format for real-time monitoring. There are two fields that have demonstrated the usefulness of sonification tools to facilitate situational awareness by overcoming limitations in divided attention: computer-network traffic monitoring and anesthesiology.

Computer network administrators must monitor flow of traffic in real-time to identify anomalous events like drops in traffic that may reflect hardware failures, or sudden increases in certain types of traffic that could reflect network intrusions (Axon, Alahmadi, Nurse, Goldsmith, & Creese, 2018). Given the large amount of data the network receives every second, the data needs to be aggregated in a way that allows for real-time monitoring. Sonification tools have been shown to be useful for this purpose, demonstrating that listeners can detect network intrusions and anomalous changes in network activity using different sonification methods (e.g., Ballora, Giacobe, & Hall, 2011; Debashi & Vickers, 2018; Vickers, Laing, Debashi, & Fairfax, 2014; Vickers et al., 2017). For example, Qi, Martin, Kapralos, Green, & García-Ruiz (2007) mapped various network traffic data to piano sounds that allowed listeners to detect different types of network intrusions and Gilfix and Couch (2000) mapped network traffic to naturalistic sounds (e.g., chirping, heartbeats) which allowed listeners to detect anomalies in network traffic.

Similarly, anesthesiologists are faced with a similar problem. They need to monitor multiple streams of information about the patient in real-time (e.g., heart rate, central venous pressure, central artery pressure, etc.), often while time-sharing between other tasks. Work in this area tends to show that anesthesiologists and non-anesthesiologists can detect changes using auditory displays as good as when they used visual displays. However, they tend to time-share between tasks better when using an auditory display (Fitch & Kramer, 1994; Loeb & Fitch, 2002; Paterson, Sanderson, Paterson, & Loeb, 2017; Seagull, Wickens, & Loeb, 2001; M. Watson & Sanderson, 2004).

Sonification can also improve situational awareness by augmenting perception. That is, sonification methods can be used to enhance the perceptual representation of our environment by providing extrasensory information. Many studies, for example, have focused on supplementing visual information for the blind using sonification. To aid in navigation, there has been success sonifying depth information (Brock & Kristensson, 2013), and the location of objects (Pavlo Bazilinskyy et al., 2016), and even one demonstration of using echolocation (Kish, 2009). Others have shown success sonifying more complex visual information like object identity (Nagarajan, Yaacob, & Sainarayanan, 2003) and line graphs (L. M. Brown & Brewster, 2003).

However, there are other examples, where extrasensory information is sonified to enhance perception. Probably, the most well-known, and most-often cited example is the Geiger counter. Developed in the early 1900’s, and still used today, the Geiger counter transforms ionization events into audible clicks allowing us to perceive radiation levels in the environment (Knoll, 2010). Another example, called the “Visor” transposes color into sounds to create artificial synesthesia (Foner, 1999). Given our visual system, different sets of wavelengths can appear as the same color, assuming you adjust the relative amplitudes accordingly. Therefore, two objects could then appear to have the same color although they have different spectra; we can perceive the color, we cannot perceive the shape of the spectrum. The visor was designed to sonify the color spectra to enable the user to discriminate colors based on the shapes of the spectrum. For example, you could hear the difference between a painting and a copy of painting, even if visually they are indistinguishable, hear camouflaged objects, or as the authors suggest, the device could be extended to allow you to hear ultraviolet, infrared, or polarized light.

Perception and action in motor skill learning

Another way in which sonification could augment cognition, is by improving perception and action in motor skill learning. That is, sound could be used to provide real-time feedback about performance in a motor task, guiding a learner towards their goal or correct performance. Enhancing motor learning has been explored using auditory alarms, sonified movement feedback, and sonified error feedback (J.F. Dyer, Stapleton, & Rodger, 2017; Sigrist, Rauter, Riener, & Wolf, 2013).

Auditory alarms have proven useful for improving motor skill learning. They are simplest form of sonification in that any movement considered an error triggers an alarm. They are easily interpreted by the learner, though they provide little information about how to correct performance. In rehabilitation, for example, auditory alarms have been used to inform patients about errors in movement (e.g., incorrect gait, unphysiological loading), and shown success in helping the learner correct the behavrio (Batavia, Gianutsos, Vaccaro, & Gold, 2001; Eriksson & Bresin, 2010; Petrofsky, 2001; Riskowski, Mikesky, Bahamonde, & Burr, 2009). Similarly, auditory alarms have facilitated motor training in gymnastics (Baudry, Leroy, Thouvarecq, & Chollet, 2006) and improving rifle movements for professional shooters (Underwood, 2009).

Sound has also been used to provide constant, real-time feedback about movement. This is considered ‘direct sonification’ because some body movement is directly mapped to sound to provide additional information and guide the learner to correct performance. For example, your location is 3D space could be mapped to amplitude and pitch of a constant sound helping you navigate through space. There is some evidence that continuous sonified feedback is beneficial in simple motor tasks; In simple reaching tasks, for example (Oscari, Secoli, Avanzini, Rosati, & Reinkensmeyer, 2012; Schmitz & Bock, 2014). Unfortunately, however, there is little direct evidence that continuous auditory feedback is beneficial in complex motor tasks. There was some success using sonified movement feedback in swimming tasks (Chollet, Madani, & Micallef, 1992; Chollet, Micallef, & Rabischong, 1988), although these effects might be explained better by increased motivation (Sigrist et al., 2013). Constant sonified movement feedback has also been incorporated in a number of different motor tasks like karate (Yamamoto, Shiraki, Takahata, Sakane, & Takebayashi, 2004), rowing (Schaffert, Mattes, & Effenberg, 2009), and skiing (Kirby, 2009), but there have not been corresponding motor learning studies to validate whether they are in fact beneficial for the learner (Sigrist et al., 2013).

There has been more success in using sonified movement error feedback to improve motor-skill learning (Oscari et al., 2012; Schmitz & Bock, 2014). Here, the sound does not directly correspond to your movements, but instead corresponds to your movements in relation to some criterion. For example, instead of directly mapping sound to your location in 3D space, you could map sound parameters to the relationship between your position and some target location (e.g., increase in pitch as you move closer to the target). Using this method has shown some benefits across different complex motor tasks such as speed skating (Boyd & Godbout, 2010) and rowing (Sigrist et al., 2011). Shooting scores during rifle training was also improved with error feedback. Here, the pitch of a pure tone was mapped to the deviation of the gun barrel to the bullseye.

Data analysis and pattern recognition.

One of the goals of datamining or data exploration is to detect hidden regularities in high dimensional data. Our ability to detect these hidden regularities is of course dependent on the representation of the data and our ability to recognize the patterns. As mentioned earlier, our auditory system excels at detecting very subtle patterns in sounds (Grond & Hermann, 2014a, 2014b; Hermann et al., 2011). The use of auditory data representations in fact has a long history, well-before there was a term for it (see Frysinger, 2005). The stethoscope, for example, still provides valuable information for a physician, and Pollack and Ficks (1954) mapped multi-dimensional data onto sound parameters to evaluate the information transmission properties of auditory stimuli (i.e., information “bits”).

Speeth (1961), provided one of the earliest studies that showed the advantages of using auditory data representations over visual for data pattern recognition. Here they were interested in using seismic measurements to discriminate between earthquakes and underground bomb blasts. The seismometer produces complex wave patterns and using visual displays of the data for categorization proved to be a very difficult task. However, once the seismic data transformed into sound, subjects could accurately classify seismic activity on 90% of the trials. Additionally, because the data was time compressed, an analyst could review up to 24 hours of data in 5 minutes.

Other early work has also shown the advantages to using auditory representations when dealing with complex multivariate data. Morrison and Lunney used sound to represent infrared spectral data (Baecker & Buxton, 1987) and Yeung (1980) used sound to represent experimental data from analytical chemistry where subjects achieved 98% classification with little practice. Similarly, Mezrich, Frysinger, & Slivjanovski (1984) used both auditory and visual components to represent multivariate time-series economic data. They found that their dynamic multi-modal display generally outperformed static visual displays.

This early work is important in that it demonstrates that some data sets are well-suited for sonification and confers pattern recognition benefits. These are often dense, multivariate data sets that can take advantage of the temporal nature of auditory representations. More recent work has expanded the range of applications of sonification for data exploration with some notable successes.

One area that has shown the usefulness of sonification is in the interpretation of brain data. For example, real-time monitoring and analysis of ectroencephalographic (EEG) data has diverse application areas including medical screening, brain computer interfaces, and neurofeedback (Väljamäe et al., 2013). Recent work shows that sonification facilitates the interpretation and categorization of EEG data (Baier, Hermann, & Stephani, 2007; Baier et al., 2007; De Campo, Hoeldrich, Eckel, & Wallisch, 2007). For example, sonified EEG data has been used to detect epilectic seizures. One study transformed EEG data into music (snapping time-frequency data to notes in a musical scale) and found that subjects could identify seizures from the auditory data alone (Loui, Koplin-Green, Frick, & Massone, 2014; Parvizi, Gururangan, Razavi, & Chafe, 2018). Similarly, positron emission topographical (PET) data has been sonified to facilitate the diagnosis of Alzheimer’s disease (Gionfrida & Roginska, 2017). Not limited to brain data, other biomedical signals like electrocardiographic (ECG) data have been sonified facilitating the detection of cardiopathic pathologies and other anomalies (Avbelj, 2012; Kather et al., 2017).

The range of fields that have begun to adopt sonification for data exploration, and have shown promising results, is in fact staggeringly diverse. From astronomical data (W. L. Diaz-Merced et al., 2011; W. L. L. Diaz-Merced, 2017; Lunn & Hunt, 2011), meterological data (George et al., 2017), oceanography (Sturm, 2005), physics (Pereverzev et al., 1997), biomedicine (Avbelj, 2012; Larsen, 2016), social sciences (Dayé & de Campo, 2006), to space exploration. During the Voyager 2 mission, the spacecraft was going through the rings of Saturn when it encountered a problem. The operators could not identify the problem through visual analysis of the data. However, once the data was played through a music synthesizer, a “machine-gunning” sound was heard, leading them to conclude that the problem was caused the problems were caused by high-speed collisions with electromagnetically charged micro-meteoroids (Barrass & Kramer, 1999). Sonification can alter our perception of the data allowing insights and pattern recognition that were not possible using visual displays.