What happens when looking at a screen is no longer the best option?
Computing has become increasingly mobile. Phones accompany people through cities, workplaces, public transport systems, shops, festivals, and countless other environments. Yet much interaction design still assumes that users can devote their attention to a display whenever information needs to be communicated. During his online guest lecture for Edinburgh Napier University, Professor Stephen Brewster challenged that assumption. Drawing on decades of research in human-computer interaction, multimodal interfaces, auditory displays, sonification, and mobile computing, he explored a deceptively simple question. What happens when information is communicated through sound rather than vision?
Brewster began by situating the discussion within a broader problem. Human beings possess multiple senses, though much digital technology continues to privilege vision above all others. Screens dominate contemporary computing. Menus, notifications, progress indicators, maps, messages, and data visualisations typically assume that users are willing and able to look. Yet many situations challenge this assumption. Someone cycling through traffic cannot continuously monitor a display. A pedestrian navigating a crowded city may already be dividing attention between multiple tasks. Bright sunlight can render screens difficult to read. Some users experience visual impairments. Others simply have more pressing demands on their attention than a device in their hand. Rather than treating these situations as exceptions, Brewster suggested they reveal a limitation in conventional interface design. If visual attention is unavailable, how else might information be communicated?
This question has shaped much of his research. Rather than viewing sound as decoration or enhancement, Brewster approaches it as a communication channel. Sound can operate while users look elsewhere. It can communicate information rapidly. It can support accessibility. It can function alongside vision rather than competing with it. The goal is not to replace screens entirely. Instead, it is to make fuller use of the sensory capabilities people already possess. Multimodal interaction, as Brewster described it, involves designing systems that acknowledge how people actually experience the world rather than assuming that vision should always dominate.
Mobile devices provided an especially important motivation throughout the lecture. Traditional desktop computing emerged within relatively controlled environments. Users sat at desks, faced screens, and focused primarily on a single task. Mobile computing transformed those assumptions. People now interact with technology while moving through complex environments filled with competing demands upon their attention. A larger display cannot solve every problem. In many situations, the challenge is not the quantity of visual information available. The challenge is finding ways to communicate information without requiring users to look at all. Brewster argued that interaction design should respond to these realities rather than simply shrinking desktop interfaces onto smaller screens.
Attention emerged as a recurring concern throughout the lecture. Many interface designs implicitly assume that information should compete for attention whenever it becomes available. Notifications flash. Windows appear. Alerts demand immediate responses. Yet everyday life rarely operates in this way. People constantly manage multiple streams of information simultaneously. Conversations continue while walking. Music plays while working. Environmental sounds remain present while attention shifts elsewhere. Brewster’s work asks whether digital systems might learn from these patterns. Rather than repeatedly interrupting users, could information move fluidly between foreground and background depending upon circumstances? Sound appears particularly well suited to this challenge. Unlike visual displays, which generally require direct attention, auditory information can remain available while users focus elsewhere. The question is not simply whether sound can communicate information. It is whether sound can communicate information without constantly demanding attention.
One reason sound becomes attractive in this context is its efficiency. Speech can communicate detailed information, though it requires time. A spoken message unfolds word by word. Non-speech audio can often communicate information much more rapidly. Brewster compared the relationship between speech and non-speech sound to the relationship between text and icons. A paragraph may describe an object in detail. An icon can often communicate a similar idea almost instantly. Carefully designed sounds can function in much the same way. Rather than reading information aloud, they communicate status, warnings, activity, trends, or relationships through concise auditory cues.
Much of the lecture explored different approaches to designing these cues. One of the earliest involved earcons, structured auditory messages built from musical elements such as rhythm, pitch, timbre, and tempo. Unlike everyday sounds, earcons are abstract. Their meaning must be learned. Yet this abstraction also provides flexibility. Brewster demonstrated how simple auditory components can be combined to create larger structures capable of communicating increasingly complex information. A particular rhythm might signal an error. Changes in timbre or pitch might identify different categories of error. Much like language, the system develops a vocabulary from smaller building blocks. Users invest effort in learning the code, though once acquired it can support sophisticated communication through relatively simple sounds.
Auditory icons take a rather different approach. Instead of relying upon abstract structures, they exploit familiar sounds drawn from everyday experience. Brewster discussed William Gaver’s influential SonicFinder project, which mapped computer operations onto recognisable environmental sounds. Selecting a folder might produce the sound of paper. Dragging an object across the desktop might generate a scraping sound. Deleting a file might end with breaking glass. Such sounds often require little training. Their meaning emerges from existing associations. Yet the approach also reveals interesting limitations. Everyday life contains only a finite number of obvious metaphors. As software functions become more specialised, finding intuitive sonic equivalents becomes increasingly difficult. What sound represents copying a file rather than moving it? What sound represents a menu hierarchy? Questions such as these expose the challenges that emerge when designers depend upon metaphor alone.
A third approach, sonification, shifts attention away from interfaces and towards data. Here, numerical values are mapped onto auditory parameters such as pitch, rhythm, or timbre. Brewster compared the process to visualisation. Graphs provide rapid access to patterns that would be difficult to identify within tables of numbers. Sonification attempts to achieve something similar through listening. By converting data into sound, listeners can often identify trends, anomalies, peaks, and relationships that might otherwise remain hidden. Rather than replacing detailed numerical information, sonification provides an overview. It allows users to perceive the broader shape of a dataset before examining specific values.
Questions from students helped illuminate this distinction further. One example involved pollen data transformed into sound through changing pitches. The goal was not to communicate precise measurements. Instead, listeners could quickly identify whether levels were increasing, decreasing, or remaining stable. Brewster argued that this reflects the real strength of sonification. A graph rarely succeeds solely through precision. It succeeds by revealing patterns. Sonification can achieve a similar outcome through auditory perception. Numerical detail remains available when required, though sound offers a rapid way of monitoring change over time.
Several studies discussed during the lecture demonstrated how even relatively simple sounds can influence interaction. One experiment examined numerical data entry on mobile devices. Participants entered information using either large visual buttons or substantially smaller alternatives. Predictably, performance declined when the buttons became smaller. Yet when simple auditory feedback was added, performance improved dramatically. Users working with the smaller controls performed almost as well as those using larger buttons. The sounds themselves were uncomplicated. Their value lay not in complexity but in the additional information they provided. By reducing uncertainty during interaction, they made the task easier to perform.
Another particularly elegant example involved progress indicators. Most software communicates progress visually through bars that gradually fill across a display. Brewster and colleagues explored whether similar information could be represented spatially through sound. As a task progressed, a sound moved around the listener’s head. Position communicated completion. Movement communicated change. Without looking at a screen, users could estimate how far a process had progressed and whether activity had stalled.
During the discussion period, students questioned whether such displays might become intrusive. Brewster responded by drawing attention to forms of ambient awareness that already exist within everyday life. People rarely focus continuously on air-conditioning systems, distant traffic, rainfall, or background conversations. Such sounds remain available without demanding constant attention. Auditory displays, he suggested, can function in a similar way. Information remains present when required, fading into the background when it is not. This idea runs through much of his research. Sound is not always most effective when it occupies the foreground. Sometimes its greatest strength lies in supporting awareness without interruption.
Spatial audio appeared repeatedly throughout the lecture as a particularly rich area for exploration. Rather than treating sound as something emitted from a single speaker, Brewster investigated how information might be organised around listeners in three-dimensional space. Progress indicators could move around the head. Calendar entries could occupy positions corresponding to times of day. Menu items could exist within an auditory environment rather than a visual one. These systems exploit the human ability to localise sound sources, transforming listening into a form of navigation. Information acquires location. Interaction becomes spatial rather than purely symbolic.
Some of the most imaginative projects discussed during the lecture extended these principles into everyday environments. AudioFeeds transformed social media activity into ambient soundscapes. Twitter, Facebook, news feeds, and other information streams occupied different locations within auditory space, represented through distinct families of sounds. Rather than repeatedly checking a screen, users could maintain a broader awareness of activity through listening. Detailed information remained available when required, though constant checking became unnecessary.
The significance of AudioFeeds extends beyond social media. The project raises broader questions about how digital information should occupy everyday life. Many contemporary systems assume that awareness requires direct inspection. Brewster’s work suggests alternatives. Awareness may emerge gradually. Information may remain peripheral until circumstances make it relevant. In this respect, auditory displays resemble many naturally occurring environmental sounds. People rarely monitor rainfall continuously, though they remain aware that it is raining. They do not focus constantly on traffic outside a window, though they often notice when conditions change. Sound supports forms of awareness that differ from the all-or-nothing relationship often associated with visual attention.
Pulse extended these ideas into urban environments. During the Edinburgh Festival, geolocated tweets became spatial audio cues distributed around the city. The project transformed social activity into something that could be heard rather than viewed. Participants were not presented with lists of events ranked by popularity, nor were they required to consult maps repeatedly. Instead, they developed a sense of where activity was occurring through listening.
One of the most interesting aspects of the project is that it occupied a space between navigation and exploration. Traditional navigation systems attempt to guide users towards predetermined destinations. Pulse encouraged discovery instead. Participants moved towards sounds that suggested activity, curiosity, or interest. Information became something encountered rather than simply retrieved. In doing so, the project demonstrated how auditory displays can support forms of engagement that differ substantially from conventional graphical interfaces.
The lecture concluded with one of Brewster’s more recent ideas: musicons. Earcons require designers to construct sounds from scratch. Musicons instead draw upon music that listeners already know. Research revealed that people consistently identify particular moments within familiar songs as especially representative. Often these moments involve vocals, choruses, or distinctive melodic features. By extracting such fragments, it becomes possible to create recognisable auditory cues from a user’s existing music collection. The appeal lies partly in familiarity. Rather than learning a completely new auditory language, users rely upon associations they already possess. Recognition emerges from memory rather than training.
Musicons reveal another recurring theme in Brewster’s work. Successful interfaces rarely begin from technology alone. They begin from existing human abilities. Earcons ask users to learn a new auditory language. Musicons exploit knowledge that listeners already possess. A few notes from a familiar song may be recognised almost instantly. Years of listening experience become part of the interface itself.
Looking across the different projects discussed during the lecture, it becomes clear that Brewster is addressing a much larger question than how to design better sounds. The deeper issue concerns the relationship between people and technology. Modern computing frequently competes with the surrounding world for attention. Screens draw the eye away from streets, conversations, environments, and other people. Brewster’s work suggests that alternative relationships may be possible.
Sound occupies a distinctive position within this discussion. It can communicate information while allowing users to continue looking elsewhere. It can support awareness without requiring constant inspection. It can reveal patterns within data, provide feedback during interaction, and create new forms of accessibility. Most importantly, it can coexist with other activities rather than replacing them.
None of this means that sound should replace vision. Brewster repeatedly emphasised the value of multimodal design rather than sensory competition. Different senses possess different strengths. The challenge for interaction designers is understanding how those strengths can complement one another. Sound becomes most useful not when it attempts to imitate visual displays, but when it contributes capabilities that vision alone cannot easily provide.
For many people, digital interaction has become almost synonymous with looking at screens. Brewster’s lecture offered a reminder that computing does not need to be confined to vision. Human beings hear, touch, move, and orient themselves within space. Designing for those abilities opens possibilities that extend far beyond the display. In that sense, the lecture was not really about sound alone. It was about recognising the full range of ways people experience the world.