Sound Design at Edinburgh Napier University

Category: Interaction Design

  • Why Game Sound Is Never Finished: Mariana Botero on Systems, Possibility, and Interactive Audio

    Mariana Botero

    What does a sound designer actually create?

    For much of the history of recorded media, the answer has seemed relatively straightforward. A sound designer creates sounds. Those sounds are edited, arranged, mixed, and eventually delivered to an audience in a finished form. Whether working in film, television, radio, or theatre, the result remains largely fixed. Every audience member encounters the same sequence of events in the same order. A soundtrack may be replayed thousands of times, though the sounds themselves do not change. Sound design, in this traditional sense, is largely concerned with creating and refining artefacts. The work may be extraordinarily complex, though the outcome is ultimately stable. Once completed, the audience receives the experience that the creators intended.

    Games operate according to a different logic. During her guest lecture at Edinburgh Napier University, Mariana Botero, Sound Designer at Criterion Games, repeatedly returned to a challenge that sits at the centre of interactive audio. Players are unpredictable. They may rush through environments that designers expected them to explore carefully. They may spend an hour in a location intended for a few minutes of gameplay. They may repeat actions endlessly, ignore carefully placed cues, or discover solutions that nobody anticipated. A sound designer may spend weeks refining a particular moment, only for players to experience it in a completely unexpected way. This unpredictability creates a fundamental difference between games and most other forms of media. Film sound designers can assume a degree of control over audience experience. A scene unfolds at a predetermined pace. Music enters at a precise moment. Dialogue arrives exactly when it is needed. Sound and image progress together through a carefully authored sequence. Games surrender much of that control. Designers can establish possibilities, though they cannot determine exactly how those possibilities will be experienced. Every player creates a slightly different path through the material.

    Botero illustrated this distinction through a deceptively simple analogy. Traditional sound design can resemble baking a cake. Once the ingredients have been combined and baked, they become a finished object. Individual components can no longer be separated. The audience receives the completed result exactly as intended. Interactive audio often works differently. The ingredients remain available. They can be rearranged, adjusted, recombined, and reshaped while the experience is unfolding. What players hear depends not only on what the designer created but also on what they choose to do. A player who rushes through an environment may hear one version of the experience. Another who explores every corner may encounter something quite different. Neither experience is incorrect. Both emerge from the same underlying system. At first glance, this may appear to be a technical distinction, though throughout the lecture it became clear that something more significant is taking place. Interactive audio challenges assumptions that have shaped sound practice for decades. Rather than creating a finished soundtrack, game audio designers increasingly create systems capable of generating many possible soundtracks. The question is no longer simply what a sound should be. The question becomes how a sound should behave.

    Implementation sits at the heart of this shift. Students often encounter implementation through software platforms such as Wwise and Unity, where attention naturally gravitates towards events, switches, parameters, states, and scripting. Botero encouraged a broader perspective. These tools are not merely technical requirements added after the creative work has been completed. They are part of the creative process itself. They provide mechanisms through which sound can respond to player actions, environmental conditions, narrative developments, and changing game states. Once audio becomes interactive, implementation ceases to be a final stage of production. It becomes one of the primary ways in which experiences are designed. Many of Botero’s examples reflected this movement from sounds to systems. Consider something as apparently mundane as footsteps. Few sounds occur more frequently in games. A footstep that appears a handful of times in a film may occur thousands of times during a single play session. What initially feels satisfying can quickly become repetitive. Players generate these sounds through their own behaviour, meaning designers cannot simply assume that repetition will remain unnoticed. Botero discussed several implementation strategies designed to address this issue. Different recordings can be selected randomly. Pitch and volume may vary subtly between repetitions. Heel and toe impacts can be separated and recombined dynamically. A relatively small collection of recordings suddenly produces a far wider range of perceived outcomes. Yet what makes these techniques interesting is not their technical sophistication. Their significance lies in how they reveal a different philosophy of authorship. Rather than crafting every individual event directly, designers create rules governing how events are generated. They design the behaviour of the system rather than the precise form of every outcome.

    Her observations about footsteps led naturally into a broader discussion about attention. Not every sound deserves equal prominence throughout an experience. Early in a game, clearly audible footsteps may help players understand movement and control. Later, those same sounds can begin to dominate the soundscape unnecessarily. As players become familiar with core mechanics, environmental details often become more valuable. A distant owl, subtle weather activity, a passing vehicle, or an unexpected environmental cue may contribute more to a sense of place than another clearly articulated footstep. This may seem like a relatively small design decision, though it reveals an important principle running throughout the lecture. Sound design is not simply about creating sounds. It is about shaping attention. Designers are constantly deciding what players should notice, what they should ignore, and how their awareness should be directed through an experience. Interactive audio therefore becomes inseparable from broader questions of perception.

    Several examples from Botero’s work on Star Wars Battlefront II illustrated this relationship particularly clearly. One challenge involved creating a convincing sensation of speed during space combat. Space provides surprisingly few visual reference points. Without roads, buildings, or passing landscapes, players can struggle to judge how fast they are moving. From a purely visual perspective, extraordinary speeds can sometimes feel unexpectedly slow. Audio therefore assumes a more active role. Changes in acceleration, orientation, proximity, and manoeuvring can all be reinforced through sound, encouraging players to feel faster than they actually are. Importantly, the objective is not deception. The objective is alignment between what players see, what they hear, and what they believe they are experiencing. Sound helps bridge the gap between game mechanics and player perception. A related challenge emerged around spatial awareness. Players navigating complex three-dimensional environments often need information that visual displays cannot always communicate efficiently. Botero described examples where reflections, pass-bys, and environmental responses helped players understand their relationship to surrounding structures. Those sounds functioned as navigational aids as much as sound effects. Players may never consciously notice them, though their absence would make environments feel less intelligible. Such examples reveal another important aspect of interactive audio. Sound is not simply representing the world. It is helping players interpret the world. A successful design often communicates information, directs attention, reinforces emotion, and supports decision-making simultaneously. The most effective implementation frequently becomes invisible. Players simply feel that the game responds naturally to their actions.

    The same principles appeared again in Botero’s discussion of interactive music. Traditional composition assumes a relatively stable structure. A beginning leads towards a middle before eventually reaching an ending. Although composers may create complex and highly sophisticated works, they generally retain control over the order in which events occur. Games rarely provide such certainty. Players may linger in one location, interrupt events unexpectedly, revisit spaces repeatedly, or trigger narrative developments in unusual orders. Music must therefore accommodate possibilities that cannot be fully predicted in advance. Rather than composing a single linear sequence, designers often create collections of musical elements capable of being reorganised dynamically while preserving coherence. Botero compared the process to building with Lego bricks. Individual pieces remain consistent, though their arrangement changes according to context. Introductory passages, transitions, loops, and endings can be connected in different ways while still feeling musically coherent. Once again, the designer is not creating a single outcome. The designer is creating a framework capable of supporting many outcomes. What matters is not only the material itself but also the relationships that determine how that material behaves under changing circumstances.

    This way of thinking extends beyond music. One particularly revealing example discussed during the lecture involved a student project built around a time-slowing mechanic. In a traditional medium, slowing time might simply involve applying predetermined processing at specific moments. Within an interactive environment, however, the relationship becomes dynamic. Real-time parameter controls allowed player actions to influence audio behaviour directly. As gameplay changed, the soundscape changed alongside it. The mechanic was not merely accompanied by sound. The mechanic became part of the sound design process itself. Audio no longer functioned as a layer added on top of gameplay. It became woven into the behaviour of the system. Examples such as these help explain why programming increasingly occupies an important place within contemporary game audio. Throughout the lecture, Botero described learning through experimentation, tutorials, practical projects, and professional experience. Programming was presented not as a replacement for creative thinking but as a means of expanding creative possibilities. The more deeply designers understand systems, the more effectively they can connect audio to player experience. Technical knowledge becomes valuable not for its own sake but for the opportunities it creates.

    One of the more interesting consequences of this shift is that game audio designers increasingly resemble system designers as much as traditional sound designers. Much of the language surrounding audio production still reflects assumptions inherited from film, television, and music. We often speak about creating sounds, mixing sounds, or arranging sounds. Botero’s examples repeatedly pointed towards a broader form of practice. Designers create relationships between sounds, player actions, environmental conditions, and game states. Their work involves determining how sounds respond, adapt, and evolve rather than simply deciding what those sounds should be. The resulting experience emerges through interaction between the player and the system. In this sense, implementation is not something that happens after sound design. Implementation is increasingly part of sound design itself.

    This spirit of exploration appeared throughout Botero’s account of her own development. Long before working professionally in games, she was already engaging deeply with sound through recording, listening, and experimentation. Growing up in Colombia, she developed a fascination with environmental sound that later evolved into a broader interest in audio design. Looking back across the lecture, one of the most striking aspects of her career trajectory is how consistently curiosity appears as a driving force. New tools, new techniques, and new technologies repeatedly emerge, though progress often begins with a simple willingness to investigate how things work. Throughout her account there was little sense of a fixed pathway into game audio. Instead, learning appeared as an ongoing process of exploration in which technical knowledge, creative practice, and experimentation continually informed one another.

    That same curiosity remains visible within professional practice. One of the most memorable examples came from Criterion’s “Sound Design Thursdays”. Team members temporarily step away from production work and undertake creative challenges built around unusual constraints. Designers exchange recordings, limit themselves to small collections of source material, or pass sounds between colleagues with each person applying a single transformation before handing them on. The outcomes can be surprising, humorous, and occasionally absurd. A camel recording may gradually evolve into something resembling a bird. Familiar sounds acquire entirely new identities. Yet beneath the humour lies a serious lesson about creativity. Unexpected results often emerge when designers deliberately create conditions that encourage experimentation. Limitations become opportunities. Constraints become creative tools. In many respects, these exercises mirror the philosophy underlying interactive audio itself. Both involve constructing systems, establishing rules, and allowing surprising outcomes to emerge from interactions between different elements. Neither depends upon complete control. Both depend upon creating environments in which interesting possibilities can arise.

    Looking back across the lecture, what emerges most clearly is a view of sound design that extends well beyond the creation of individual sounds. Footsteps, adaptive music, spatial cues, implementation systems, real-time parameters, and creative experiments all point towards the same conclusion. Interactive audio requires designers to think in terms of relationships rather than isolated assets. Sounds gain meaning through how they respond to players, environments, and changing circumstances. The challenge is no longer simply to create a soundtrack. The challenge is to create a framework capable of supporting many different soundtracks.

    Perhaps this is what makes game audio such a distinctive area of contemporary sound practice. Film sound designers craft experiences that audiences receive. Game sound designers craft possibilities that audiences help create. Every play session unfolds differently. Every player encounters a slightly different combination of events. Every interaction generates new relationships between sounds, systems, and behaviours. The sounds matter. What matters just as much are the rules that determine what those sounds might become.

    In game audio, the work is never truly finished. It simply waits for the player to decide what happens next.

  • Why Do the Sounds of Don’t Starve Feel So Alive? Matthew Marteinsson on Experimentation, Voice, and Play

    Matthew Marteinsson

    Many games strive for realism. They aim to reproduce the sound of the world as accurately as possible, carefully modelling spaces, materials, physics, and behaviours so that players feel immersed in a believable environment. Don’t Starve takes a rather different approach. Its world is filled with living scarecrows, walking trees, giant spiders, impossible creatures, and surreal landscapes that seem to have escaped from the pages of a dark storybook. Very little about it appears realistic in any conventional sense. Yet despite this, the game feels remarkably alive.

    Matthew Marteinsson’s guest lecture explored how that happened. Although the talk covered specific technical systems, recording techniques, production challenges, and implementation details, a broader idea repeatedly emerged beneath them. The sounds of Don’t Starve do not feel convincing because they imitate reality. They feel convincing because they remain connected to physical performance, playful experimentation, and a constant willingness to explore unexpected possibilities.

    During the early development of Don’t Starve, Marteinsson was effectively the sole audio designer working alongside two composers, with no dedicated audio programmer and no substantial audio department behind him. Development moved rapidly, content changed constantly, and there was little opportunity for elaborate production pipelines. Rather than treating these limitations as obstacles, the team repeatedly used them as opportunities to find simpler and more creative solutions. Constraints were not merely something to overcome. They actively shaped the character of the game’s sound world.

    The game’s character voices provide an excellent example. Traditional voice acting would have required large quantities of dialogue recording, scheduling actors, and continuously updating content as the game evolved. Such an approach was difficult to reconcile with the speed at which the project was being developed. Yet the characters still needed personality, emotional expression, and identities that players could immediately distinguish. Instead of using spoken language, Marteinsson turned to musical instruments. Inspired partly by the adults in Peanuts cartoons and partly by Peter and the Wolf, where different instruments represent different characters, each character in Don’t Starvereceived its own instrumental voice. Wilson’s distinctive muted trumpet became the starting point, with subsequent characters developing from their own carefully chosen instrumental identities. What began as a practical solution ultimately became one of the most recognisable features of the game.

    Human vocal performance appeared repeatedly as a creative tool throughout the lecture. Many memorable sounds originated not from extensive libraries or complex synthesis chains but from experimentation with the voice itself. The spiders, for example, were largely built from Marteinsson’s own vocal performances combined with processing. The Gobbler, one of the game’s most beloved creatures, began with attempts to gather suitable turkey recordings. After examining the animation, however, he found himself instinctively making a strange vocal sound that immediately felt more appropriate than any authentic turkey call. The library recordings were discarded and the vocal performance became the creature. As he noted, the deliberately exaggerated human performance communicated personality far more effectively than realism alone could have achieved.

    Realism and believability emerged as distinct ideas within Marteinsson’s approach to sound design. A perfectly accurate turkey recording might have sounded more realistic, though it may not have felt more alive. The Gobbler succeeds precisely because it occupies an unusual space between animal, caricature, and performance. Players are not simply hearing a creature. They are hearing a performance of a creature. The sound communicates character as much as biology.

    Personality often seemed more important than realism throughout the lecture. Many of the creatures in Don’t Starveexist within a visual world that is intentionally exaggerated, stylised, and slightly absurd. Conventional fantasy sound design might have felt strangely out of place. Marteinsson instead described grounding many creatures in a “weird reality”, where recognisable physical behaviours remain present but become filtered through performance, humour, and experimentation. Human vocalisations proved especially valuable in this regard. Audiences are extraordinarily sensitive to nuances in human expression. Even heavily processed vocal sounds can communicate intention, emotion, vulnerability, aggression, or curiosity in ways that are difficult to achieve through purely synthetic or animal-based recordings.

    Environmental audio presented an equally interesting challenge. Procedurally generated worlds create difficulties that traditional environmental sound design rarely encounters. Designers cannot assume where players will travel or which environments they will encounter. Marteinsson described a system that continuously examines the terrain surrounding the player, identifies the dominant biome types within the immediate area, and dynamically blends the corresponding ambiences. Grasslands, forests, marshes, and other environments continuously mix together according to what the player is actually seeing at that moment. Rather than creating a fixed soundtrack for a predetermined world, the system responds to the world being generated in real time.

    What makes this system particularly interesting is that the underlying idea remains remarkably simple. Players should hear the world they are looking at. Technical sophistication only becomes valuable when it strengthens the player’s experience. Systems matter not because they are complex but because they help players understand the world around them. Throughout the lecture, Marteinsson repeatedly demonstrated a preference for elegant solutions that serve a clear experiential purpose.

    The broader design philosophy became especially clear during the discussion following the lecture. Marteinsson argued that game audio should generally perform one of two functions: it should either build the world or inform the player. If a sound accomplishes neither, its value becomes questionable. Such a statement sounds straightforward, though it carries considerable implications for design practice. Many games accumulate audio over time, layering additional sounds onto already crowded mixes. The result can be confusion rather than clarity. Marteinsson instead advocates careful consideration of why a sound exists and what purpose it serves. Sound is not decoration. It is communication.

    Small details often became surprisingly important within this design philosophy. During the lecture, he discussed how player feedback during early access revealed complaints about a particular pickup sound. Some players even requested a dedicated option to disable it. Rather than immediately changing the sound itself, Marteinsson investigated further and discovered that the underlying issue was simply that the sound was mixed too loudly. Once its level was adjusted, the complaints disappeared. The lesson was not that players were wrong. Rather, it highlighted the importance of identifying the underlying problem rather than accepting proposed solutions at face value. Players are often very effective at identifying areas where something feels wrong. Determining why it feels wrong remains part of the designer’s responsibility.

    Recording sessions often sounded closer to scientific experiments than conventional sound production. Music boxes, improvised instruments, jelly, pudding, toys, unusual household objects, mines, and novelty items discovered in shops all found their way into Marteinsson’s recording collection. A music box originally intended for composing melodies eventually became the basis for the unsettling sounds associated with the Shadow Hand. A visit to a local mining museum produced unique underground ambience recordings for the game’s cave systems. Strange objects were collected not because a specific project required them, but because they might become useful in the future.

    Playfulness often appeared not as a break from the work but as part of the work itself. Making strange noises while watching an animation, experimenting with unusual objects, collecting sounds without a specific purpose in mind, or exploring unexpected combinations of recordings all reflect a willingness to follow curiosity wherever it leads. Listening to these stories, it became increasingly clear that creativity often depends upon creating opportunities for surprise. The value of an unusual object or recording does not necessarily become apparent immediately. A sound designer may encounter something intriguing, record it, store it away, and only discover its purpose years later.

    Technical decisions rarely appeared separate from creative ones during the lecture. Recording techniques, implementation systems, middleware, debugging tools, and production constraints were all discussed in detail. Yet none of these elements were treated as separate from creativity itself. Debug tools existed to facilitate experimentation. Procedural systems existed to strengthen immersion. Recording techniques existed to discover new forms of expression. Technology remained important throughout the talk, though it rarely appeared as the primary source of innovation.

    Reflections on game development brought many of the lecture’s themes together. Marteinsson acknowledged the challenges facing the industry, including long hours, instability, and periods of significant uncertainty. Yet his reflections consistently returned to enthusiasm, curiosity, and the joy of creating experiences that players genuinely care about. That optimism felt closely connected to the ideas that had surfaced throughout the lecture. The sounds of Don’t Starve emerged not from a search for perfection but from a willingness to experiment, adapt, collaborate, and occasionally embrace absurd ideas simply to see where they might lead.

    Perhaps that helps explain why the world of Don’t Starve feels so distinctive. Its sounds rarely seem trapped by expectations about what things ought to sound like. A spider may begin as a human vocal performance. A terrifying shadow creature may emerge from a modified music box. An iconic turkey may owe more to an impulsive noise made while watching an animation than to any field recording. Throughout the lecture, Marteinsson repeatedly demonstrated that memorable sound design often emerges when curiosity is allowed to guide the process.

    Rather than attempting to recreate reality exactly, Don’t Starve constructs a world that feels alive through performance, experimentation, and play. Many of the sounds discussed during the lecture began as accidents, improvisations, constraints, or strange ideas that simply seemed worth exploring. What emerged from that process was not merely a collection of sound effects but a coherent sonic world. Listening to the lecture, it became difficult to separate the sound of Don’t Starve from the spirit in which it was created. Both are defined by curiosity.

    In doing so, Marteinsson offered a useful reminder that some of the most memorable sounds are not discovered by following established rules. They emerge when designers remain willing to ask a simple question: what happens if we try this?

  • Speaking into Spaces: Lou Mallozzi on Language, Sound, and Public Architecture

    Lou Mallozzi

    We often think of speech as a way of transmitting information from one person to another. Someone speaks, someone listens, and meaning moves between them. Language therefore appears relatively straightforward. Words describe things, explanations clarify ideas, and speech communicates intentions. Yet speaking also does something else that often receives much less attention. Voices establish relationships. A person standing quietly in a crowded room occupies space in one way, though speaking suddenly reorganises that same environment. Attention shifts. Distances feel altered, while public and private boundaries become less stable. Speech does not simply move through spaces. It also changes them.

    This broader relationship between sound, language, and space formed a recurring concern throughout an online guest lecture delivered by Lou Mallozzi, a Chicago-based sound artist whose work moves across performance, installation, moving image, and public intervention. Although the projects discussed during the lecture varied considerably in form, common questions repeatedly surfaced beneath them. What happens once language stops functioning simply as communication? What happens when speech becomes material rather than message? How do voices alter relationships between people and places?

    Mallozzi described some of these ideas through the notion of an “architectonics of public speech”. Rather than treating speech as something moving cleanly between speaker and listener, this perspective considers speech as something capable of creating structures around it. Speaking establishes relationships, gathers attention, and reshapes environments around it. Architecture repeatedly appeared throughout the lecture, though not always in conventional ways. Buildings establish boundaries physically through walls, rooms, doors, and passages. Sound establishes different kinds of structures. Voices create temporary boundaries of attention, while amplified speech can reorganise atmospheres entirely. A room filled with silent individuals does not feel identical to one organised around speech. The physical dimensions remain unchanged, though experiences of proximity, distance, and social relationships begin shifting once voices enter the environment.

    These ideas became particularly visible in discussion of La Patria Patrisci Patri Partiti, translated as The Fatherland Gives Birth to Departed Fathers. The work emerged partly through reflections on Mallozzi’s father’s experiences growing up in fascist Italy before emigrating to the United States. Historical documents, fascist texts, and autobiographical material became intertwined within the work. Materials themselves were not simply presented directly. Mallozzi instead described subjecting language to processes of subtraction through the removal of letters taken from his own family name.

    Initially this process appears procedural, though the consequences become more complicated once language itself begins breaking apart. Certain words survive while others collapse into fragments. Meaning becomes unstable. Speech no longer functions simply as a transparent carrier of ideas. Language begins behaving differently once removed from its usual role. Words become sounds, rhythms, repetitions, and interruptions. Fragments acquire physical presence independent of straightforward meaning. Listeners are no longer simply absorbing information. They become aware of language itself as material.

    Performance introduced another layer to these relationships through carefully controlled microphone feedback generated from microphones positioned within the mouth itself. Feedback was not presented simply as an acoustic effect. Mouths function as resonant chambers, while rooms similarly become chambers of resonance. Internal and external forms of architecture begin reflecting one another. The body itself begins appearing as another form of acoustic space rather than simply the source from which speech emerges.

    Questions surrounding public space appeared again through works such as Outpost, Peers, and Didact, which approached different environments though repeatedly returned to similar concerns involving observation, attention, and collective listening. Outpost involved Mallozzi standing above public spaces equipped with a telescope and amplified microphone system while verbally describing individuals moving below in real time. Descriptions themselves often remained entirely ordinary. Someone reached into a pocket, adjusted clothing, or walked across an open area. Little of obvious importance occurred.

    Yet amplification transformed the situation entirely.

    Nothing new was being revealed. Individuals already knew what they were doing, while others nearby could already see it happening. Information itself therefore became almost irrelevant. Attention had become the important factor.

    Public spaces frequently depend upon a delicate balance between visibility and anonymity. Most of the time these arrangements remain unnoticed precisely because they operate so quietly. People move through stations, streets, galleries, and public squares while occupying an unusual position between presence and absence. We recognise that others exist around us, though frequently without examining them closely. Shared environments therefore involve a form of social agreement in which attention remains distributed and unstable. Outpost disturbed that agreement. Someone who had previously existed as part of the background suddenly became the centre of attention.

    Surveillance formed part of this discussion, though not entirely in familiar ways. Contemporary surveillance systems frequently operate quietly from concealed positions. Cameras observe without drawing attention to themselves. Outpostinstead exaggerated observation until it became impossible to ignore. The telescope almost became absurd in its visibility. Observation itself became public.

    Related works such as Peers and Didact returned to similar concerns through multiple simultaneous voices. Speech that would normally appear as private reading or individual statements instead became layered, collective, and unstable. Meanings drifted in and out of focus as voices overlapped, while explanations gradually became textures and descriptions became atmospheres. A single voice often carries assumptions of coherence and individuality. Multiple voices instead create shifting relationships where language repeatedly moves between intelligibility and abstraction.

    Later works extended these concerns into moving image and impossible tasks. Screenplay 1-1 involved watching films with the sound removed while attempting to verbally describe everything appearing on screen in real time. Such a task immediately creates difficulty. Language struggles to keep pace with visual experience. Camera movements, gestures, editing, and events unfold more rapidly than speech comfortably allows.

    Mallozzi described many projects of this kind as involving impediments or impossible conditions. Failure therefore ceases to become something requiring avoidance. Contemporary discussions surrounding creative work often assume that limitations are problems waiting to be removed. Better technologies frequently promise greater control, speed, and precision. The projects discussed throughout the lecture repeatedly suggested something slightly different. Constraints sometimes generate possibilities that unrestricted situations would never produce. Impossible tasks force unexpected decisions. Gaps emerge between intention and outcome. Material appears that nobody initially planned.

    Throughout the lecture, speech repeatedly occupied a position between structure and instability. Voices organise attention while simultaneously disrupting it. Language communicates while also becoming detached from communication itself. Sound establishes relationships while continually reshaping them.

    Perhaps one of the more striking ideas emerging from the lecture involved recognising that speaking rarely involves transmitting information alone. Voices connect bodies, spaces, histories, and listeners around them. Speech therefore becomes something more than information moving between individuals. Voices gather people together, establish temporary structures of attention, alter atmospheres, and occasionally make familiar environments seem unfamiliar again.

    Much of this activity normally passes unnoticed. Conversations emerge and disappear. Announcements briefly reorganise public spaces. Voices drift across rooms before fading into the background once more. Listening often feels passive, though Mallozzi’s work repeatedly suggested something rather different. Speaking changes environments, while listening changes our relationships with them. Spaces may appear stable around us, though many of the ways we experience them are continually being reorganised through sound.

  • Creating Sounds for Things We Cannot See: Kenny Young on VR, Music, and Guiding Attention

    Kenny Young

    Many forms of media depend upon controlling attention. Films decide where audiences look through editing, framing, and camera movement. Theatre guides attention through staging and movement. Conventional games frequently do something similar through interface design, visual effects, or camera behaviour. Important information rarely appears entirely by accident. Designers often decide where attention should go long before audiences realise those decisions are being made. Most of this guidance becomes invisible precisely when it works well. Players rarely stop to think about how often games quietly redirect their attention from one place towards another. Experiences simply feel natural. Objectives appear at appropriate moments, important events seem difficult to miss, and information arrives when required.

    Kenny Young’s guest lecture explored what happens once some of these assumptions begin disappearing. Virtual reality introduces a relatively simple change that gradually creates much larger consequences. Players control the camera continuously. Looking left means physically turning left. Looking upwards requires physically raising the head. Looking away from something important may simply mean missing it altogether. Initially this sounds like a relatively minor alteration, though the consequences begin spreading surprisingly far once control over attention starts shifting away from designers themselves.

    Imagine hearing something important happening behind you in a conventional game. Designers possess numerous methods for ensuring that players notice it. Cameras may shift automatically, indicators can appear around the screen, and control may even be briefly interrupted. Decades of game design have produced increasingly sophisticated methods for solving these problems. Virtual reality complicates many of these solutions. Fixed interface elements become intrusive, large overlays can weaken immersion, and information existing outside the player’s field of view can remain entirely unnoticed. Questions therefore begin emerging around how players discover important information once designers can no longer simply place it directly in front of them.

    Young suggested that sound changes role at precisely this point. Human vision behaves selectively. We actively choose where to direct our eyes and ignore much surrounding information. Hearing functions rather differently. Sounds continue arriving whether or not we intentionally seek them out. A player may choose not to look towards something important, though hearing something nearby can still trigger an immediate response. Sound therefore begins moving away from a supporting role attached to visible events and towards something more active.

    Players do not simply hear sounds in games. They gradually learn them. Initially a sound may exist only as another event occurring within a larger environment. Yet repeated exposure slowly changes its role. Through familiarity, sounds begin accumulating meaning. This process often happens without players consciously noticing it. A sound that initially appears neutral gradually becomes linked with expectations, actions, and outcomes. Eventually hearing the sound no longer involves interpreting something unfamiliar. Players instead recognise patterns they have already learned.

    Young discussed the familiar alert sound from Metal Gear Solid as an example. During early encounters players hear a brief cue alongside visual information, though repeated exposure gradually changes the relationship. Eventually players stop hearing the sound as a sound effect at all. Instead, it begins behaving almost like language.

    Language may not be entirely the right word, though the comparison becomes useful. Words themselves do not naturally contain meaning. People gradually learn relationships between sounds and ideas through repeated experience until recognition becomes almost immediate. Something similar begins happening within games. A short musical cue or brief sound effect acquires meaning through use rather than explanation. Players are not consciously translating sounds each time they hear them. Recognition simply becomes increasingly automatic.

    Nobody pauses a game to explain that a particular sound means danger. Players learn these relationships through repeated experience. Over time certain sounds become linked with expectations, actions, and outcomes until responses begin occurring almost automatically. Listening changes in subtle ways once these associations form. Players stop consciously analysing what they hear, as attention begins shifting before deliberate thought catches up. Sound therefore becomes something more than feedback occurring after an event. It starts creating expectations about what might happen next.

    Music introduces another layer to these learned relationships. Discussions around game music frequently focus on emotion, atmosphere, and immersion. Players may notice tension increasing during combat, emotional themes returning around familiar characters, or changing musical textures supporting movement through a world. Young explored another possibility entirely. Under certain conditions, music may also begin operating as information.

    Much of his work on Tethered explored whether these kinds of relationships could be developed within virtual reality environments. Strategy games already involve unusually large amounts of simultaneous information. Resources require management, environments continue changing, threats emerge unexpectedly, and events occur across multiple locations at once. Conventional interfaces frequently solve these problems visually. Players monitor maps, indicators, menus, and notifications distributed around the screen. Translating these expectations into VR introduced a more difficult question. How can players remain aware of a world once they can comfortably see only part of it at any given moment?

    Rather than functioning purely as atmosphere or emotional support, musical phrases could gradually become learned signals recognised through repeated interaction. Certain sounds became associated with changing conditions, important events, or emerging situations. Initially these sounds carried little meaning beyond existing as recognisable musical gestures. Over time something rather different happened. Players were not simply listening to music accompanying a world. They were gradually learning the world itself.

    Listening consequently begins developing an unusual relationship with navigation. Physical landmarks help people orient themselves within real environments, though players may also begin constructing sonic landmarks. Certain sounds become associated with places, behaviours, or changing conditions. Listening therefore starts becoming part of understanding how a world behaves. Particular musical phrases began functioning almost like landmarks within an environment. Certain combinations of sounds became associated with emerging threats or opportunities requiring attention. Over time players could respond before consciously thinking about what had changed. Listening therefore became intertwined with understanding the behaviour of the world itself.

    Examples such as these begin shifting the discussion slightly. Rather than asking whether music sounds appropriate or emotionally effective, another question begins appearing. How do people learn sonic environments? Under what circumstances do sounds stop behaving like sounds and begin behaving more like information? Underlying processes of this kind may already exist across many forms of game audio, even if virtual reality makes them easier to recognise.

    Extending these ideas into working systems introduced additional challenges. Sounds needed to remain distinctive while fitting comfortably alongside underlying music. Delays had to remain short enough that players still connected events with their causes, while multiple simultaneous events could create confusion or dissonance. Initial solutions often resolved one issue only to reveal another elsewhere. Technical constraints, musical decisions, and player behaviour continually interacted throughout development. Creative work therefore emerged less as a process of executing perfect ideas and more as a continual process of adjustment.

    Running throughout the lecture was a broader observation concerning the role of sound itself. Discussions surrounding game audio frequently emphasise realism, emotion, and atmosphere. These remain important concerns, though Young’s work suggested something slightly different. Once familiar methods for directing attention become less reliable, sound begins taking on responsibilities traditionally associated with cameras and interfaces.

    Virtual reality may therefore reveal something that has existed quietly within games for much longer. Sound has rarely functioned only as decoration or atmosphere. It has also shaped where players look, what they notice, and how they organise experiences around them.

    Perhaps the more interesting question is not whether sound helps players understand virtual worlds. It may instead involve asking how much of our experience has always depended upon sound guiding us in ways we barely notice. Once designers lose many familiar methods for directing attention, sound begins moving from the background towards the centre of interaction itself.

  • Creating Sounds for Worlds That Refuse to Sit Still: Malin Arvidsson on Game Audio and Interactive Design

    Sound in games often feels invisible when it is working well. Players notice visual worlds immediately. Landscapes stretch into the distance, characters move through environments, and stories unfold through action and dialogue. Sound tends to arrive more quietly. Footsteps simply seem to belong beneath a character, background ambiences appear to exist naturally around us, and a creature’s voice feels inseparable from its personality. Everything seems to fit together so naturally that the work behind these experiences often disappears from view.

    Yet creating sound for games involves a challenge that differs fundamentally from many other forms of media. Film and television unfold through fixed sequences of events. A sound designer working on a film knows exactly when a door opens, when dialogue occurs, when music begins, and when tension rises. Audiences experience those moments in the same order every time. Games behave rather differently. Players stop unexpectedly, move in different directions, repeat actions endlessly, ignore objectives, or spend long periods interacting with things designers never anticipated would receive much attention. Some players rush directly through environments while others investigate every possible corner of a world. A sound designer may know what can happen inside a game, though cannot always know what will happen, when it will happen, or how often particular experiences will occur. Sound therefore cannot simply be attached permanently to images and left alone. It must continue adapting long after the designer has stepped away.

    During an online guest lecture, Malin Arvidsson explored this challenge through reflections on her own experiences working across game audio. Throughout projects involving children’s games, procedural systems, and large-scale interactive worlds, a recurring idea gradually emerged. Game audio frequently involves building systems rather than constructing isolated sounds. Designers create frameworks, relationships, and behaviours that continue operating within worlds that remain unpredictable.

    Arvidsson described discovering games somewhat unexpectedly. Having decided at an early age that she wanted to work with sound, she initially pursued sound engineering and recording work before later encountering opportunities in game production. Games had not necessarily appeared to be an obvious destination at the time. Film and television perhaps felt more visible as career directions, while game audio remained relatively unfamiliar. Yet after joining Audio Interactive and working on early projects, games gradually became something much larger than a temporary opportunity. Part of this attraction appeared to emerge from constant change. Technologies evolve rapidly, development processes shift, while projects rarely require exactly the same approaches twice. Many creative fields involve continual learning, though games introduce an additional layer of complexity through their combination of artistic decisions and technical systems. Sound designers are often required to think simultaneously about recording, editing, implementation, behaviour, memory, interaction, and player experience.

    Some of the earliest examples discussed during the lecture illustrated how dramatically workflows have changed over time. While working on Action Man: Jungle Storm, implementation tools remained extremely limited compared with contemporary systems. There were no dedicated audio middleware environments, no simple methods for previewing sounds directly within gameplay, and no convenient ways of rapidly testing ideas. Implementation frequently involved manually replaying sections of gameplay while attempting to synchronise sounds externally. Looking back, the process appears cumbersome and time-consuming. Yet despite those limitations, hearing newly created sounds finally appearing inside the game still produced a strong sense of satisfaction.

    Later projects introduced another challenge as assumptions taken from linear media no longer translated effectively into interactive environments. Arvidsson described work on Republic: The Revolution, where large numbers of character animations required accompanying sounds. Initial approaches appeared straightforward enough. Individual animations were paired with carefully designed sounds in much the same way they might be within film production. Footsteps, movements, and interactions each received specific audio elements designed to support visual actions. Problems quickly appeared once these sounds entered gameplay. Memory limitations immediately became one issue, with thousands of individual files consuming valuable resources. Yet another issue proved equally important. Players repeatedly encountered exactly the same actions throughout long periods of gameplay. A movement animation viewed once might feel entirely convincing, though hearing precisely the same sound attached to the same movement hundreds of times gradually became distracting rather than believable.

    This problem reveals something broader about realism itself. Human beings often tolerate variation without noticing it consciously, while exact repetition becomes highly noticeable. Everyday experiences rarely unfold identically from one moment to another. Footsteps change subtly according to movement, surfaces, speed, and context. Someone walking across gravel rarely produces exactly the same sound twice. Objects interact slightly differently each time they collide, while environmental sounds fluctuate continuously. We generally ignore these small differences, though their absence can become surprisingly noticeable. Once a sound begins repeating with complete consistency, attention gradually shifts away from the world itself and towards the system generating it. Perfect consistency can therefore begin feeling less realistic than controlled variation.

    Solutions required a different form of thinking. Rather than attaching one sound permanently to one action, sounds became collections of possibilities. Footsteps could exist within larger groups of variations, different surfaces could trigger different responses, and small adjustments in pitch, timing, and volume could introduce subtle differences between repetitions. Players no longer heard identical events replaying endlessly. Instead, they experienced systems capable of producing varied outcomes.

    Arvidsson reflected on this through an observation extending beyond the immediate technical problem. She noted that changing sounds can sometimes create the impression that animations themselves are changing. Sound was therefore no longer simply accompanying visual information. It had begun influencing how visual information itself was interpreted.

    Repetition emerged again through examples involving dialogue. While working on Evil Genius, background conversations between characters introduced similar difficulties. Real dialogue becomes recognisable very quickly once repeated frequently, though replacing speech with meaningless placeholder sounds created worlds that felt strangely artificial. The eventual solution involved constructing thousands of vocal recordings using invented forms of structured nonsense speech. Colleagues recorded large collections of vocal performances resembling language without becoming meaningful dialogue. The purpose was not literal realism. Players were not expected to understand these conversations or extract semantic meaning from them. Instead, the objective involved creating evidence that activity continued occurring around the player. Worlds rarely feel alive merely through visual detail alone. People often listen for small signals suggesting that environments continue existing independently of their own actions. Background conversations, distant movement, as well as changing environmental activity all contribute to the impression that spaces continue functioning whether or not the player directly observes them.

    Memory constraints returned in a different form during discussion of LittleBigPlanet. Storage restrictions within the PSP version introduced significant constraints compared with larger console releases. Some reductions remained relatively straightforward. Numbers of variations could be lowered and certain content could be simplified, though environmental soundscapes proved more difficult. Long ambient recordings consumed considerable amounts of memory, while straightforward looping solutions introduced repetition problems of their own. Instead, Arvidsson described constructing simpler environmental foundations combined with shorter sound fragments including birds, insects, and environmental details. Individual elements could then appear according to changing probabilities and timings while introducing subtle variation. Rather than hearing static recordings replaying continuously, players experienced environments appearing more dynamic and less predictable.

    Examples such as these suggested that technical limitations did not merely reduce possibilities. Constraints frequently redirected attention towards different forms of design thinking. Rather than storing larger quantities of material, systems could generate richer experiences from fewer resources.

    Increasingly interactive systems introduced another layer of complexity. Physics systems created situations where players themselves generated outcomes that designers could not fully predict beforehand. Within LittleBigPlanet, players could construct objects using different combinations of materials and structures. Objects then collided using changing amounts of force under varying conditions. Questions that initially appeared simple quickly became more complicated. Which material should dominate when metal collides with sponge? Should paper dominate plastic? What happens when multiple materials contribute simultaneously? Questions such as these reveal how game sound often shifts away from designing isolated sounds towards establishing behaviours and rules. Designers create relationships and systems, allowing games themselves to determine outcomes dynamically.

    Broader reflections on working within the industry also appeared near the end of the lecture. Networking, persistence, and long-term relationships emerged repeatedly throughout these discussions. Freelancing across games, film, and television introduced uncertainty alongside flexibility, requiring continual adaptation as projects, collaborators, and opportunities changed over time. One comment near the conclusion captured this relationship clearly. Arvidsson described game sound design as roughly forty percent creativity and sixty percent technical implementation and problem solving.

    Initially this ratio may appear unexpected. Sound design often seems associated primarily with creativity and artistic expression. The examples discussed throughout the lecture suggested something slightly different. Creativity within games frequently emerges through solving problems. Memory restrictions, implementation systems, player unpredictability, and technical limitations all shape the final experience.

    Players rarely notice these systems directly. They simply hear worlds that feel alive. Background conversations seem to continue without them, environments appear to change naturally, as movement feels connected to the spaces around it. Much of the underlying complexity disappears beneath the experience itself.

    Perhaps that invisibility forms part of the achievement. Successful game audio may involve more than creating individual sounds. It may involve building worlds capable of continuing to surprise players long after the designer has stepped away. Rather than asking whether a sound works in isolation, a broader question may involve whether an entire system continues behaving convincingly once players begin doing things nobody predicted.

  • Listening to the Future: Dr Justyna Maculewicz on Sound Design for Intelligent Vehicles

    Dr Justyna Maculewicz

    Sound in vehicles often becomes noticeable only when something goes wrong. Most people can immediately recall an irritating warning tone, an intrusive navigation prompt, or repetitive notifications during a journey. These sounds tend to interrupt rather than accompany experience, appearing briefly to signal danger, demand attention, or communicate instructions before disappearing again. Much less attention is usually given to the wider role sound plays in shaping how journeys actually feel. Yet vehicles already communicate continuously through sound, although many of these interactions become so familiar that they disappear into the background of everyday travel. Indicators click rhythmically beside us, seatbelt reminders demand attention, parking systems announce approaching obstacles, and navigation systems guide movement through spoken instructions. A largely invisible conversation already exists between people and vehicles, though most of it remains unnoticed until something becomes irritating or disruptive.

    As vehicles become increasingly intelligent and potentially autonomous, this relationship begins changing in important ways. Traditional vehicles rely heavily on direct control. Drivers steer, brake, accelerate, and make continual decisions throughout a journey. Future vehicles may shift some of these responsibilities towards automated systems, creating a rather different experience. Attention may move away from the road itself and towards work, conversation, entertainment, or rest. Questions therefore begin emerging around whether sound should continue acting primarily as interruption or whether it might instead become a quieter form of support that helps people feel informed, comfortable, and connected to the actions of a vehicle.

    These questions formed the basis of an online guest lecture delivered by Dr Justyna Maculewicz, whose work explored user-centred approaches to sound design for future vehicles. Rather than beginning with technological possibilities alone, her work started with people and their experiences. The emphasis throughout the lecture repeatedly returned to an important principle: understanding users before designing sounds.

    Research presented during the lecture involved interviews with drivers and passengers across a range of commuting contexts. Participants discussed their daily experiences, frustrations, routines, and emotional responses during travel. The purpose was not simply to determine whether participants liked particular sounds but to understand how people experienced travel itself and where sound might play meaningful roles within those experiences.

    Findings suggested that travelling involves far more than moving physically from one place to another. A commute can become preparation for a working day, a brief period of quiet after a stressful afternoon, or one of the few moments available for concentration and reflection. Someone travelling home after a long day may seek quietness and reassurance, while another person beginning a working day may value engagement and awareness. A parent travelling with children may experience entirely different priorities from someone commuting alone. Expectations and needs therefore change continuously across situations.

    One of the more interesting aspects of the work involved moving away from rigid user categories and towards behavioural patterns. Three broad behavioural tendencies emerged from the interviews. One group preferred control and active engagement with driving experiences. Another sought reassurance and clarity, valuing confidence in the behaviour of systems around them. A third group prioritised comfort and productivity, viewing travel time as an opportunity to focus on other activities.

    Importantly, these were not treated as fixed personality types. Maculewicz emphasised that individuals could move between different behaviours depending on context, mood, fatigue, weather conditions, or travel purpose. Someone who normally enjoys driving may prefer a calmer and more supportive experience after a stressful day. Equally, a passenger travelling during unfamiliar conditions may suddenly seek additional reassurance and information. Behaviour therefore appeared dynamic rather than static.

    This distinction had important consequences for sound design. Traditional systems often assume that one solution should work equally well for everyone. Yet if user needs change over time, sound design may also need to become adaptive rather than fixed.

    For users seeking active engagement, richer sonic environments appeared more appropriate. Additional information and more expressive interactions could support a sense of control and awareness. Those seeking reassurance instead preferred clearer and calmer forms of communication that reduced uncertainty. Meanwhile users focused on work or productivity often preferred quieter interactions providing only essential information while avoiding unnecessary interruption. Rather than creating a single universal sound environment, the work explored whether future systems might adapt according to changing experiences and needs.

    A broader design framework was then introduced that organised vehicle interaction into multiple layers. These included perception, intention, current actions, required responses, strategy, and emotional context. Emotional framing operated across these categories rather than existing separately, helping shape the overall experience rather than acting as an isolated feature.

    What made this framework particularly interesting was that it treated sound as something larger than isolated alerts. Traditional warning systems often appear only during particular moments requiring immediate attention. In contrast, this approach considered how sound might support an ongoing relationship between users and vehicles. Instead of simply reacting to problems, sounds could help explain behaviour, communicate intentions, and create a sense of continuity throughout a journey.

    Among these ideas, intention sounds emerged as one of the most distinctive aspects of the lecture. Conventional warning sounds typically communicate information after an event has occurred or immediately before danger appears. Intention sounds operated rather differently. Rather than announcing what had already happened, these sounds communicated what a vehicle was about to do.

    Sounds associated with acceleration, braking, or turning were introduced slightly before physical movements occurred. Although this difference initially appears relatively small, it has interesting implications for perception. Human beings continuously anticipate actions and outcomes within everyday experience. When travelling in a vehicle driven by another person, passengers often prepare unconsciously for changes in movement based on visual information, driver behaviour, or expectations formed through experience. Autonomous systems may reduce some of these familiar cues.

    Without anticipation, even small delays between expectation and movement can create discomfort. This issue becomes particularly important when people are no longer focused directly on driving tasks. Someone reading, working, or looking away from the road may have fewer signals available for predicting changes in movement.

    Findings presented during the lecture suggested that intention sounds could help address this problem. Participants gradually became accustomed to these cues, often reporting that they stopped consciously noticing them over time. Yet despite becoming less consciously aware of the sounds, behavioural effects remained present. Participants reported greater comfort, improved trust, and reductions in motion sickness.

    This aspect of the work suggests an interesting possibility. Effective sound design may sometimes involve creating sounds that gradually disappear from conscious awareness rather than continually demanding attention. Successful design may occasionally involve fading into the background, allowing people to feel supported without constantly being reminded of the system itself.

    Trust formed another important theme running throughout the lecture. Autonomous systems raise practical questions concerning safety and reliability, though they also introduce psychological questions involving confidence and reassurance. People may intellectually understand that a system functions correctly while still feeling uncomfortable or uncertain.

    Sound therefore becomes important not only for transmitting information but also for shaping emotional responses. Perception sounds and intention sounds appeared capable of supporting trust while remaining acceptable during longer periods of use. Rather than overwhelming users with constant warnings or large quantities of information, carefully designed sonic interactions helped establish a feeling that the system remained understandable and predictable.

    Another particularly interesting aspect involved the methods used early within the design process itself. Maculewicz described vocalisation exercises in which participants and researchers used their own voices to explore sound concepts before detailed design work began. Instead of immediately creating polished digital sounds, people experimented using simple vocal expressions to communicate movement, intention, and emotional qualities.

    Although these exercises initially appeared playful, they served an important purpose. They helped clarify what sounds were intended to communicate before investing significant effort into production and implementation. Questions surrounding function and meaning could therefore be explored before technical decisions became fixed.

    Running throughout the lecture was a broader shift in thinking about the role of sound within vehicles. Traditional systems frequently focus on isolated moments of interruption and attention. Future sound interaction may instead become something quieter and more continuous, operating as an adaptive layer supporting comfort, anticipation, trust, and wellbeing throughout travel.

    Vehicles may therefore communicate with us in increasingly subtle ways. Sound within future systems may gradually move away from functioning as collections of warnings and alerts towards becoming a quieter layer of interaction that helps people understand not only what a vehicle is doing, but also how they relate to it.

  • Echo Location: Navigating Sonic Interaction Design with Professor Myounghoon Jeon

    Professor Myounghoon “Philart” Jeon, a professor at Virginia Tech, recently delivered an engaging online guest lecture on sonic information design, where he explored the intersection of auditory perception, cognitive science, and interactive sound design. His research spans auditory displays, human-computer interaction, and affective computing, with applications in assistive technologies, automotive interfaces, and interactive performance. Throughout the lecture, he shared detailed insights into the process of designing and evaluating auditory cues, explaining how specific sound design choices impact usability, accessibility, and engagement.

    Myounghoon "Philart" Jeon

    The Evolution of Sonic Information Design

    Professor Jeon introduced sonic information design as a field that integrates sonification, auditory displays, auditory user interfaces, and sonic interaction design. While sound design has historically been guided by artistic intuition, his work highlights a shift towards scientific, data-driven approaches. This transition ensures that auditory interfaces are both intuitive and efficient, optimising interaction in hands-free, visually demanding, or multi-tasking environments.

    One example of this approach is his development of “Spindex” (Speech Index), an auditory menu navigation system that enhances efficiency by using compressed speech cues instead of full words. Instead of users listening to long, spoken menu options, Spindex provides shortened speech cues, allowing them to scan options quickly. Through user testing, he found that people could navigate menus more effectively when exposed to a combination of compressed speech and indexed categories, rather than traditional text-to-speech output. The decision to use speech compression without pitch alteration ensured that the information remained intelligible while increasing the speed of interaction.

    Applications of Auditory Displays

    Professor Jeon discussed a range of applications where sound enhances usability and accessibility, particularly in assistive technology, automotive sound design, and interactive exhibitions. One of his most practical and tested projects focused on indoor navigation for visually impaired users. His team developed a wearable navigation system that incorporates ultrasonic belts providing both tactile and auditory feedback. The sound design choices involved creating gradual frequency shifts to indicate proximity to obstacles. Low-pitched tones signalled distant objects, while higher-pitched tones and increasing intensity indicated closer obstructions, ensuring users could interpret spatial information efficiently.

    His work in automotive auditory interfaces examined how sound can improve situational awareness for drivers. One project involved designing warning systems for railway level crossings, where drivers might overlook visual alerts due to distraction. His team conducted experiments using different auditory cues, testing whether short, rhythmic pulses or long, sweeping alerts were more effective at conveying urgency without overwhelming the driver. Findings showed that spatialised auditory warnings, where sounds were positioned to indicate the direction of an approaching train, helped drivers respond more accurately than traditional beeping tones.

    Professor Jeon also highlighted his work on interactive sonification in public exhibitions, including the Accessible Aquarium project, which used computer vision to track fish movements and convert them into sound and music. The sound design process for this project involved defining sonic mappings that correlated with fish speed, size, and position. Large fish were assigned deep, resonant tones, while smaller fish produced higher-pitched sounds. The system was further refined by introducing dynamic panning, so the audio reflected the fish’s position within the tank, allowing visually impaired visitors to perceive their movements in real-time.

    The project was later expanded by introducing audience interaction through motion-tracking technology. Visitors could use arm movements to mimic fish, triggering musical patterns that followed their gestures. The decision to incorporate layered harmonic structures ensured that overlapping user-generated sounds remained cohesive rather than chaotic, maintaining an aesthetically pleasing experience while preserving informational clarity.

    Designing Effective Auditory Cues

    Throughout the lecture, Professor Jeon provided detailed insights into sound design decision-making, particularly in branding, interaction design, and auditory icons. In his work with LG Electronics and Samsung, he developed sound profiles for home appliances, ensuring that product sounds were both functional and emotionally resonant. His research explored how users interpret different tonal qualities and how sound frequency influences perceived urgency and pleasantness. In one experiment, he tested whether major-key melodic notifications were perceived as more friendly and reassuring than atonal, percussive alerts.

    Another innovative area of his research involved the development of lyricons (lyrics-based earcons), a novel approach where melodic speech reinforces functional commands. Instead of using generic tones, this system integrated spoken words into short musical motifs, making auditory cues more memorable. For example, turning a device on or off could be represented by a short, ascending or descending melodic phrase, rather than a simple beep. His studies demonstrated that users recalled lyricon-based auditory cues more accurately than traditional earcons, highlighting the potential of music as a tool for reinforcing interaction memory.

    In his dance-based sonification research, Professor Jeon explored how motion-capture technology can translate body movements into real-time music generation. His team designed a system where dancers wore infra-red motion sensors, allowing spatial position and gesture dynamics to control auditory parameters. The sound mappings were carefully structured so that slow, fluid movements produced soft, sustained tones, while sharp, rapid gestures triggered percussive elements. By fine-tuning these interactions, the system ensured that each performance remained expressive yet predictable, allowing dancers to intentionally shape the evolving musical landscape.

    The Future of Sonic Interaction

    Looking forward, Professor Jeon discussed how artificial intelligence, machine learning, and real-time sound generation are shaping next-generation auditory interfaces. One of his projects in this area involves music-based social robots for children with autism, where robotic agents use music to enhance social communication. The system was designed with emotion-sensitive audio cues, allowing the robot to modulate its voice and musical output based on the child’s mood. His team experimented with different musical scales and rhythmic patterns, determining that gentle, repetitive melodic structures were the most effective at capturing attention without overwhelming the child.

    His lecture provided a comprehensive and technically rich exploration of sonic information design, demonstrating how scientific principles, auditory perception, and interactive sound technologies continue to shape human-computer interaction. By combining rigorous research with creative experimentation, his work highlights the growing impact of auditory interfaces in accessibility, engagement, and multisensory experiences across multiple fields.

     

  • Andrew Spitz: Crafting Soundscapes, Interactivity, and Innovations

    In the evolving world of design and technology, Andrew Spitz’s career serves as an inspiring example of how creativity and experimentation can lead to unique and impactful innovations. From sound design to interactive media and the art of prototyping, Andrew’s journey offers insights into building meaningful user experiences through multidisciplinary approaches. Andrew Spitz shared his experiences and knowledge during an online guest lecture, offering a glimpse into his journey and expertise.

    Andrew Spitz, Frolic Studio

    The Journey: From Linear Sound Design to Interactive Media

    Andrew Spitz started his career in the world of sound design, where his primary focus was creating immersive audio experiences for films. This phase of his work was marked by linear storytelling—designing soundscapes that enhanced the narrative of visual media. For example, Andrew recorded the sounds of African wildlife to bring animated characters to life, showcasing the meticulous effort involved in capturing authentic audio.

    However, this linear approach left him yearning for more dynamic ways to engage audiences. His desire to explore interactivity led him to Edinburgh, where he delved into interactive sound design during his Master’s programme. Here, tools like Max/MSP opened new doors, allowing Andrew to experiment with dynamic soundscapes that responded to user interactions.

    This transition marked a pivotal shift in his career—from designing sounds that followed a fixed storyline to creating experiences where users could shape the narrative. It was a move from being a storyteller to an enabler, allowing audiences to co-create their journey.

    Interactive Media: Bridging Empathy and Technology

    One of Andrew’s key insights into interactive media is the importance of empathy. As an interaction designer, he emphasises the ability to step into the user’s shoes. Whether it’s designing physical installations or digital interfaces, understanding the emotional and functional needs of users drives successful designs.

    In his work with prototypes and concepts, Andrew explores how technology can evoke emotions and foster connections. For instance, a project for BMW involved recreating the exhilarating experience of walking into a packed rugby stadium, complete with crowd noise and synchronised visuals. This installation not only showcased technological prowess but also highlighted how sensory design can forge powerful emotional connections.

    Andrew also stresses that great interaction design isn’t just about logic and utility; it’s about creating delight and emotional resonance. Products that succeed are those that strike a chord with users, making them feel connected and understood.

    The Art and Impact of Prototyping

    Andrew believes that “doing is the new thinking.” Prototyping is at the heart of his creative process, enabling him to turn abstract ideas into tangible experiences. He advocates for quick, iterative prototyping as a means to test concepts, gather feedback, and refine designs efficiently.

    One of his standout projects, Paper Note, involved turning sound into physical sculptures. What began as playful experimentation with materials like cornstarch and sand evolved into a compelling visualisation of sound frequencies. This process underscores how unstructured exploration can lead to innovative applications.

    Andrew also highlights the importance of embracing imperfection during prototyping. By failing fast and cheap, designers can refine their intuition and adapt to users’ real needs. Whether building a functional prototype like *Ice Cube*, a tangible music player, or creating tools for interactive sound, the goal remains to make ideas accessible, testable, and impactful.

    Lessons from Andrew Spitz’s Journey

    Andrew Spitz’s work offers several takeaways for anyone interested in sound design, interaction design, or creative innovation:

    1. Experiment Freely: Many of Andrew’s breakthroughs came from playful experimentation with new tools and ideas. Don’t be afraid to explore without a clear goal.
    2. Embrace Empathy: Understanding the user’s perspective is key to designing experiences that resonate emotionally and functionally.
    3. Prototype Iteratively: Start small, test often, and refine based on feedback. Prototyping is as much about learning as it is about building.
    4. Merge Creativity and Technology: Use technology as a tool to tell stories, evoke emotions, and create connections, rather than as an end in itself.

    Andrew Spitz’s career illustrates the power of curiosity and creativity in pushing the boundaries of what’s possible in design and technology. His work continues to inspire by showing how sound, interaction, and prototyping can come together to craft experiences that truly engage and delight.