Category: Interaction Design

  • Creating Sounds for Worlds That Refuse to Sit Still: Malin Arvidsson on Game Audio and Interactive Design

    Sound in games often feels invisible when it is working well. Players notice visual worlds immediately. Landscapes stretch into the distance, characters move through environments, and stories unfold through action and dialogue. Sound tends to arrive more quietly. Footsteps simply seem to belong beneath a character, background ambiences appear to exist naturally around us, and a creature’s voice feels inseparable from its personality. Everything seems to fit together so naturally that the work behind these experiences often disappears from view.

    Yet creating sound for games involves a challenge that differs fundamentally from many other forms of media. Film and television unfold through fixed sequences of events. A sound designer working on a film knows exactly when a door opens, when dialogue occurs, when music begins, and when tension rises. Audiences experience those moments in the same order every time. Games behave rather differently. Players stop unexpectedly, move in different directions, repeat actions endlessly, ignore objectives, or spend long periods interacting with things designers never anticipated would receive much attention. Some players rush directly through environments while others investigate every possible corner of a world. A sound designer may know what can happen inside a game, though cannot always know what will happen, when it will happen, or how often particular experiences will occur. Sound therefore cannot simply be attached permanently to images and left alone. It must continue adapting long after the designer has stepped away.

    During an online guest lecture, Malin Arvidsson explored this challenge through reflections on her own experiences working across game audio. Throughout projects involving children’s games, procedural systems, and large-scale interactive worlds, a recurring idea gradually emerged. Game audio frequently involves building systems rather than constructing isolated sounds. Designers create frameworks, relationships, and behaviours that continue operating within worlds that remain unpredictable.

    Arvidsson described discovering games somewhat unexpectedly. Having decided at an early age that she wanted to work with sound, she initially pursued sound engineering and recording work before later encountering opportunities in game production. Games had not necessarily appeared to be an obvious destination at the time. Film and television perhaps felt more visible as career directions, while game audio remained relatively unfamiliar. Yet after joining Audio Interactive and working on early projects, games gradually became something much larger than a temporary opportunity. Part of this attraction appeared to emerge from constant change. Technologies evolve rapidly, development processes shift, while projects rarely require exactly the same approaches twice. Many creative fields involve continual learning, though games introduce an additional layer of complexity through their combination of artistic decisions and technical systems. Sound designers are often required to think simultaneously about recording, editing, implementation, behaviour, memory, interaction, and player experience.

    Some of the earliest examples discussed during the lecture illustrated how dramatically workflows have changed over time. While working on Action Man: Jungle Storm, implementation tools remained extremely limited compared with contemporary systems. There were no dedicated audio middleware environments, no simple methods for previewing sounds directly within gameplay, and no convenient ways of rapidly testing ideas. Implementation frequently involved manually replaying sections of gameplay while attempting to synchronise sounds externally. Looking back, the process appears cumbersome and time-consuming. Yet despite those limitations, hearing newly created sounds finally appearing inside the game still produced a strong sense of satisfaction.

    Later projects introduced another challenge as assumptions taken from linear media no longer translated effectively into interactive environments. Arvidsson described work on Republic: The Revolution, where large numbers of character animations required accompanying sounds. Initial approaches appeared straightforward enough. Individual animations were paired with carefully designed sounds in much the same way they might be within film production. Footsteps, movements, and interactions each received specific audio elements designed to support visual actions. Problems quickly appeared once these sounds entered gameplay. Memory limitations immediately became one issue, with thousands of individual files consuming valuable resources. Yet another issue proved equally important. Players repeatedly encountered exactly the same actions throughout long periods of gameplay. A movement animation viewed once might feel entirely convincing, though hearing precisely the same sound attached to the same movement hundreds of times gradually became distracting rather than believable.

    This problem reveals something broader about realism itself. Human beings often tolerate variation without noticing it consciously, while exact repetition becomes highly noticeable. Everyday experiences rarely unfold identically from one moment to another. Footsteps change subtly according to movement, surfaces, speed, and context. Someone walking across gravel rarely produces exactly the same sound twice. Objects interact slightly differently each time they collide, while environmental sounds fluctuate continuously. We generally ignore these small differences, though their absence can become surprisingly noticeable. Once a sound begins repeating with complete consistency, attention gradually shifts away from the world itself and towards the system generating it. Perfect consistency can therefore begin feeling less realistic than controlled variation.

    Solutions required a different form of thinking. Rather than attaching one sound permanently to one action, sounds became collections of possibilities. Footsteps could exist within larger groups of variations, different surfaces could trigger different responses, and small adjustments in pitch, timing, and volume could introduce subtle differences between repetitions. Players no longer heard identical events replaying endlessly. Instead, they experienced systems capable of producing varied outcomes.

    Arvidsson reflected on this through an observation extending beyond the immediate technical problem. She noted that changing sounds can sometimes create the impression that animations themselves are changing. Sound was therefore no longer simply accompanying visual information. It had begun influencing how visual information itself was interpreted.

    Repetition emerged again through examples involving dialogue. While working on Evil Genius, background conversations between characters introduced similar difficulties. Real dialogue becomes recognisable very quickly once repeated frequently, though replacing speech with meaningless placeholder sounds created worlds that felt strangely artificial. The eventual solution involved constructing thousands of vocal recordings using invented forms of structured nonsense speech. Colleagues recorded large collections of vocal performances resembling language without becoming meaningful dialogue. The purpose was not literal realism. Players were not expected to understand these conversations or extract semantic meaning from them. Instead, the objective involved creating evidence that activity continued occurring around the player. Worlds rarely feel alive merely through visual detail alone. People often listen for small signals suggesting that environments continue existing independently of their own actions. Background conversations, distant movement, as well as changing environmental activity all contribute to the impression that spaces continue functioning whether or not the player directly observes them.

    Memory constraints returned in a different form during discussion of LittleBigPlanet. Storage restrictions within the PSP version introduced significant constraints compared with larger console releases. Some reductions remained relatively straightforward. Numbers of variations could be lowered and certain content could be simplified, though environmental soundscapes proved more difficult. Long ambient recordings consumed considerable amounts of memory, while straightforward looping solutions introduced repetition problems of their own. Instead, Arvidsson described constructing simpler environmental foundations combined with shorter sound fragments including birds, insects, and environmental details. Individual elements could then appear according to changing probabilities and timings while introducing subtle variation. Rather than hearing static recordings replaying continuously, players experienced environments appearing more dynamic and less predictable.

    Examples such as these suggested that technical limitations did not merely reduce possibilities. Constraints frequently redirected attention towards different forms of design thinking. Rather than storing larger quantities of material, systems could generate richer experiences from fewer resources.

    Increasingly interactive systems introduced another layer of complexity. Physics systems created situations where players themselves generated outcomes that designers could not fully predict beforehand. Within LittleBigPlanet, players could construct objects using different combinations of materials and structures. Objects then collided using changing amounts of force under varying conditions. Questions that initially appeared simple quickly became more complicated. Which material should dominate when metal collides with sponge? Should paper dominate plastic? What happens when multiple materials contribute simultaneously? Questions such as these reveal how game sound often shifts away from designing isolated sounds towards establishing behaviours and rules. Designers create relationships and systems, allowing games themselves to determine outcomes dynamically.

    Broader reflections on working within the industry also appeared near the end of the lecture. Networking, persistence, and long-term relationships emerged repeatedly throughout these discussions. Freelancing across games, film, and television introduced uncertainty alongside flexibility, requiring continual adaptation as projects, collaborators, and opportunities changed over time. One comment near the conclusion captured this relationship clearly. Arvidsson described game sound design as roughly forty percent creativity and sixty percent technical implementation and problem solving.

    Initially this ratio may appear unexpected. Sound design often seems associated primarily with creativity and artistic expression. The examples discussed throughout the lecture suggested something slightly different. Creativity within games frequently emerges through solving problems. Memory restrictions, implementation systems, player unpredictability, and technical limitations all shape the final experience.

    Players rarely notice these systems directly. They simply hear worlds that feel alive. Background conversations seem to continue without them, environments appear to change naturally, as movement feels connected to the spaces around it. Much of the underlying complexity disappears beneath the experience itself.

    Perhaps that invisibility forms part of the achievement. Successful game audio may involve more than creating individual sounds. It may involve building worlds capable of continuing to surprise players long after the designer has stepped away. Rather than asking whether a sound works in isolation, a broader question may involve whether an entire system continues behaving convincingly once players begin doing things nobody predicted.

  • Listening to the Future: Dr Justyna Maculewicz on Sound Design for Intelligent Vehicles

    Dr Justyna Maculewicz

    Sound in vehicles often becomes noticeable only when something goes wrong. Most people can immediately recall an irritating warning tone, an intrusive navigation prompt, or repetitive notifications during a journey. These sounds tend to interrupt rather than accompany experience, appearing briefly to signal danger, demand attention, or communicate instructions before disappearing again. Much less attention is usually given to the wider role sound plays in shaping how journeys actually feel. Yet vehicles already communicate continuously through sound, although many of these interactions become so familiar that they disappear into the background of everyday travel. Indicators click rhythmically beside us, seatbelt reminders demand attention, parking systems announce approaching obstacles, and navigation systems guide movement through spoken instructions. A largely invisible conversation already exists between people and vehicles, though most of it remains unnoticed until something becomes irritating or disruptive.

    As vehicles become increasingly intelligent and potentially autonomous, this relationship begins changing in important ways. Traditional vehicles rely heavily on direct control. Drivers steer, brake, accelerate, and make continual decisions throughout a journey. Future vehicles may shift some of these responsibilities towards automated systems, creating a rather different experience. Attention may move away from the road itself and towards work, conversation, entertainment, or rest. Questions therefore begin emerging around whether sound should continue acting primarily as interruption or whether it might instead become a quieter form of support that helps people feel informed, comfortable, and connected to the actions of a vehicle.

    These questions formed the basis of an online guest lecture delivered by Dr Justyna Maculewicz, whose work explored user-centred approaches to sound design for future vehicles. Rather than beginning with technological possibilities alone, her work started with people and their experiences. The emphasis throughout the lecture repeatedly returned to an important principle: understanding users before designing sounds.

    Research presented during the lecture involved interviews with drivers and passengers across a range of commuting contexts. Participants discussed their daily experiences, frustrations, routines, and emotional responses during travel. The purpose was not simply to determine whether participants liked particular sounds but to understand how people experienced travel itself and where sound might play meaningful roles within those experiences.

    Findings suggested that travelling involves far more than moving physically from one place to another. A commute can become preparation for a working day, a brief period of quiet after a stressful afternoon, or one of the few moments available for concentration and reflection. Someone travelling home after a long day may seek quietness and reassurance, while another person beginning a working day may value engagement and awareness. A parent travelling with children may experience entirely different priorities from someone commuting alone. Expectations and needs therefore change continuously across situations.

    One of the more interesting aspects of the work involved moving away from rigid user categories and towards behavioural patterns. Three broad behavioural tendencies emerged from the interviews. One group preferred control and active engagement with driving experiences. Another sought reassurance and clarity, valuing confidence in the behaviour of systems around them. A third group prioritised comfort and productivity, viewing travel time as an opportunity to focus on other activities.

    Importantly, these were not treated as fixed personality types. Maculewicz emphasised that individuals could move between different behaviours depending on context, mood, fatigue, weather conditions, or travel purpose. Someone who normally enjoys driving may prefer a calmer and more supportive experience after a stressful day. Equally, a passenger travelling during unfamiliar conditions may suddenly seek additional reassurance and information. Behaviour therefore appeared dynamic rather than static.

    This distinction had important consequences for sound design. Traditional systems often assume that one solution should work equally well for everyone. Yet if user needs change over time, sound design may also need to become adaptive rather than fixed.

    For users seeking active engagement, richer sonic environments appeared more appropriate. Additional information and more expressive interactions could support a sense of control and awareness. Those seeking reassurance instead preferred clearer and calmer forms of communication that reduced uncertainty. Meanwhile users focused on work or productivity often preferred quieter interactions providing only essential information while avoiding unnecessary interruption. Rather than creating a single universal sound environment, the work explored whether future systems might adapt according to changing experiences and needs.

    A broader design framework was then introduced that organised vehicle interaction into multiple layers. These included perception, intention, current actions, required responses, strategy, and emotional context. Emotional framing operated across these categories rather than existing separately, helping shape the overall experience rather than acting as an isolated feature.

    What made this framework particularly interesting was that it treated sound as something larger than isolated alerts. Traditional warning systems often appear only during particular moments requiring immediate attention. In contrast, this approach considered how sound might support an ongoing relationship between users and vehicles. Instead of simply reacting to problems, sounds could help explain behaviour, communicate intentions, and create a sense of continuity throughout a journey.

    Among these ideas, intention sounds emerged as one of the most distinctive aspects of the lecture. Conventional warning sounds typically communicate information after an event has occurred or immediately before danger appears. Intention sounds operated rather differently. Rather than announcing what had already happened, these sounds communicated what a vehicle was about to do.

    Sounds associated with acceleration, braking, or turning were introduced slightly before physical movements occurred. Although this difference initially appears relatively small, it has interesting implications for perception. Human beings continuously anticipate actions and outcomes within everyday experience. When travelling in a vehicle driven by another person, passengers often prepare unconsciously for changes in movement based on visual information, driver behaviour, or expectations formed through experience. Autonomous systems may reduce some of these familiar cues.

    Without anticipation, even small delays between expectation and movement can create discomfort. This issue becomes particularly important when people are no longer focused directly on driving tasks. Someone reading, working, or looking away from the road may have fewer signals available for predicting changes in movement.

    Findings presented during the lecture suggested that intention sounds could help address this problem. Participants gradually became accustomed to these cues, often reporting that they stopped consciously noticing them over time. Yet despite becoming less consciously aware of the sounds, behavioural effects remained present. Participants reported greater comfort, improved trust, and reductions in motion sickness.

    This aspect of the work suggests an interesting possibility. Effective sound design may sometimes involve creating sounds that gradually disappear from conscious awareness rather than continually demanding attention. Successful design may occasionally involve fading into the background, allowing people to feel supported without constantly being reminded of the system itself.

    Trust formed another important theme running throughout the lecture. Autonomous systems raise practical questions concerning safety and reliability, though they also introduce psychological questions involving confidence and reassurance. People may intellectually understand that a system functions correctly while still feeling uncomfortable or uncertain.

    Sound therefore becomes important not only for transmitting information but also for shaping emotional responses. Perception sounds and intention sounds appeared capable of supporting trust while remaining acceptable during longer periods of use. Rather than overwhelming users with constant warnings or large quantities of information, carefully designed sonic interactions helped establish a feeling that the system remained understandable and predictable.

    Another particularly interesting aspect involved the methods used early within the design process itself. Maculewicz described vocalisation exercises in which participants and researchers used their own voices to explore sound concepts before detailed design work began. Instead of immediately creating polished digital sounds, people experimented using simple vocal expressions to communicate movement, intention, and emotional qualities.

    Although these exercises initially appeared playful, they served an important purpose. They helped clarify what sounds were intended to communicate before investing significant effort into production and implementation. Questions surrounding function and meaning could therefore be explored before technical decisions became fixed.

    Running throughout the lecture was a broader shift in thinking about the role of sound within vehicles. Traditional systems frequently focus on isolated moments of interruption and attention. Future sound interaction may instead become something quieter and more continuous, operating as an adaptive layer supporting comfort, anticipation, trust, and wellbeing throughout travel.

    Vehicles may therefore communicate with us in increasingly subtle ways. Sound within future systems may gradually move away from functioning as collections of warnings and alerts towards becoming a quieter layer of interaction that helps people understand not only what a vehicle is doing, but also how they relate to it.

  • Echo Location: Navigating Sonic Interaction Design with Professor Myounghoon Jeon

    Professor Myounghoon “Philart” Jeon, a professor at Virginia Tech, recently delivered an engaging online guest lecture on sonic information design, where he explored the intersection of auditory perception, cognitive science, and interactive sound design. His research spans auditory displays, human-computer interaction, and affective computing, with applications in assistive technologies, automotive interfaces, and interactive performance. Throughout the lecture, he shared detailed insights into the process of designing and evaluating auditory cues, explaining how specific sound design choices impact usability, accessibility, and engagement.

    Myounghoon "Philart" Jeon

    The Evolution of Sonic Information Design

    Professor Jeon introduced sonic information design as a field that integrates sonification, auditory displays, auditory user interfaces, and sonic interaction design. While sound design has historically been guided by artistic intuition, his work highlights a shift towards scientific, data-driven approaches. This transition ensures that auditory interfaces are both intuitive and efficient, optimising interaction in hands-free, visually demanding, or multi-tasking environments.

    One example of this approach is his development of “Spindex” (Speech Index), an auditory menu navigation system that enhances efficiency by using compressed speech cues instead of full words. Instead of users listening to long, spoken menu options, Spindex provides shortened speech cues, allowing them to scan options quickly. Through user testing, he found that people could navigate menus more effectively when exposed to a combination of compressed speech and indexed categories, rather than traditional text-to-speech output. The decision to use speech compression without pitch alteration ensured that the information remained intelligible while increasing the speed of interaction.

    Applications of Auditory Displays

    Professor Jeon discussed a range of applications where sound enhances usability and accessibility, particularly in assistive technology, automotive sound design, and interactive exhibitions. One of his most practical and tested projects focused on indoor navigation for visually impaired users. His team developed a wearable navigation system that incorporates ultrasonic belts providing both tactile and auditory feedback. The sound design choices involved creating gradual frequency shifts to indicate proximity to obstacles. Low-pitched tones signalled distant objects, while higher-pitched tones and increasing intensity indicated closer obstructions, ensuring users could interpret spatial information efficiently.

    His work in automotive auditory interfaces examined how sound can improve situational awareness for drivers. One project involved designing warning systems for railway level crossings, where drivers might overlook visual alerts due to distraction. His team conducted experiments using different auditory cues, testing whether short, rhythmic pulses or long, sweeping alerts were more effective at conveying urgency without overwhelming the driver. Findings showed that spatialised auditory warnings, where sounds were positioned to indicate the direction of an approaching train, helped drivers respond more accurately than traditional beeping tones.

    Professor Jeon also highlighted his work on interactive sonification in public exhibitions, including the Accessible Aquarium project, which used computer vision to track fish movements and convert them into sound and music. The sound design process for this project involved defining sonic mappings that correlated with fish speed, size, and position. Large fish were assigned deep, resonant tones, while smaller fish produced higher-pitched sounds. The system was further refined by introducing dynamic panning, so the audio reflected the fish’s position within the tank, allowing visually impaired visitors to perceive their movements in real-time.

    The project was later expanded by introducing audience interaction through motion-tracking technology. Visitors could use arm movements to mimic fish, triggering musical patterns that followed their gestures. The decision to incorporate layered harmonic structures ensured that overlapping user-generated sounds remained cohesive rather than chaotic, maintaining an aesthetically pleasing experience while preserving informational clarity.

    Designing Effective Auditory Cues

    Throughout the lecture, Professor Jeon provided detailed insights into sound design decision-making, particularly in branding, interaction design, and auditory icons. In his work with LG Electronics and Samsung, he developed sound profiles for home appliances, ensuring that product sounds were both functional and emotionally resonant. His research explored how users interpret different tonal qualities and how sound frequency influences perceived urgency and pleasantness. In one experiment, he tested whether major-key melodic notifications were perceived as more friendly and reassuring than atonal, percussive alerts.

    Another innovative area of his research involved the development of lyricons (lyrics-based earcons), a novel approach where melodic speech reinforces functional commands. Instead of using generic tones, this system integrated spoken words into short musical motifs, making auditory cues more memorable. For example, turning a device on or off could be represented by a short, ascending or descending melodic phrase, rather than a simple beep. His studies demonstrated that users recalled lyricon-based auditory cues more accurately than traditional earcons, highlighting the potential of music as a tool for reinforcing interaction memory.

    In his dance-based sonification research, Professor Jeon explored how motion-capture technology can translate body movements into real-time music generation. His team designed a system where dancers wore infra-red motion sensors, allowing spatial position and gesture dynamics to control auditory parameters. The sound mappings were carefully structured so that slow, fluid movements produced soft, sustained tones, while sharp, rapid gestures triggered percussive elements. By fine-tuning these interactions, the system ensured that each performance remained expressive yet predictable, allowing dancers to intentionally shape the evolving musical landscape.

    The Future of Sonic Interaction

    Looking forward, Professor Jeon discussed how artificial intelligence, machine learning, and real-time sound generation are shaping next-generation auditory interfaces. One of his projects in this area involves music-based social robots for children with autism, where robotic agents use music to enhance social communication. The system was designed with emotion-sensitive audio cues, allowing the robot to modulate its voice and musical output based on the child’s mood. His team experimented with different musical scales and rhythmic patterns, determining that gentle, repetitive melodic structures were the most effective at capturing attention without overwhelming the child.

    His lecture provided a comprehensive and technically rich exploration of sonic information design, demonstrating how scientific principles, auditory perception, and interactive sound technologies continue to shape human-computer interaction. By combining rigorous research with creative experimentation, his work highlights the growing impact of auditory interfaces in accessibility, engagement, and multisensory experiences across multiple fields.

     

  • Andrew Spitz: Crafting Soundscapes, Interactivity, and Innovations

    In the evolving world of design and technology, Andrew Spitz’s career serves as an inspiring example of how creativity and experimentation can lead to unique and impactful innovations. From sound design to interactive media and the art of prototyping, Andrew’s journey offers insights into building meaningful user experiences through multidisciplinary approaches. Andrew Spitz shared his experiences and knowledge during an online guest lecture, offering a glimpse into his journey and expertise.

    Andrew Spitz, Frolic Studio

    The Journey: From Linear Sound Design to Interactive Media

    Andrew Spitz started his career in the world of sound design, where his primary focus was creating immersive audio experiences for films. This phase of his work was marked by linear storytelling—designing soundscapes that enhanced the narrative of visual media. For example, Andrew recorded the sounds of African wildlife to bring animated characters to life, showcasing the meticulous effort involved in capturing authentic audio.

    However, this linear approach left him yearning for more dynamic ways to engage audiences. His desire to explore interactivity led him to Edinburgh, where he delved into interactive sound design during his Master’s programme. Here, tools like Max/MSP opened new doors, allowing Andrew to experiment with dynamic soundscapes that responded to user interactions.

    This transition marked a pivotal shift in his career—from designing sounds that followed a fixed storyline to creating experiences where users could shape the narrative. It was a move from being a storyteller to an enabler, allowing audiences to co-create their journey.

    Interactive Media: Bridging Empathy and Technology

    One of Andrew’s key insights into interactive media is the importance of empathy. As an interaction designer, he emphasises the ability to step into the user’s shoes. Whether it’s designing physical installations or digital interfaces, understanding the emotional and functional needs of users drives successful designs.

    In his work with prototypes and concepts, Andrew explores how technology can evoke emotions and foster connections. For instance, a project for BMW involved recreating the exhilarating experience of walking into a packed rugby stadium, complete with crowd noise and synchronised visuals. This installation not only showcased technological prowess but also highlighted how sensory design can forge powerful emotional connections.

    Andrew also stresses that great interaction design isn’t just about logic and utility; it’s about creating delight and emotional resonance. Products that succeed are those that strike a chord with users, making them feel connected and understood.

    The Art and Impact of Prototyping

    Andrew believes that “doing is the new thinking.” Prototyping is at the heart of his creative process, enabling him to turn abstract ideas into tangible experiences. He advocates for quick, iterative prototyping as a means to test concepts, gather feedback, and refine designs efficiently.

    One of his standout projects, Paper Note, involved turning sound into physical sculptures. What began as playful experimentation with materials like cornstarch and sand evolved into a compelling visualisation of sound frequencies. This process underscores how unstructured exploration can lead to innovative applications.

    Andrew also highlights the importance of embracing imperfection during prototyping. By failing fast and cheap, designers can refine their intuition and adapt to users’ real needs. Whether building a functional prototype like *Ice Cube*, a tangible music player, or creating tools for interactive sound, the goal remains to make ideas accessible, testable, and impactful.

    Lessons from Andrew Spitz’s Journey

    Andrew Spitz’s work offers several takeaways for anyone interested in sound design, interaction design, or creative innovation:

    1. Experiment Freely: Many of Andrew’s breakthroughs came from playful experimentation with new tools and ideas. Don’t be afraid to explore without a clear goal.
    2. Embrace Empathy: Understanding the user’s perspective is key to designing experiences that resonate emotionally and functionally.
    3. Prototype Iteratively: Start small, test often, and refine based on feedback. Prototyping is as much about learning as it is about building.
    4. Merge Creativity and Technology: Use technology as a tool to tell stories, evoke emotions, and create connections, rather than as an end in itself.

    Andrew Spitz’s career illustrates the power of curiosity and creativity in pushing the boundaries of what’s possible in design and technology. His work continues to inspire by showing how sound, interaction, and prototyping can come together to craft experiences that truly engage and delight.