Not All Sound Is Created Equal
Sound professionals are constantly faced with the daunting task of creating sound that can be easily consumed on multiple devices by the mass market. Consumers listen to sound on all types of devices: radios, smartphones, speakers, soundbars, headphones, studio monitors, etc. The difference in visual content between a VR video game and a television sitcom is astronomical. This variety of output demands different approaches to sound making and consuming as well.
It's important that we start paying attention to the way our world sounds. By breaking down how sound is made for VR, Television, Film, and Video Games, we can begin to see just how far we have strayed from capturing the world the way we hear it.
Hollywood has taken us to Outer Space, Jurassic Park, Under the Sea, Oz, and beyond. With such epic journeys, the aural has to keep up with the visual.
For years, a punch in film has never been just a punch. It's a piece of wood cracking, mixed with a catcher's mitt slapping a frozen piece of meet inside an echo chamber in the middle of a gymnasium, run through a compressor so tight it'll make your eardrums pop. The crazy thing is, our brains have come to expect a punch to sound this way.
Take the iconic film Rocky, for example. The punches in Rocky are huge and epic, completely different from the sounds a real life punch makes. The filmmakers wanted it this way. The punches in Rocky are unlike anything you've ever heard because Rocky Balboa is a man unlike you've ever seen.
There are so many real world sounds regularly sweetened for the silver screen: cars, kisses, animals, thunderstorms, cell phones. Even menial tasks like jacket zippers, car door openings, walking and running -- all sweetened and heightened. The industry has even created a paid position for a person who reenacts these exaggerated human tasks: a foley artist. A foley artist is responsible for any sounds that come in contact with a living human on screen.
The need for foley is a pretty understandable one. On set, the main focus is capturing dialogue, because who has time to ADR an entire film? So often, the small organic sounds are missed when shooting on set. When these sounds are added later on in the mixing process, however, they're so loud and compressed that when viewers asks themselves, does this sound like it would if I was there? Almost 100% of the time the answer is No.
We've so over designed and amplified sound in film that consumer audiences come to expect this type of sonic experience, this HOLLYWOOD sound. Even films without monsters or adventure or explosions are forced to follow this sonic psychology. How many living room dramas sound like a Metallica track?
Rain is a great example when talking about film sound. In the early days of motion pictures, microphone technology wasn't advanced enough to capture the full frequency spectrum of rainfall. When it comes to rain, every frequency is key. So in the early days of rain sound creation, sound artists would use rice or grains to recreate rain. When rice falls, the high frequency content is much brighter than normal rain, but since sound microphone technology was lacking in the upper frequency spectrum, this overly bright rice actually evened things out when played back.
When Supervising Sound Editor Craig Henighan was working on Darren Aronofsky's Blockbuster Noah, he was set with the difficult task of appropriately placing different rain in a film that is close to 85% rain scenes.
The trouble with rain is that it becomes like white noise very quickly, so you look for specific rain sounds: rain on wood, rain on grass, rain on trees, rain on cement, rain on mud -- the list can go on and on and on. It takes a lot of listening, a lot of EQ'ing, sort of pitching sounds as you lay them up. You could record the best-sounding rain in the world, but in context with music, dialog, the sounds of the ark and the animals, it has to read as rain. You can tell pretty quickly if it turns into hissy white noise.
The main thing was how to make the ark feel it like it was raining constantly, but still hear the rain, have it be interesting. You don't want it to be annoying, but you don't want people to forget it's there, either. Darren was very, very focused on wanting to kind of drive people crazy with hearing the rain, no matter where you were in the ark. That's the psychology of it. When the rain stops, it's a relief for the characters, but also for the people in the theater
Laugh track!? Thankfully it's gradually disappearing, but it's important to understand why television and television only incorporated this cheap gag.
Think of the perspective of a sitcom show. The key is to make you, the consumer, feel like you're there in the live studio audience from the comfort of your living room. Sitcom television essentially brought the Broadway proscenium stage to the small screen.
So visually, you feel like you're there. Check. What are you missing? The sounds of the rest of the audience around you. For what is looking like I'm there from my Lazy Boy if it doesn't also sound like I am there?
The laugh track accomplishes this task of making you feel like you're there in the studio, like your watching a play through a screen. Sitcoms use the fourth wall like a theatrical proscenium. The only difference is the ability to cut between 3 to 4 camera angles and zoom. The permanently installed set often calls for a permanently installed sound system. Microphones are often permanently installed in a television set to properly broadcast the sound on stage and a boom mic is used for picking up dialogue. The result is a sound experience quite similar to a Broadway play, just coming through your laptop speakers from your living room in Illinois.
Unlike film and television, video game sound is sourced and pulled from catologues, then placed upon visuals to imitate what that image would sound like had it been in a realistic environment. Whether it's human or not, sounds from all over the world are placed upon visuals in video games to imitate a specific sound.
This leaves a lot to the imagination and gives a designer complete control over the game's sonic world. This also puts consumer ears at the will of the creator.
Video games, like television and film, are consumed through a two dimensional screen, though they're often (and increasingly) heard on headphones. Gamers love headphones. This plays a big part in how we hear during gameplay: our sound perspective, like our visual perspective, is forced just being the character on screen, meaning we are not the character but the puppeteer. We're able to hear a little further and louder because of this, but our ability to localize sounds on screen is limited because we are not in the game. We are omnisciently watching over the game.
What we miss as puppeteers in videos games we regain in Virtual Reality. Now we are not the puppeteer but the actual character. We hear what they do, see what they do. Now the sound designer is creating sound from a POV perspective.
There are many software companies behind this spatialization experience. Current VR experience is very much a consumption experience. Few people have access to the tools required to make VR content and not a lot of people have devices capable of playing it back. This leaves a lot of power with software and studios interested in designing highly controlled VR experiences. The result is an experience that makes you feel like you're there. No matter the world.
What happens, though, when we consumers start making our own VR content the way we take our own videos and photos on our smartphones now? Will VR sound the same, or will it need to pivot?
Though the quality and aesthetic of sound across VR, television, video games and film differ, we must always remember that at the end of the day the consumers are consuming it.
Whether headphones, speakers, soundbars, or smartphones, I'd like the sound in our virtual worlds to sound a little more like the sound in our actual world. We hear the world in 3D, let's capture, design, and broadcast it that way on VR, television, video games, and film. Let's pay attention to natural sounds as we continue to push these mediums into different corners of sonic aesthetic.
Virtual Reality has a very exciting opportunity for this: it has the ability to define, from its foundation, how consumers conceptualize sound. We need to set this standard sooner rather than later. Without real sound, virtual reality isn't reality. It's just virtual.
From One Ear To Another,