Xbox Series X: Project Acoustics 3D Audio Using Wave Physics with Custom Hardware

[bcatwilly]

Everyone always talks about the great graphics that we get with games and of course with these next generation consoles, but I really like the focus by Sony on their Tempest 3D audio solution that they announced and by Xbox with their continued push in 3D audio based on their years of Microsoft Research with Project Acoustics. They have a dedicated custom audio hardware block to process Project Acoustics wave physics based audio. Their GDC talk last year is actually really well done IMO if you are into this type of technical thing, showing how easily a developer can do the computationally intensive simulations of the wave physics in the scene in the cloud during development and then use that result to tailor within Unity or Unreal Engine while building their game. What seems to be a real differentiator here is that a full and accurate simulation of the audio in the scene relative to the geometry and materials in the scene can be generated and then tweaked as desired by the developer because fully "real" isn't always what they want to portray in their game, rather than just placement of sound sources with no regard to the wave physics at play in the scene.

These awesome advances in audio on both consoles are going to really help with immersion for sure.

From the detailed Xbox Series X glossary at https://news.xbox.com/en-us/2020/03/16/xbox-series-x-glossary/, which is really a nice technical summary of the new features in the console:

Project Acoustics
– Incubated over a decade by Microsoft Research,
Project Acoustics
accurately models sound propagation physics in mixed reality and games, employed by many AAA experiences today. It is unique in simulating wave effects like diffraction in complex scene geometries without straining CPU, enabling a much more immersive and lifelike auditory experience. Plug-in support for both the Unity and Unreal game engines empower the sound designer with expressive controls to mold reality. Developers will be able to easily leverage Project Acoustics with Xbox Series X through the addition of a new custom audio hardware block.

Click to shrink...

Spatial Audio
– Spatial Audio delivers deeply immersive audio which enables the player to more accurately pinpoint objects in a 3D play space. With full support for Dolby Atmos, DTS:X and Windows Sonic, Xbox Series X has custom audio hardware to offload audio processing from the CPU, dramatically improving the accessibility, quality and performance of these immersive experiences.

Click to shrink...

Great GDC video talking about Project Acoustics at GDC last year:


More detailed writeup on Project Acoustics that includes links to the Unity and Unreal Engine plugins and samples, and given that many of us are in more due to this coronavirus I may play around with the Unity things myself in Visual Studio if I get a chance :)
https://docs.microsoft.com/en-us/azure/cognitive-services/acoustics/what-is-acoustics

 

[Klobrille]

With Hellblade II being a thing and Ninja Theory now being an Xbox first-party studio, I'd bet quite some good amount of hazelnuts that this game will offer an industry-wide reference audio experience coming next-generation. Xbox Series X will only support that even further.

 

[bcatwilly]

With Hellblade II being a thing and Ninja Theory now being an Xbox first-party studio, I'd bet quite some good amount of hazelnuts that this game will offer an industry-wide reference audio experience coming next-generation. Xbox Series X will only support that even further.

Click to shrink...

Hellblade with headphones is literally the single greatest gaming audio experience that I have ever had personally, and the thought of Hellblade II using Project Acoustics and the dedicated audio hardware on Series X is just insane and exciting!

 

[Theorry]

Ninja Theory

"It's extremely exciting," senior sound designer Daniele Galante said of the new console. "We're going to have a dedicated chip to work with audio, which means we finally won't have to fight with programmers and artists for memory and CPU power."





The studio's audio lead David Garcia added: "We take for granted that graphics are powered by their own video cards. But in audio, we haven't had anything like that. Now we have some power dedicated to us."

Click to shrink...

 

[PurpleMoustache]

Hellblade with headphones is literally the single greatest gaming audio experience that I have ever had personally, and the thought of Hellblade II using Project Acoustics and the dedicated audio hardware on Series X is just insane and exciting!

Click to shrink...

As someone who has been low-key obsessed with Binaural recordings since I stumbled upon an old iPhone app with Binaural nature recordings (Naturespace, still up if people are interested), it was exciting to see people FINALLY come around to Binaural recordings with Hellblade

 

[bsigg]

I'm sure this thread won't get much attention because we all know why but considering MS has been working on this for a while, I'm really interested to see how 343i, Turn10 and Ninja Theory all use it in their upcoming games.

All 3 studios have released games with some really great audio and given additional tech and horsepower now available to them, it'll be interesting to hear what they've done.

 

[Navidson REC]

Yeah, Hellblade with headphones was an unreal experience. The second game on Series X is gonna be insane with all these advances and such a capable team behind it.

 

[Prine]

To have Ninja Theory to exclusively demonstrate this tech is pretty great. Thanks for posting this, had no idea there's hardware support.

 

[Biggzy]

With Hellblade II being a thing and Ninja Theory now being an Xbox first-party studio, I'd bet quite some good amount of hazelnuts that this game will offer an industry-wide reference audio experience coming next-generation. Xbox Series X will only support that even further.

Click to shrink...

Ooooooooo I had not put those two together and now I am salivating what craziness Ninja Theory has in stall for us.

 

[Convasse]

343i should do good work on this front, they invest quite heavily in the audio side of their productions and Halo Infinite looks to be no different.

 

[Philippo]

With Hellblade II being a thing and Ninja Theory now being an Xbox first-party studio, I'd bet quite some good amount of hazelnuts that this game will offer an industry-wide reference audio experience coming next-generation. Xbox Series X will only support that even further.

Click to shrink...

Yeah, I already felt unsettled playing Hellblade, don't think my heart will resist the sequel, too much anxiety.

Horror games next gen are to become real horrors.

 

[christocolus]

This was built for project Mara.. wasn't it? :)

 

[DeadmansTheory]

I'm really curious to see what Rare will do in this area with their next game Everwild but really excited to see what others will also do as well.

 

[bcatwilly]

I'm really curious to see what Rare will do in this area with their next game Everwild but really excited to see what others will also do as well.

Click to shrink...

Everwild looks so interesting to me, love the art style. And there could be some very cool use of audio in that world for sure.

 

[Black_Stride]

I'm really curious to see what Rare will do in this area with their next game Everwild but really excited to see what others will also do as well.

Click to shrink...

I legit forgot this was announced.
Damn this is gonna sound so good.
I can almost already hear the everwilds just from looking at the gifs.

Glad my trusty DT770s are still going strong, might cop a set of DT990s for when im playing more open games.
With the XSX not even having optical audio the deal I found on Z906s will save me a good few bucks I can reinvest into nextgen.

<<-- Lives in a small apartment

 

[Black Mantis]

This was built for project Mara.. wasn't it? :)

Click to shrink...

No, it's been in the works for the past decade.

 

[Deleted member 56752]

as someone with tinnitus, i'm always a little annoyed by focus on audio - but the feature Is cool

 

[DeadmansTheory]

I legit forgot this was announced.
Damn this is gonna sound so good.
I can almost already hear the everwilds just from looking at the gifs.

Glad my trusty DT770s are still going strong, might cop a set of DT990s for when im playing more open games.
With the XSX not even having optical audio the deal I found on Z906s will save me a good few bucks I can reinvest into nextgen.

<<-- Lives in a small apartment

Click to shrink...

I've been waiting for more info about it but Rare definitely has my attention. Like you, small apartment, headphones are my best option when gaming. I've got a pair from Turtle Beach(5.1 wireless/optical) but planning on upgrading when the XSX comes out.

I'm a little bummed that it won't have a separate port for optical as well but it seems the industry is going that way now when next gen systems hit retail.

 

[space_nut]

Ahhhhhh XSX is a beast! Love the tech MS has built

 

[christocolus]

No, it's been in the works for the past decade.

Click to shrink...

Yeah, I know that. I'm suggesting that the tech will be a good fit for project Mara.

 

[Black_Stride]

as someone with tinnitus, i'm always a little annoyed by focus on audio - but the feature Is cool

Click to shrink...

Focus?
The NT devs talk specifically about how there has effectively never been a focus on audio when it comes to gaming.

A good portion of Cernys talk was again on a lack of focus on Audio.
Whens the last time there was hype about Audio......if ever.

P.S Weird flex to enter a thread about audio to tell us how much you hate focus(non-existent) on Audio.

 

[Deleted member 56752]

Focus?
The NT devs talk specifically about how there has effectively never been a focus on audio when it comes to gaming.

A good portion of Cernys talk was again on a lack of focus on Audio.
Whens the last time there was hype about Audio......if ever.

P.S Weird flex to enter a thread about audio to tell us how much you hate focus(non-existent) on Audio.

Click to shrink...

Focus was an incorrect usage of the word. I personally avoid topics related to audio. It's always been a sore spot for me. It's literally the opposite of a flex.

 

[solis74]

Ahhhhhh XSX is a beast! Love the tech MS has built

Click to shrink...

indeed :)

 

[KillaKap]

wow this is fire. this demonstration really showcases this tech well. Props on the post

 

[Soliloquy of a Dogge]

Fascinating stuff and great tech from Microsoft. Can't wait to see the implementation of 3D audio going forward and how it evolves the gaming experience. Games should be as immersive as possible and audio is a huge part of that equation.

I'm sure this thread won't get much attention because we all know why but considering MS has been working on this for a while, I'm really interested to see how 343i, Turn10 and Ninja Theory all use it in their upcoming games.

Click to shrink...

With the craziness that's transpired here over the past week this thread will probably come as a surprise to some. Seen a lot of misinformation claiming Microsoft had no solution in this area.

 

[TuMekeNZ]

Its a battle for our eyes and ears!!!!!

 

[ShapeGSX]

The difference in Gears in that demo with this tech turned on is incredible. I can't wait to hear this on my 7.2.2 Atmos system!

 

[Lukas Taves]

This was built for project Mara.. wasn't it? :)

Click to shrink...

Not really, it was an old research, they started used in gears 4 and now are making available to everyone through gamestack

 

[solis74]

wow this is fire. this demonstration really showcases this tech well. Props on the post

Click to shrink...

indeed

 

[EvilBoris]

Improved audio for both consoles is really welcome

 

[Transistor]

Hellblade with headphones is literally the single greatest gaming audio experience that I have ever had personally

Click to shrink...

You and me both

 

[LukasHeinzel]

This sounds incredibly impressive.

 

[zedox]

Really cool stuff. Can't wait to get my hands to dev with it :)

 

[christocolus]

Not really, it was an old research, they started used in gears 4 and now are making available to everyone through gamestack

Click to shrink...

I know. read my post above.

 

[KillaKap]

im eager to try out horror games with these new consoles. taking spooky to a new level.

 

[Fuchsia]

as someone with tinnitus, i'm always a little annoyed by focus on audio - but the feature Is cool

Click to shrink...

I'm sorry but this post strikes me as odd. I also have tinnitus. It almost seems by "focus on audio" you mean 'loud sounds that could make your tinnitus worse.' If that's the case, simply turn down the loudness. I know that sometimes things are mixed very loud in a game or film, and the quality depends on the individual mix job, but you as the listener also have agency in controlling your overall listening level.
In general, audio is kind of an unsung hero in entertainment. It's often taught that audio is doing its job in a film or game when you *don't* necessarily notice it - when it's so seamless that your brain accepts it and pays attention to other information. Most people don't really notice a great UI SFX set, for example, or a wonderful ambience composition job. Even music can sometimes be unnoticed because it's so seamless with the experience.

However, creating those elements take a lot of time and skill. Implementing them in game does as well. Research is being done to give audio engineers new and exciting tools to create experiences with sound far beyond what was capable before. For audio people of all kinds (both creators and players), these are truly wonderful developments that will push things forward in terms of sound and how it affects the experience, and thus the player. Especially for sound designers, the attention being given to audio now with these initiatives is lovely.

It just seems odd to come in and complain about a focus on audio when these developments by Sony and Microsoft are an injection of exciting vitality into the interactive audio engineering landscape.

Maybe I'm misinterpreting your post.

 

[Deleted member 56752]

I'm sorry but this post strikes me as odd. I also have tinnitus. It almost seems by "focus on audio" you mean 'loud sounds that could make your tinnitus worse.' If that's the case, simply turn down the loudness. I know that sometimes things are mixed very loud in a game or film, and the quality depends on the individual mix job, but you as the listener also have agency in controlling your overall listening level.
In general, audio is kind of an unsung hero in entertainment. It's often taught that audio is doing its job in a film or game when you *don't* necessarily notice it - when it's so seamless that your brain accepts it and pays attention to other information. Most people don't really notice a great UI SFX set, for example, or a wonderful ambience composition job. Even music can sometimes be unnoticed because it's so seamless with the experience.

However, creating those elements take a lot of time and skill. Implementing them in game does as well. Research is being done to give audio engineers new and exciting tools to create experiences with sound far beyond what was capable before. For audio people of all kinds (both creators and players), these are truly wonderful developments that will push things forward in terms of sound and how it affects the experience, and thus the player. Especially for sound designers, the attention being given to audio now with these initiatives is lovely.

It just seems odd to come in and complain about a focus on audio when these developments by Sony and Microsoft are an injection of exciting vitality into the interactive audio engineering landscape.

Maybe I'm misinterpreting your post.

Click to shrink...

Yeah. It's a weird point. I'm not mad at anyone. I just avoid the topic because it bothers me. That's all.

 

[Ushay]

This sounds very promising having a dedicated hardware block.

What I'd really like to know is how will this affect head set compatibility.
We saw a LOT of high end headsets (Hi Res) miss the Xbox this gen due to their chat protocol. I hope this gets address this gen.
I'm talking about headsets like Arctic Pro Wireless and Corsair Virtuoso.

 

[SeanMN]

I was reading the Digital Foundry Scorpio piece from 2017, and this stood out to me:

The audio processor in Xbox One is fully transplanted across to Scorpio and gains new functionality - spatial surround, effectively adding a 'height' component to the existing 7.1 set-up. Scorpio is set to receive support for Dolby Atmos for gaming, Dolby Atmos for headphones plus a Microsoft proprietary format called HRTF, developed by the Hololens team. Because the APB (audio processor block) hardware is basically identical to that found in Xbox One, it means that all existing iterations of the console will get the spatial surround upgrade.

Click to shrink...

HRTF's are one of the highlights or Mark Cerny's PS5 talk last week. It seems as though if Xbox was some level of this tech in their 2017 console, there will be advancements upon that in the XSX.

 

[Prine]

Nice find that. Nice to see cross company usage of it.

 

[TameLucas]

Hellblade with headphones is literally the single greatest gaming audio experience that I have ever had personally,

Click to shrink...

Same for me. Playing Hellblade with headphones is a must.

I never felt so immersed in a game like I did on Hellblade. Not even on VR. For me, audio is simply the best immersion I can possibly get. So happy to see that this is going to be a big thing on next generation for both consoles.

 

[FinKL]

Hellblade with headphones is literally the single greatest gaming audio experience that I have ever had personally, and the thought of Hellblade II using Project Acoustics and the dedicated audio hardware on Series X is just insane and exciting!

Click to shrink...

Man, I REALLY need to sit down and play this, I keep misplacing my adapter that plugs into the Xbox controller to connect my headphones. The new controllers with 3.5mm preinstalled on the controllers sound soft to me whereas the adapter definitely seems like the audio is amplified.

 

[Pargon]

I missed this thread when it was originally posted - that's a really interesting talk and demos.
It seems like Microsoft's approach is to bake in the acoustic environment rather than trying to simulate it in real-time - which is probably going to allow for more complex/accurate environmental audio, and with less setup from the game developer.

It seems as though if Xbox was some level of this tech in their 2017 console, there will be advancements upon that in the XSX.

Click to shrink...

The use of HRTF in game audio is nothing new; it's been in PC gaming since the late '90s and is how tech like virtual surround works.

Focus?
The NT devs talk specifically about how there has effectively never been a focus on audio when it comes to gaming.

A good portion of Cernys talk was again on a lack of focus on Audio.
Whens the last time there was hype about Audio......if ever.

Click to shrink...

It may not have been a focus on consoles, but I would say that audio certainly had attention back in the mid-'90s to mid-'00s on PC, with lots of exciting tech developments.
Windows Vista largely killed that by discontinuing DirectSound3D hardware acceleration support, and audio has languished ever since.
Sound design has been improving a lot, but audio tech has stagnated until now. There are still so many games which don't even simulate occlusion, or have no height information at all.
I often see Alien: Isolation praised a lot in audio discussions. Its sound design might be really good, but the actual audio tech is lackluster. You can't tell if the xenomorph is right next to you or two rooms over. Or if that stomping around you hear is actually from the floor above you.
I'm very excited to see what both consoles bring to the table.

I never felt so immersed in a game like I did on Hellblade. Not even on VR. For me, audio is simply the best immersion I can possibly get. So happy to see that this is going to be a big thing on next generation for both consoles.

Click to shrink...

I haven't played the game, but my understanding is that Hellblade was only playing back pre-recorded binaural tracks for the "voices in her head".
That's not really "game audio" as much as it is a neat trick. It's not dynamic and doesn't change in relation to the player's movement/actions.

 

[Deleted member 64241]

Very impressive.

 

[Deleted member 4761]

I was reading the Digital Foundry Scorpio piece from 2017, and this stood out to me:


HRTF's are one of the highlights or Mark Cerny's PS5 talk last week. It seems as though if Xbox was some level of this tech in their 2017 console, there will be advancements upon that in the XSX.

Click to shrink...

Yep.
Xbox One already has superb audio chip.
Can't wait for the XSX evolution.

 

[Lady Gaia]

I love love love that audio is finally getting the kind of attention it deserves in the new consoles. I've always found sound to be an essential companion to any visual experience and look forward to seeing how this plays out.

 

[Ushay]

I love love love that audio is finally getting the kind of attention it deserves in the new consoles. I've always found sound to be an essential companion to any visual experience and look forward to seeing how this plays out.

Click to shrink...

Hoping we get a high end headset from it as well, Hi Res hopefully.

 

[EvilBoris]

Hoping we get a high end headset from it as well, Hi Res hopefully.

Click to shrink...

I'd like for the surface headphones to become compatible

 

[defaltoption]

I'd like for the surface headphones to become compatible

Click to shrink...

Yeah all of the companies stuff really should be compatible even cross brand, I really wish both Sony and Microsoft or even a third party like Logitech or whatever so it's cross platform would work with an audio company to develop a high end wireless solution someone like a Senheisher, Audio-Technica, or Beyerdynamic, yeah a wireless solution will never be as good as a wired option but it's close enough for most people we just need someone to do it. I don't want a wireless pair of crappy headset from a company that knows nothing about audio and just wants to market rgb.

The same way Logitech is doing a Herman Miller chair collaboration I want a Microsoft x Sennheiser or Logitech x Audio Technica solution. I feel like a market is there and only touched by Astro which many know aren't actsully great for the money audio wise compared to most high end audio companies