How Oodle Kraken and Oodle Texture supercharge the IO system of the Sony PS5

MrFox · Sep 25, 2020

Crap, I missed that thread!

If they need to tune the stream to prevent decompressor bottleneck, does it mean it wouldn't automatically do 22GB/s from any 4:1 compression data?

What I'm understanding here is that the stream must not become too processing intensive in a way that it can no longer pull 5.5GB/s continuously because the decoder is bogging down. So in those situations they remove some layers of compression to keep it at maximum bandwidth all the time.

gofreak · Sep 25, 2020

MrFox said:
Crap, I missed that thread!

If they need to tune the stream to prevent decompressor bottleneck, does it mean it wouldn't automatically do 22GB/s from any 4:1 compression data?

What I'm understanding here is that the stream must not become too processing intensive in a way that it can no longer pull 5.5GB/s continuously because the decoder is bogging down. So in those situations they remove some layers of compression to keep it at maximum bandwidth all the time.

This is an offline/encode process, not dynamically at runtime. So if the content is, say, 4:1 after the PS5 targeted encode, then 4:1 is what you'll get, consistently. I think when you put the explanation about encode options and decode provisioning and over provisioning next to that explanation, it basically means the decoder has been provisioned to provide up to 4x compression without bottlenecking the ssd speed. That's the degree of over provisioning available there, and the PS5 encode will twiddle its options to remain within, or take advantage of, that budget.

Jawmuncher · Sep 25, 2020

Makes Sense

iceblade · Sep 25, 2020

I guess this tech is going to give us oodles of free space >:D

Lionheart · Sep 25, 2020

Consoles biggest weakness was their HDD bottleneck, now its their greatest strength.

MrFox · Sep 25, 2020

gofreak said:
This is an offline/encode process, not dynamically at runtime. So if the content is, say, 4:1 after the PS5 targeted encode, then 4:1 is what you'll get, consistently. I think when you put the explanation about encode options and decode provisioning and over provisioning next to that explanation, it basically means the decoder has been provisioned to provide up to 4x compression without bottlenecking the ssd speed. That's the degree of over provisioning available there, and the PS5 encode will twiddle its options to remain within, or take advantage of, that budget.

What throws me off is that they turn on "slower modes" (so modes that are more computationally intensive on the decoder side) as long as the stream can still be decoded at wire-rate. It seems that the tuning is more about computational cost provisions, not target compression ratio nor the maximum output limit of 22GB/s.

The various bit stream options do well on different types of data, and they have different performance trade offs in terms of decoder speed vs compression ratio. On the Sony PS5 we use this to make encoded bit streams that can be consumed at the peak SSD bandwidth so that the Kraken decoder is never the bottleneck. As long as the Kraken decoder is running faster than 5.5 GB/s input, we can turn on slower modes that get more compression. This lets us tune the stream to make maximum use of the time budget, to maximize the compression ratio under the constraint of always reading compressed bits from the SSD at full speed. Without this ability to tune the stream you would have very variable decode speed, so you would have to way over-provision the decoder to ensure it was never the bottleneck, and it would often be wasting computational capacity.

There's a lot of questions about how it differs for other RDO tools, I'm curious if this is exclusive to Oodle Texture and Kraken, or if other method are also going in that direction:

Orthogonality of bit stream options is a game changer; it means we can try N boolean options in only O(N) time by measuring the benefit of each option independently. If you had to re-encode for each set of options (as in traditional monolithic compressors), it would take O(2^N) time to find the best settings.

Expy · Sep 25, 2020

Kraken plus Oodle Texture can double previous compression ratios

👀

Deleted member 51539 · Sep 25, 2020

Bold One said:
Someone explain this to me in DBZ terms.

We were like "yeah, Burter is the fastest in the universe", but now Goku came to Namek.

Deleted member 1003 · Sep 25, 2020

iceblade said:
I guess this tech is going to give us oodles of free space >:D

The best comment on this thread! Thank you.

eathdemon · Sep 25, 2020

Crayon said:
Whoa does sc require nvme? I thougmt it was just sata ssd. I dream squadron 42 comes to console (doubt) because my pc is pretty modest as i dont like throwing too much money at hardware.

ssd at this point means nvme to most people.

Hey Please · Sep 25, 2020

gofreak said:
This is an offline/encode process, not dynamically at runtime. So if the content is, say, 4:1 after the PS5 targeted encode, then 4:1 is what you'll get, consistently. I think when you put the explanation about encode options and decode provisioning and over provisioning next to that explanation, it basically means the decoder has been provisioned to provide up to 4x compression without bottlenecking the ssd speed. That's the degree of over provisioning available there, and the PS5 encode will twiddle its options to remain within, or take advantage of, that budget.

So, as a lay person, can I take this to mean that in the case of pre-arranged loading operations (like fast travel, loading new levels and scripted gameplay sequences) can take advantage of the highest compression rate, i.e. greater throughput of up to 22GB/s w/o bottlenecking the system as opposed to player driven data streaming (like trapesing around open world and somesuch)?

gofreak · Sep 25, 2020

Hey Please said:
So, as a lay person, can I take this to mean that in the case of pre-arranged loading operations (like fast travel, loading new levels and scripted gameplay sequences) can take advantage of the highest compression rate, i.e. greater throughput of up to 22GB/s w/o bottlenecking the system as opposed to player driven data streaming (like trapesing around open world and somesuch)?

There's no difference between the two scenarios. In the end the compression ratio you'll get for a given piece of data is up to the compressibility of the data, within the bounds of the decode hardware's decode budget.

What I was trying to say is, the compression ratio doesn't vary dynamically at runtime in response to emergent factors at runtime...the behaviour, the ratios across the bitstream, are fixed up front for a given piece of data during the encode. Various variables in the (one-time, offline) encode process are varied to ensure the decode time will not bottleneck (*relative to the ssd read speed). One of the variables is how aggressive the compression ratio is. In the end, whatever compression ratio the encoder achieves is designed to guarantee it is decodable at runtime consistently, without bottlenecks from the decoder.

It's kind of a minor detail in terms of the end result - in the end, what you see is what you get in terms of whatever compression ratio the encoder achieves - but I think they're promoting the granularity at which settings can be varied in kraken to tune for decode speed, compared to some other algorithms.

Hey Please · Sep 25, 2020

gofreak said:
There's no difference between the two scenarios. In the end the compression ratio you'll get for a given piece of data is up to the compressibility of the data, within the bounds of the decode hardware's decode budget.

What I was trying to say is, the compression ratio doesn't vary dynamically at runtime in response to emergent factors at runtime...the behaviour, the ratios across the bitstream, are fixed up front for a given piece of data during the encode. Various variables in the (one-time, offline) encode process are varied to ensure the decode time will not bottleneck. One of the variables is how aggressive the compression ratio is. In the end, whatever compression ratio the encoder achieves is what you'll get at runtime, consistently, without bottlenecks from the decoder.

Ah, gotcha. Thanks.

Dizastah · Sep 25, 2020

Interesting tech. PS5 is going to be blazing fast.

Lukemia SL · Sep 25, 2020

s y said:
never understood why "enter every building" always comes up. what's the point of having every building be enterable? what kind of content would justify the time spent creating some procedurally generated room system or the 20 year dev time it would need to have every room building interior be specifically designed?

I with you on the for the most part. I remember back in the day reading magazines and seeing Vice City and being like "wow you can go in this hotel and enter that nightclub". Then I was like "yeah I've gotta leave this hotel and get out of this nightclub".

You can enter lots of buildings in shenmue 2 but as long as there much of nothing to do inside these places then what's the point in it. Even RDR2 has nothing worth doing in buildings besides scrounging.

bi0g3n3sis · Sep 25, 2020

rntongo said:
Here's a solid post by someone working on PS5 texture compression and proof of why 11GB/s what you'll get! Still impressive but no where near the 22GB/s people have taken out of context.

How Oodle Kraken and Oodle Texture supercharge the IO system of the Sony PS5

The Sony PS5 will have the fastest data loading ever available in a mass market consumer device, and we think it may be even better than yo...

cbloomrants.blogspot.com

I've mentioned it that thread higher numbers ( 12 GB/s on average ) and it can go higher, man.

rntongo said:
I hope we can all now put the 22GB/s figure to rest. 11GB/s is still really good and much faster than what you'll get on almost any other platform.

So, we can pull XSX SSD 6 GB/s theoretical maximum to the rest too. If PS5's theoretical max. 22 GB/s figure can be put to the rest, same can be applied for XSX. Or we won't do this, hm? Since you mentioned you prefer Direct Storage.

Cerny stated the decompression block on the PS5 is capable of 22GB/s "if the data happened to compress particularly well". So, PS5 SSD can go above 11 GB/s without a sweat since it's the average number. For your sanity, you should reconsider your approach to this topic. There are members here who understands topic of this.

platocplx · Sep 25, 2020

bi0g3n3sis said:
I've mentioned it that thread higher numbers ( 12 GB/s on average ) and it can go higher, man.

So, we can pull XSX SSD 6 GB/s theoretical maximum to the rest too. If PS5's theoretical max. 22 GB/s figure can be put to the rest, same can be applied for XSX. Or we won't do this, hm? Since you mentioned you prefer Direct Storage.

Cerny stated the decompression block on the PS5 is capable of 22GB/s "if the data happened to compress particularly well". So, PS5 SSD can go above 11 GB/s without a sweat since it's the average number. For your sanity, you should reconsider your approach to this topic. There are members here who understands topic of this.

Yeah. They are really really trying hard to make it seem cerny is wrong in his assertions. Lol. McFly is a saint for trying to explain to them how wrong they are. and even if it doesn't reach in most scenarios it's still a massive speed in general across multiple types of files.

stryke · Sep 26, 2020

too bad it's only coming around now
at least with this we can save a bit more space on the drive with games coming next year

McFly · Sep 26, 2020

GDC 2021 would be very illuminating for a lot of next Gen stuff. Hopefully this covid situation should be better.

platocplx · Sep 26, 2020

McFly said:
GDC 2021 would be very illuminating for a lot of next Gen stuff. Hopefully this covid situation should be better.

Yeah def is gonna be some interesting presentations

AllBizness · Sep 26, 2020

McFly said:
GDC 2021 would be very illuminating for a lot of next Gen stuff. Hopefully this covid situation should be better.

Nope, when POTUS says April for vaccine really means its further out then that.

mentok15 · Sep 26, 2020

Interesting. So, without trying to stare any console war stuff, is this a big advantage for the PS5 over the XSX?

Xeonidus · Sep 26, 2020

mentok15 said:
Interesting. So, without trying to stare any console war stuff, is this a big advantage for the PS5 over the XSX?

It's what Matt said a long time ago. Nothing the Xbox series x can do to make up the speed advantage of the PS5. However, nothing the PS5 can do to make up the gpu power of the Xbox series x. In this case yes, the ps5 has a clear advantage

rntongo · Sep 26, 2020

bi0g3n3sis said:
I've mentioned it that thread higher numbers ( 12 GB/s on average ) and it can go higher, man.

So, we can pull XSX SSD 6 GB/s theoretical maximum to the rest too. If PS5's theoretical max. 22 GB/s figure can be put to the rest, same can be applied for XSX. Or we won't do this, hm? Since you mentioned you prefer Direct Storage.

Cerny stated the decompression block on the PS5 is capable of 22GB/s "if the data happened to compress particularly well". So, PS5 SSD can go above 11 GB/s without a sweat since it's the average number. For your sanity, you should reconsider your approach to this topic. There are members here who understands topic of this.

I think people have been going in circles over this same thing over and over. The bottom line is the PS5 SSD is significantly faster and has much higher throughput. And at the same time, the XSX has an incredibly fast SSD as well.

BreakAtmo · Sep 26, 2020

stryke said:
too bad it's only coming around now
at least with this we can save a bit more space on the drive with games coming next year

Well, the Warframe devs are going the extra mile and updating their game to include it and drop the file size. Maybe others will do the same.

rntongo · Sep 26, 2020

bi0g3n3sis said:
I've mentioned it that thread higher numbers ( 12 GB/s on average ) and it can go higher, man.

So, we can pull XSX SSD 6 GB/s theoretical maximum to the rest too. If PS5's theoretical max. 22 GB/s figure can be put to the rest, same can be applied for XSX. Or we won't do this, hm? Since you mentioned you prefer Direct Storage.

Cerny stated the decompression block on the PS5 is capable of 22GB/s "if the data happened to compress particularly well". So, PS5 SSD can go above 11 GB/s without a sweat since it's the average number. For your sanity, you should reconsider your approach to this topic. There are members here who understands topic of this.

On the other hand explain to me how >6GB/s means the theoretical max is 6GB/s. All we know is the typical throughput is above 4.8 and above 6GB/s is possible. So no one is expecting much further than 6, but the theoretical max for the XSX could be something crazy as well for all we know. The PS5 as far as we know is currently rated at 9GB/s but once Oodle is ready for the system it will reach a 2:1 ratio which is 11GB/s. Thats what will be the typical throughput. Maybe a bit higher but making it seem like its 4:1 or 3:1 ratio typically is a bit untrue.

platocplx · Sep 26, 2020

rntongo said:
On the other hand explain to me how >6GB/s means the theoretical max is 6GB/s. All we know is the typical throughput is above 4.8 and above 6GB/s is possible. So no one is expecting much further than 6, but the theoretical max for the XSX could be something crazy as well for all we know. The PS5 as far as we know is currently rated at 9GB/s but once Oodle is ready for the system it will reach a 2:1 ratio which is 11GB/s. Thats what will be the typical throughput. Maybe a bit higher but making it seem like its 4:1 or 3:1 ratio typically is a bit untrue.

It's not.

Cipherr · Sep 26, 2020

Some of the coolest shit Ive heard and read about next gen so far.

rntongo · Sep 26, 2020

platocplx said:
It's not.

Okay! lets agree to disagree. Otherwise its exciting what they've done with the PS5 SSD, a true generational leap.

platocplx · Sep 26, 2020

rntongo said:
Okay! lets agree to disagree. Otherwise its exciting what they've done with the PS5 SSD, a true generational leap.

No it's not agree to disagree dude. You are objectively wrong. There is absolutely Nothing in the XSX that closes that gap. The system is objectively faster in I/O as the XSX is more powerful. These systems have different ways of being balanced and Sony put more bias in the speed of the console and MS put more in the Power.

rntongo · Sep 26, 2020

platocplx said:
No it's not agree to disagree dude. You are objectively wrong. There is absolutely Nothing in the XSX that closes that gap. The system is objectively faster in I/O as the XSX is more powerful. These systems have different ways of being balanced and Sony put more bias in the speed of the console and MS put more in the Power.

I didn't say the XSX closes the gap. All I said typically the throughput should be between 4.8-6. it can even go further than that and MSFT hasn't mentioned a theoretical max like Sony did. It's clear that the PS5 has significantly higher throughput and the recent post by the Oodle developers proves that it will on average have a 2:1 compression ratio. That's 11GB/s.

I understand that this compression ratio will depend on what kind of data and what not which is impressive that some data will have all the way up to a 4:1 ratio.

platocplx · Sep 26, 2020

rntongo said:
I didn't say the XSX closes the gap. All I said typically the throughput should be between 4.8-6. it can even go further than that and MSFT hasn't mentioned a theoretical max like Sony did. It's clear that the PS5 has significantly higher throughput and the recent post by the Oodle developers proves that it will on average have a 2:1 compression ratio. That's 11GB/s.

It's not going to go further than that. that's what we are saying. There is not enough software in the world. When the hardware constraint is 2.4 raw. and even the. All of the higher numbers is all based on the mix of compression it's never just all textures. just stop saying it's higher than what MS stated. it's not correct.

d3ckard · Sep 26, 2020

Anybody can give me a scenario when you need to reread your whole RAM every two seconds? Because only then this enormous throughput would bring any real improvement. It's awesome and exciting from technological point of view, but practically it seems to be an overkill. performance doesn't automatically transform into user noticeable features.

PS5 SSD looks like it will be utilized only by first party games and even then by a few of them, since I can't even fathom any gameplay that would take advantage of this. I expect most multiplat engines to optimize streaming to the lowest possible factor (way lower than XSX theoretical peak), to increase availability on PC SSDs. They will go higher when necessary, but we're talking here about over an order of magnitude change in memory reading coupled with double the memory size and 5 times the performance. They will not saturate that bandwidth.

James Sawyer Ford · Sep 26, 2020

d3ckard said:
Anybody can give me a scenario when you need to reread your whole RAM every two seconds? Because only then this enormous throughput would bring any real improvement. It's awesome and exciting from technological point of view, but practically it seems to be an overkill. performance doesn't automatically transform into user noticeable features.

PS5 SSD looks like it will be utilized only by first party games and even then by a few of them, since I can't even fathom any gameplay that would take advantage of this. I expect most multiplat engines to optimize streaming to the lowest possible factor (way lower than XSX theoretical peak), to increase availability on PC SSDs. They will go higher when necessary, but we're talking here about over an order of magnitude change in memory reading coupled with double the memory size and 5 times the performance. They will not saturate that bandwidth.

Only needing less than one second in RAM has pretty huge implications in terms of RAM efficiency and utilization, less things that need to "hang around just in case"

behOemoth · Sep 26, 2020

I really look forward to loading time experiences like back in the SNES and N64 days. I can imagine how turtoring it will be playing old disc based games after becoming accustomed to these loading times.

d3ckard · Sep 26, 2020

James Sawyer Ford said:
Only needing less than one second in RAM has pretty huge implications in terms of RAM efficiency and utilization, less things that need to "hang around just in case"

Sure. But the drive speed will be ~30x increase for XSX and ~69x increase for PS5. This is coupled with universal RAM size increase by a factor of 2(even less when we count One X) and processing power 8,6x for XSX and 5.7 for PS5. This is enormous disparity which makes the possibility of any of those consoles bandwidth bottlenecked extremely unlikely. Once you get THIS much of an increase all the problems you had to optimize for go away. At the same time, even the PS5 doesn't have nearly enough bandwidth and latency to do things like loading assets every frame, which could change paradigm.

I know this is a clear marketing win for Sony and good for them, but I fail to see how would you utilize it.

James Sawyer Ford · Sep 26, 2020

No that's not how things work. Just because there's a big leap for one doesn't mean that an even greater leap for another isn't as influential

The SSD certainly isn't "over engineered"

d3ckard · Sep 26, 2020

James Sawyer Ford said:
No that's not how things work. Just because there's a big leap for one doesn't mean that an even greater leap for another isn't as influential

The SSD certainly isn't "over engineered"

This is so convincing argument I bow in awe.

Look, this is exactly like with cashiers in supermarket. Once you have more cashiers than customers, adding more doesn't improve processing speed.

gofreak · Sep 26, 2020

d3ckard said:
At the same time, even the PS5 doesn't have nearly enough bandwidth and latency to do things like loading assets every frame, which could change paradigm.

Do we know that?

We may well be down at levels where view-dependent loading of data - within a small number of frames if not a single frame - is possible. UE5 is more or less intimating that.

Bandwidth isn't necessarily about cycling out your whole RAM multiple times a second. It has a relationship with latency - the faster the transfer, the lower the latency - which would be relevant for any less predictable, relatively time sensitive, data access to data sets that are too big to be resident in memory. If, in whatever you're doing with this data, there is a perceptual limit to acceptable latency, higher bandwidth may translate to more data arriving within that limit, which might mean higher resolution data, or more view dependent lookup tables, or whatever. We're not talking gigs of fresh data - we're talking relatively small, unpredictable slices of data from sets that are too big to be in memory, but that need to come and go fast. Or in cases where you're caching data from disk, having a worst case cache miss penalty that leaves a technique usable i.e. being able to handle spikes in demand within certain timeframes. Minimising the worst case can often unlock the use of things.

In cases like that, the difference between faster and slower access to that kind of data might not always manifest in breaking ways, but that doesn't mean it wouldn't be noticeable, or be something improving hardware shouldn't address.

d3ckard · Sep 26, 2020

gofreak said:
Do we know that?

We may well be down at levels where view-dependent loading of data - within a small number of frames if not a single frame - is possible. UE5 is more or less intimating that.

Bandwidth isn't necessarily about cycling out your whole RAM multiple times a second. It has a relationship with latency - the faster the transfer, the lower the latency - which would be relevant for any less predictable, relatively time sensitive, data access to data sets that are too big to be resident in memory. If, in whatever you're doing with this data, there is a perceptual limit to acceptable latency, higher bandwidth may translate to more data arriving within that limit, which might mean higher resolution data, or more view dependent lookup tables, or whatever. We're not talking gigs of fresh data - we're talking relatively small, unpredictable slices of data from sets that are too big to be in memory, but that need to come and go fast. Or in cases where you're caching data from disk, having a worst case cache miss penalty that leaves a technique usable i.e. being able to handle spikes in demand within certain timeframes. Minimising the worst case can often unlock the use of things.

In cases like that, the difference between faster and slower access to that kind of data might not always manifest in breaking ways, but that doesn't mean it wouldn't be noticeable, or be something improving hardware shouldn't address.

Latency is not in relationship with bandwidth, latency is a constant you need to add on top of size/transfer speed, to get realistic real life timings.

Anyway, let me go with another example. You remember those corridors in God of War they had to put for loading? When Kratos squeezed through one for 10 seconds they were able to load around 800 megs of data. XSX will do it in 0.33 second(not including data compression and not having to optimize for serial reads, so even faster in practice), PS5 in 0,15 second. You really think those 183 ms make much of a difference, when over 9 whole seconds have been shaved?

PS5 would have to render much faster and on bigger data to saturate that bandwidth and its not going to happen. I would be willing to bet a small amount it will be a prime example of over engineering in a near future.

Benny Del Toro · Sep 26, 2020

d3ckard said:
This is so convincing argument I bow in awe.

Look, this is exactly like with cashiers in supermarket. Once you have more cashiers than customers, adding more doesn't improve processing speed.

That assumes that you have the same amount of customers every time though.

MrKlaw · Sep 26, 2020

d3ckard said:
Latency is not in relationship with bandwidth, latency is a constant you need to add on top of size/transfer speed, to get realistic real life timings.

Anyway, let me go with another example. You remember those corridors in God of War they had to put for loading? When Kratos squeezed through one for 10 seconds they were able to load around 800 megs of data. XSX will do it in 0.33 second(not including data compression and not having to optimize for serial reads, so even faster in practice), PS5 in 0,15 second. You really think those 183 ms make much of a difference, when over 9 whole seconds have been shaved?

PS5 would have to render much faster and on bigger data to saturate that bandwidth and its not going to happen. I would be willing to bet a small amount it will be a prime example of over engineering in a near future.

You're talking effectively about loading screens - a corridor is just a 'live' loading screen. In those cases, a tiny difference may not be noticable.

The bigger improvements may come from engine integration at a deeper level. Extreme example - you look at the postmortem discussions about spideman and having to duplicate assets 400x on the HDD because the physical location on the disc matters to optimise load/stream times - so their engine can hit their desired design goals and let them build a world at the density and quality desired.

In general a fast SSD with fast HW decode should dramatically improve this. The design team won't be constantly fighting (as much) with the tech team about their requirements for world detail, and the tech team won't need to be burning valuable time carefully optimising asset locations etc to get over the limitations of the storage medium.

Its possible (entirely hypothetical point incoming) that if you have a 2x increase in that SSD, it may surface some additional useful benefits at a lower level - something as simple as needing half as much ram buffer becuase you can pull it from storage fast enough could lead to more capacity for assets etc on screen; or tricks like the view-dependent streaming of assets becomes more viable - maybe on a slower SSD you have to restrict player rotation speed, or have more buffer for the edges to help as you turn.

Maybe small things that we might not even notice. But every little helps, and as a baseline may help to push things forward further for next-next gen too. Heck even if an open world game is visually idential to me as a player, its still helpful if the improved tech means a saving of X% developer time becuase they don't need to work around that particular technical limitation as much. Can mean the game comes out quicker, or they can get a different feature in, or maybe less crunch

gofreak · Sep 26, 2020

d3ckard said:
Latency is not in relationship with bandwidth, latency is a constant you need to add on top of size/transfer speed, to get realistic real life timings.

Sorry, I didn't mean to say bandwidth affected seek latency. I was talking about the total end to end time - the total latency - between asking for a bit of data and getting it in memory. The bandwidth of the transfer is a factor in that function at least beyond certain data sizes.

Re. your example. it's effectively a 'load time' case - the difference between SSDs of different speeds on upfront loading is indeed a matter of seconds vs fewer seconds. I'm not sure anyone's arguing that the difference in that case is much more than a matter of convenience for the end user (not to diminish that, mind you). But what I was talking about was related to more granular, less predictable runtime access to data, frame to frame, rather than designed-in loading periods. In the former case, talking about the human perception of seconds, that's one thing. In the latter case, we're talking about much smaller windows of time and potentially spikey and unpredictable access to data.

In the end it depends on how developers want to use storage, and it'll take time for devs to make the most of it, but faster storage will invite them to think about pushing more data out to storage with smaller worst case access penalties, free up more RAM etc. etc. Once devs do shift into that, there's probably a long road ahead of usable SSD performance improvement, well past PS5's speeds.

Deleted member 20297 · Sep 26, 2020

d3ckard said:
Latency is not in relationship with bandwidth, latency is a constant you need to add on top of size/transfer speed, to get realistic real life timings.

Anyway, let me go with another example. You remember those corridors in God of War they had to put for loading? When Kratos squeezed through one for 10 seconds they were able to load around 800 megs of data. XSX will do it in 0.33 second(not including data compression and not having to optimize for serial reads, so even faster in practice), PS5 in 0,15 second. You really think those 183 ms make much of a difference, when over 9 whole seconds have been shaved?

PS5 would have to render much faster and on bigger data to saturate that bandwidth and its not going to happen. I would be willing to bet a small amount it will be a prime example of over engineering in a near future.

I have to agree with you here to a certain degree. All three consoles are orders of magnitude better with io in comparison to current gen. This is due to both hardware and clever software engineering. The difference between the three consoles is not an order of magnitude. How that will translate to differences in games will have to be seen but coming from this gen and seeing how the lowest spec console can suspend one game and resume another one in eight seconds, cutting this number in half might be half the time but we are talking about seconds here, not minutes. I know some people will pretend how important that difference is to them but objectively the difference is not big, especially compared to waiting times we are used to.

d3ckard · Sep 26, 2020

MrKlaw said:
You're talking effectively about loading screens - a corridor is just a 'live' loading screen. In those cases, a tiny difference may not be noticable.

The bigger improvements may come from engine integration at a deeper level. Extreme example - you look at the postmortem discussions about spideman and having to duplicate assets 400x on the HDD because the physical location on the disc matters to optimise load/stream times - so their engine can hit their desired design goals and let them build a world at the density and quality desired.

In general a fast SSD with fast HW decode should dramatically improve this. The design team won't be constantly fighting (as much) with the tech team about their requirements for world detail, and the tech team won't need to be burning valuable time carefully optimising asset locations etc to get over the limitations of the storage medium.

Its possible (entirely hypothetical point incoming) that if you have a 2x increase in that SSD, it may surface some additional useful benefits at a lower level - something as simple as needing half as much ram buffer becuase you can pull it from storage fast enough could lead to more capacity for assets etc on screen; or tricks like the view-dependent streaming of assets becomes more viable - maybe on a slower SSD you have to restrict player rotation speed, or have more buffer for the edges to help as you turn.

Maybe small things that we might not even notice. But every little helps, and as a baseline may help to push things forward further for next-next gen too. Heck even if an open world game is visually idential to me as a player, its still helpful if the improved tech means a saving of X% developer time becuase they don't need to work around that particular technical limitation as much. Can mean the game comes out quicker, or they can get a different feature in, or maybe less crunch

Spiderman example is perfect. 400x repetition is due to slow random reads on HDD,a problem that is solved with any SSD. World detail is a solved problem as well, since PS5 doesn't have enough horsepower to process all the data it could theoretically pull from disk.

If any of you done any sort of performance optimization, you know that the value of improving any subsystem speed gets marginally smaller as you improve. When you take something that took a second and make it take a half, it's a bigger win than subsequent taking a half and making it a quarter. Why? Because the impact of the subsystem on the workflow was cut in half the first time you optimized it, so now you're only optimizing something that's twice less important.

I think we should make it clear that every conceivable data streaming problem from last gen is effectively solved. Every developer that had to go for artistic concessions due to this, doesn't anymore.

Yes, probably there can be scenarios when you could utilize it fully(super fast racer with extremely detailed environment?), but it will be exceptions and even then, difference is most likely to manifest in still screens, as the rate of motion will make it hard to discern the details while playing.

MrKlaw · Sep 26, 2020

d3ckard said:
Spiderman example is perfect. 400x repetition is due to slow random reads on HDD,a problem that is solved with any SSD. World detail is a solved problem as well, since PS5 doesn't have enough horsepower to process all the data it could theoretically pull from disk.

If any of you done any sort of performance optimization, you know that the value of improving any subsystem speed gets marginally smaller as you improve. When you take something that took a second and make it take a half, it's a bigger win than subsequent taking a half and making it a quarter. Why? Because the impact of the subsystem on the workflow was cut in half the first time you optimized it, so now you're only optimizing something that's twice less important.

I think we should make it clear that every conceivable data streaming problem from last gen is effectively solved. Every developer that had to go for artistic concessions due to this, doesn't anymore.

Yes, probably there can be scenarios when you could utilize it fully(super fast racer with extremely detailed environment?), but it will be exceptions and even then, difference is most likely to manifest in still screens, as the rate of motion will make it hard to discern the details while playing.

I generally agree with you on the marginal gains and that the 'general 'issue with storage is solved. However the devil may be in the detail and we may need a few years and game postmortems to properly understand if that is the case.

Devil's advocate - what if storage transfer speed differences are marginal, but for the same engine end up sitting either side of a 60Hz framerate update. PS5 SSD provides assets within a 60Hz frame budget, XSX SSD takes just a few ms longer but thats enough to drop you to 30Hz. Now obviously the devs or engine would likely scale back resolution/assets/asset density to get back under budget but it can mean a practical impact.

And PS5 not having the power to process all the data doesn't matter - its about availablity of data. Not necessary frame to frame but over a few frames you want availability of data to pull from and that availablity is capable of being broader on PS5.

Madjoki · Sep 26, 2020

gozu said:
So, with Oodle texture compression, when we're decompression textures we go from the 8-9GB/s average to to ~~16.5~~ 15 GB/s average ?

Sounds too good to be true. Textures are the bulk of game data usually.

You don't and can't decompress Oodle textures. It's encoded to native format the GPU uses to render. Smaller filesize, smaller bandwidth use AND smaller memory usage.
The details it removes, are gone forever, but you (assuming developer doesn't over do it) probably will never notice it.

gofreak · Sep 26, 2020

d3ckard said:
I think we should make it clear that every conceivable data streaming problem from last gen is effectively solved.

I wouldn't disagree with that, I think if you plug last gen data management engines into any SSD, and they'll fly, they'll easily be saturated.

The question I have is what does faster storage unlock in terms of new approaches to data management. When you get into the kind of usage where sub-seconds and milliseconds matter (vs multi-second preloading of data, or even the low-bandwidth streaming we saw last gen). That's an open question, but I wouldn't bet against a relatively long road of storage IO improvement, and software scaling to use that, as RAM increases stall relatively speaking.

spwolf · Sep 26, 2020

All of this will lead to smaller game sizes, especially on PS5 where (presumably) stronger compression can be used due to hardware decoder built in.

This will also be built into PS dev tools and very easy to integrate. Slowest part of new tool integration will be testing, so expect to see it in many games soon enough.

Also, fact that they are engineering for 8-9Gbs mean that has been their target set by Sony team. They expect to use this in games otherwise they would increase compression for smaller sizes.

All that variable compression encoder does is ensures that above speed can be reached and picks best compressed sample that meets that speed.

If that game someone mentioned dropped from 66GB to 50GB, expect more with new tools on PS5, maybe significantly more.

So target I/O speeds have been set already, this is about making games smaller at same target I/O speed.

Zok310 · Sep 26, 2020

"Kraken compression acts as a multiplier for the IO speed and disk capacity, storing more games and loading faster in proportion to the compression ratio."

This sounds like magic, it literally makes the i/o AND storage better, and this is without Oodle Texture.
Its any wonder why devs cant shut up about developing on PS5.

How Oodle Kraken and Oodle Texture supercharge the IO system of the Sony PS5

VFX Rendering Pipeline Developer

Crisis Dino

VFX Rendering Pipeline Developer

User requested account closure

2020 Member Elect

2020 Member Elect

“Fuck them kids.”

2020 Member Elect

2020 Member Elect

2020 Member Elect

User requested account closure