Amazon AWS and Nintendo Systems presents about how Nintendo modernized the back-end of the Nintendo eShop into a microservice-based architecture

[delete12345]

View: https://www.youtube.com/watch?v=grdawJ3icdA

Close if old.


UPDATE:

Main takeaways:

• The frontend of the Nintendo eShop on the Nintendo 3DS and the Nintendo Switch are both written in React (likely ReactJs) as a single-page application.
  • It is unknown how they are able to handle memory usage on the embedded devices.
• The plan is for Nintendo to maintain the Nintendo eShop for at least 20+ years starting from 2011 (to achieve long-term maintainability), when the Nintendo 3DS launched.
• They are using the Domain Driven Design Principle to help split the modular monolithic backend into an infrastructure that can be more manageable and maintained easily while allowing them to support the increasing requirements as time went by.
• Their microservice team is split into Service teams and a large Embedded team, for compliances and governance requirements.
  • Each Service team has a group of developers, which handles the applications and a group of DevOps which handle the infrastructure.
  • The DevOps group in the Service team collaborates with the Embedded DevOps team to fulfill the infrastructure deployments.
  • Each team are autonomous and independent of each other, as they handle their respective services.
• As they went from 2017 to 2020, it was then realized it is difficult for Nintendo to transition the DevOps to other projects, since they are very engrained into the services. So, in 2020, they began looking to "Platform Engineering", and has continued to do this to this day (2020 ~ 2023).
  • Platform engineers help to develop toolchains and workflows, enabling self-service capabilities for software engineering organizations, and bundling them into an internal developer platform, so that the Service team developers can use that platform while reducing the cognitive loads.
  • In other words, the Service team developers do not need to concern about the DevOps and backend integration with Amazon AWS as they develop their services.
  • As a result, each Service team has their own independent AWS account, allowing the Nintendo Systems to be freed from the limitations of using a single AWS account. Many services, such as Nintendo SDK, network services, various QA processes for each titles in development, and so on, all benefit from this.
• Due to internal organizational constraints, there isn't a dedicated platform engineering role. However, this lack of a dedicated team allowed Nintendo Systems to enable the creation of a lightweight platform sufficient for their needs.
• Instead of using Amazon EKS to manage Kubernetes clusters, they went with Amazon ECS, and let the service teams manage their ECS clusters independently. This allowed the developers to not be concerned about managing the clusters, and to be more interested in their services.
• The way the microservice and platform engineering works is:
  • It dynamically checks the user's digital assets (games, DLCs, add-ons, etc.) when the user launches a game on the game console (e.g., Nintendo Switch).
  • It is designed so that the service itself is accessed frequently in a short period of time on launch days of major titles.
  • To achieve that availability and scalability to use the service under heavy traffic spikes, the entire internal developer platform (IDP) is built as a microservice itself, using IDP components in a self-service manner.
  • It uses Amazon ECS Fargate, DynamoDB and S3 for storage, and CloudHSM for DRM-related processes.
• During the launch of The Legend of Zelda: Tears of the Kingdom:
  • They scaled the number of ECS tasks to around 120 tasks, and (unknown quantities) of AWS CloudHSM instances.
  • These numbers are based on sales forecasts and actual pre-order numbers.
  • DynamoDB capacity mode (used for provisioning based on predictable traffic) was changed to on-demand mode during this period of time to accommodate the unpredictable traffic.
  • Within a matter of minutes, the traffic was several times higher than normal.
• About the e-Commerce API Platform:
  • This platform is what powers the Nintendo eShop, My Nintendo Store, Year-in-Review site, mobile apps and more.
  • Third party developers engaging in the e-Commerce API platform are increasing as a result to demand.
  • Nintendo is currently pursuing a business strategy for its e-Commerce API Platform.

 

[Oliver James]

The slides Watson, what do they mean?

 

[Wrexis]

Thanks for posting this. This is a fascinating insight.
Slightly scary that in Phase 2 they had dumped everything into a single AWS account/VPC lol. But hence Phase 3.

Good for gaming peeps: they built the eShop to be maintainable for at least 20 years.

EDIT: I'm watching the whole thing because this is actually relevant to what I do at work lol.
This is a really good presentation.

 

[Disco Infernal]

Backend might be good, but holy shit that front end is atrocious.

 

[cw_sasuke]

Backend might be good, but holy shit that front end is atrocious.

Click to shrink...

They probably didn't expect the eShop to host 10k games and more.

Expecting a massive revamp on Switch 2.

 

[kirbyfan407]

They probably didn't expect the eShop to host 10k games and more.

Expecting a massive revamp on Switch 2.

Click to shrink...

Hopefully! I'm surprised at how slow the UI is in both the NSO app and eShop on Switch, but I imagine speeding that up will be a priority for their next console and iteration of the OS.

 

[logan_cadfgs]

Hopefully! I'm surprised at how slow the UI is in both the NSO app and eShop on Switch, but I imagine speeding that up will be a priority for their next console and iteration of the OS.

Click to shrink...

I like how the NSO app straight-up tells you "uhh yeah we know it's slow, maybe try closing your game lol?"

 

[Kromis]

I love microservices. That is all.

 

[genericbrand]

They probably didn't expect the eShop to host 10k games and more.

Expecting a massive revamp on Switch 2.

Click to shrink...

My memories have me feeling like it was pretty non-performant even back in 2017

 

[Secret Bambino]

Thanks for sharing, OP. I'm definitely curious in watching all of it, since half my job is setting up infrastructure lol

 

[Pineapple]

That's pretty neat but yeah, I hope the easy and quickness of the shop is an important focus in the next system. Scrolling through the big sale right now is such a pain in the ass and I wasn't even able to load 500 games of the 4k+ games on sale.

Thank god for Deku Deals, at least that way I can see everything on sale, but I shouldn't have to use it. Also add a damn shopping cart.

 

[boris_feinbrand]

Backend might be good, but holy shit that front end is atrocious.

Click to shrink...

I actually suspect the Switchs CPU to just be too slow to handle the shop while any game is open.

 

[logash]

They probably didn't expect the eShop to host 10k games and more.

Expecting a massive revamp on Switch 2.

Click to shrink...

Can't wait for that. As someone who has been digital since 2020 I find the current eShop damn near unusable. Or at least just functional enough to make a purchase. Not stable enough to have a good time browsing it.

 

[PumpkinPieAddict]

I have to watch this all but AWS isn't a bad bet for the Nintendo Eshop. If I recall correctly Sony is using MS Azure? At least for the Playstation Now side. Kinda have to pick one of the two big dogs. I mean you could just pull a Nintendo and use Google lol.

 

[Jahranimo]

Insight into ANYTHING Nintendo behind the scenes is intriguing. Adding to watch later and will definitely give it a watch...later!

 

[Amnixia]

I have to watch this all but AWS isn't a bad bet for the Nintendo Eshop. If I recall correctly Sony is using MS Azure? At least for the Playstation Now side. Kinda have to pick one of the two big dogs. I mean you could just pull a Nintendo and use Google lol.

Click to shrink...

Nothing inherently wrong about using either (I predominately use AWS for work).

Everything really hangs on your own programs.

 

[Lumination]

My memories have me feeling like it was pretty non-performant even back in 2017

Click to shrink...

Yeah it was never good. And any competent pagination method makes it not matter if you're listing 10, 100000, or 100000000 games. I think just the base UI is not optimized or has too many frills for the CPU.

 

[Alvis]

I have to watch this all but AWS isn't a bad bet for the Nintendo Eshop. If I recall correctly Sony is using MS Azure? At least for the Playstation Now side. Kinda have to pick one of the two big dogs. I mean you could just pull a Nintendo and use Google lol.

Click to shrink...

You can also build your own server infrastructure, like Valve did

 

[behOemoth]

I have to watch this all but AWS isn't a bad bet for the Nintendo Eshop. If I recall correctly Sony is using MS Azure? At least for the Playstation Now side. Kinda have to pick one of the two big dogs. I mean you could just pull a Nintendo and use Google lol.

Click to shrink...

I remember rackspace being one supplier for cloud services for Sony. I wouldn't be surprised that they are using a mix.

 

[Iztok]

We dropped AWS in favor of Azure but this is still very interesting to me, it all hits very close to home.
But I have to admit, it somewhat breaks the illusion, knowing what's behind the scenes.

 

[radardi]

Better than the MK1 panel. Very interesting they embed devops with their microservices team to deploy.

 

[HotsauceDragon]

All that cloud power and they still can't spare enough cycles to have an mp3 playing in the background.


View: https://www.youtube.com/watch?v=yyjUmv1gJEg

 

[onpoint]

I heard the eShop on the Switch is basically just running a web browser, any truth to this?

Either way. Runs like trash and needs better sorting options. It's truly the worst aspect of the Switch.

 

[Meelow]

All that cloud power and they still can't spare enough cycles to have an mp3 playing in the background.


View: https://www.youtube.com/watch?v=yyjUmv1gJEg

Click to shrink...

This music makes me want to buy games haha.

I heard the eShop on the Switch is basically just running a web browser, any truth to this?

Either way. Runs like trash and needs better sorting options. It's truly the worst aspect of the Switch.

Click to shrink...

I'm pretty sure its true, like other posters said I expect a full revamp for Switch 2.

 

[delete12345]

The panel revealed the frontend Nintendo eShop is written in React as a single-page application. How are they able to manage using little amounts of memory size on the Switch to execute and run the React applet, is quite amazing.

Nintendo plans to maintain the Nintendo eShop for at least 20+ years, replacing and embracing new technologies as necessary.

 

[BaconHat]

The panel revealed the frontend Nintendo eShop is written in React as a single-page application. How are they able to manage using little amounts of memory size on the Switch to execute and run the React applet, is quite amazing.

Click to shrink...

Holy shit, i honestly never would have guessed.

 

[CubeApple76]

If switch 2 has an eShop that runs fast that'll be the reason I upgrade lol

 

[delete12345]

Holy shit, i honestly never would have guessed.

Click to shrink...

It does make sense for the app to use the RESTful services to interact with the backend microservices.

 

[Yoshimitsu126]

I only ever buy eshop games from its nintendo store page on my phone because of how slow it is on Switch console.

 

[808s & Villainy]

Slightly scary that in Phase 2 they had dumped everything into a single AWS account/VPC lol. But hence Phase 3.

Click to shrink...

You can actually get quite a bit done in a single AWS account. We had a project where we had to hit a massive scale and we had a meeting with some AWS/Twitch engineers and they told us within a single account you can request some of the default limits for various services (APIGW, Lambda, etc) to be scaled quite a bit, then replicate that across regions, and only after you've hit limits doing that do you really need to consider multiple accounts

But anyway, this type of stuff is why I'm less nervous about the "fears" people have of Nintendo starting from scratch again. With the Switch they built things the proper, modern way

Holy shit, i honestly never would have guessed.

Click to shrink...

Why? We knew it was basically a web app and React is pretty common as far as building interactive UIs nowadays

I heard the eShop on the Switch is basically just running a web browser, any truth to this?

Either way. Runs like trash and needs better sorting options. It's truly the worst aspect of the Switch.

Click to shrink...

This is 100% because the Switch reserves an extremely tiny amount of RAM for the eshop. Remember it has to run while games are suspended along with the rest of the OS, so less than a gig of memory. It's probably altering the DOM, aggressively unloading and lazy loading everything, and the page sizes it's loading are probably fairly small as a result. It's also why the experience on any other device is drastically different

You can also build your own server infrastructure, like Valve did

Click to shrink...

I doubt it's worth it for Nintendo. Steam is Valve's main business, unlike the eShop just being part of it for Nintendo. Even Netflix uses AWS rather than their own infrastructure

 

[Juryvicious]

Will give this a watch tomorrow morning. Thanks for posting this.

All that cloud power and they still can't spare enough cycles to have an mp3 playing in the background.


View: https://www.youtube.com/watch?v=yyjUmv1gJEg

Click to shrink...

I immediately want to buy video games now. lol

 

[Eoin]

If I recall correctly Sony is using MS Azure? At least for the Playstation Now side.

Click to shrink...

Sony signed an MoU with Microsoft regarding exploring the possibility of using Azure for PlayStation Now, and they renewed that MoU later, but as far as we've heard there's been no further developments. The streaming service (formerly PlayStation Now) is not currently running on Azure.

I heard the eShop on the Switch is basically just running a web browser, any truth to this?

Click to shrink...

Yep. If the service (or your connection) isn't working properly it'll even show some browser errors. It's purely HTML. There is some sense to that but it means that it's quite heavy relative to the portion of the Switch's resources that are devoted to running it.

 

[Wrexis]

You can actually get quite a bit done in a single AWS account. We had a project where we had to hit a massive scale and we had a meeting with some AWS/Twitch engineers and they told us within a single account you can request some of the default limits for various services (APIGW, Lambda, etc) to be scaled quite a bit, then replicate that across regions, and only after you've hit limits doing that do you really need to consider multiple accounts

Click to shrink...

Actually I wasn't thinking of that, we hit the limits at work all the time on some accounts and just request new limits to be upped.

Basically having everything in one AWS isn't best practice at all any more from a security perspective - one account is considered to have a large blast radius if you have a malicious actor. It also means all your logs and other important audit data are all in the same account, which can be dangerous.

 

[delete12345]

Do note that I'm currently updating OP as I watch the presentation. If I have anything written incorrectly, please let me know so I can update the OP. (I'm somewhere around 24:45 as of writing.)

 

[DominicanGlory]

The eshop is atrocious. Lags like crazy

 

[808s & Villainy]

Actually I wasn't thinking of that, we hit the limits at work all the time on some accounts and just request new limits to be upped.

Basically having everything in one AWS isn't best practice at all any more from a security perspective - one account is considered to have a large blast radius if you have a malicious actor. It also means all your logs and other important audit data are all in the same account, which can be dangerous.

Click to shrink...

Ah yes, security wise that makes sense. Although I haven't watched the whole presentation (but have seen the slide about Phase 2) it's possible that just details how it's structured for a single account and they do actually duplicate it across multiple accounts

The eshop is atrocious. Lags like crazy

Click to shrink...

Again, this isn't an infrastructure issue. The eShop is built as a SPA (single page app), meaning all the javascript code for rendering various pages and interactivity is downloaded to the browser, with the server only providing the data. It's very likely the slowness is a result of the small RAM budget allocated for the eShop meaning it's constantly having to unload and load new data.

 

[BaconHat]

Why? We knew it was basically a web app and React is pretty common as far as building interactive UIs nowadays

Click to shrink...

Never knew personally it was a web app, and had to work with react devs for a few years now, so it was a fun surprise to see it was also used in such a way with the eshop.

 

[G-X]

Hopefully! I'm surprised at how slow the UI is in both the NSO app and eShop on Switch, but I imagine speeding that up will be a priority for their next console and iteration of the OS.

Click to shrink...

from what i have gathered there is little to no overhead in the switch system OS, which is why when things run simultaneously with it like the eshop it runs like hot trash, and also why we have seen little to no extra features that most want. I don't know if its a matter of the hardware tdp being limited when in OS or the OS being that much of a resource hog/poorly optimized, but that has been what I have seen from multiple "engineering" type people

 

[Hazz3r]

If they're already using React then swapping over to Next seems like a fairly common sense move if it's continuing to lag.

 

[OtakuCoder]

All that cloud power and they still can't spare enough cycles to have an mp3 playing in the background.


View: https://www.youtube.com/watch?v=yyjUmv1gJEg

Click to shrink...

That's something they'd have to do client-side and I don't think they'd want to tax the Switch's CPU any more than they already are!

 

[JSRF]

I am legally required to post this

View: https://www.youtube.com/watch?v=y8OnoxKotPQ

 

[TonyBaduy]

Ah yes, security wise that makes sense. Although I haven't watched the whole presentation (but have seen the slide about Phase 2) it's possible that just details how it's structured for a single account and they do actually duplicate it across multiple accounts


Again, this isn't an infrastructure issue. The eShop is built as a SPA (single page app), meaning all the javascript code for rendering various pages and interactivity is downloaded to the browser, with the server only providing the data. It's very likely the slowness is a result of the small RAM budget allocated for the eShop meaning it's constantly having to unload and load new data.

Click to shrink...

Since the Switch 2 will probably have 12GB of RAM, hopefully that means they allocate 2GB for the OS and with it's faster CPU it allows them to have a blazing fast eShop.

 

[delete12345]

Okay, finally finished watching the presentation. I skipped the Amazon AWS solutions architectural concepts in the slides, and only grabbed what may be interesting to ResetEra users. The OP is updated as a result.

If there's any incorrect info, please let me know. I'm not a solutions architect, so there are a lot of possible unknowns.

 

[808s & Villainy]

If they're already using React then swapping over to Next seems like a fairly common sense move if it's continuing to lag.

Click to shrink...

I doubt server side rendering would really help things. I think the issues are only solved by more RAM for the eShop, which the Switch 2 will likely have

Since the Switch 2 will probably have 12GB of RAM, hopefully that means they allocate 2GB for the OS and with it's faster CPU it allows them to have a blazing fast eShop.

Click to shrink...

Yeah, it should be significantly less "laggy"

Okay, finally finished watching the presentation. I skipped the Amazon AWS solutions architectural concepts in the slides, and only grabbed what may be interesting to ResetEra users. The OP is updated as a result.

If there's any incorrect info, please let me know. I'm not a solutions architect, so there are a lot of possible unknowns.

Click to shrink...

I'm fairly certain they meant ReactJS, as you assumed. If they had meant React Native, they probably would've explicitly said so

 

[delete12345]

I'm fairly certain they meant ReactJS, as you assumed. If they had meant React Native, they probably would've explicitly said so

Click to shrink...

Awesome. Thanks for this.

 

[TonyBaduy]

Yeah, it should be significantly less "laggy"

Click to shrink...

I do wonder how much RAM they allocated to the eShop on Switch though. Plus here is something to consider: It's laggier if you have any game or app open. That means it either has a variable amount of RAM allocated, less when a game is open and more when it isn't or it's being CPU limited. Neither the RAM nor the CPU should be a problem next gen, so it hopefully fixes this issue with the eShop... they should still make an algorithm that suggest games based on what you downloaded to improve discoverability and improve the front page to help with it too.

 

[808s & Villainy]

I do wonder how much RAM they allocated to the eShop on Switch though. Plus here is something to consider: It's laggier if you have any game or app open. That means it either has a variable amount of RAM allocated, less when a game is open and more when it isn't or it's being CPU limited. Neither the RAM nor the CPU should be a problem next gen, so it hopefully fixes this issue with the eShop... they should still make an algorithm that suggest games based on what you downloaded to improve discoverability and improve the front page to help with it too.

Click to shrink...

So we've heard the entire Switch OS only has 800MB reserved for it and the eShop only has a fraction of that, so probably not a large amount...the additional lag when games are suspended is probably because of the small CPU budget they both have to share (as the CPU is responsible for making the API requests and actually running the javascript code)

But I'm sure they're just as frustrated having to deal with such limited resources so I'm confident Switch 2 will reserve more for that stuff

 

[boontobias]

This is literally my job im glad I know half those words

 

[Tathanen]

The eshop performance is so god awful that I wonder about their selection of react for building it, like if the processor is too wimpy or they aren't giving it enough RAM or something for a front end JavaScript framework to be able to handle a simple list of scrolling rectangles maybe they should have built it using an older suite of technologies or something. Hell just moving focus from the left sidebar to the main content area stutters and chugs, the switch can't handle whatever they built.

I know the slides in the OP have nothing to do with the front end but still, I question their decision to use front end web technologies instead of something more native given the performance we see.

 

[Qassim]

Sounds similar to the setup at my company, it's a fairly common pattern these days, we tend to not have dedicated DevOps people in teams though (there's a few devops engineers that float around teams that need help), our software devs build and manage the infrastructure for their own software.