Yamauchi didn't understand the market anymore and made a series of near-fatal mistakes.
It's why he retired & Iwata took over, who course corrected the company to the DS & Wii
Hiroshi Yamauchi actually was fully behind and backed Nintendo's new direction.
The reason he stepped down as CEO was that by 2002 he was way too old to be the CEO of a multinational company (he was 75 years old!)
How I know he backed Iwata's tenure? Because he was the one that came out with the dual screen concept and because he was vocal, during interviews around the time there was the pass of the baton, on the direction he wished the company followed:
"Coincidental to my leaving the company, I would like to make one request: that Nintendo give birth to wholly new ideas and create hardware which reflects that ideal. And make software that adheres to that same standard. Furthermore, this software should attract consumers as new and interesting. Lastly, and of equal importance, is completing these products quickly and at a cost comparable to today's current market. I imagine most people question the feasibility of my request, but Nintendo has always pursued those objectives. I'd ask that the company continue to follow this goal as my final and only request to the new management staff. I can't say what these new types of software will be, but I'm sure they'll release it during my lifetime."
Under Yamauchi Nintendo collected successes and failures but the old man still had an incredible business acumen until the very end.
I always like to recall the answer he gave to the japanese magazine Dengeki when asked about the current state of the japanese console industry back in 2002 (Sony was dominating with PS2):
"Tomorrow, they could lose that strength, as reversals of fortune are part of this business. Taking into account the things I've encountered in my experiences as Nintendo president, I have come to the conclusion that it requires a special talent to manage a company in this industry. I selected Iwata-san based on that criteria. Over the long-term I don't know whether Iwata-san will maintain Nintendo's position or lead the company to even greater heights of success. At the very least, I believe him to be the best person for the job. "
And what happened in Japan a few years later was that in 2006 the Nintendo platforms regained the number one spot for the highest number of software sold in the year and never looked back, with the sole exception of 2011.
Nowadays Nintendo is in such a strong position in Japan that Nintendo's first-party software outsold the total third-party software sales combined on PlayStation platforms for 4 straight years (Media Create data for the whole 2020 isn't yet available however 2020 was even more lopsided in favor of Nintendo than previous years).