The [almost] complete lack of Golden Sun mentions in this so far is mind-boggling to me.
Multilevel types of puzzles which switch from vertical to horizontal. Puzzles needing the player to use their magic in the field. Zelda-esque dungeon centered setpieces. Fun secrets like trying to find hidden weapons or djinn. Secret summon bosses. Just absolutely spectacular. There were random encounters here, but I never felt like it was overbearing in that regard. The flaws of that, such as they are, come more from the weaknesses of the GS combat systems, like the slackening need to use psyenergy or general ease of it all. The dungeons themselves, however, are as top notch as they come.
Alternatively, I'd also point to the likes of Divinity: Original Sin 2. Here, there are no random encounters at all, each one instead being carefully placed and crafted, but I think of stuff like Wrecker's Cover, and I remember what makes it great. A ton of different ways in. The party being tossed into different scenarios with each needing a different way of escape - picking a lock, having to be rescued by everyone else, fighting your way out, or sneaking your way past enemies. Dozens of secrets large and small, between quest progression or just teleporting a chest into reach. In the inverse of GS, the strengths here come from the spectacular inventive nature of DOS2 systems, but it can be broadly approximated elsewhere.
So what makes these work? I can say what I'd want to see:
- Puzzles. Real thinker type puzzles too, if you can swing it. Ones where you need to pay attention to context and capabilities of the party to succeed, not just "find a key somewhere else". Maybe not La Mulana hard, but I want to see designers try.
- Different scenarios. Boss rushes. Splitting the party up and needing to get everyone together. Maybe have two groups which are on a timer to complete two halves of the same goal. Maybe one group needs to rescue another from increasingly tough waves of enemies. Getting lost is entertaining exactly once.
- Different approaches. Maybe there are different ways into a dungeon. Maybe those above scenarios can be solved differently. Maybe the powersets of the characters can be applied in different ways. Maybe a boss can be brute forced, whereas another player can poison them or whatever.
- If you *have* to do a maze, don't have it just be...a maze. Switch up planes of travel, or ways to look at the shape of the maze, or moving pieces around. Maybe have one where you need to make a map for yourself, sure, but if that's the case, do not just keep repeating the trick. Warp dungeons or the like are about as fun as pick-a-pipes in Mario maker.
Random encounters can go either way. In the immediate sense, sure, I've had times when I wished an encounter wasn't random, or that I could avoid the enemy. But as someone pointed out, "enemy on field" types have a tendency to become like an action game, which you're not trying to do here. Randomly being attacked by a foe is part of the oppression of a dungeon in the first place. I think it's best if there are ways to mitigate frequency, or better yet, if things are carefully placed ala the CRPG world. That's far harder to do, so in the meantime, I'll just say - there can be a time and place for random encounters.
The absolute worst dungeons in the world are ones where you just wind up getting lost, with nothing gained or variety experienced. Dungeons like the ones in Dragon Quest 8, for instance, absolutely suck. If the only thing you're doing in a dungeon is walking around and fighting, that's a bad dungeon. Seeing praise for stuff like SMT dungeons, ie, just sprawling, pointless mazes with nothing to do except leave...blows my mind.