The crux of the issue is that for you "engaging" and "downtime" are mutually exclusive. While there's definitely bloat that you're describing, and it's never a sustained rollercoaster the way Among Thieves is, each section definitely has a rise in tension and escalation of combat of environmental obstacles, as well as an emotional arc. It's more of a "chunked" up game (which makes sense given the length) where each major area has a build and crescendo. There are more stretches of silence and banter than Lost Legacy because of the expanded scope, but there are more moments that illuminate Sam and Nate's character than just that island section. And I think the writing for the more illuminating character moments is stronger than The Lost Legacy, which I found to be a bit on the nose with regards to its periodic character info dumps.
"Engaging" and "downtime" aren't mutually exclusive to me. Village chapter in 2 is engaging because it's special first and foremost—walking around a small residential area is not common and repeated throughout the game. From a mechanical standpoint, it's filled with new, personal interactions (greeting people, kicking the football/playing hide-and-seek with kids, petting a cow) which contrast with all the bombast and violence preceding it. From a story perspective, the player is put right in the protagonist's shoes and are naturally drawn to explore to figure out where they are and/or how they got there.
It's why I always try define 4's downtime as being "forced," "scripted," and/or "frequent," to delineate it from, say, slower moments in Resident Evil 4 where the player has options on whether they move, gather items/intel, upgrade weapons, and so forth. Even stuff like finding the mass grave in Uncharted 2's Borneo chapter involves a novel "use blue flame to follow blood trail" mechanic to help make it standout—even when downtime isn't left up to the player, it can still have something novel or clever to keep it from feeling like "push forward until cutscene."
4's frequent downtime is often lengthy climbing up a cliff face with nothing distinct about it relative to all the other similar minutes-long segments. Or it has the player looking through drawers for some bolt cutters, and you know they're always in the last drawer, or Nate cuts the player off during a puzzle because the scripted sequence wants you to go look at paintings, even if you already know the solution. If the climbing was going to be risk-free, it could've at least allowed the player agency to choose where to climb or have different branching paths to the next area. If the player's forced to wait for an NPC to find a bauble that leads into a cutscene, give them useful items to find in the meantime—anything to keep it from feeling like going through motions that amount to QTEs (minus timing/tension).
And I don't doubt there are some more bits of illuminating dialogue between Nate and Sam outside of when they first reunite, the boating segment, and their sit down in Libertalia, but it's diluted among fluff of restrictive downtime and traversal. What character moments are there doesn't justify or enhance all that empty, numbing pap at all.
I purposefully don't frame the issues with UC4 as merely being "it's too slow" or "there's not enough action," because the slower, more grounded segments could've been better had they not also been restrictive, or if they all offered more to the player than nice scenery.
There's plenty of action in 4. I even consider many of its battle arenas to be way better than what's in Lost Legacy (chapter 14's, the one from the early gameplay demo shown years before release, is exquisite) but the balance between it and the downtime is off, and that downtime takes away much of the agency the player is afforded in the combat arenas and setpieces. I can choose to sneak around enemies until they spot me, clamber up a tower to start sniping, leap out as they toss grenades and swing around to drop on the last enemy, which can be completely different from what another player did, or different from another segment later in the game. But then ... I have to push a crate over to another wall, or I have to climb up this single path on a cliff face with nary an opportunity to mess up in any way. The latter sequences give me near-zero agency, but it's about as common as the former, more engaging one, even though they didn't have to be so dull.
Again, I avoid dumbing down an experience with a game to "is it full of action or not" because a game chock full of frequent action could also be dull if it doesn't mix things up or give the player a lot of choice, or at least meaningful choice. If a sequence in a game feels meaningful, it can be complicated, simple, action-packed, slow, grand, or intimate. However, 4's downtime often feels meaningless, as of it's some formality, or vestigial element from a previous build that couldn't be outright removed.