It is cheaper per transistor or they wouldn't do it (this is a fundamental 'law' of the business) - it wouldn't make business sense. Simply by shrinking the node you get more chips per wafer, which directly reduces cost proportional to the shrink, provided your yields are decent. Even if yields start so-so, they'll improve to where they'll make greater profit. Otherwise you just stick with whatever node you're on.
Navi 1 wasn't able to undercut Nvidia because their GPU design isn't as advanced.
The reduction in cost per transistor has been slowing. 7nm cost per transistor was estimated at roughly 70% of what it was on "10nm", but this also comes with added thermal considerations, and the cost per transistor on a larger chip would have been even worse than that because this is an averaged value that doesn't scale linearly with arbitrarily sized dies. AMD had public slides showing that the cost per mm^2 (which is a correlate of cost per transistor, although they obviously aren't identical values) on 7nm was not very favourable.
The economic downsides of using 7nm for large dies were a major factor in why the Radeon 7 (331mm^2) was unable to meaningfully undercut the RTX 2080 (545mm^2). It's a much smaller die, so usually that would imply big savings, but 7nm was just a much more expensive process, even at that scale. Later, the 251mm^2 Navi was only able to undercut Nvidia by a modest margin, even without dedicated RT and AI acceleration on the wafer.
This is not
just technology of AMD versus Nvidia, this is also the reality of using modern processes to produce large dies. It used to be that every shrink was denser, cheaper, faster, and more power efficient. They're still getting denser, but the other three factors are in constant contention with each other.