"Given the opportunity, players will optimize the fun out of a game." - Soren Johnson, Lead Designer on Civ IV
I get where you're coming from OP. Accessibility is nice, but mechanics that dramatically alter a game in the name of accessibility should be opt-in (or at least opt-out). Or if you don't even want to allow the player to opt out, you could at least give some inconsequential reward for not using the feature (though I think this is the weakest solution).
Contrast this with Celeste, which has an excellent suite of options to make the game easier. In Celeste, you have to opt-in, the game tells you, "This is fine, but not intended," and you get a little emblem on your save file saying that you used the accessibility modes. None of those are a big deal, and shouldn't deter someone who wants/needs those features, but it makes a clear line that a player has to cross. That line is important, because without it, players who want the difficult experience are primed to rob themselves of it. At that point, you're making the game better for one group while making it worse for another, when it would be easy to design it so that both groups have a good experience.