First of all the bolded is false; there's no tactile sensation to the symbols on the buttons at all. It is blatantly not part of this and it shows a bit of ignorance as to what it actually is which is an accessibility aid for visual impairment, something a flat symbol printed onto a button (often reinforced with an on-screen visual aid) does nothing to aid with.
Tactile sensation isn't a requirement for a tangible symbol system, simply the association of a object/image to a concept.
It makes sense that when you have been culturally engrained to associate the concept of an O to mean affirmation/positive, and the X to mean rejection/negative, that you don't need to then reintroduce the concept when someone is first exposed to video games. They should intuitively look at the controller, see "O" and "X", and be able to parse which one is probably the positive/confirm and which one is negative/go back. Instead now they're being told the opposite is true for this one case, something that I'm sure people "can" learn to deal with, but still seems ultimately unnecessary to change.
So I honestly don't know what you're going on about because tangible symbol systems are specifically designed for people with general communication issues, not just visual issues. It's literally built around the idea that you need to be able to see the symbols to interpret them for a number of examples, with tactile only being needed when visual issues are brought in.
Looking at my XBO controller, it's a green A in the bottom position and a red B to the right. Standard Western thought surrounding yes/no questions, have them as yes/no and not no/yes. So hitting "A" to confirm (where the "X" on a PlayStation controller is), ascribes to the Western logic of left-to-right reading, of A being primary to B, and of green being affirmative to red's negative.
Button colors were really inconsistent over the years, until companies either abandoned them or stuck with the Microsoft layout.
The Super Famicom controller had A as Red, B as Yellow, Y as Green and X as Blue, with B being bottom and A being right.
N64/Gamecube used A = Green, B = Red and C= Yellow in their unique layouts.
PS1 (and onwards) used O = Red, X = Blue, Triangle = Green, and Square = Purple.
Dreamcast controller was A = Red, B = Blue, Y = Green and X = Yellow, and swapped the X/Y and A/B positions relative to the SNES (so A was bottom). The color scheme actually lines up with the PS line when you think about how it functions in japan, but in the wrong (i.e western) positions.
Xbox controller had A = Green, B= Red, Y=Yellow and X=Blue, but kept the Dreamcast layout (A bottom, Y top), and it seems they're the last holdouts.