There are significantly fewer things a car can do than say a NPC human or creature which makes tweaking its AI, even when it's a "black box", much simpler in that regard
Yeah, but then that becomes a fundamentally different game. The current impositions on the number of actions of enemies, where they show up and where they can roam are all carefully designed to create specific combat encounters with specific difficulties. For instance, if enemies could just follow you wherever you could easily end up in a situation where enemies overcrowd you in an area, or specifically designed "rests" between combat encounters becomes broken, etc.
In terms of your second point, that's also fundamentally different from how these enemies are designed. These action games are fundamentally about pattern recognition. Recognizing an enemy's startup animation, recognizing its range/reach and any quirks in its hitbox, recognizing its recovery time. All these aspects mean that each type of attack involves carefully tweaking of every level of its animation. If an enemy with a spear doesn't have a way to cover for its weakness, that's not a failing of AI or an oversight. It was designed that way to allow users to find that weakness and exploit it. Now yes, there is something to be said about giving enemies enough actions that finding their pattern isn't super simple, but you're never going to get a game like a From game where the enemy can perform arbitrary movements of their weapons, because that's not the game they design and there would be no sense of the player actually improving or getting better at the game if that were the case
I guess my point is that the "good" AI, as you define it (in this situation at least) isn't a matter of devs not being able to, it's a matter of them not wanting to. It breaks the game flow in some cases. It breaks the level design in some cases. It breaks the idea behind specific types of combat and how the games are designed. That isn't to say there isn't room for it. Back to that second point, if an enemy has too few patterns it doesn't become interesting to fight either. So there's definitely a balance between too few, and so many that there's no way to learn their patterns.