I doubt the game being on Xbox One is the main issue. Phil stressed a lot how cross gen games wouldn't be held back and I don't think he would have used definitive statements like "Frankly, held back is a meme that gets created by people who are too caught up in device competition" if Infinite's problems were down to the game coming out on current gen.
Phil is clearly wrong then.
You can't make a cross gen game that has to perform at 60fps on the base Xbox One and not expect it to hold back the next gen version.
Infinite would look substantially better without having to conform to last gen standards. Even just dropping 60fps from the Xbox One version would probably help things enormously.