As has been mentioned, Adventure EXP is not wasted since it's accumulated and you will jump to whatever level you would have got once you do level up. The other downsides are that rewards are worse at lower world levels (less money / books from ley lines, slightly less / lower rewards from world bosses), and that you can't access certain domain levels with a low world level.
In general I would strongly recommend to keep your world level where you're confortable with it, and only raise it when you start to feel like world enemies are boringly easy. That's what I did: I kept it at like WL5-6 for a couple months until I felt ready and increased it (I'm now WL8 / AR56). Fortunately you can now dial back one world level (only one, sadly) if you raise it and feel the game is too challenging. Hell, if you're finding the game a challenge right now, you can even dial the world back one level and see if that helps.
It's a game, do what's the most fun for you. Don't just increase world level because you "should" or to turn it into some dumb dick measuring contest.
Nah, this aint it chief. I knew all of these things from day one but the game was still challenging (fuck you, Oceanid) until I rolled a decent number of characters, raised their levels / talents, and decked them with decent weapons and artifacts. People just conveniently forget how long that takes once they're steamrolling enemies later on.
No offense but I will never understand this mentality. "Grit your teeth and play in a way you find unconfortable so that... you can win some race with strangers and burn through the game in less time?".
