Fast forward to the release of Avengers 1. Suddenly everyone was gushing over the movie. And I had to admit, watching the trailer it looked fun. But apparently you had to watch all other previous movies to really "get the experience" so to speak. Yeah, I had no interest and doing that, especially since I thought Ironman was mediocre and the others movies were apparently worse.
I always found it funny how much the crossover was emphasized, because in hindsight you really didn't need to watch the rest of Phase 1 to understand Avengers 1, just Iron Man. I saw it without seeing Cap, Thor or Hulk's movies and it was surprisingly coherent. Captain America? You're told that he was from the past during the movie. Thor? Thor and Loki are gods and brothers, boom, explained in movie. Hulk? He's a different character here, his movie is basically irrelevant. Those movies are kind of fun for context, but the only thing you need to know about them is that their title characters are in the movie. There's a detail here and there that might not make a ton of sense (Thor knows Dr. Selvig? Oh ok) but there's nothing from those previous films that affects the story. Both with and without those other movies, the plot of Avengers 1 is effectively "a super powerful god wants to rule humanity, so his brother and a couple of superheroes assembled by the government attempt to stop him."
The only character that didn't have setup was Iron Man, because the movie kind of assumes you know who he was (he was the most popular of the four at the time). The biggest thing with Phase One was that they weren't confident anyone was going to watch all four movies despite Iron Man's success, and it shows in how Avengers takes a lot time telling you about everyone EXCEPT Tony Stark, because he already had two movies to himself.
To answer your question, if you haven't watched past Phase One, watch whichever movies interest you from it. In fact, I'll deviate from a lot of recommendations and say that if you want to watch Age of Ultron without the four movies leading up to it, you actually don't miss much. Without spoiling it, Ultron pretty much pretends that the ending of Iron Man 3 (and in a few ways, Winter Soldier) may as well not have happened, Thor 2 isn't referenced from what I can tell, and GOTG is a standalone film, so aside from a few lines here and there you could theoretically watch UIltron without seeing the rest of the phase. I do think it's a fun movie, it's just a bit of a mess.
Personally I'd recommend watchin Guardians of the Galaxy or Winter Soldier (a fun, mostly standalone, action-adventure movie and an MCU plot driver featuring characters from the Avengers). Of the movies in Phase 2, these are the sort of films to expect from the MCU going forward. I'd also recommend Iron Man 3, but that's just because I think it's neat. It's actually a good movie, but again, the rest of the series basically pretends it didn't happen (and some people are still pissed about how its villain was handled).