YUP Madison honestly is what kills season 2 for me. It seems that most people agree that season 1 was the best, season 2 was the worst, and 3/4 were kinda tied, it just depends on what you like.
Now that I'm thinking of it, if it werent for EA I would probably like season 4 more beause I loved the ephram/andy bits and them getting along. i loved the complete LACK of madison around and that entire storyline. I loved to see E/A coming back in the end. I loved the kyle/ephram relationship as well... so now I'm not sure again.
But i think season 3 just felt more cohesive and maybe in general the episodes were more interesting over all? IDK.
I agree that every season of the show was amazing though, I never wanted to quit watching like I have with some of my other shows lol. And with 4 seasons it definitely didnt drag on. It sucks we didnt get more, but honestly, it was a good amount of episodes.