I did the same thing!
I discovered Buffy during the summer after its first season aired.
So, yeah, I didn't like Xander all that long either, except that the soul bit was at the end of Season 2. So there's one season where I liked him.
Didn't really hate him when he was with Cordelia either. But, you know, I would have prefered that he spill the beans.