Not just that, but he was also still with Buffy when spuffy started getting big, so both sides just hated him - he literally couldn't have been unluckier when it came to timing.
Still, he was a good guy at heart who wanted to protect the world as best as he could, & he was exactly what Buffy needed at that time in her life whether people want to admit that or not