Agreed
If you're going to create conflict, then at least have a basis for it.
Cordelia expressed deep jealousy over Buffy/Xander in 'Killed By Death', and even Angelus brought it up ("you still love her"), so I would have found Xander/Cordy's break-up much more believable had it happened over the long-term Buffy issue.