Well it wasn't so much that he wanted to leave all that time, like "Ugh, I don't wanna do this, get me away from these people!!".
But when NBC/Warner Bros. had come to the cast about extending their contracts and getting more money, George basically said "No thanks. I'll honour my original contract and I think 5 years will be enough for my character" which it was. John Wells has said they had no plans for Doug after Season 5.