Or maybe he's a bit afraid of spending time with them.
I mean, he seems very uptight, and a bit cut off (although still completely lovable), and I get the impression that he's a bit afraid of letting his emotions show. I mean, the only really intense moment of unhappiness he's shown was in TFL, when he was so overcome he had to turn his head away so his daughter wouldn't see him cry. Interesting thoughts......