I agree - I think he would've seen Lily smiling at him, letting him know that he was forgiven. It's losing her friendship that devasted him, then when she died ... it broke him.
Anna: I'm not sorry, you know. I'd marry you now if I wasn't already your wife.
Bates: I'm not sorry either. Maybe I should be, but no man can regret loving as I have loved you.
BATES & ANNA (Downton Abbey)