I loved that too!
Because when he left I could understand it after all he went through hell and it was a chance to get back to his family, but a part of me was disappointed that he would just leave Brienne there. Then when he turned back I really liked him, and we could see that he actually wasn't just a bad guy