I was a little surprised to learn that Sly's main problem with his father was that Kenneth never connected to him because he wasn't the son he wanted and that essentially he was an absentee father. Based on statements Sly made in earlier episodes and even his fearful reaction at the beginning of Sun of a Gun, I always assumed his father was more overtly abusive. Not that failing to accept and be there for his son isn't awful, but for someone as risk-averse as Sylvester to make the decision to leave home and be on his own as a teenager, I would have thought his childhood was even worse.