That still doesn't mean you didn't have an impact on him. Some kids develope a bond and will never show it.
All I can say is I can't imagine how tough it must be on the child in a single parent home. I grew up in a very loving family. Nobody I ever even knew got a divorce until after I was grown. It was hard for me to understand where these kids were coming from at first. But I was never involved long enough to make much of an impact on them so I'm sure I was more of a negative in their lives than a positive.