It is your choice how to take it. You can choose be offended or have your feelings hurt or just consider the source of the comment and choose to let it slide. It is completely up to you. You can't control what others do but you can control how you react.
Yep, no issues with him strangely, others seem to, maybe he's high IQ aswell and you just don't get him