Why not talk to him about it? :) That way, you'll have a better understanding of his side and you two could start working things out. Based on my experience, when I let disturbing remarks like that slide, the conflicts just got worse because I didn't air them out when I should have.