Or, he could just be pulling your leg with an obtuse rumor comment. I wouldn't worry about it much.
I know what I'd be thinking. I'd be thinking, "this person is responsible for my grade in this course?"
I faced a similar situation many years ago, in graduate school. One of my instructors liked to make depreciating remarks about Macs and Mac users while instructing in the classroom. I didn't mind her being an idiot about that particular issue so much as I minded her having anything to do with my grade in the course.