I feel like I'm being singled out at work?

Have you sat down with your boss and asked him this question, face-to-face? If you just pretend there's no problem then he doesn't have to admit he's a part of it. If this doesn't work, who's your boss' boss?