Privilege can be a hard thing to talk about.
Oftentimes, when it's implied or stated that someone is "privileged," they can feel defensive or upset. They may have worked very hard for what they have accomplished and they may have overcome many obstacles to accomplish it. And the word "privilege" can make a person feel as though that work is being diminished.
The key point about privilege, though, is that it doesn't mean that a person was raised by wealthy parents, had everything handed to them, and didn't have to do much other than show up.