There is nothing wrong at all with being Vegan, Kabby. And from my point of view, if you have found a lifestyle that suits you and you feel comfortable with then hey, who am I to judge..??
I can see where you're coming from though when you use strong language such as "hate". There are people around who will despise you for what you've done. Don't ask me why they do such things. Their thought processes confound reason.
I've experienced hate crime too - though not for veganism - and I know how unpleasant that is. You haven't done anything to hurt anybody, your way of life doesn't adversely affect anybody and you're not "Ramming it down anybody's throat".
Perhaps though (for example), you may go into a restaurant with some friends and you ask the waiter if there is a vegan option. This gets overheard on another table and it's enough to set some people off. You weren't trying to shine some sort of light on yourself.... sort of "Oh, I'm a vegan so I'm superior to anybody else". But that's what you'll be accused of doing. Just because you made a reasonable request for an alternative menu in a restaurant. You can attract hate crime that easily when you're a little different to other people.
I once had a beer can thrown at me because I wore a rainbow ribbon on the lapel of my jacket. I wasn't flaunting my homosexuality. I was on my way to a Student Union meeting to hear a speaker give a lecture on LGBT campaigns in the media. But wearing the ribbon was enough and I ended up with a bruise on my face and beer splattered all over my jacket. I was going about my lawful business and not affecting or hurting anybody. I've suffered other assaults but all they do is steel my resolve to be myself and not hide from anyone.
So I empathise with you and understand your confusion.
I don't think you should hide your veganism. It isn't anything to be ashamed of. Don't keep it in the closet as if it was some kind of dirty secret.
If you want to wear a vegan symbol, then do so. It's your lawful right. You may attract some adverse comments but hopefully, no violence.
Ignore the idiots who taunt or call you names. They're more to be pitied than scorned. It isn't easy to do this but for what it's worth, I respect your right to live as you choose and I would suggest that the vast majority do too.
In time, more and more people will come to accept vegans. For sure, there will always be morons but ever was it thus so. Don't let them grind you down. Don't let them win.
Little by little, the LGBT community here in UK is becoming more and more accepted. There are still homophobic incidents, but they are diminishing year on year. Take heart from that.
Lastly, I'll say "Good for you" that you've found a way of life that you're happy with and you feel is good for you. I admire and respect you for that.
One last thought. When it comes to the haters, all you have to do is be bigger than them.
Because you'll almost certainly outlive them..!!