i understand what you mean and i think its only a natural feeling. but you can't say for sure how people will react. its just a chance you have to take. you can be the most beautiful person in the world but a beast lives inside ! I think we all know atleast one person like that.
all you can do is just be you. you have a PWS. thats just a part of you but its not all that you are. Try not to let what others may think concern you. Its your life to live not theirs.