Sometimes it's hard being handsome and charismatic?

So many girls ask me to stuff like dances, to go out, to hang out and most are flirty in general, some even straight out come onto me. I am nice to them but I am definitely not the player type of guy. I don't date multiple girls or cheat. What is difficult about it is I inadvertently hurt girls' feelings sometimes... what should I do to handle this in the smoothest way possible? I never like to see a girl's feelings hurt but I don't want to lead them on.