How do you feel about people taking your picture in a public place?

I take a lot of pictures, have several nice cameras and travel quite a bit. Sometimes I will feel that someone has a unique look or is near something that I find interesting but hesitate to take the picture or don't take it at all for fear that people will object to having their picture taken. I'm rethinking this, however, because I'm missing a lot of good shots. How would you feel if it were you in my camera's lens in the aforementioned public place?