Tell me this isnt loitering at the gym?

Why aren't you complaining to a manager? If it's as bad as you say, then you should definitely complain. However, I find it very odd that this would happen to you all the time and nobody else. Could it be that you are staring at people without realizing it? That would make me very uncomfortable. If not, then you should speak to the manager or go to another gym.