Usually if there's another person standing in front of a crosswalk when I get there, I don't push the button, because that seems kind of insulting. It's basically saying to that person, "hey, I think you're probably too stupid to have pushed the button before waiting for the light, so I'm going to push it again." But, of course, that ends immediately once I've waited through three green lights waiting for the walk signal, and then I give up on not trying to be insulting for a while.
If a light came on when the button was activated, this thing wouldn't be a problem. Many of the lights around here even have a light, but it's only on when the button is depressed.