But why is panhandling not so restricted?

Panhandling is generally seen as a freedom of speech. Ironically, working (or attempting to work) is not. This bothers me a bit on principle.