Where's the Logic?

It is commonly held among modern philosophers that we human beings do not really have free will. Every choice that we make, so they say, is dictated by a personal nature that we did not choose and by circumstances beyond our control. From this premise comes the argument that we ought not to punish criminals, because they cannot act otherwise than they do and are therefore not responsible for their actions. Do you see a flaw in this? Why do we as punishers have an option when the criminal does not? Wouldn’t we simply punish him or pardon him according to our circumstances and our unchosen nature? If we are able to choose, so is he. The premise cannot withstand the argument.