In 1930s Shanghai, petty crooks Sing and Bone aspire to join the notorious Axe Gang, which rules the town with an iron fist under the leadership of the cold-blooded Brother Sum. The two visit Pigsty Alley, a rundown slum, where they claim to be Axemen and attempt to extort the residents before being chased off by the slum’s Landlady. In an attempt to bluff them, Sing throws a firecracker, which blows up the hat of a minor Axe Gang boss passing by. Sing blames the residents for throwing the firecracker, and the boss attacks a barber, only to be struck by an unseen assailant. The gang then calls for reinforcements while three of the tenants — Coolie, Tailor, and Donut — reveal themselves to be kung fu masters and triumph over the gang members. Fearing the Axe Gang’s retaliation, the Landlady quickly evicts the trio.
Infuriated by the loss of his men, Brother Sum captures Sing and Bone, intending to kill them for posing as gang members. Sing narrowly frees themselves with his lock-picking skills, impressing Brother Sum, who allows them to join the gang once they commit manslaughter. Lamenting his failure, Sing relives his childhood to Bone: he was duped into believing he was a natural-born kung fu genius by a vagrant selling a martial arts pamphlet, and as a result he gave the vagrant his meager savings to purchase the pamphlet. Sing practiced the Buddhist Palm technique and attempted to save Fong, a mute girl, from bullies, only to be beaten himself and humiliated when the bullies point out the pamphlet costs a mere 10 cents. Sing then became adamant the good guys never win and resolved to be a villain.
The two return to Pigsty Alley to kill the Landlady, though their plan backfires and they barely escape. Seeking safety in a traffic pulpit, Sing is surprised as his body rapidly heals from the injuries sustained during the escape; the pain causes him to strike the sides of the pulpit with extreme force, covering the surface with hand-shaped impressions.
Brother Sum hires The Harpists — two assassins who use a magical guqin – to kill the Pigsty Alley masters. They arrive at the alley undetected at night and overwhelm the three masters, who were in the process of leaving. The Landlady and her husband, the Landlord, intervene and reveal themselves to be kung fu masters as well. They defeat the Harpists and send a stern warning to Brother Sum before evacuating Pigsty.
Frustrated with his increasing failures, Sing attempts to rob an ice cream vendor but inadvertently discovers that she is actually Fong. She offers him a lollipop as she did in the past, but he swats it away, ashamed of how far he has fallen. After shooing Bone away, Sing is brought to Brother Sum. The gang leader offers Sing instant membership into the gang if he can free the legendary assassin Beast from an asylum. Sing is successful, and the Beast is brought to the gang’s headquarters.
Brother Sum is initially skeptical of the Beast, whose flippant attitude and sloppy appearance annoys him. However, the Beast quickly reveals himself to be a lethal and skilled fighter, and he destroys half of the casino next door to confront the Landlady and Landlord, who are waiting to settle the score with the gang. The three engage in a vicious battle that ends in a stalemate when the three are immobilized through a inter-joint lock. Sing, pressured by Brother Sum to attack the Landlord and Landlady to help the Beast, has a change of heart and attacks the Beast instead. Infuriated, the Beast pummels Sing, who is narrowly rescued by the Landlord and Landlady and the three escape. The Beast casually kills Brother Sum for berating him, and assumes control of the Axe Gang.
Back at Pigsty Alley, the Landlord and Landlady treat Sing’s wounds and are surprised when he recovers at top speed. Sing then treats the two’s wounds and takes them into a quiet room to rest before meeting the Axe Gang outside, who are waiting for them. The Landlady deduces Sing is indeed a natural-born kung fu genius when he effortlessly dispatches the gang members before confronting the Beast. The Beast initially gains the upper hand and sends Sing flying high into the sky. Sing comes into inner peace by envisioning Buddha in the clouds before arriving back down to incapacitate the Beast. The Beast admits defeat and bows down to Sing.
With the Axe Gang disbanded, the townsfolk return to their peaceful lives. Sing and Bone open a candy store with Fong’s lollipop as their logo. When Sing sees Fong outside, he happily invites her in to relive their childhood memories. The same vagrant who sold Sing the pamphlet entices another child just outside the store, only this time he is selling multiple pamphlets teaching different styles.