Harry Caul (Gene Hackman) is a surveillance expert who runs his own company in San Francisco. Caul is obsessed with his own privacy; his apartment is almost bare behind its triple-locked door and burglar alarm, he uses pay phones to make calls, claims to have no home telephone and his office is enclosed in wire mesh in a corner of a much larger warehouse. He has no friends, his mistress Amy knows nothing about him, and his one hobby is playing along to jazz records on a tenor saxophone in the privacy of his apartment.

Caul insists that he is not responsible for the actual content of the conversations he records or the use to which his clients put his surveillance activities. However, he is racked by guilt over a past wiretap job which resulted in the murder of three people. This sense of guilt is amplified by his devout Catholicism.

Caul, his colleague Stan and some freelance associates have taken on the task of bugging the conversation of a couple as they walk through crowded Union Square in San Francisco, surrounded by a cacophony of background noise. Amid the small-talk, the couple discuss fears that they are being watched, and mention a discreet meeting at a hotel room in a few days. The challenging task of recording this conversation is accomplished by multiple surveillance operatives located in different positions around the square. After Caul has merged and filtered the different tapes, the final result is a sound recording in which the words themselves are clear, but their meaning remains ambiguous.

When the client is not in his office, Caul refuses to leave the tape with the client’s assistant. The assistant warns him not to get involved, telling him that the tapes are “dangerous.” Caul feels increasingly uneasy about what may happen to the couple once the client hears the tape. He plays the tape again and again, gradually refining the recording. Using a filter, he reveals a key phrase hidden under the sound of a street musician: “He’d kill us if he got the chance.”

Caul avoids handing in the tape to the aide of the man who commissioned the surveillance. Afterward, he finds himself under increasing pressure from the client’s aide and is himself followed, tricked, and bugged. The tape of the conversation is eventually stolen from him in a moment when his guard is down. He goes to the client (“the Director”, played by Robert Duvall) to find he has received the tapes, and learns that the woman in the recording is the client’s wife, apparently having an affair with the other man in the tapes.

Caul books a hotel room next to one mentioned in the recording of the conversation. He uses equipment to overhear the client in a heated argument with his wife. When he goes to the balcony to watch the events through the windows out of curiosity, he sees what he believes to be the wife being murdered and retreats in shock.

Caul later attempts to confront the client, but the client is absent. While departing, Caul catches sight of the wife, alive and unharmed, in a limo. He learns that his client was killed in an “accident,” and discovers the truth: the couple were talking about killing the woman’s husband, and the murder Caul witnessed was actually that of his client and not the wife. It is revealed to the viewer that the man in the recording actually said, “He’d kill us if he got the chance.”

Caul gets a phone call from his client’s assistant, who tells him not to look any further into the matter, and says, “We’ll be listening to you.” Caul goes on a frantic search for a listening device, tearing up his apartment to no avail. He sits amid the wreckage, playing the only thing in his apartment left intact: his saxophone.