The story opens in an exiled Shanghainese community in British Hong Kong in 1962. Chow Mo-wan (Tony Leung), a journalist, rents a room in an apartment of a building on the same day as Su Li-zhen (Maggie Cheung), a secretary from a shipping company. They become next-door neighbours. Each has a spouse who works and often leaves them alone on overtime shifts. Due to the friendly but overbearing presence of a Shanghainese landlady, Mrs. Suen, and their bustling, mahjong-playing neighbours, Chow and Su often find themselves alone in their rooms. Their lives continue to intersect in everyday situations: a recurring motif is the loneliness of eating alone. The film documents the leads’ chance encounters, each making his and her individual trek to the street noodle stall, sometimes intersecting without connecting.
Chow and Su each nurse suspicions about their own spouse’s fidelity; each comes to the conclusion that their spouses have been seeing each other. Su wonders aloud how their spouses’ affair might have begun. Su and Chow re-enact what they imagine might have happened.
Chow soon invites Su to help him write a martial arts serial for the papers. Their neighbours begin to take notice of Su’s prolonged absences. In the context of a socially conservative 1960s Hong Kong, friendships between men and women bear scrutiny. Chow rents a hotel room away from the apartment where he and Su can work together without attracting attention. The relationship between Chow and Su is platonic, as there is the suggestion that they would be degraded if they stooped to the level of their spouses. As time passes, however, they acknowledge that they have developed feelings for each other. Chow leaves Hong Kong for a job in Singapore. He asks Su to go with him; Chow waits for her at the hotel room and then leaves. She can be seen rushing down the stairs of her apartment, only to arrive at the empty hotel room, too late to join Chow.
The next year, Su goes to Singapore and visits Chow’s apartment. She calls Chow, who is working for a Singaporean newspaper, but she remains silent when Chow picks up. Later, Chow realises she has visited his apartment after seeing a lipstick-stained cigarette butt in his ashtray. While dining with a friend, Chow relays a story about how in older times, when a person had a secret that could not be shared, he would instead go atop a mountain, make a hollow in a tree, whisper the secret into that hollow and cover it with mud.
Three years later, Su visits her former landlady, Mrs. Suen. Mrs. Suen is about to emigrate to the United States, and Su inquires about whether the apartment is available for rent. Some time later, Chow returns to visit his landlords, the Koos. He finds they have emigrated to the Philippines. He asks about the Suen family next door, and the new owner tells him a woman and her son are now living there. He leaves without realising Su is the lady living there.
The film ends at Siem Reap, Cambodia, where Chow is seen visiting Angkor Wat. At the site of a ruined monastery, he whispers for some time into a hollow in a ruined wall and then plugs the hollow with mud.