Few houses in Savannah have a darker reputation, and nearly all of it comes from the 1960s. As the stories go, the move went badly from the start. Part of the neighboring structure collapsed and killed a workman. Retellings have since raised the toll to three, though Destination Ghost Tours notes that documented accounts mention a single fatal accident. Workers were also said to have found an old tabby crypt, half full of water, when they lifted the house. They told Williams it was empty and covered it back over.

During the restoration, the legend continues, workers heard voices and footsteps in the empty house, and some refused to come back. In a story told on many tours, Williams and friends heard footsteps upstairs, climbed toward them, and heard them move above them again, all the way up to the widow's walk. Then the sounds came from below. Another tale tells of a friend who went up alone to investigate. He was found face-down on the top floor, soaked in sweat, saying he had felt plunged into icy water and pulled toward the open chimney shaft. One local telling says a neighbor once phoned Williams, who was away, to report what sounded like a party in the empty house. Some still claim to glimpse a man in a dark suit at the upper windows.

The most famous part of the legend is the exorcism. According to these accounts, on December 7, 1963, Bishop Albert Rhett Stuart of the Episcopal Diocese of Georgia performed a 45-minute rite at the house. Within a week or so, the stories say, the footsteps and voices were back. Ever since, Hampton-Lillibridge has been called one of the most haunted houses in America, and it is a regular stop for trolley and walking tours.