The Andrew Low House, at 329 Abercorn Street, is the oldest building on Lafayette Square. It was designed by the New York architect John S. Norris, who also designed Savannah's Custom House, and completed in 1849. Andrew Low II had come from Scotland as a teenager to work in his uncle's cotton firm, and he became one of the wealthiest merchants in Georgia. His first wife, Sarah Cecil Hunter, and their young son died before the house was finished, according to Ghost City Tours. Within a few years he married Mary Cowper Stiles, daughter of the diplomat William Henry Stiles. The house is Italianate in style, with ornate cast-iron balconies and railings.

The English novelist William Makepeace Thackeray stayed here on his American lecture tours in the 1850s, and Robert E. Lee visited the Lows. The house's most famous resident was Juliette Gordon Low, Andrew's daughter-in-law, who founded the Girl Scouts of the USA in 1912. The carriage house next door, at 330 Drayton Street, was the Girl Scouts' headquarters until 1913. She died in the house on January 17, 1927.

Today the Andrew Low House is a museum open to the public, with guided tours, gardens and one of the South's finest collections of 19th-century silver, porcelain and furnishings. Tours also tell the stories of the enslaved men and women who lived and worked on the property. The museum focuses on history, not ghosts, so save the ghost talk for an evening tour.