The Olde Pink House stands on the northwest trust lot of Reynolds Square, at 23 Abercorn Street. Begun in 1771 and completed in 1789, it was first called Habersham House after its owner, James Habersham Jr., an important early cotton factor and the son of the colonial merchant and acting royal governor James Habersham. James Jr. and his brothers Joseph and John supported the Revolution against their Loyalist father. Joseph famously arrested Royal Governor James Wright in 1776. James Jr. helped finance the cause, served as Speaker of the Georgia General Assembly in 1782 and 1784, and sat on the 1785 board that founded the University of Georgia. He lived in the house until his death in 1799. The house survived the fire of 1796, which destroyed 229 buildings.

In 1812 the house became the Planters Bank, the first bank in Georgia, and the Doric portico and a north wing were added. The stucco over the red brick kept bleeding through as pink, and the color eventually became the house's identity. After the Civil War it was an architect's office, a law office, a bookstore and, from 1930 to 1943, Alida Harper Fowlkes's Georgian Tea Room. Jim Williams bought and restored it in 1968. Herschel McCallar Jr. and Jeffrey Keith bought it in 1970, jacked up the building to repair its foundation, uncovered the original 18th-century cooking fireplaces in the basement (now the heart of Planters Tavern), and opened the restaurant in 1971.

Today it is one of Savannah's best-known restaurants, with thirteen dining rooms and the cellar Planters Tavern. It reopened after a December 2018 fire in the upstairs ballroom closed it for four months. Reservations are recommended.