The Scarbrough House at 41 Martin Luther King Jr. Boulevard (old West Broad Street) was built in 1819 for William Scarbrough. He was a North Carolina–born shipping merchant educated at Edinburgh, and he became president of the Savannah Steamship Company, whose SS Savannah became the first steamship to cross the Atlantic that year. His architect was William Jay of Bath, England, then only in his twenties. The house, which Scarbrough called 'The Castle,' is the earliest example of domestic Greek Revival architecture in the Deep South and one of the few surviving works by Jay. Scarbrough hosted President James Monroe there in 1819.

The steamship was not a commercial success. In 1820 Scarbrough was declared an insolvent debtor, and the house was sold at auction to his brother-in-law, Robert Isaacs, who let the family stay on. Julia Scarbrough became legendary for her lavish dinner parties. Ownership fights, involving their son-in-law Godfrey Barnsley, went all the way to the U.S. Supreme Court. Ships of the Sea's historical interpreter notes the house was sold in 1850 to the O'Byrne family, who lived there until 1865.

In 1878 the house was given to the Board of Education, and it served as the West Broad Street School for African American children until 1962. After years of vacancy, the Historic Savannah Foundation rescued it, and it became a National Historic Landmark in 1974. Since a restoration in 1996–97 it has been the Ships of the Sea Maritime Museum, with ship models, maritime art and large gardens. It is open to the public with admission.