Ghost City Tours says the fort should be 'treated with great respect,' and visitors who come only for the history sometimes leave convinced it's haunted. Many describe a sudden chill and an uneasy feeling that someone might step out at any moment. The most common report is phantom footsteps on the brick and along the walls. Believers attribute them to soldiers still on watch over the river.

The fort's best-known story dates to the last days of Confederate control. As the legend goes, an Irish private named Patrick Garrity resented his lieutenant, George Dickerson. Some tellers say Dickerson mistreated the Irish Catholic men under his command. One day, while Garrity stood guard at the drawbridge, Dickerson returned to the fort, and Garrity attacked him with his musket. He is said to have struck so savagely that he fractured the officer's skull in four places and broke the musket in half. When the lieutenant's men came running, Garrity leapt into the moat to escape and drowned. Dickerson survived but never fully recovered, and he gave up his command.

Over the years, according to the tour, people have reported a figure in the spot where the attack took place, and many believe it is Garrity. One witness, Ghost City Tours says, was the fort's site manager at the time, Greg Starbuck, who had an encounter there late one day.