His second wife, Annie E. Wigg — a Charleston schoolteacher he had married in 1890, five years after Hannah’s death in 1883 — had predeceased him in November 1895. His daughter Sarah Voorhies, born in 1863 and living until 1920, outlived him. His son William Robert Smalls, born of his second marriage in 1892, lived until 1970.
The Planter itself had outlasted the war but not by much. On March 25, 1876, while towing a grounded schooner, the ship sprung a plank and was beached for repairs. Stormy seas that night damaged it further as the tide rose, and it was abandoned. It sank on March 26. Smalls is reported to have said that losing it felt like losing a family member. In May 2014, NOAA announced it believed it had located the wreck.
The house at 511 Prince Street, where his mother Lydia had worked as an enslaved servant and where he had raised his family after the war, is now the Robert Smalls House, designated a National Historic Landmark. It stands in Beaufort, where it has stood since before he was born.
His pension from the federal government, which took until 1897 to secure through a special congressional act, was set at $30 a month — the equivalent of a Navy captain’s pension. The 1883 bill that would have placed him on the Navy retired list had been halted, with accounts at the time attributing the blockage to racial bias. The 1897 pension represented a partial and belated recognition by the federal government of the wartime service he had rendered.