The Colleton Museum is currently housed in the "Old Jail," which is the third jail in the county's history. The two-story neo-gothic structure resembles a castle and sits like a jewel overlooking one of Walterboro's main thoroughfares. Constructed in 1855-56, it is said to be built of brick from the old Jacksonboro Courthouse. The jailer and his family lived in the structure to oversee the care of the prisoners. The building was remodeled in 1937, a side porch was removed, the walls were stuccoed and painted, and rooms were added to the back.