In the early 1900s the islands were accessible only by boat. It was said that only lovers made the effort to get to this romantic island, thus the name Lovers Key. A road to the island was built in 1965. Local legend hints that Black Island got its name from Black Augustus, a pirate who had been captured by authorities, and later escaped, making this island his home for the remainder of his life. Fish camps were located on Black Island from the early 1900s until the late 1950s.
In the 1960s and 1970s, the four barrier islands were slated for development. Preparations for development damaged the islands. Mangrove swamps were altered to uplands by dredging a canal through Black Island.
In 1983, the state acquired the islands and in 1996, merged with adjacent Carl E. Johnson County Park to become Lovers Key Carl E. Johnson State Park.