More than a dozen forts were built to defend New York Harbor at the time of the War of 1812. The Southwest Battery was constructed on the rocks off the tip of Manhattan Island between 1808 and 1811. Although fully armed and staffed, the fort never had occasion to fire upon an enemy. In 1817, the fort was renamed Castle Clinton in honor of DeWitt Clinton, Mayor of New York City. The army vacated the fort in 1821 and the structure was deeded to New York City in 1823. In the summer of 1824, a new restaurant and entertainment center opened at the site, now called Castle Garden. A roof was added in the 1840s and Castle Garden served as an opera house and theater until 1854.

On August 3, 1855, Castle Garden, now leased to New York State, opened as an immigrant landing depot.

During the next 34 years, over 8 million people entered the United States through Castle Garden, until it was closed on April 18, 1890. The building was altered once again and reopened as the New York City Aquarium on December 10, 1896. It was one of the city's most popular attractions until it closed in 1941.