Beginning in 1859, a burgeoning menagerie was located in and around the Arsenal. Gifts or loans of animals by the likes of impresario P. T. Barnum, financier August Belmont and Civil War general William Tecumseh Sherman could be seen in outdoor cages and in the basement of the building. The "great insecurity and danger" of this arrangement, as well as the stench, resulted in the removal of the interior cages by 1871.
A Gallery of Art, later relocated to the former Convent of Mount St. Vincent at the park's north end, graced the first floor space. Prior to moving to the park's Belvedere Castle, the Municipal Weather Bureau's instruments were atop the Arsenal roof from 1869 to 1918.
Various designs for remodeling the Arsenal emerged over time. Jacob Wrey Mould, the Architect responsible for much of the ornamental detail at Bethesda Terrace and elsewhere in the park, remodeled the Arsenal interior in 1870. An Arsenal restaurant existed early in the 20th century; plans for its renovation appear in an official parks report of 1916. Also in that year the Parks Department considered demolishing the building altogether and relocating the weather bureau and police station to the Belvedere and Sheepfold (later Tavern on the Green) respectively.
From 1914 to 1924 the Manhattan Parks Department operated out of the newly built Municipal Building in Lower Manhattan. During this period the Arsenal deteriorated to such an extent that in 1922 a headline in the New York Times read: "Parks Arsenal a Near Ruin." The paper went on to report that this neglected landmark had a leaking roof, broken windows, and missing brickwork among other structural defects.
At that time the City appropriated $75,000 for an overhaul of the Arsenal, including new central turrets and a clock, basement storage for "trees, plant and shrubs," and a conference room for park researchers (later a library).
Most of the space was set aside for Parks Department offices. During reconstruction an underground spring and a secret passage were discovered (the latter possibly from the building's days as an arsenal, for the undercover transfer of arms). The restoration was completed in 1924.