Public School Number Four was constructed on the site of the former Riverside Park School. The original wood-framed schoolhouse had multiple additions since being built in 1891. It was eventually considered unsafe and closed after being deemed a fire hazard. In 1915, Duval County voters passed legislation to build a dozen new brick schoolhouses. Public School Number Four was completed in 1918 at a cost of $250,000. At the time, the school had a beautiful view of Riverside Park.
Public School Number Four was designed by architect Rutledge Holmes and built by Florida Engineering and Construction Company. The entire two-story building is solid concrete making it virtually fireproof. The Neo-Classical design features massive columns prominently placed at the front and side entrances. Classrooms were located upstairs with the library, auditorium, and administration offices on the first floor. The schoolhouse featured large windows and high ceilings with a fireplace in the cafeteria.
In 1999, Foundation Holding Inc. purchased the property and planned to demolish the school building. In its place would be a retirement facility named Lytle House Condominiums. Some of the units were sold before construction even began. After public outrage and pressure from multiple historic groups, the City of Jacksonville designated the school a historic landmark in 2000, halting plans for demolition indefinitely.