At the time of early English settlers, Leesylvania was believed to be the site of an Algonquian village, overlooking Neabsco CreekHenry Lee II settled on the land from 1747 until his death in 1787. He and his wife had eight children at their home, including Henry "Light Horse Harry" Lee, Revolutionary War hero and future father of Civil War general Robert E. Lee. George Washington mentions visiting the Lee House three times in his diaries. In 1825 the property was sold to Henry Fairfax, and later passed to John Fairfax in 1847. Fairfax later served as a staff aide to Confederate Lt. General James Longstreet.

The site was Fairfax's boyhood home, and he returned to live on the property in late 1875, remaining there until his death in 1908. The land was also used as a small Confederate force and gun emplacement during the Civil War. The Freestone Point Confederate Battery was listed on the National Register of Historic Places in 1989.[2] The battery engaged with vessels of the US Navy's Potomac Flotilla on September 25, 1861. There were no casualties on either side, but the Federal vessels withdrew at the conclusion of the fighting. (Ref: Official Records of the Union and Confederate Navies in The War of The Rebellion)

Today, only a small cornerstone of the Lee House remains. The house and its path were completely bulldozed in the 1950s to make way for a road. A restored chimney of the Fairfax House remains. Henry Lee II and his wife, along with Henry Fairfax and his third wife are buried on the property. The sites and the cemetery are accessible by trail. The Leesylvania Archeological Site was listed on the National Register of Historic Places in 1984.[2]