The house has changed ownership, structure, and purpose often since its construction in 1685. After the Olmsteds sold the property in 1866, the house served as a home to many families, as a boarding house, and as a resort. The last private owners of the house were naturalist and educator Carlton Beil and his wife, Louise, who purchased the house in 1955.