The first manor at Cogges was established by the River Windrush in the 11th century, though the current Manor House dates back to the 13th century and sits next to a perfectly preserved farmyard with a variety of historic farm buildings. Latterly the site has become famous as the location of Downton Abbey's Yew Tree Farm.

In 1667 Francis Blake, a woollen draper, purchased Cogges for £8,000; either he or his son William rebuilt part of the house at about this time. After William's death the house passed eventually to a cousin, Daniel Blake, who was also a woollen merchant.

The house was lived in until the 1970s, when the farm was sold to Oxfordshire County Council. Cogges today is now run by a charitable trust, ensuring public access through a team of volunteers. The site now measures about 20 acres of historic buildings and farmyard, walled garden, grounds and grazing land.