The remains of the 'Bishop's Palace' are situated in the grounds of Mount House, just east of St Mary's Church. Inreality the ‘palace' was a manor house, one of the earliest and largest of its type, the working centre of the estates owned the Bishops of Winchester. Archaeological excavations have revealed a series of stone buildings arranged around a courtyard and surrounded by a wall and moat.

Bishops and royalty frequently visited the manor house in the early 13th century, with King John's visit in 1208-9 even prompting the building of new garderobes (medieval toilets), but these visits gradually petered out. By 1453, the bishops had largely given up their interests in sheep farming allowing local farmers and merchants to enter the trade.

Just as their earlier interest had stimulated the wool trade in Witney, it seems that their later disinterest may also have proved of long-term benefit. Free from manorial control and the limited trade restrictions found in many other areas, many master weavers seem to have chosen to practise their business in Witney.

The surviving medieval buildings of the Bishop's Palace were demolished and replaced by a new house, probably just after 1757. This in turn was replaced by the existing Mount House, in about 1905, probably for J.E. Marriott, owner of the Mount Mills blanket factory which once stood behind the house.