The site dates back as far as the Iron Age with Walton Castle set upon a hill fort. Reference to it is also made in the Doomsday Book as belonging to “Gunni The Dane”. However the construction as we now see it took place, according to Pevsner Encyclopedia, between 1615 and 1620. It was designed as a Hunting Lodge for Lord John Poulett, a wealthy and renowned Somerset land owner and Member of Parliament.
The land surrounding Walton Castle had been acquired by marriage. The English Civil War marked a decline in such enormous and valuable holdings by absentee landlords and then by 1791, Walton Castle lay abandoned and derelict. The roof and floors then fell in and parts of the building became a convenient home for dairy cattle. Despite its exposed position,the structure remained meriting Pevsner`s description in 1957 as “remarkable as a piece of ornamental planning”. Restoration by the largest landowners in the area was abandoned and taken up by a city financier who returned Walton Castle to its former glory.