Hestercombe is situated on the southern slopes of the Quantock Hills with extensive views over the Vale of Taunton to the Blackdown Hills beyond.

Much of the character of the buildings and gardens at Hestercombe is derived from the local stone that lies close to the surface and frequently outcrops. Dark pink diorite, a dense igneous rock, which was used both in the eighteenth and nineteenth century gardens and to face the west front of Hestercombe House, occurs only rarely in Somerset. It is complimented by silvery-grey morte slate, a sedimentary sandstone, which is used to striking effect in the Formal gardens.

Hestercombe was first mentioned in an Anglo-Saxon charter of 854 and from 1391 until 1872 was continuously owned by one family, the Warres. The first reference to a garden at Hestercombe is from the estate accounts of Sir Francis Warre for, ironically, pulling down a garden hedge in 1698. In 1731 John Bampfylde, MP for Exeter, who had married Sir Francis's daughter Margaret in 1718, commissioned plans for a garden from a Mr Brown of London.

Coplestone Warre Bampfylde designed and laid out the Landscape garden we know today after inheriting the estate from his father in 1750.