Photo Charles Taylor

The site has been occupied for at least 3000 years. The Romans built a signal station in the 4th century to warn of invaders and the outline remains near the cliff. Construction of a stone wall around the site started in 1136 and the 100 foot high keep was added in the 1160's. The outer wall of the inner bailey was extended along the cliff overlooking the town at the beginning of the 13th century and it had become one of the most important castles of its time. At he start of the Civil War it was held by Parliament but changed sides twice. After seiges in 1645 and 1648, during which bombardment caused half the tower to be brought down it finally fell to Parliament. It was planned to slight it but the people of Scarbrough persuaded Parliament to keep it. The 18th century barracks that had been added were destroyed in 1914 by German naval action.

Leased to Scarborough Council in 1906, it is now managed by English Heritage.