The Castle Hills originally formed part of the Castle's outer defences in the mediaeval period. Known as the Outer Bailey, it was also used as a recreational area and for growing fruit trees, herbs and vegetables.
When the Castle was no longer inhabited from the 15th century the site continued to be used for recreation, and partly for grazing animals. In 1840, a section of it was allocated for a cattle market. Repository sales commenced in 1884, and old photographs depict fences around the site so that animals were penned in.
In the early 20th century, gravel was dredged from the area, resulting in the steep slope between the two mounds, and the flat grassed area between them.
In 1938, the Town Reeve Rosalind Messenger acquired the Hills for the Town Trust, as a public recreation area. The wrought iron gates depicting the Bigod Lion crest were erected at the entrance. The site is now a Scheduled Monument in the care of English Heritage.
Over the years children's play equipment has been installed, but due to regular vandalism, and English Heritage's strict rules about digging footings for equipment on their properties, the policy is no longer to provide it.
The current arrangement is to create a balance between using the area for recreation, but also maintaining sections to conserve wildlife and the natural environment. The flat grassed area is for play and picnics, and benches and tables are provided. Dog owners are welcome, as long as they act responsibly, and clean up after their pets using the bins at the two entrance gates.
Benches are also provided on the two mounds, creating good views over the beautiful Waveney valley marsh and grazing land. The smaller mound to the left of the entrance gates, was formerly a look-out post for Air-Raid Wardens during WW2, and the current intention is to provide a plaque commemorating its importance.
The verges and slopes are maintained in accordance with advice from Suffolk Wildlife Trust, to ensure that plants, wild flowers, insects, and birds can flourish. In particular, the Trust is keen to ensure that the Hills can provide a natural habitat for bees which are currently in decline.
These areas will normally only be cut twice a year, to allow spring and summer growth to flourish. A tree-warden provides regular maintenance of all the trees on the site.