In 1853 the castle was the headquarters for the Isle of Wight Artillery Militia and some of the buildings were used for the storage or arms and ammunition. Much of the castle was in a poor state of repair.
In 1856 the Castle came under the control of the Office of Works and some restoration work then commenced.
The part time soldiers of the Isle of Wight Artillery Militia used the castle for training purposes. Within ten years there were over 300 Militia, with their principal role being the coastal defence batteries and forts around the Island. 64-pdr Rifle Muzzle Loading guns were used for training of militia intended to services the coastal artillery.