The medieval castle became a military stronghold during the Wars of Religion
The castle witnessed the Wars of Religion in the 16th and 17th centuries. It suffered several sieges by both sides and historical traces and engravings (especially the siege of 1573) remain. The medieval defensive system was reinforced by the construction of several buildings including a square tower and a bastion. Barracks were built to house soldiers. These were added to older buildings turned into prisons: the Bermond tower, the Montlaur tower (in the lower part as the top had been destroyed), the chapel and accommodation. Protestants, especially women, were imprisoned in the royal jails.
The revocation of the Edict of Nantes by Louis XIV in 1685 fanned the flames of religious conflicts. The Camisards revolt in the Cévennes at the beginning of the 18th century was one of the consequences. The king's dragoons were instructed to capture Protestants and imprison them in the castle at Sommières. Graffiti in the royal chapel remain as traces of their presence. Common law prisoners were also imprisoned in the castle until the end of the 18th century.
The royal fort lost its strategic importance in the 18th century and a fair number of the defensive works fell into ruin for want of maintenance. There was a succession of redevelopment projects but none were completed. Fortunately, the army sappers' archives contained precise plans that make it possible to understand the organisation of the fortress at the time, and in particular those of Jacques Philippe Marechal who also laid out the Jardins de La Fontaine in Nîmes.