The original seat of the castle dates back to Norman times, while Frederick occupied the castle from 1225 to 1228. During the Crusades the castle became a regular shelter for knights returning or departing for the Holy Land. It was here that Frederick II held the famous Diet, given his imminent departure on the sixth Crusade, which included his directives for the time he was away and specific instructions should he die.
On initiation of the works, Frederick announced the birth of Corrado in the presence of barons and other prelates. In 1308, during the Angionian Reign, the Templars from the Supreme Priory of Southern Italy were arrested and imprisoned in the castle. They were imprisoned here until 1312, when the Order was abolished.
The Aragons took possession from 1458 to 1481, reinforcing the walls and on the order of Charles V, the castle took on its symmetrical appearance with four corner lancet embankments with radial defence slits and apertures along the curtain walls, adapting the castle to the defence canons of that era.
During the first Spanish occupation from 1502-1503 the castle hosted the Great Captain Consalvo of Cordova and part of the Spanish army. It was during this period that the castle experienced its glory years, on occasion of the Dual of Barletta in which the castle was the location of many events that accompanied this dispute.