The Home Office said it planned to completely vacate the site by the end of the month to hand it back to the Ministry of Defence, ahead of a transfer in March to a housing developer that purchased the site.
It said it did not comment on individual asylum accommodation sites or on operational arrangements around those sites.
However, it said the government would close every asylum hotel.
The government had previously announced discussions had begun to use barracks in Crowborough in East Sussex and Inverness in Scotland to house about 900 male asylum seekers in total.
Napier Barracks housed more than 300 asylum seekers at its peak, but was believed to have fewer than 100 residents ahead of its closure.
The 130-year-old military site was hit by the allegations of poor conditions following numerous inspections since February 2021.
The charity Care4Calais said it completed its final clothing distribution at the barracks on Sunday.
A spokesperson said: “We were never allowed indoors, so we worked outside in all weathers.
“It was not how we wanted to work, but our volunteers did it with care, dignity, and determination.”