The visitors started the brighter and Aaron Ramsey’s shot from range tested Millwall keeper Max Crocombe early on.
At the other end, Leicester keeper Stolarczyk used his legs smartly to keep out the in-form Azeez.
Just before the break, Casper De Norre dropped deep on halfway and played a long ball up to Azeez, who left Luke Thomas floundering before lashing the ball into the back of the net – his fourth goal in as many games.
The Foxes were dealt a further blow moments after the half-time break when Tuesday’s goal-scorer Ramsey was forced off with an injury and replaced by Argentine loanee striker Julian Carranza.
Azeez was then tripped in the box by Foxes midfielder Harry Winks but Serbian striker Ivanovic’s well-struck spot-kick was superbly saved down to his right hand side by Stolarczyk.
That deprived the struggling visitors of the lift they needed and Azeez went close again, but his left-foot shot from outside the box went just wide.
Off the bench, substitute Carranza’s attempt to grab his first goal for Leicester saw his overhead kick sail over the bar.
Deep into six minutes of additional time, fellow sub Patson Daka was denied a dramatic leveller when his header was saved by Lions keeper Crocombe as the hosts hung on for a well-earned three points.