Timo Werner scored two times as and Germany will now face Mexico in the semis. But Cameroon were seething at the dismissal of Mabouka, who was initially spared after a case of mistaken identity.
The video review system was called upon when Mabouka was sent off on 64 minutes for a dangerous tackle as he went in studs up on Liverpool's Emre Can.
Colombian referee Wilmar Roldan showed Sebastien Siani the red card at first, before his error was rectified and Mabouka received his marching orders instead after intervention from the VAR.
The decision left Cameroon reeling and Germany's Werner scored his first goal just a minute after the dismissal.
"The referee didn't give any explanation," fumed Belgian Broos when asked about the bizarre sequence of events.
"First of all he gave a red card against Siani, then the video ref called him back and told him it was the wrong player, so Mabouka was sent off.
"I think both players (Can and Mabouka) had their legs high, so I don't see why he had to give a red card.
"I think everyone is confused, also me, I didn't know what was happening, I didn't understand it and I still don't understand it now.
"I think this is something for the referee, he and he alone can explain what happened in that incident."
After Kerem Demirbay fired Germany in front just after the break, Werner struck either side of Vincent Aboubakar's consolation effort for Cameroon on 78 minutes.
Earlier on, Portugal sent off New Zealand with a devastating 4-0, while hosts Russia were beaten by Mexico, and Chile and Australia drew 1-1.
Semi finals are to be played on Wednesday June 28 between Portugal-Chile and Germany-Mexico.