Hollande's popularity rose after he took a tough approach on security following the November 13 jihadist attacks on Paris which killed 130 people.
But three months later, the problem that has plagued him since he came to power in 2012 -- France's stubbornly high unemployment -- has dragged him back down to popularity ratings of around 19 percent.
Hollande is haunted by the pledge that he made at the start of his mandate that he would not run again if he did not improve the jobless rate.
Now he finds himself under fire, not only from the opposition right-wing Republicans of his predecessor Nicolas Sarkozy, but increasingly from within the ranks of his own Socialist Party.
The right accuses Hollande of stealing its clothes, partly with his moves to increase powers for the security services following the attacks, and partly with his attempts to liberalise the economy through the government's rising star, 38-year-old Economy Minister Emmanuel Macron.
Fabius, 69, will almost certainly exit the cabinet to chair the Constitutional Court, which rules on the constitutionality of laws. Reports say Hollande -- never an ally of his -- will announce the move on Wednesday.
"You can see it in the way he is acting and his staff are already packing up their things," one source close to Fabius said.
Segolene Royal, Hollande's former partner and mother of their four children, has been tipped to leave her role as environment minister and take over the foreign brief.
Royal, a failed presidential candidate in 2007, remains a largely popular figure to Socialist Party supporters.
Her departure could allow Hollande to bring a member of the Greens back into his cabinet, a move which would also allow him to extend his appeal to green-minded voters.
But Hollande has to play a careful game because his right-leaning prime minister, Manuel Valls, is hated by many ecologists.
Moving Royal to the foreign ministry would also restore the gender balance in the cabinet that was disturbed when Christiane Taubira quit last week as justice minister over her opposition to the government's moves to strip French nationality from bi-nationals convicted of terrorist offences.
Other names in the frame are former prime minister Jean-Marc Ayrault and, perhaps more surprisingly, Matthias Fekl, the 38-year-old foreign trade minister.