PSG, who are seeking their sixth consecutive League Cup crown, were pegged back midway through the second half when Joseph Lopy's right footed shot from 25 metres beat PSG goalkeeper Gigi Buffon.
Substitute Moussa Diaby hit the winner against the second division outfit 10 minutes from time.
Orléans boss Didier Ollé-Nicolle said: "We mustn't underestimate what happened. PSG are quite capable of putting four or five goals past first division sides. They play in the Champions League and have players who are up for the Ballon d'Or.
"We could have been thrashed. Everyone came to see the PSG stars but they ended up seeing a football match."
Ollé-Nicolle's PSG counterpart, Thomas Tuchel, said he was impressed with the attitude of a team who are 11th in Ligue 2.
"It's good that we played in a serious way with a good mentality. I really didn't like those last few minutes after we scored the second goal. They really came at us. But we held on even though it was hectic stuff."
Ollé-Nicolle added: "The match was a great sporting moment for the city and for the club. My only regret is that we weren't able to pull off the shock of the century. Just imagine a second division club beating PSG when no one in the first division can get past them. That would have got everybody talking for years."