Feminist groups called for a boycott of the Césars awards after Polanski was chosen to preside and threatened to demonstrate outside the ceremony, which is on 24 February.
An online petition had collected 61,500 signatures by Tuesday morning when Polanski made his announcement.
Women's Rights Minister Laurence Rossignol described the choice of Polanski to preside as "surprising" and "shocking" but Culture Minister Audrey Azoulay refused to criticise it, insisting that the Césars Academy was free to make its own choices and pointing out that the case, while "particularly serious", dates back four decades.
Still wanted in US
The director of Tess, Chinatown and Rosemary's Baby, now 83, spent 47 days behind bars in the US in 1977 but fled the country when he was released on bail.
Since then the victim, Samantha Geimer, has dropped charges after a financial settlement but Polanski, who was born in France to Polish parents, is still subject to an international arrest warrant.
The uproar has "profoundly saddened Roman Polanski and affected his family", a statement issued by his lawyer, Hervé Termine, said, so he had "decided not to accept the invitation".
Polanski, who was born in Paris to Polish parents, became a French citizen in 1976 and lives in France.