After a 63-year-old woman was killed at a roadblock in the French Alps on Saturday morning, 409 people were injured during the protests, 14 of them seriously, according to Castaner.
The number rose overnight, as about 3,500 continued to demonstrate at 87 sites.
"There were attacks, fights, stabbings," Castaner told RTL radio. "There were fights between Yellow Vests. There was a great deal of alcohol in some places and that lead to idiotic behaviour that can lead to violence and that becomes unacceptable."
A total of 282 people were arrested on Saturday, 73 of them overnight.
With inexperienced organisers, most of the demonstrations, which involved disrupting traffic on motorways or road junctions, had not been notified to the authorities, making it difficult to police them.
"People who claim to organise and who in fact want to disorganise the country, don't want to respect democracy, have to face up to their responsibilities," stormed the minister.
Protests to continue
Demonstrators had vowed to continue the protests on Sunday at 150 sites, according to Castaner.
About 100 protests were taking place on France's motorways at midday, according to operator Vinci.
Some 200 people tried to block access to the Disneyland amusement park near Paris but failed to do so.
Far-right leader Marine Le Pen hailed a "great success" when interviewed by RTL on Sunday and accused Castaner of trying to play down its significance.
"Accidents can happen and everyone regrets that," she said when asked about the high number of injuries, adding that the "great majority" of participants demonstrated peacefully.
She called on the government to listen to the movement's demands.
But several ministers on Sunday morning declared that there would be no change in the government's line.
"So far as ecological taxation is concerned, we'll carry on as planned," Environment Minister François de Rugy said. "Not to do so would be irresponsible."
The wave of discontent was sparked by the rise in petrol and diesel prices, partly because of an increase in an environmental tax.
But for many involved it has become a protest against the rising cost of living and poor public services, especially in rural areas and small towns, as well as the government's taxation policy.
President Emmanuel Macron's popularity ratings have dropped to 25 percent, a fall of four points since December, according to an opinion poll published on Sunday.
That leaves him higher than his predecessor Socialist François Hollande's 20 percent at the same point in his term but lower than Hollande's predecessor, right-winger Nicolas Sarkozy, who was at 44 percent.