"I have no one to blame but myself, and I really hope that no one else got hurt because of me," said Van Garderen in a statement released by his team Education First on Friday night.
The fracture in the bone at the base of his thumb was diagnosed after the finish of Friday's stage in Chalon-sur-Saone.
Van Garderen hit a traffic island seven kilometres into the longest stage of the Tour de France. After a medical inspection, he got back in the saddle with blood running down his face and completed the 230-km ride.
"The crash was a result of a personal error," van Garderen said.
"I was looking down at my bike because I saw something caught up in it, like a piece of paper, so I was looking down and I hit a median."
After the finish, Van Garderen had an x-ray which showed a broken first metacarpal in his left thumb.
"We will miss having him in the team," said Jonathan Vaughters, the head of the American team. "He has showed great form coming into the race."
"We're not only concerned about Tejay doing lasting damage to the fractured bone if he were to continue riding, but we are also concerned for the safety of others, too," said Vaughters.
"Full use of your hand is important when racing alongside 180 riders."
The 30-year-old, who placed fifth in the Tour in 2012 and 2014, was riding in support of team-mates Rigoberto Uran and Michael Woods.
"All I'm thinking about now is the disappointment, less for myself and more for the team," said van Garderen.
"Rigo and Woods, they both have a big chance to podium, to win stages, even to win the whole damn Tour. I would have loved to have been a part of that."
"Unfortunately, as all cyclists have become accustomed to saying, these things happen," said the American.