Authorities appeared near-certain that there was no Islamist connection to the attack which saw 20 people injured, six of them seriously.
While stressing that the investigation was still ongoing, North Rhine-Westphalia state interior minister Herbert Reul said the perpetrator was believed to be a German citizen "and not, as has been claimed everywhere, a refugee or anything like that".
It was not clear whether the 48-year-old man hoped to commit a "murder-suicide" or had political motivations.
"Two restaurant goers died, two Germans aged 51 and 65 died. Because of the severe nature of the injuires to at least six other people, they [the authorities] are expecting the death toll to rise," correspondent Nik Martin told RFI.
"The incident happened in the historical part of the town, very popular with tourists, which was full due to the unseasonably warm weather. It was around 25°C, compared to 11 the day before, so many people were milling around here".
Martin says it is still unclear how the suspect's car managed to enter the area, which was supposedly accessible only to pedestrians.
"While there was some relief that the number of victims in this attack was much lower than others we've seen, it's impact is likely to be felt for a long time," he went on to say.
German media reports said the driver, not yet named, had a history of petty crime and drug dealing as well as mental health problems and had already attempted suicide before.
The ZDF news channel reported that he had had links to far-right groups.
Der Spiegel magazine reported that an assault rifle was discovered in the man's apartment, while Die Welt daily reported that police found a pistol tied to a wire in the floor of the van, and fearing an explosive device, sent a bomb squad to the scene.
Previous vehicle attacks
Germany has been on especially high alert after a string of terrorist attacks using vehicles across Europe, several claimed by the Islamic State group.
In the Berlin Christmas market attack, Tunisian asylum-seeker Anis Amri hijacked a truck and murdered its Polish driver before killing another 11 people and wounding dozens more by ploughing the heavy vehicle through the festive market in central Berlin.
He was shot dead by Italian police in Milan four days later while on the run.
In July 2016 Mohamed Lahouaiej-Bouhle drove a lorry into crowds on the seafront of the Riviera town of Nice on Bastille Day.
Germany's security services estimate there are around 10,000 Islamic radicals in Germany, some 1,600 of whom are suspected of being potentially violent.
Chancellor Angela Merkel said she was "deeply shaken" by the incident and "everything possible will be done to determine what was behind this act and to help the victims".
The presidents of Russia and France, Vladimir Putin and Emmanuel Macron, both sent their condolences.