Having been beaten 4-0 during the Ashes, England won with seven balls to spare in the opener of the five-match ODI series against Australia.
Set 305 to win, Roy's daring 180 helped England make 308 for five.
The successful chase was also a record for a one-day international at the MCG, eclipsing Australia's 297 for four against England in 2011, and vindicated captain Eoin Morgan's decision to bowl first.
The 27-year-old passed teammate Alex Hales's England record knock of 171 against Pakistan in 2016 with a chip just over the head of a despairing Australian fielder.
Roy was still coming to terms with his achievement after the match.
"Incredibly special, I don't have too many words right now," he said. "Halesy was pretty disappointed."
Australian captain Steve Smith praised Roy for his bold stroke.
"He didn't give us a chance, and he chanced his arm," Smith said. "They got off to an absolute flyer and we couldn't pull them back."
Roy had a willing partner in Test captain Joe Root, who had been tormented all tour by the Australians and was in no mood to let a rare victory slip.
Root made an unbeaten 91 and he and Roy put on 221 -- an English ODI record -- for the third wicket.
Aaron Finch's ninth one-day international century was the cornerstone of the Australian total of 304 for eight.
Finch's 107 was backed by half-centuries for Mitchell Marsh (50) and Marcus Stoinis (60) as Australia recovered from a shaky start in which the tourists finally managed to get bogeyman Steve Smith cheaply.
Fresh from scoring three centuries as Australia won the Ashes 4-0, Smith looked in commanding touch again as he scored at better than a run a ball but was caught behind off spinner Adil Rashid for just 23.