Fears over the Zika virus have been cited by a number of top players for pulling out of Rio, despite such concerns not appearing to trouble athletes in other sports or indeed the world's leading female golfers.
But McIlroy has also admitted that the Olympics just don't mean as much to golfers as other athletes, a point he hammered home as he spoke at Royal Troon in Scotland, where this week's British Open will be held.
"I'll probably watch the Olympics, but I'm not sure golf will be one of the events I watch," said the 27-year-old during a press conference at the British Open.
When asked what he would turn his television on for during the Games, McIlroy added: "Probably the events like track and field, swimming, diving, the stuff that matters."
The sport has been brought back into the Olympic fold for the first time since the 1904 Games in St Louis and International Golf Federation chief Peter Dawson had voiced his hope that the new exposure given by the Olympics would help the game become more popular.
But McIlroy sees things differently.
"I didn't get into golf to try and grow the game," said the Northern Irishman, a four-time major winner. "I got into golf to win championships and win major championships."
The build-up to this week's Open on Scotland's west coast has to some extent been overshadowed by the Olympic debacle.
Jordan Spieth spent most of his time in front of the media explaining why he made a last-minute call to withdraw on Monday, just ahead of the deadline.
But the Texan prodigy, who won the Masters and the US Open last year, indicated that pulling out of Rio was the "hardest decision" he has ever made and said going to the next Olympics in Tokyo in 2020 remains a "significant goal".
"I'm very passionate and very much a supporter of the Olympics and Olympic golf," he said in a press conference. "I do hope to play in four or five, you name it, Olympics representing the United States in the future, if I have the opportunity."
That raises the question as to why Spieth chose so late not to go to Rio, all the more so as he denied that his withdrawal was specifically for fears of catching the mosquito-borne Zika disease.
He also refused to confirm what medical advice he had been given to help him reach such a decision.
Two-time Masters champion Bubba Watson, Henrik Stenson, Rickie Fowler, current Masters champion Danny Willett and Justin Rose are the biggest names who will feature in Rio.
"I think the Zika risk is going to be one of those things that we look back at and think it's a non-event, hopefully," said Rose, the number-11 ranked former US Open champion. "You're going to get down there and you're probably not going to see a mosquito in sight."