The absence of the 17-times Grand Slam winner was announced a day after world number three Rafa Nadal withdrew from the August 4-12 tournament with a knee injury and robs the event of two of its biggest attractions.
"I have always enjoyed playing there but after a long stretch of tournaments, I will need some time to recover," Federer said in a statement. "I look forward to coming back to Canada next year."
Federer's decision came shortly after he defeated Argentine Juan Martin del Potro 3-6 7-6 19-17 in the Olympic men's semi-final on Friday, which was the longest men's three-set match played in the professional era.
In an bid to avoid withdrawals stemming from participation in the London Games, the Toronto field was trimmed to 48 players from 56 with the top 16 seeds receiving first-round byes.