The Stratford venue has been the centre of a glorious fortnight of sport, culminating in Sunday night's spectacular closing ceremony.
But its future has long been the subject of debate, with the Hammers initially denied tenancy last October following a legal dispute involving Tottenham and Leyton Orient.
The need to preserve a home for athletics was central to London 2012's legacy aims, but Allardyce insists the stadium will fall into decline without Premier League football.
He told the Times: "If they keep it for athletics, the only time it will ever get a big crowd again after the Olympics is when they hold the World Championships there in five years' time."
He went on: "You only have to look around the world at some of the Olympic Stadiums that have been built. They're white elephants now. There are weeds growing there. Look at the 'Bird's Nest' in Beijing. Look at Athens.
"The Olympic Park is a fantastic place, but it can't be left to rack and ruin.
"If a club with the history and fanbase and potential of West Ham don't go there, the concern would be that the park is left empty the vast majority of the time.
"I can understand some West Ham fans feeling that they don't want to leave the history and tradition of Upton Park, but it's an incredible opportunity for the club to move forward."