Youngster Barrie McKay opened his Rangers account with what should have been a goal to savour, before second half goals from Rory McAllister and Scott McLaughlin saw the home side surge into the lead.
But Andy Little grabbed a late equaliser and a share of the points for Ally McCoist's men. The match at Balmoor was the first ever league fixture played by the Glasgow giants outside the top flight of Scottish football.
Rangers opened the scoring after 26 minutes and the honours went to McKay. A long ball from Lewis McLeod was knocked on by Lee McCulloch for the 17-year-old, who kept his cool to drive a low shot into the back of the net.
McAllister then had a great chance to draw level for Peterhead with a shot from six yards that he dragged across the face of goal and wide of the upright. McKay could have added to his own tally before the break but the teenager was also off target and Rangers had to settle for a slim advantage at half-time.
Rangers goalkeeper Neil Alexander produced an impressive one-handed save after the restart to prevent a lovely swerving effort from McAllister from finding the top corner. But the forward had been a menace throughout the match and was finally rewarded with a goal after 64 minutes when he did well to hold off Carlos Bocanegra before lashing into the bottom corner.
Rangers threw Francisco Sandaza into the action for McKay but it was Bocanegra who could have restored the lead when he met Ian Black's free-kick only to nod wide from close range. Former St Johnstone striker Sandaza then tried to chip Paul Jarvie but lofted into the goalkeeper's arms instead, before Dean Shiels was denied by a save at the near post.
But Rangers were left reeling when Peterhead had the net bulging again with nine minutes to go when Ryan Strachan's corner was cleared into the path of McLaughlin and he rifled past a stunned Alexander.
The Light Blues were back on level terms with a minute to go when substitute Kevin Kyle rose above everyone else in the box to meet a Shiels corner and although his header came back off the bar Little was on hand to score.