Spain were by far the better team but had to come from behind twice to earn a 2-2 draw after extra-time.
Willian gave Brazil the lead in the 25th minute but Spartak Moscow striker Rodri levelled matters in the 57th minute.
In extra-time Dudu thought he had won it for Brazil 10 minutes from time but Espanyol striker Alvaro Vazquez bundled in an equaliser just two minutes later.
That sent the match to spot-kicks and while Brazil scored four from four, missed spot-kicks from Sergi Roberto and Vazquez ruined Spain's chances.
Cruzeiro midfielder Dudu added to his goal in regulation by scoring the winning penalty and Brazil will now face Mexico in the semi-finals as they look to win a fifth U20 World Cup.
France qualified for the semi-finals for the first time in their history with a 3-2 win over Nigeria in Cali after extra-time.
Alexandre Lacazette had given France the lead early in the second-half but Maduabuchi Ejike levelled matters right at the death to send the game into extra-time.
France though started extra-time like a house on fire as Gueida Fofana and a second from Lacazette put them 3-1 up and they then held on for the win despite another goal from Ejike nine minutes before the end of extra-time.
France will now play Portugal in the semi-finals.