The first three sets were decided by single breaks of serve, with the fourth going to a tiebreak as neither player could break through in a tight match played in hot, sunny conditions at Arthur Ashe Stadium.
Isner, winner of 13 of his last 14 tiebreakers in U.S. Open play, including three in his four-set victory on Thursday over 12th seed Gilles Simon of France, looked poised to take Murray to a fifth set.
But the 24-year-old Scotsman beat the 22nd-ranked Isner at his own game, sharpening his play as the towering 6-foot-9 (2.06 m) American began to crumble.
Isner, playing in his first Grand Slam quarter-final, double-faulted on the third point of the decider to give Murray the early advantage and netted an easy forehand volley to fall behind 5-2. Two points later he hit a serve return wide to end the three-hour 24-minute match.
"It's so frustrating playing against him," Murray told a centre court crowd that included U.S. First Lady Michelle Obama, who was sitting in a box with former tennis great Billie Jean King.
"You feel like you're playing good tennis, but it's so tough to break him. He started serving unbelievable."
The 24-year-old Scotsman lost to Nadal at the French Open and Wimbledon semi-finals and trails the Spaniard 12-4 in career meetings, but Murray is confident on the hard court.
"Well, I've beat him before in the semis," noted Murray, who overcame Nadal on his way to the 2008 US Open final.
"I won against him also at the Australian Open," the Briton said, referring to a quarter-finals test last year. "And when I've won against him, it's always been on hard courts. It's a good surface for me to play him on."
Murray is 3-3 against Nadal on hard courts dating back to 2008.
A three-time Grand Slam finalist, Murray is trying to become the first Briton to win a Grand Slam singles crown since Fred Perry won his third US Open in 1936.
Murray needed three hours 24 minutes to eliminate Isner and will have to bounce back Saturday to play Nadal in a schedule compressed due to back-to-back wash-outs this week.
"I'm feeling fine now. A lot of times that's the case. It's kinda how you wake up the next day," Murray said.
"It was hot today. It was a pretty long match. You know, it's not like you are you into a classic endurance when you play against him. You have to be so explosive.
"You're not like having 30-shot rallies that you might have against Rafa. You're having to react all the time and stretch for balls and chase down like drop volleys. It's a different sort of fitness against him.
"I think on the American hard courts, it's a close, close match-up," he said.
British doubles duo Colin Fleming and Ross Hutchins recommenced their postponed quarter-final match from Thursday but were soon bounced out by fifth seeds Rohan Bopanna and Aisam-Ul-Haq Qureshi, the Indian-Pakistani partnership sealing a 7-5 2-6 7-5 victory.
In the junior tournament, Britain's boys continued to dominate with George Morgan and Oliver Golding joining a third Brit Kyle Edmund in the semi-finals.
Morgan registered a 6-7(3) 6-4 6-4 win over Adam Pavlasek of the Czech Republic, while Golding also progressed after a 5-7 6-4 6-1 success against Slovakia's Filip Horansky.
Morgan now meets Golding for a place in the final, while Edmund, who beat Egypt's Karim Hossam 6-2 6-4, plays number one seed Jiri Vesely.