Murray, the defending tournament champion who pulled out of last week's Toronto Masters with a knee injury after his opening match, never looked himself as he was broken in the seventh game of the first set by Chardy, who is in the draw as a lucky loser.
Neither player could hold serve in the opening five games of the second set but Chardy held firm to go up 4-2 and never looked back in the 98-minute match.
Chardy, ranked 38th, has been in good form of late beating American Andy Roddick in the first round in Cincinnati after defeating compatriot and world number six Jo-Wilfried Tsonga in Toronto last week.
The 25-year-old took full advantage of Murray's less-than-potent serve and unforced errors for his first win against the Scotsman in five attempts.
"I didn't serve particularly well. I got broken three times in the second set, which isn't good enough. I broke him a couple of times in the second," said Murray.
"I had a lot of close games on his serve in the first and second set and didn't convert enough chances, and, yeah, I didn't serve well enough."
Murray will head to the August 27-September 9 US Open with just three hardcourt games under his belt since his Olympic triumph on the grass courts at the All England Club but he says he is not worried about lack of preparation before Flushing Meadows.
"I won't play any more matches. Going into the big tournaments, sometimes I've won tournaments in the build-up and it hasn't helped me and then this year Wimbledon I lost in the first round at Queen's and made the final there for the first time," said Murray.
"So it doesn't normally have that much bearing. But, you know, obviously I would have liked to have done a bit better this week."