Banco do Brasil SA, the nation's largest bank by assets, described the remark as "unfortunate" while the National Confederation of Financial Industry Workers (Contraf) said its members had been "disrespected".
"The Banco do Brasil, along with all the Brazilian people, wishes good luck to Luiz Felipe Scolari in his new challenge as coach of the Brazil national team," said the bank in a statement.
"However, the Banco do Brasil regrets the unfortunate comment...and states that it is proud of counting on 116,000 workers who wear the colours of the bank, with the colours of Brazil, every day and work with dedication and commitment to look after the needs of our customers and clients.
"For the BB family, planning, respect and organisation are the secrets for a strategy of success."
Contraf "repudiated" Scolari's comments. "Felipao (Big Phil) not only disrespected bank workers but also showed completely lack of knowledge about the reality of work in the financial system," it said.
It said that around 1,200 workers a month had to take time off work for health reasons due to the pressure on them to obtain results and targets which were often unrealistic.
"We hope he is not as out of touch with football as he is about the working situation in the banks," it added.
Scolari made his comment after being asked about the pressures of leading Brazil in a World Cup on home soil, where nothing less than winning the trophy in 2014 will be considered satisfactory.
"If you don't like pressure, it's better to go and work in the Banco do Brasil, or outside on the corner or sit in an office and do nothing," he told a news conference.
Scolari is beginning a second stint as Brazil coach, having led his country to World Cup victory in 2002.
The bank later said that Scolari, who is one of their customers, had contacted them to apologise.
"He said he did not intend to offend the employees and had expressed himself badly."