Eliot Richards rifled a low 25-yard shot inches wide of a post before Lee Brown put the Pirates ahead in the ninth minute, drilling his shot past goalkeeper Adam Bartlett into the far corner of the net.
Rovers goalkeeper Michael Poke then came to his side's rescue when he turned a rasping 25-yard shot from Will Evans around a post.
But Hereford levelled four minutes into the second half through striker Delroy Facey, who latched onto a long ball from Benoit Dalibard before beating Poke from five yards.
Hereford were reduced to 10 men in the 55th minute when Dalibard was sent off by referee Carl Boyeson for a dangerous foul on Richards.
Bulls substitute Nathan Elder had an overhead effort cleared off the line by Andy Dorman before Scott McGleish came off the bench to head home the winner for Rovers in the 72nd minute following a Richards cross.