Bishop sparked new life into Chris Casper's side when he came on as a second-half substitute for James Dean. And he should have added to last Saturday's opening-day double at Milton Keynes after the home side created some decent chances.
Instead, visitors' keeper Phil Barnes kept the Mariners in the lead with some excellent saves including from Glynn Hurst and Richie Baker who threatened the visitors' goal.
But it was 19-year-old Parrish, in only his second season with the Shakers, who showed his team mates how to finish. Pushed up field for a Nicky Adams free-kick 19 minutes from time, the tall six-footer Bolton born defender got the vital touch.
It was nothing more than the home team deserved but a home win, which has now eluded them since last December, still would not come.
Grimsby, also unbeaten after two matches, were at their best in the opening 45 minutes. Paul Bolland gave them the lead after keeper Jim Provett pushed away a Tom Newey free-kick.
James Dean wasted two good opportunities to level for Bury before the break. And with his patience tested once too often, manager Casper replaced the former Northwich and Halifax player with Bishop.
"The opposition know they are in for a tough time when they see Bishop," said Casper. "There were some worried looks on their faces."
Opposite number Alan Buckley added: "In the first half we looked far and away the best team.
"But Bishop came on and changed the game. Bury won the majority of the battles in the second half so 1-1 was probably the right result."