Gordon Brown's leadership was under fresh pressure after a Cabinet minister branded efforts to silence rebel MPs "ridiculous".
Work and Pensions Secretary James Purnell - tipped by some as a successor to Mr Brown - also admitted he shared some of their concerns.
The comments emerged as the Prime Minister struggled to recover from a torrid spell of party infighting, which has seen four Government figures leave in protest at his performance in less than a week.
There was relief among Labour loyalists that there had been no further departures on Wednesday after the damaging resignation of Scotland Office minister David Cairns.
However, former minister George Howarth did step up the attack on Mr Brown by insisting he was the least popular British PM since Neville Chamberlain made a doomed bid to appease Adolf Hitler.
And Mr Purnell risked inflaming tensions further by echoing Mr Cairns' warning that Downing Street should not dismiss critics as having "taken leave of their senses".
In an interview with the New Statesman, Mr Purnell refused to condemn those who voiced doubts about the Prime Minister.
"I think it would be ridiculous to pretend that you can't complain when you're worried," he said. "I mean, I'm worried that we're 20 points behind (in the polls). "I'm not going to condemn people or question their motives. (But) I don't agree with what they did."
Mr Purnell also warned that voters had "turned away" from Labour over recent years, and there was no "magic bullet" that could bring them back. Mr Purnell also refused to rule out supporting a future leadership challenge, saying merely that the question was "hypothetical".
"One of the great wisdoms of politics is not to answer hypothetical questions and that's a hypothetical question," he said. "The job of the Cabinet is to support the Prime Minister and that's what we're going to do."
Read More
MORE ABOUT