Justice Secretary Jack Straw has signalled he is preparing to "rebalance" the Human Rights Act amid concerns it had become a "villains charter".
Mr Straw - who introduced the act 10 years ago when he was Home Secretary - said he had become frustrated with the way that it is operating.
He suggested that the rights in the legislation - which enshrined the European Convention on Human Rights in British law - could be balanced with new "responsibilities" to obey the law and to be loyal to the country.
In an interview with the Daily Mail, he said: "In due course I could envisage that there could be additions made to work in the issues of responsibilities."
Mr Straw acknowledged there were genuine public concerns about the way the act had been used in some cases by prisoners to avoid punishment or to prevent the deportation of Islamic extremists.
He blamed "nervous" judges for refusing to accept assurances from ministers that such removals were in the national interest.
"There is a sense that it's a villains' charter or that it stops terrorists being deported or criminals being properly given publicity," he said.
"I am greatly frustrated by this, not by the concerns but by some very few judgments that have thrown up these problems."
The Mail said Mr Straw's proposals to reform the act could be a key plank of Labour's next general election strategy.
