Millions of pensioners are paying too much tax because of the complexity of the system and their reluctance to seek advice, according to the National Audit Office.
Some 1.5million older people overpaid income tax because of processing errors, while about 3.2million were failing to take up their allowance entitlements, the watchdog found.
It also estimated that 2.4million were paying more than they needed to on their savings because of their tax arrangements.
The overpayments ran into hundreds of pounds for many pensioners, who are particularly affected by losing the cash because of their relatively low incomes.
There were also 500,000 older people who had underpaid tax, by an average £207.
The auditors said errors occurred because the tax for pensioners was often more complicated than for the rest of the population.
It said the average income tax overpayment due to error was £171 — having a disproportionate effect on a group whose mean income was just £16,000.
Failure to claim allowances on their tax cost pensioners up to the 4% they would otherwise increase their incomes by.
The audit office said older people were less likely to seek help from HMRC, even though 36% did not understand their tax liabilities.
It urged the HMRC to “rethink its approach to ensure that older people get the financial support to which they are entitled” and work with other departments to provide a “more coherent” service.
Spokeswoman Amyas Morse said: “Older people want to pay the right amount of tax but too many pay more than they need to because they do not claim allowances to which they are entitled, and because of errors.
“By providing a more coherent service, HMRC could make substantial savings as the number of inquiries from older people about their tax affairs would reduce – a win-win situation for all.”
Edward Leigh, chairman of the public accounts committee, said: “Older people no less than younger ones deserve to be charged the right amount of tax and to get the advice they need.
“HMRC needs to rethink its approach to dealing with the tax affairs of older people. It should work with other government departments to make sure that there is a system in place which enables older people to pay the right level of tax and get the allowances they are entitled to in a simple manner.”