We all have regrets, but hopefully not over our pension, because making the wrong decisions about your retirement savings could leave you thousands of pounds worse off.

About 39 per cent of retired people said they had regrets about their pension, according to a survey of 500 people by the investment platform Hargreaves Lansdown in May.

About 3 per cent wished they had kept a closer eye on their investments, 10 per cent said they wished they had boosted their contributions earlier, and 15 per cent regretted not putting together a plan for their retirement sooner.

So what are the most common regrets, and how can you avoid them?

Many people do not realise until it is too late that they have not saved enough for retirement. Under auto-enrolment, all workers aged 22 and over who earn more than £10,000 a year are signed up to their workplace pension scheme. Most workers contribute a minimum of 5 per cent of their salary and their employer puts in 3 per cent. But experts say this is not enough for a comfortable retirement.

A single person needs an estimated income of £31,300 a year, after tax and housing costs, for a moderate standard of living in retirement, according to the Pensions and Lifetime Savings Association (PLSA), an industry body. This allows about £55 a week on groceries and a two-week holiday abroad each year. A couple would need about £43,100 a year between them for this standard of living.

The earlier you start saving, the less you need to set aside. To maintain a moderate lifestyle in retirement you would need a pot worth about £494,000, according to the wealth manager Evelyn Partners. If you started saving aged 20 you would need to contribute £298 a month to achieve this by state pension age. A 30-year-old would need to set aside £484 and a 40-year-old £840 a month.

A comfortable standard of living in retirement, as defined by the PLSA, would require an income of £43,100 after tax for a single person and £59,000 for a couple. This works out at a savings pot of £794,000 — which would need monthly contributions of £479 if starting at 20, £779 at 30 and £1,352 at 40.

About four in five workers are not on track for a moderate lifestyle, according to the PLSA. “If you increase your pension contributions every time you get a pay rise this can be a relatively painless way of boosting your pension,” said Helen Morrissey from Hargreaves Lansdown.

Paying too much

Savers who don’t keep an eye on fees could come to regret it, experts warn. You will be charged for the fund you invest in, by the company that manages your pension, and for financial advice if you take it. These fees can vary significantly, and can quickly add up and eat into your investment returns.

“The annual fees charged on pensions can be an absolute killer. You are literally giving money away. Paying less could mean a complete lifestyle upgrade in retirement,” said Ian Millward from Candid Financial Advice.

Customers pay 1.9 per cent a year on average for advice and portfolio charges, according to the Financial Conduct Authority, the City watchdog. The advice website Boring Money compared the fees of six firms.

It found that Vanguard was cheapest for someone with £100,000. It charges a platform fee of 0.15 per cent a year.

If you have £250,000, the cheapest option is Interactive Investor. It charges a fee of £12.99 a month plus £3.99 each time you buy or sell a fund.

Whoever you choose you’ll also have to pay an annual fee for the fund you invest in on top.

• Are you on track for your dream retirement?

Check how your money is invested. Most workplace pension schemes start moving your money away from stocks and shares and into assets that are considered safer, such as government bonds, as you approach retirement. This is known as de-risking. If you don’t want your money to be de-risked you can choose a higher growth fund or set a later expected retirement date.

Not making a plan

You need to start thinking about retirement at least ten years before you get there, but the earlier the better. Take into account your workplace pension pots, savings, investments and state pension entitlement to work out what your retirement income will be, said Gianpaolo Mantini from the wealth manager Saltus. You could consider downsizing your home to free up money to contribute to your retirement pot.

Plan how you will spend your time in retirement too. “One of the biggest regrets people have is not thinking about how they will spend the first few months of retirement,” Mantini said. “When you are at work, your life is often structured. Some people stop work and have no hobbies or aspirations so they feel bored and go back to work.”

Taking out money early

Most people can access their pension from the age of 55 (rising to 57 in April 2028), but it may not be a good idea to raid your savings straight away.

Stephen Lowe from the insurer Just Group said: “Having what looks like a large pool of money at hand can be a temptation, but anyone thinking of accessing cash early needs to consider the long-term consequences.”

You can withdraw 25 per cent of your pension pot as a tax-free lump sum, and while it might make sense to do this if you need to pay off your mortgage or make a big purchase, there is no point leaving the money in your bank account. Leaving the money invested means it can continue to grow.

Withdrawing the lump sum in stages can also pay off. If you had a £400,000 pension pot you could choose to take £100,000 tax free in one go. But if you withdrew the money over ten years as a series of lump sums, of which 25 per cent of each was tax-free, you could take £125,779 in total because it had had more time to grow, Mantini said.

“If you take money out of your pension pot too soon, you are giving up tax-free growth unnecessarily,” Lowe said. “If you leave it in the bank, you could be taxed on the interest the money earns, particularly if you are still earning an income from work.”

• Retirement planning tool

Some 28 per cent of over-55s took money out of their pension before they stopped work, according to a survey of more than 1,000 people by the insurer Just Group. But about 8 per cent said they regretted doing so.

If you plan to leave an inheritance, it can make sense to leave your pension untouched for as long as possible and use up other savings first. Pension pots do not normally count towards the value of your estate for inheritance tax purposes. If you die before age 75, withdrawals made by your beneficiaries through drawdown are tax-free. If you die after 75, your beneficiaries will typically pay income tax on any withdrawals.

“It has become more common for people to drain their Isas and sell second properties, if they have them, to fund their retirement before touching their pension pot,” said Jason Hollands from the wealth manager Evelyn Partners. “The risk is that the government might remove the favourable inheritance tax treatment of pension pots in its upcoming budget.”

‘I wish I had taken control sooner’

Robert Trott regrets not managing his pension pot himself. For ten years Trott, 54, paid fees of 1.88 per cent for his £445,000 pension pot to be invested with the help of a financial adviser — about £8,370 a year based on his pension’s present value.

In November 2022 he decided to go it alone and invest in a low-cost fund, and he now pays about £1,135 a year.

“I was losing thousands of pounds a year in fees. It’s a lot easier than you would imagine to manage your money yourself,” said Trott, who lives in Bath. “The savings definitely make it worthwhile.”

• The best Sipp providers

Trott is a pilot and started looking at ways to save money when he was made redundant during the pandemic.

He set up an account with the firm Interactive Investor, which costs £12.99 a month, and invested his pension pot in the Vanguard LifeStrategy Fund, which comes with a charge of about 0.22 per cent a year. He regrets not doing this sooner.

“I’m saving thousands every year,” said Trott, who is looking forward to travelling and going on more bike rides with his wife, Lucy, 54, when he stops working. “Now my pot is invested in about 80 per cent equities, but at my age it’s worth it. I’m not planning to retire for a while.”