Americans are growing more anxious about not having enough money to live comfortably after they retire, and that anxiety is translating to poorer physical health.

Only about 6 in 10 American workers are confident that they can retire with enough money to keep up with inflation and the cost of living in retirement, down from 67% last year. It’s the most insecure they’ve felt since 2017, according to a new retirement confidence survey.

For retirees, the concern is also inching up.

“The most interesting finding from this year’s report is that there’s a correlation with how people are feeling about both their finances and their physical health,” said Craig Copeland, director of wealth benefits research at Employee Benefit Research Institute (EBRI), which published the findings. “Not only are Americans rating their financial well-being lower, but also their physical health.”

Two years ago, more than 60% of people said their physical health was good. Now only 45% do.

“The pressures of increased debt, raising housing prices, and prices in general are creating increased uneasiness about the future,” Copeland said.

Debt is a ‘major’ problem

Debt is a problem for nearly two-thirds of workers and a major problem for a quarter.

Half of workers have credit card debt, and nearly 4 in 10 have more than $25,000 in non-mortgage debt. Two in 5 retirees report debt as a problem, but it’s not as pressing — more than half have less than $1,000 in non-mortgage debt.

Meantime, many retirees say they retired earlier than planned. Most stepped out of the workplace before they turned 65, and most often the reason was out of their control, from a health issue to a job loss.

There’s also a gap between how workers think they will stop working and the way it actually works out. While almost half of workers think they’ll make a gradual transition to retirement, for 3 in 4 retirees, it was a hard stop.

Retirees ‘spending money how they want’

Not all is grim. A majority of retirees say their standard of living is at least good, with half reporting that it’s excellent or very good.

Three in 4 say they’re able to spend money how they want within reason. “They’re enjoying it,” Copeland said.

But for many, expenses in retirement are steeper than expected.

There is “a striking difference by income,” Copeland said. Among those with annual incomes less than $35,000, more than half rate their lifestyle as fair or poor, compared with only 10% among those with incomes of $75,000 or more.

Kerry Hannon is a Senior Columnist at Yahoo Finance. She is a career and retirement strategist and the author of 14 books, including “Retirement Bites: A Gen X Guide to Securing Your Financial Future,” “In Control at 50+: How to Succeed in the New World of Work,” and “Never Too Old to Get Rich.” Follow her on Bluesky and X.

Sign up for the Mind Your Money newsletter

Click here for the latest personal finance news to help you with investing, paying off debt, buying a home, retirement, and more

Read the latest financial and business news from Yahoo Finance