Taxpayers are on the losing end of inadequate funding and bigger workloads at the Internal Revenue Service, the National Taxpayer Advocate reported today. As a result, they're suffering small indignities such as unanswered phone calls and larger insults that represent an erosion of their legal rights.

In her annual report to Congress, National Taxpayer Advocate Nina Olson outlined a number of issues that have made life harder and potentially less fair for American taxpayers. They include:

• Fewer calls answered, slower responses. From fiscal year 2004 to 2011, the percentage of phone calls the IRS answered from taxpayers requesting assistance dropped from 87 percent to 70 percent. Over the same period, the percentage of taxpayer correspondences that hadn't been addressed after 44 days increased by 309 percent.

• Dangerous dependence on automation. The IRS is increasingly relying on software to flag and adjust inaccurate returns, a process that that can end with taxpayers paying more. At the same time, the agency isn't classifying many of those cases as audits, so taxpayers don't have the same rights they would have with a full audit. In 2010 alone, the IRS contacted about 15 million taxpayers to readjust their tax liability, but treated only 10 percent of those cases as audits.

• Unnecessary "math error" letters. The IRS spotted 10.6 million "math errors" in 2010, up from 4 million in 2004. But in its correspondences with taxpayers, it often isn't clear about the problem, making it difficult for taxpayers to respond. And in many cases, the IRS needn't pester taxpayers about this issue at all; in a study of 2010 math errors, the advocate's Taxpayer Advocate Service found that the IRS actually had data on hand to immediately resolve more than half of the "math errors" it had identified.

• Delays in large refunds. Among taxpayers who contacted the Taxpayer Advocate Service when their refunds were withheld due to suspicion of fraud, 75 percent eventually got their money. But they had to wait an average of six months overall, for refunds averaging $5,600. Olson said those waits were of particular concern because a delay in refunding such large sums could adversely affect many households' finances, and because the IRS's stepped-up automation has the potential to incorrectly flag more claims as fraudulent.

• Accusations of fraud without due process. A new IRS program in 2011 identified 900,000 returns as fraudulent, but didn't send notices to taxpayers indicating suspicion of fraud. Olson said the procedure, which automatically voided the tax returns, "violates fundamental notions of due process," which gives individuals suspected of fraud notice and the opportunity to respond before the government takes adverse action.

• Burden of the "tax gap." Because the IRS is underfunded, it isn't able to go after all the money owed. That "tax gap" means that taxpayers who do pay their fair share shoulder a surtax of sorts, estimated by the IRS at $3,400 per household in 2006.

The National Taxpayer Advocate, appointed by and reporting to the Commissioner of Internal Revenue, is the official ombudsman for U.S. taxpayers.

Among the advocate's recommendations to Congress was a 10-point "taxpayer bill of rights," and a five-point list of taxpayer responsibilities. Read the report's executive summary here.