In all, members of Congress sent at least eight letters to the IRS over the past two years, asking about complaints from conservative groups that they were being harassed by the IRS. None of the IRS responses acknowledged that conservative groups were targeted.
Miller was scheduled to testify Friday at a Ways and Means hearing. A committee aide said Wednesday evening that Miller was still expected to attend the hearing.
"More than two years after the problem began, and a year after the IRS told us there was no problem, the president is beginning to take action," said Senate Republican leader Mitch McConnell of Kentucky. "These allegations are serious - that there was an effort to bring the power of the federal government to bear on those the administration disagreed with, in the middle of a heated national election. We are determined to get answers."
The Justice department opened its criminal investigation on Friday, Holder said.
"I can assure you and the American people that we will take a dispassionate view of this," Holder told the House Judiciary Committee at a hearing Wednesday. "This will not be about parties, this will not be about ideological persuasions. Anybody who has broken the law will be held accountable."
But, Holder said, it will take time to determine if there was criminal wrongdoing.
Legal experts, however, said it could be difficult to prove that IRS officials or employees knowingly violated the civil rights of conservative groups. If there is a violation, the experts said, investigators can sometimes prove more easily that officials made false statements or obstructed justice in some other way.
"I think it's doubtful that any of these knuckleheads who engaged in the conduct that gave rise to this controversy knowingly believed that they were violating the law," said David H. Laufman, a former Justice Department lawyer. "But that remains to be seen. That's what investigations are for."
The IRS started targeting groups with "Tea Party," `'Patriots" or "9/12 Project" in their applications for tax exempt status in March 2010, the inspector general's report said. The criteria later evolved to include groups that promoted the Constitution and the Bill of Rights.
Wednesday's hearing was the first of several in Congress that will focus on the issue.
The House Oversight Committee announced Wednesday that it would hold a hearing May 22, featuring Lois Lerner, the head of the IRS division that oversees tax exempt organizations, and Shulman, the former commissioner.
The Senate Finance Committee announced a hearing for next Tuesday.
Colleen M. Kelley, president of the National Treasury Employees Union, said Wednesday that no union employees had been disciplined, as far as she knew. She noted that the IG's report said agents were not motivated by political bias.
Kelley told The Associated Press that low-level workers could not have specifically targeted conservative groups for long without the approval of supervisors. However, she noted, there are many levels of supervisors at the IRS.
"No processes or procedures or anything like that would ever be done just by front-line employees without any management involvement," Kelley said. "That's just not how it operates."