He said he ordered the FBI to investigate Friday - the day the IRS publicly acknowledged that it had singled out conservative groups.
"Those (actions) were, I think, as everyone can agree, if not criminal, they were certainly outrageous and unacceptable," Holder said. "But we are examining the facts to see if there were criminal violations."
Numerous congressional committees already are investigating the IRS for singling out tea party and other conservative groups during the 2010 congressional elections and the 2012 presidential election. But Holder's announcement takes the matter to another level, if investigators are able to prove that laws were broken.
Holder said he wasn't sure which laws may have been broken.
In an opinion piece in Tuesday's editions of USA Today, acting IRS Commissioner Steven Miller conceded that the agency demonstrated "a lack of sensitivity to the implications of some of the decisions that were made." He said screening of advocacy groups is "factually complex, and it's challenging to separate out political issues from those involving education or social welfare."
Miller said the agency has implemented new procedures that will "ensure the mistakes won't be repeated."
On Monday, the IRS said Miller was first informed on May, 3, 2012, that applications for tax-exempt status by tea party groups were inappropriately singled out for extra scrutiny.
At least twice after the briefing, Miller wrote letters to members of Congress to explain the process of reviewing applications for tax-exempt status without disclosing that tea party groups had been targeted. On July 25, 2012, Miller testified before the House Ways and Means oversight subcommittee but again did not mention the additional scrutiny - despite being asked about it.
Miller's op-ed did not address why he did not inform Congress after he was briefed.
Miller was a deputy commissioner at the time. He became acting commissioner in November, after Commissioner Douglas Shulman completed his five-year term. Shulman had been appointed by President George W. Bush.