The IRS targeting certain groups for harassment because of their politics would be unfair. If we found out the NSA was keeping special tabs on everyone who worshiped at a mosque or took a Bible trip through the Middle East, you’d have an uprising.
It doesn’t take a majority to make a rebellion; it takes only a few determined leaders and a sound cause.