No, you didn't. The IRS will never cold call you. They will always send you at least one letter first, usually up to three, and if a phone call is needed they will schedule it so you call them. You got called by a scammer who is trying to scare you into giving away your credit card information. This is true for any cold call threatening you with legal action unless "you pay right now over the phone!!" Real organizations work through certified letters so they can prove you received the notices.
Here is a link to the IRS website about the scam: