AI Fraud is About To Get Unreal
The real problem isn’t AI fooling humans. It’s much worse.
“Joe, do you know how to remove Anydesk?”
Uh-oh.
The scene I’m about to set up couldn’t be more cliché.
The scam happened to my mother-in-law. It’s the scam where a friendly email notifies her of a payment on her subscription service and asks her to call a number if she suspects an error. Then she winds up talking to an even friendlier guy in a foreign country.
What happened next surprised me though. Right before it scared the crap out of me.
Not Today, Scammer
I need to give my mother-in-law a lot of credit here. She had actually just paid a final payment on the exact service the scam email was referencing, so she called to clear up what appeared to be an everyday duplicate charge. I get those emails once a month because Capital One can’t figure out I have twins.
Also, the minute she heard “gift cards,” she flashed back to a conversation we had a while ago about the “blue-haired guy on the YouTubes” and she immediately hung up.
But not before she installed Anydesk. So we uninstalled Anydesk and I made sure she had opened no other vulnerabilities. It took a while but no harm done. It seemed like a good time to change all her passwords anyway, and we left it at that.
Fraud Is Inevitable, Like Thanos
Also to her credit, she didn’t panic or get embarrassed. And when I commended her on her steely-eyed demeanor, she laughed and said, “It’s inevitable these days.”
A few minutes later, after we had hung up, it hit me.
Call me what you will, but the techie in me thought, “I bet if I was a scammer I could figure out how to use agentic AI to bypass my mother-in-law completely and get to those sweet gift cards.”
Oh shit. I got pretty far. The only firewall I had left to knock over was the bank that can’t figure out why I have to buy two of the same thing sometimes.
The problem with AI fraud isn’t AI fooling humans, but the fact that humans build the systems we already use.
How long before the scammers start using AI to fool the systems?
And that’s the moment when I realized that if I ”just figured it out,” they’re already doing it.
I’ll wait here while you go and change all your passwords. But come back. I’ve got more to say.
We’re Focused on the Wrong Thing
Look. It’s 2025. I’m a super nice guy but I need to be direct. If someone is in a position to be swayed to do something stupid by a deepfake in any forum where a deepfake could be dropped, let’s just say there’s a gap going on between them and the world around them, and I can’t do anything proactively about that. Neither can you. Neither can anyone else.
We can all get fooled. It can happen. I can also get struck by lightning, but I still have the courage to leave the house when it’s raining.
I’m not trying to be callous. Just honest. And I’m not alone here. In your heart of hearts, you probably believe the same thing. You’re not gonna fall for the banana in the tailpipe.
But in a sense, that kind of fear is a red herring. I’m not saying there’s a conspiracy or anything, but maybe if we feel confident about not falling for fraud and scams when they come directly at us, maybe we get too comfortable thinking we’re automatically safe when the fraud doesn’t come directly at us.
We’re not safe. At all. Again, I got pretty far completely bypassing my mother-in-law in ten minutes on the back of a napkin. And I’m no David Lightman.
War Games? Matthew Broderick? High school kid almost starts a nuclear war hacking into the DoD?
Automatic Seatbelts for AI
I’m not going where you think I’m going with this argument. But I’m going exactly where we need to go. I’ve been screaming for years about the backlash against a growing enshitification of the implementation of this faster computer processing we’re all slyly calling “AI.”
So, techies, CEOs, futurists, and all you leaders of industry.
Your systems aren’t secure enough for you to be pushing AI as hard as you are. The mistake was made over a decade ago when we absolutely, positively had to have “one-click” transactions.
So I ask you this.
Do you remember automatic seatbelts?
Very few of you. I think you have to be like 50 or older.
But you might have overheard old folks either marvel at or complain about how long we were allowed to zip around the highway without a seatbelt and it was perfectly cool.
Then they stepped in and gave us automatic seatbelts. That solved the problem in the dumbest way possible.
If anyone thinks we’re heading into a brave new world of convenience and technical enlightenment without consequences or pushback, just remember that in the short history of tech, every action produces an unequal and opposite overreaction.
So while we come up with novel ways to reduce the number of mother-in-laws sending gift cards to Kolkata, maybe we should instead be tamping down some of the blind rush to AI implementation and spend that energy putting a little friction back into our one-click systems before we let their AI talk our AI into sending our money to another AI.
We need to do this before someone comes in and makes us do it in the dumbest way possible.
If you enjoyed this, you should join my email list. I promise never to ask you for a gift card, but I do accept them.
canonical=https://www.inc.com/joe-procopio/ai-fraud-is-about-to-get-unreal/91201820