The AI You're Using Might Just Be Some Dude In the Philippines
Why tech startups fake it but don’t make it
We’ve reached the very public stage of the AI startup fraud cycle.
In fact, I don’t even need to write a funny opener for this post because it writes itself. Here’s what I want to take the contrarian side on:
TL;DR: The founder and former CEO of “Nate,” an AI shopping app that raised over $50 million from investors including Coatue, Forerunner Ventures, and Renegade Partners, was charged with defrauding investors earlier this month by the U.S. Department of Justice, who claims that the actual automation used by Nate was, wait for it, “effectively 0%.”
Who saw that coming?
But look, I’m not here to kick a fintech guy who used a bunch of low-cost human labor to fake AI while he’s down. Instead, I’m going to recklessly speculate on why this happened, why it wasn’t a stupid move, and what anyone using AI or building tech or founding a company can learn from it.
Yeah. That’s right. I said it wasn’t a stupid move. OK, the massive fraud part was. Let’s dive in.
How Does This Happen?
The story of Nate isn’t exactly new news. Back in 2022, The Information did an investigation on Nate. What’s news now is that the DOJ says that after $50 million in funding, there was, apparently, no AI to speak of.
So basically the AI was all just dudes in the Philippines, furiously “surfing the web” for the best fuzzy purple sweater. Not a horrible job, if you can get it. I wonder if the human shoppers had to doctor their responses so they sounded like a human but not too much like a human, consciously trying to emulate that drony, vanilla conversational voice that most AI leans on.
“Good news! I found four fuzzy purple sweaters exactly like you described, ranging in price from $34.99 to SPACE ICE CREAM MONKEY CIRCUS. Would you like to buy one in size Medium and have it shipped to your address in Nebraska?”
“Well, Nate, a little hallucination there, but here’s your check for $50 million.”
So, like always, the answer is money. But…
The thing is, these aren’t the first AI dudes from the Philippines
Not only is Nate not new news, it’s not the first human-run AI to get called out by the press, not even the first one employing the good people of the Philippines (and my apologies, “dudes from the Philippines” just sounds funnier).
The article references another article, back in December, 2023, about another AI startup that got caught using human labor in the Philippines to take “70 percent” of drive-thru orders from Del Taco and Carl’s Jr. And that article references another article about “AI-coded” apps being coded by humans in August of 2019, which references even more examples of humans faking AI.
Again. The answer is money. But the real question is: Why does it take $50 million before they figure out the ruse?
Well, I mean, for the same reason no one ever says, “Hey, I think I’ll start a Ponzi scheme! What could go wrong?”
Ok, someone probably has said that. But…
Humans are a better solution until they’re not
This is the first of two reasons why starting an automation build with human labor is not stupid.
A company I worked with employed folks from the Philippines for our overnight support center, and the economics worked out pretty well. They took phone calls, texts, and emails and made basic changes to our next-day schedule. They weren’t super cheap, but they were quite reliable and worth it.
We had been mucking about with chatbots and automation but this was a better alternative for the light load of overnight calls and emails we were getting. Even with our most liberal projections for growth, the ROI on an AI solution for that scenario was not worth the effort.
So you have to ask yourself: Unless you’re Amazon or Walmart, is AI worth it for shopping?
The answer is yes, but only at a critical mass of customers. The act of shopping, with all its options and outliers, is also a difficult process to automate. In fact, Amazon is just now taking a real stab at it.
So you need the money to pull off the difficult thing, and you need a critical mass of customers to get the money, but you need the difficult thing built to get a critical mass of customers.
Death spiral.
Let’s manually buy 100 purple sweaters
The second reason why using humans is not stupid is that AI is still stupid. Or, I guess “nascent” is a better word. Or “early.”
Back when we were automating data-driven articles at Automated Insights very early, in 2010, one of the things we did to get a handle on the complexity of any article we were going to automate was to read, analyze, and even hand-write hundreds of example articles covering the same topic. This was before LLMs were ready for prime time, but we knew if we stayed niche in our topics and focus, we could code up “lite” versions of LLMs on our own, and self-train them.
We coded all of it. We never hand-wrote anything for production. We couldn’t. Our first customer was Yahoo Fantasy Football which required over 13 million articles completed in three hours. But people still accused us of using humans at first.
I don’t know. What is that? 13 million monkeys on 13 million typewriters? That would do it.
This also made it very clear that our “magic” wasn’t magic, it was just damn good code.
But I can understand, with that kind of pressure and the death spiral swirling, having humans available to catch all the outliers or when the AI hallucinates is a nice-to-have escape hatch. It’s better than saying “try again later.”
That’s not stupid. But…
AI could be magic, it just isn’t yet
So let’s just not tell anyone.
That’s my guess as to how things like this happen.
NFTs aren’t really worth anything but let’s just pretend like they are because we’ve sunk all this cash into them.
Crypto isn’t feasible as a mainstream currency but what’s a little gas between friends?
This magic blood test isn’t real but what if nobody, you know, knew that?
That’s what I’m afraid of with AI. All it’s going to take is a few more stories like these before the tech loses the public entirely. And the more we use the words “magic” around this technology, the more we gloss over the fact that this is just really fast processing and should only be used that way, and the more investor money goes up in flames and the less likely the best use cases come to the forefront.
AI fails when it tries to be everything to all people for any reason. Don’t be a part of that death spiral. Start niche. You can even use humans if you want. Then figure out which parts are ripe for AI and limit your scope to that spec. It’s slower and a lot less exciting, but at least it’s not a human pyramid waiting to fall over.
I’ll keep trying to throw sensible lifelines through the chaos, so please join my email list and follow along.
canonical=https://www.inc.com/joe-procopio/the-ai-youre-using-might-just-be-some-dude-in-the-philippines/91176936" /