The Magic Runs Out

The Other $85 Trillion series | redebuter.com

In 2015, my team at Happen set out to teach a machine to feel.

We formed a joint venture with a specialist technology partner to build a tool that could scrape social media and read something more useful than positive or negative sentiment. What we actually wanted was frustration and excitement, specifically. Frustration, because it's the earliest available signal of an unmet need, the thing people complain about before anyone has built the product that fixes it. Excitement, because it's the earliest signal that something new is starting to land, before the sales numbers catch up and prove it. We called the product Starmaker. The thesis underneath it was one I still hold: innovation starts with a real human insight, not a framework, and the two most honest places to find that insight are the moments people are annoyed and the moments they're delighted.

This was pre-AI, or near enough. Off-the-shelf natural language processing at the time could tell you whether a post was broadly positive or negative and not much else. To get to something as specific as frustration or excitement, we, or rather our very brilliant technology partner, had to hand-build elaborate coding structures we called lenses, that told the system what to look for and how to weigh it. There was a lot of hard work involved in building the lenses, but it worked. The scraping itself was challenging at the time too. Data came back messy, duplicated, half-formed, and with no AI to help clean it, a meaningful chunk of every project was just tidying up before the analysis could start.

We used it mostly with FMCG clients, but not only. Several technology clients used it too, including a major smartphone manufacturer, for launch analysis: what people actually felt in the days after a new phone landed, rather than what the press release said they should feel. We got the business to its first £1m of revenue. That's a real number, and I'm proud of it. What I'm more honest about now is that we never took it much further than that. Starmaker made money as a consulting tool, but we didn't manage to scale it.

There were lots of reasons why. We were early and trying to create a market. We probably weren't bold enough. We weren't clear enough about how we were taking it to market, and we didn't give it enough focus. But underneath all of that sat three more fundamental problems. Two of them were technical: the emotion-reading took a huge effort, and the scraping was messy. The third wasn't technical at all, and at the time we didn't clock how different it was from the other two.

Every fresh scrape looked, for a few weeks, like magic. Real, specific, usable clues would surface: a recurring frustration nobody at the client had named, an early flicker of enthusiasm for something that hadn't been marketed yet. It felt like discovery. Then, particularly in low involvement markets, the well ran dry. The same handful of genuine signals would repeat themselves in slightly different language, and everything after that was noise that had to be dressed up to look like insight. Real emotional signal, it turns out, is rare. Most of what people say online is repetition, cliché, and echo. The interesting bit is a thin seam in a very large amount of rock, and once you've mined the seam, mining harder doesn't necessarily produce more of the interesting bit.

We had found a way, with smart technology and a lot of manual labour, to solve two genuine engineering problems. We had not solved, because it isn't solvable, the fact that genuinely new human signal is scarce.

Which brings me to now. The two problems that were genuinely technical in 2015, reading emotion properly and cleaning messy scraped data, are close to solved. Modern AI does it faster, and without the manual labour, although there is still a lot of value in using lenses in certain use cases, but that's another story. If Starmaker were a decision I was making today, that half of the problem would barely register.

A good chunk of what I see now, in the businesses wrestling with retrieval-augmented generation, or RAG, to get more out of their market insight data, is exactly this: Starmaker's easy half, solved, and its hard half, quietly assumed away.

Because the third problem hasn't gone anywhere, and I think it's about to matter a great deal more than it did to us. Genuine signal was never scarce because the tools were bad. It was scarce because most human expression, most of the time, isn't novel. Faster tools don't manufacture more of it. What they do is burn through the existing scarce signal faster, and with far less friction than we had. Our tech was so slow that its own limitations gave us a kind of forced honesty: we could feel the well running dry because finding anything took real effort. Today's tools are quick enough, and confident enough in their own output, to disguise repetition as fresh discovery.

A RAG built on scraped social data has exactly this blind spot. It retrieves the nearest matching chunk with complete confidence, whether that chunk is the first genuinely fresh signal in the corpus or the fortieth repetition of it wearing different words. Retrieval isn't judgment. It's just fast enough now, and quiet enough about its own limits, that nobody notices when the corpus has nothing left to say.

If there's a practical takeaway from a decade-old product that didn't scale, it's this. The question worth asking of any AI insight tool, RAG included, isn't how much it's finding. It's how quickly it runs dry, and whether anyone would notice if it did. Genuine scarcity should look like scarcity, even to a fast tool. A system that keeps surfacing something new indefinitely isn't a sign the well is deep. It's usually a sign nobody's checking.

That's the warning. The more interesting question, and one I want to come back to, is whether it has to stay that way. An exhausted corpus doesn't have to be a wall you quietly stop mentioning to the client. There may be a way to build a RAG that knows when it's mining an old seam, and goes looking for a new one instead of repeating itself with a straight face. I think there is. That's a piece for another day.

We ran out of magic at Starmaker long before we ran out of client interest in believing there was more of it. I wonder how many teams running AI over their own data right now are heading for the same moment, just considerably faster than we were.

How would you know if your AI insight tool had already run dry, and simply hadn't told you yet?

#Leadership #Innovation #AI #Strategy #Redebuter