Posts

GPT-5 achieves 94.6% accuracy on AIME 2025, suggesting near-human mathematical reasoning.

Yet ask it to query your database, and success rates plummet to the teens.

The Spider 2.0 benchmarks reveal a yawning gap in AI capabilities. Spider 2.0 is a comprehensive text-to-SQL benchmark that tests AI models’ ability to generate accurate SQL queries from natural language questions across real-world databases.

While large language models have conquered knowledge work in mathematics, coding, and reasoning, text-to-SQL remains stubbornly difficult.

Perplexity AI just made a $34.5b unsolicited offer for Google’s Chrome browser, attempting to capitalize on the pending antitrust ruling that could force Google to divest its browser business.

Comparing Chrome’s economics to Google’s existing Safari deal reveals why $34.5b undervalues the browser.

Google pays Apple $18-20b annually to remain Safari’s default search engine¹, serving approximately 850m users². This translates to $21 per user per year.

The Perplexity offer values Chrome at $32b, which is $9 per user per year for its 3.5b users³.

In 1999, the dotcoms were valued on traffic. IPO metrics revolved around eyeballs.

Then Google launched AdWords, an ad model predicated on clicks, & built a $273b business in 2024.

But that might all be about to change : Pew Research’s July 2025 study reveals users click just 8% of search results with AI summaries, versus 15% without - a 47% reduction. Only 1% click through from within AI summaries.

Cloudflare data shows AI platforms crawl content far more than they refer traffic back : Anthropic crawls 32,400 pages for every 1 referral, while traditional search engines scan content just a couple times per visitor sent.

GPT-5 launched yesterday. 94.6% on AIME 2025. 74.9% on SWE-bench.

As we approach the upper bounds of these benchmarks, they die.

What makes GPT-5 and the next generation of models revolutionary isn’t their knowledge. It’s knowing how to act. For GPT-5 this happens at two levels. First, deciding which model to use. But second, and more importantly, through tool calling.

We’ve been living in an era where LLMs mastered knowledge retrieval & reassembly. Consumer search & coding, the initial killer applications, are fundamentally knowledge retrieval challenges. Both organize existing information in new ways.

2025 is the year of agents, & the key capability of agents is calling tools.

When using Claude Code, I can tell the AI to sift through a newsletter, find all the links to startups, verify they exist in our CRM, with a single command. This might involve two or three different tools being called.

But here’s the problem: using a large foundation model for this is expensive, often rate-limited, & overpowered for a selection task.

What force could dethrone AWS after more than a decade of unchallenged dominance?

For years, Amazon Web Services ruled the cloud infrastructure market. It was the default choice without a question for every startup.

Then OpenAI released GPT-4. Microsoft’s exclusive partnership with OpenAI transformed Azure from a second-place player into the obvious choice for AI-first companies. With this week’s earnings, we are seeing the ultimate impact of that strategic decision.

Why is the sub-$5 million seed round shrinking?

A decade ago, these smaller rounds formed the backbone of startup financing, comprising over 70% of all seed deals. Today, as Jacob Robbins of PitchBook News reports, that figure has plummeted to less than half.

The numbers tell a stark story. Sub-$5M deals declined from 62.5% in 2015 to 33% in 2024. This 29.5 percentage point drop fundamentally reshaped how startups raise their first institutional capital.

The internet is about to look a whole lot more like the online advertising world.

No, I don’t mean there’ll be more ads. In fact, I think there’ll be far fewer.

But the technology stack for content distribution will mirror the architecture that has been implemented in the online ad world.

As we’ve reached the AI search tipping point, publishers face an existential challenge : ensuring AI systems use their content in answers to maintain relevancy.

OpenAI receives on average 1 query per American per day.

Google receives about 4 queries per American per day.

Since then 50% of Google search queries have AI Overviews, this means at least 60% of US searches are now AI.

It’s taken a bit longer than I expected for this to happen. In 2024, I predicted that 50% of consumer search would be AI-enabled.

But AI has arrived in search.

If Google search patterns are any indication, there’s a power law in search behavior. SparkToro’s analysis of Google search behavior shows the top third of Americans who search execute upwards of 80% of all searches - which means AI use isn’t likely evenly distributed - like the future.¹

In working with AI, I’m stopping before typing anything into the box to ask myself a question : what do I expect from the AI?

2x2 to the rescue! Which box am I in?

On one axis, how much context I provide : not very much to quite a bit. On the other, whether I should watch the AI or let it run.