Posts

Today all inference is real-time. A human types, a model responds, & the clock starts over. The infrastructure is built for someone waiting on the other end. Every millisecond of latency costs money because the serving stack optimizes for cold-start, not throughput.

As we built internal AI systems at Theory, we embraced queueing. Parallelize ten agents on a single task, let them run for hours, & the productivity gains are enormous. It is the product of token-maxxing,¹ pushing every dollar of compute to do more work. But the cost was unsustainable.

On Thursday, July 9th at 9 AM Pacific / 12 PM Eastern, Office Hours will host Sunil Agrawal, CISO at Glean, for a conversation on what security readiness looks like now.

We’ll talk about :

• AI compresses the time required to understand a target, map the attack surface, and personalize the first move.
• The grammar, tone, and context clues that once revealed attacks are disappearing.
• Deepfake calls and synthetic media change the control plane for approvals, payments, and trust.
• Security teams will need new processes, tools, and organizational muscle to respond at the pace of model-driven attacks.

The format. 15 minutes online. One topic. Call-in questions live. No slides. No pre-written questions. Just a real conversation.

Crypto is the quietest part of the venture capital market. Funding is at multi-year lows. The narrative says the space is moribund.

But beneath the surface, something structural is happening.

Crypto is now a top-10 holder of US government debt. Stablecoin issuers hold $165b in US Treasury bills. That is 2.5% of the total $6.1t T-bill market.

To put that in perspective, stablecoin issuers now hold more T-bills than China, Norway, or Switzerland. They rank behind only Japan among foreign holders.

The fastest-growing companies in AI are either selling inference or reselling it. They’re its first derivative. But reselling inference at cost is a zero-margin business : a payment rail, not a software company.

So how do you keep 30 points of gross margin or more?

It comes down to the same distinction every sales pitch makes : cost-plus pricing versus value-based pricing.

The chart shows the two cost-based mechanics. The solid orange line is cost-plus : customer price rides 30% above. The dotted green line is optimization : delivered cost starts near the inference line & falls away as the engine compounds. Value-based pricing isn’t on the chart : it’s decoupled from the inference line entirely.

The gap between Databricks and Snowflake was $490m in March. It’s $1.6b today.

Databricks announced it has crossed $6.9b in annualized recurring revenue, up 80% year over year.¹ Snowflake’s latest quarter puts them at roughly $5.3b ARR, up 34%.

Each quarter adds more distance.

Today, a Hacker News thread asked a simple question : “Has anyone replaced Claude/GPT with a local model for daily coding?”¹ 500+ comments later, a clear picture emerged of the local coding stack.

Qwen 3.6 35B-A3B dominates model mentions at 33%, followed by the 27B variant at 20%. DeepSeek Pro & Gemma4 31B round out the top four. The common thread : mixture-of-experts architectures that run fast on consumer hardware.²

We’re entering the golden age of AI applications. Three recent developments confirm it.

The Fable retraction shows regulatory risk. Nadella’s thesis shows strategic consensus. Salesforce’s acquisition shows market validation.

First, the US government shut down Fable access¹ & the software ecosystem roared with many responses : Bring it back! Open-source & local models have become essential! Don’t rely on a single model!

Satya Nadella published an AI ecosystem thesis.² He argued that for a healthy ecosystem, the moat can’t be the model. Instead, human expertise & the system around the model (the harness³) must be the moat.

SpaceX IPOs today. One hallmark of the largest IPO in history : Elon Musk’s astoundingly low cost of capital. Despite raising 25x more than the typical founder, Musk retained ownership in the top decile.

Some founders raise $2m for an idea. Others raise $15m. Yet others raise hundreds of millions.

We’ve reached the upper bound of AI.

Not in the sense that performance won’t improve. On the contrary, AI will improve AI.

But Anthropic’s Fable release has imposed a glass ceiling. How do you release the most powerful model in the world to everyone without destroying kingdoms?

Strong guardrails. It’s easy to trigger a gentle reminder of verboten topics : ask for a description of a plant cell or a detailed description of a modern large language model or question about software security.

Three forces are reshaping the AI cost structure :

1. Foundation labs are moving up the stack into applications,^{1 2}
2. Frontier model prices keep rising for the smartest models,³
3. Open-source models have crossed the good enough threshold for most use cases.^{4 5}

The natural response from AI buyers is substitution.

Coinbase⁶ :

At Coinbase we’re working hot on routing prompts to cheaper models where appropriate, & in some cases have been able to keep costs roughly flat, while token usage continues to grow exponentially.