It all started when I knocked on a door in Palo Alto.
I noticed the little llama icon on the door & remembered how much I loved the product. Michael opened the door & welcomed me in. That is how I became friends with the Ollama founders, Jeff Morgan & Michael Chiang.
Since then, I have migrated most of my workloads to Ollama ; it turns out I’m not alone.
I still remember when my agent would forget what I said mid-sentence.
Context size is not the ceiling. Memory architecture is.
I have been experimenting with a memory architecture that runs preflight instructions. A pilot plans the route before takeoff. My agent does the same.
AI companies have committed $9.75b in 12 months to forward-deployed engineering. The FDE model, embedding engineers inside customers to deploy AI, has gone from a Palantir signature to an industry default.
That commitment is one quarter of Accenture’s annual labor cost.1
Three structural models are emerging.
The Economist ran 25 frontier AI models through the World Values Survey1, the questionnaire that has mapped the moral beliefs of 100 countries since 1981. For this 2x2, there are two axes : first, traditional (religious) to secular. Second, survival, with a focus on collective basic needs, to self-expression & individualism.
Most models sit in the self-expression half of the map, which makes sense given the training data.
Most teams building agents pick the model first & the architecture second. That is backwards. The model choice is the last decision, not the first.
What matters is the router, a small piece of code that decides which tier of model handles each request. Get the router right & 70-80% of traffic runs on local models that cost nothing per call, or on async models1 that reduce AI spend by 90%+.
The CIO’s priorities are clear. The public markets reveal them.
Two of five public software sectors are up over the last year. The other three are bleeding. The buying pattern is consistent : fund the AI stack, cut everything else.1
Anthropic spends 2.3x its payroll on compute.1 With ~5,000 employees & roughly $10b in inference & training spend in 2026, that works out to about $2m of compute per employee per year against a likely all-in comp of $500k+.2
The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI, 40% of a fully-loaded $224k senior engineer salary3.4 The median spends $137. That is the gap : 2.3x at the frontier, 0.4x at the top of the market, near zero at the median.
Every founder receives the question on slide three. What is your moat? They answer with technical differentiation. A model, a dataset, an architecture. At the application layer, that answer dissolves in a year.
What if there is no immediate moat? What if the moat is earned?
Leading moats exist at founding. Technical differentiation, a novel architecture, a proprietary dataset. You can point to them in the seed deck. They are most common at the infrastructure layer, where the product is the technology.
Today all inference is real-time. A human types, a model responds, & the clock starts over. The infrastructure is built for someone waiting on the other end. Every millisecond of latency costs money because the serving stack optimizes for cold-start, not throughput.
As we built internal AI systems at Theory, we embraced queueing. Parallelize ten agents on a single task, let them run for hours, & the productivity gains are enormous. It is the product of token-maxxing,1 pushing every dollar of compute to do more work. But the cost was unsustainable.
On Thursday, July 9th at 9 AM Pacific / 12 PM Eastern, Office Hours will host Sunil Agrawal, CISO at Glean, for a conversation on what security readiness looks like now.
We’ll talk about :
The format. 15 minutes online. One topic. Call-in questions live. No slides. No pre-written questions. Just a real conversation.
