Posts

The future of LLM evaluations resembles software testing more than benchmarks. Real-world testing looks like this, asking LLMs to produce Dad jokes like this zinger : I’m reading a book about gravity & it’s impossible to put down.

Machine learning benchmarks like those published by Google for Gemini2 last week, or precision and recall for classifying dog & cat photos, or the BLEU score for measuring machine translation provide a high-level comparison of relative model performance.

Mongo announcing earnings yesterday concludes this quarter’s index of software buyers.

As the chart above shows, the software market is plateauing & some companies are rebounding.

Why is a flat growth rate something to cheer?

First, 30% growth is now top quartile.

Second, each of these businesses has scaled, doubling or more in the last three years. A 30% growth on twice as large a revenue base signifies many buyers are ready to spend.

These are the top 10 posts by engagement for 2023 at tomtunguz.com.

Reflecting on them, they cover the doldrums of the first half of the year, the euphoria of AI reinvigorating the second half, & the valuation environment. The next five posts all mentioned major exits : Klaviyo, Loom, & the many take-privates in 2023.

I’m grateful to everyone who read, commented, debated, & shared these posts & others.

0. Launching Theory! We launched the firm this year & announced our first fund. I’m grateful to the many people who helped us achieve this milestone.
1. The Typical Startup Saw a 24% Increase in Sales Cycle in 2023. The first two quarters of this year were rough for startups. Lengthening sales cycles created pipeline shocks & instability in sales forecasting.
2. How Markets Value SaaS in 2023. The public market shifted from valuing absolute growth, to valuing profitability, to valuing efficient growth - all in less than 12 months.
3. Why Every Startup Needs an AI Strategy. ChatGPT & AI upended the ecosystem. Much like mobile technology became a de facto part of every startup, AI is now no longer a category but the core or a component of every product.
4. Every Customer is a Design Partner - Leading Your Sales Motion with Sales Engineering. I argued that the best sales processes are sales engineering sales. Founder sales start the company this way & great sales teams continue this practice.
5. The 4 Questions Startups Should Ask Themselves about Building with Generative AI. In this presentation at Saastr, we broke down the major questions for startups building with AI : the layer, the moat, the market, & the depth of AI to use.
6. The Paradox of AI and Data Roles: How Automation Will Increase Demand for Data Professionals. As data becomes critical to developing products, the need for data professionals only grows, even if AI automates rote data analysis & retrieval.
7. The Implications of Increased Regulatory Scrutiny for Startup Acquisitions. The anti-trust pressure from the US & European governments eliminates about 70-75% of the purchasing power for M&A in Startupland.
8. Which Increases Productivity More : The Advent of Personal Computer or a Large-Language Model?. The PC increased GDP in the US by an immeasurably small amount. On the other hand, AI has the potential to boost US productivity enormously, boosting GDP growth by 50% or more.
9. Which AI Model Should You Pick for Your Startup?. The early days of the constellation of models thesis which is starting to appear in buyers’ architectures. Most AI products will be built with a series of models: one large general purpose, several smaller models, & a router in front to optimize cost & performance.
10. Operating in the Dark. A rallying cry during the dark days of 2023, throughout all the layoffs, unpredictability, & dysfunctional fundraising market.

On December 14th at 9.30 am Pacific, Office Hours will host Philip Zelitchenko, VP of Data at ZoomInfo to discuss Managing Data as Product.

Recently, Philip shared his management techniques to run a data team like a standard product software development function with some key nuances.

I asked him to join us on Office Hours to share his perspective because we believe data and software engineering practices converge.

Snowflake announced earnings yesterday, echoing the strength within software companies more broadly & underscoring the shift in enterprise buyer behavior to normal after three quarters of extensive cost cuts.

“Q3 product revenue grew 34% year-over-year to reach $698 million. Non-GAAP adjusted free cash flow was $111 million, representing 7% year-over-year growth. Results reflect strong execution in a broadly stabilizing macro environment.”

The resurgence of strength is a recent phenomenon.

“For 3 weeks, consumption grew faster than any other period in the past 2 years. Consumption continued to grow in the month of October…Consumption trends have improved.”

Recently, Guy Podjarny, President of Snyk & I talked through an investor’s view of AI’s impact on security & data.

What’s changed with AI?

A reckless data security leak threatens the career of a head of data, just as much as a mismanaged security incident ends a CISO (chief information security officer)’s. Clorox, Uber, SolarWinds, Target, JPMorgan, Facebook, Equifax, & CapitalOne : All these CISOs have been fired in the last ten years for security issues.

I can remember at each career inflection point the person who took a bet on me.

When I joined Google it was Kim. When I moved to product, Marissa & Brian. Then to Redpoint, it was the early stage team : Geoff, Tim, Chris, & Scott. Lloyd & Frank at Looker for their Series A. Today, it’s the team & the LPs at Theory.

That’s the very beginning of the list.

Last week, OpenAI suffered management team infighting over the key direction of the company. The schism inside the company resulted in Sam Altman’s firing & Greg Brockman’s departure.

This outcome injects uncertainty in the market including key partners (Microsoft & investors) & employees about the company’s future.

Microsoft, recognizing risk with a key technology partner in one of their most important markets, stabilizes its future by hiring Altman, Brockman, & others. Microsoft has rights to the OpenAI models, has diversified its algorithm providers beyond OpenAI & counts among its staff several of the most critical people in AI.

How much does AI cost? It depends on a a few different dimensions, but for a typical use case where a user enters a query & expects about a 200 word response, the cost varies from 0.03c to 3.60c.

Different models cost different amounts. Also, the size of the context window is an important factor.

The context window is a fancy way of saying the quantity of background information the user can provide to the AI.

12 years of zero-interest rate policies & then a sudden increase to 5.5% woke us all up to the Wall Street mantra : Don’t fight the Fed.

I’ve been reading to understand the history of the Fed & found some surprising facts from these two books : The Repo Market & Broken Money.

The US Federal Reserve Banking system is about 115 years old. It was created after the Panic of 1907. In October of that year, companies couldn’t borrow money because a trading firm called the Knickerbocker Trust went bankrupt trying to corner the copper market.