Posts

I’ve been using large-language models (LLMs) most days for the past few months for three major use cases : data analysis, writing code, & web search¹.

Here’s what I’ve observed:

First, coding incrementally works better than describing a full task all at once. Second, coding LLMs struggle to solve problems of their own creation, turning in circles, & debugging can require significant work. Third, LLMs could replace search engines if their indexes contain more recent or evergreen data for summarization searches but not for exhaustive ones.

I’m watching public company earnings to identify early trends in the software market to inform startups’ plans for 2023. Earlier, Mongo DB announced earnings. I’m adding Mongo to the list of tracked companies for this series.

The economic slowdown which started one year ago has reduced growth rates by 31% across this group from 46% annually to 31% annually.

Ten years ago, Nvidia’s market cap hovered around $4b, down from its previous high of $13b in 2008.

Today, Nvidia is the fifth most valuable technology company in the US, worth nearly $1t - just 20% less valuable than Amazon.

In a decade, the business increased in value about 250x, compounding at about 74%.

Nvidia’s most recent breakout occurred in the last two years.

The top publicly traded software companies have surged in the last few weeks.

MongoDB (MDB), Snowflake (SNOW), DataDog (DDOG), Cloudflare (NET), & ServiceNow (NOW) have watched their forward multiples surge by 42% to 112% since the beginning of the year. The bond market has priced in a roughly 1% to 1.25% cut in the next 24 months. When interest rates fall, high-growth software surges.

Yesterday, the Enterprise Tech 30 List was revealed. Congratulations to all the winners (especially MotherDuck, Hex, & Omni)!

I analyzed the headcount patterns within these companies to shed light on three questions :

1. How are these top companies changing their headcount through the downturn?
2. What percent of headcount is in product & engineering?
3. What percent of headcount is in sales & marketing?

The typical company grew headcount by 57%. The mid-stage businesses grew fastest at 88%, next early companies at 62%, & the later companies by 43%. The Giga companies held headcount flat over the past year.

A product manager today faces a key architectural question with AI : to use a small language model or a large language model?

The pace of innovation in the field clouds the answer. Each day, researchers publish novel findings on performance, discover new techniques to implement, & surface new challenges to wrestle with.

This is my current mental model of when to choose a large or small model :

When to choose a large model :

Great companies transform a technology innovation into a go-to-market advantage.

In the last five major cycles (internet, social, mobile, cloud, web3), startups seized the new technologies of the era to create a distribution advantage.

Google led to search-engine optimization, which startups like Expedia & Zillow grew to become public companies. Mastering social distribution (sheep-throwing in the early days) on MySpace & Facebook’s propelled YouTube & Zynga, not to mention the social networks themselves.

I’ve been tinkering with LLMs.

First, I created a chatbot using all my blog posts. Then I created a model to produce blog entries based on my writing.

As I went through this process, I asked myself some questions along the way :

• Does the ML model ingest my posts & keep them? What if I’d like to remove them from the logs or training set or output for others?
• The blogbot often returned no answers for questions I thought would be straightforward. How can I ensure the response rate is 99%+ before launching?
• Which post is better : hand-written, ChatGPT, or custom-trained models? How would I judge? More views? Written in my voice with alliteration & rhetoric? Use of the & instead of “and”? Injects links to other related posts?
• When is a blogbot a better user experience than search? With search, I can know that I’ve plumbed the depths of the blog, looking at every relevant post for an answer. How do I do that with a bot?
• The latency for both is significant : 5 to 25 seconds depending on the query. Google observed 400ms increase in latency decreases traffic by 20%. Will users be more patient with bots than with search?
• What if I reimagined the home page of tomtunguz.com as a chatbot interface rather than a list of all posts? That UI would personalize the experience for each visitor & each session, but it would hamper browsing? Which is the more important use case?

I imagine many product teams are asking analogous questions about how to leverage these new models.

Which database generated a -$1.8b loss & within a year produced $130m in profits? Ethereum.

Last year, I compared MongoDB & Ethereum growth to draw a parallel between two major database companies. This year, web3 activity has fallen dragging revenues along for the roller coaster ride.

Ethereum revenues have collapsed 82%, which makes the earnings change that much starker. What other business suffered a loss of 80% of revenue & went from burning $2b per quarter to producing more than a hundred million in profit?

I asked ChatGPT about the numbers 1 & 4. Which one is bigger?

Sometimes, 1 was bigger. Othertimes, 4 was bigger. Sharon Zhou ran this experiment at scale to showing the order of yes & no matters in the response.

This is called a non-deterministic or stochastic answer. Similar inputs do not consistently produce identical outputs. The answers have inconsistent logic.