TLDR
Software has a popular trend called shift left. This describes the process of moving testing, security checks, and quality assurance to the early stages of development to catch risks as early as possible. However, my impression of the AI industry as a whole is that it is shifting right: it is moving away from fields that are result-oriented and data-driven. Instead, LLM companies are shifting towards problems that are (A) harder to quantify or (B) so challenging that results are measured over years rather than months.
AI Eats the (Early-Stage Funding) World
In 2023, the viral spread of ChatGPT altered the landscape of Venture Capital investing and early-stage startups. Y Combinator’s company profiles provide a mechanism to explore this change. Take a look at an overview of YC company descriptions before vs. after the launch of ChatGPT.
In the first quarter of this decade, you can see a focus on a broad set of fields, from finance to healthcare to marketplace solutions (click into any of the inner circles to get a closer look at the more granular topics). Data Analysis is maybe the 4th most popular category, but it is one of many in the long tail of interests.
This sunburst is an aggregation over many batches. Areas of focus crest and fall with each batch. That being said, compare this diversity to YC’s focus over the past 2.5 years:
Normally I would perform a more rigorous trend comparison, but your own eyes reveal a massive shift in focus within early-stage startups. AI really ate the early-stage software world. (If you are so inclined, you can dig deeper into these sunbursts yourself →)
However, similar to the pre-2023 model, this aggregation of topics does not capture the crests and falls of topics batch by batch, year over year. And I think the crests and falls of areas of focus for AI tell an extremely compelling story.
Crests and Falls
In 2023, AI startups were attempting to automate some of the most high-leverage, highly paid sectors on the market. All of a sudden there was a PhD-level researcher in your pocket. Startups began popping up attempting to automate financial analysis, insurance audits, real-time logistics, and sales outreach. It made sense: these are high-leverage fields that have often been at the cutting edge of technology and machine learning. They are highly data-driven, would be able to measure any change on the margins, and the margins are critical. However, when we jump from 2023 to 2024, these are the same fields that fell off the map at YC.
Now there’s a chance that the companies that chased these applications were so dominant there was no point in even attempting to compete with them. However, talking with anyone in insurance or finance will quickly reveal their day-to-day workflows are little changed.
There is a second, more likely explanation: that large language models are simply not reliable enough to perform meaningful data-driven analysis. And the more data-driven and measurable a field is, the more quickly LLMs will prove ineffective for automation. If you look at the bottom right corner of the slope plot above, it is littered with use cases with clear problem definitions that would be incredibly lucrative for AI to automate. However, they are all in fields that are too results-driven to be swayed by a cool demo.
In 2024, there is a rise in slightly fuzzier use cases such as telehealth patient care, manufacturing problem detection, data synchronization, market analysis, therapeutic bioengineering, and robotics. All of these fields have tempered expectations by scaling back on full automation and by providing harder-to-define success criteria. It is much harder to measure if an LLM platform is effective at supporting telehealth patient care than it is to determine if sales numbers go down when AI is on the other end of the phone. Harder, but still not hard enough.
In 2025, all of these fields fall off the map. 2025 continues the previous trends of shifting to fuzzier applications and definitions of success. In fact, a major focus of 2025’s companies is addressing the limitations of previous batches’ technology. The highest growth topics focus on AI Context Management, Artificial Society Simulation, Data Access Controls, AI Security, Revenue Leakage Prevention, Access Control Policy, and Data Security Compliance. There are also some moonshots such as AI-Accelerated Clinical Trials.
Parting Thoughts
Thus far, I’ve primarily focused on directly presenting the results of my data mining (admittedly with an editorial tilt). However, I want to extrapolate a little bit.
I have listened to thousands of customer service and sales calls in my career and in the process annotated thousands of conversations. I have been consistently amazed at how excellent human customer service representatives are. I have also been equally amazed at how undervalued and actively handicapped these reps are by the organizations for which they work. Working with humans is nuanced and challenging, whether it be as a customer support agent or a sales rep for Boeing. Working with data is nuanced and challenging, whether it be as a data annotator or data scientist.
My technical view has always been that LLMs struggle with handling out-of-distribution inputs. The empirical data reveals that LLM applications are shifting right to more vague and more broad goals. There is a clear set of use cases for LLMs: massaging data for presentation or writing heavily supervised boilerplate-adjacent code. LLMs need to be able to fully automate key decisions to justify the current investment. While data-driven fields have been able to pick up on their limitations, I fear there is a broad set of applications (Customer Service, Data Analysis, Health Care Automation) for whom flawed automations will have a limitted effect on short term metrics but a large effect on longer term value.
Footnotes
[1] This data was organized by leveraging hierarchical mixture models.
[2] The slope plots are accessible here.
[3] The annotated dataset is publicly accessible at index_c50d731139f74898b4c004937ae77a83. This is most easily accessed using sturdy-stats-sdk. Additional documentation.