The Bypass: When SQL Injection Grants System-Wide Access

Consider the dangerous implicit default for handling API keys. If you treat a machine token like a user password, the implementation seems obvious:

1. Generate a random secret.
2. Hash it (e.g., SHA-256).
3. Store the hash in an api_keys table alongside an org_id.
4. When a request comes in, hash the provided secret and look for a matching row.

This feels secure because the database only holds hashes, not plaintext secrets. But let’s look at what happens if an attacker discovers a blind SQL injection vulnerability anywhere else in the application.

The attacker doesn’t need to read the database or invert any hashes. An attacker can simply register a legitimate account and generate his own valid API key. Since the key is both legitimate and legitimately his, he knows the plaintext secret, and he knows the application will successfully verify it. So far, this is entirely licit.

Now let’s imagine the attacker finds a SQL injection exploit. By executing a simple UPDATE statement, he can copy the hash of his key and paste it over the hash of the victim’s key. When the attacker sends a request attempting to access the victim’s data, he uses his own (valid) key. The API middleware hashes it, finds the victim’s row (which now contains the attacker’s pasted hash), and grants access.

Boom. Now it doesn’t matter how many other security measures we have in place; the attacker has just bypassed the API security perimeter and the entire key-based tenant boundary layer. His attack uses a valid key. Because the application blindly trusts the database’s record of the hash, a simple database write translated into complete tenant takeover.

The Fix: Cryptographic Binding

Securing the database against SQL injection is obviously a priority, but the “bypass” nature of this attack means we need Defense in Depth comparable to what we apply at the perimeter. We must separate the security of the authentication system from the security of the database.

To achieve this, Sturdy Statistics does not store simple hashes of API keys. Instead, we use a server-side cryptographic pepper – a high-entropy secret TPM-sealed to the backend – to compute an HMAC-SHA512 signature.

Crucially, we don’t just sign the secret. We sign the structural context of the key:

Stored Hash = HMAC-SHA512(Pepper, api-key-id
                                  || rotation-version
                                  || org-id
                                  || secret)

Where do these pieces come from? The api-key-id, rotation-version, and secret are parsed directly from the incoming client token. The org-id is strictly pulled from the requested URL path parameter. The database provides the stored hash to check against, along with its record of the api-key-id, rotation-version, and org-id. Thus, the secret and stored hash each appear only once. The other identifiers are each stored in two locations. This “double-book accounting” allows us to check for consistency.

If an attacker manages to write to the database and swaps a hash from Organization A into Organization B’s row, the attack fails immediately. When our authentication middleware verifies the incoming token, it pulls Organization B’s org-id from the URL path and injects it into the HMAC calculation. Because the hash encodes the wrong org-id, the hash changes, the signature is rejected, and the attacker is locked out.

Crucially, because this signature requires the Pepper – which lives exclusively in backend memory and never touches the database – even an attacker with full DB read and write access is thwarted. It is impossible for him to mint or modify keys without the pepper. The database no longer decides identity on its own; it stores verifiers that only the backend can validate. Thus the backend’s defense layers remain intact and can’t be bypassed via the DB.

Paranoid Tenancy: Four Layers of Verification

Cryptographically binding the API key to the tenant is the first step, but Defense in Depth requires overlapping security boundaries. In a multi-tenant environment, cross-tenant data leakage is the most common security failure identified by OWASP.

To ensure absolute isolation, we enforce tenancy checks repeatedly throughout the lifecycle of every single request:

1. The Routing Layer: Tenancy is explicitly declared in every request path itself (as a path parameter), ensuring the routing infrastructure knows exactly which logical boundary is being accessed before any application code executes.

2. The Authentication Layer: As detailed above, API key verification is cryptographically tied directly to the org-id.

3. The Application Layer: Every read and write operation at the application level enforces tenant scoping. Business logic functions do not accept naked identifiers; they require the authorized tenant context to proceed.

4. The Database Schema Layer: As the final failsafe, we enforce tenancy at the lowest possible level using database TRIGGERs. Even if a logic error in the application code attempts a cross-tenant write, the schema rejects it.

Graceful Security: Zero-Downtime Rotation

Security mechanisms that are hard to use usually end up being bypassed. Key rotation is notoriously painful, often requiring coordinated downtime between the API provider and the client.

Because we explicitly control the exact mechanics of our hashing and database schema, we built zero-downtime rotation directly into the architecture. Our api_keys table includes a prev-hash column and a grace-period-expires-at timestamp.

When a user rotates a key, we generate a new secret, increment the rotation version, and compute the new primary hash. The old hash is demoted to the prev-hash column and given a strict time-to-live (e. g., 24 hours). During this window, the backend checks will authorize either key. This allows distributed client systems to update their environment variables asynchronously without dropping a single licit request.

If you’re paying attention, this re-introduces the rollback vulnerability: an attacker with DB write access and access to an expired key could resuscitate the key by overwriting the grace period. We solve this with another TRIGGER: the grace period can only be extended when the rotation version is also being incremented.

Containment without Containers

At Sturdy Statistics, we don’t believe that adding complex layers of distributed infrastructure is the only way to achieve security.

True security comes from architectural simplicity and mathematical rigor. By moving the trust anchor out of the database and enforcing strict cryptographic boundaries at the application edge, we ensure that a failure in one layer remains precisely that: a contained failure, not a catastrophic breach.

Nota bene: Throughout this post, I used the language of SQL injection and database TRIGGERs because those are the familiar terms of relational database architecture. Under the hood, Sturdy Statistics actually uses Datomic. Datomic’s immutable ledger means authentication state is never simply overwritten in place, which adds protection against attacks that depend on rewriting stored data. While Datomic does not use SQL triggers, we enforce the same one-way ratchets and boundary conditions through Datomic transaction functions. I chose SQL terminology here for accessibility, but the core lesson is the same regardless of storage engine: database state alone should not be able to grant authentication.

Equation solvers are closed worlds

When solving physical equations (extended magnetohydrodynamics in my astrophysics research) or statistical ones (probabilistic graphical models at Sturdy Statistics), the program is essentially an executable version of a mathematical theory. In such systems:

• the equations are known,
• the variables are precisely defined, and
• the relationships between quantities are fixed.

In other words, the model’s structure is predetermined. The program exists to faithfully implement a set of mathematical relationships. As a result, formal correctness is everything.

Mistakes in this sort of code are subtle and very difficult to find. A wrong index or an incorrect normalization constant won’t trigger compiler warnings or static checks. Moreover, the calculations are often too complex, interconnected, and (formally) chaotic to allow for meaningful unit tests. Since these are research topics, your code is sui generis: there is no other way to solve the equation than the one you’re building. If your code does not work, you have nothing to compare it to. And to top it off, incorrect results frequently appear perfectly plausible.

If you’re lucky, a bug produces NaNs or unstable convergence. More often it simply produces numbers that look reasonable but are wrong. Debugging these systems can be painstaking. It often means tracing numerical behavior through hundreds or thousands of iterations and checking results with pen and paper. It is not for the faint of heart.

In one case I’d prefer to forget, an error that subtly broke detailed balance took months to track down. The mistake was so subtle that I’m still amazed I caught it. Even so, it rendered the entire model useless.

This is where closed-world reasoning shines. When the structure of the system is fixed, you want to enforce as many invariants as possible and constrain the code tightly around the mathematical model. In my experience, the most important thing in these programs is discipline: clear structures, explicit invariants, and assertions everywhere. Once the implementation matches the mathematics, the program becomes a faithful machine for executing the theory.

Web systems are open worlds

Web services live in a completely different universe.

They deal with HTTP requests and responses, JSON payloads, database rows, user input, logs, and third-party APIs. Rather than implementing a fixed theory, these systems are managing information in a constantly evolving environment.

In these systems:

• schemas change
• fields appear and disappear
• integrations evolve
• user behavior is unpredictable

The program must accommodate data whose structure is only partially known, and which will inevitably change over time. In this environment, flexibility is paramount. Data structures that can grow and evolve naturally tend to work well here. The system must be able to ingest new information, adapt to changing schemas, and tolerate incomplete or messy inputs.

Fortunately, failures in these systems are usually much easier to diagnose. When a web service fails, the problem is not a deep conceptual mystery because you can see it: you have something like a stacktrace or a database inconsistency. Debugging involves inspecting data rather than tracing intricate numerical behavior.

Unlike a subtle “detailed balance” error, you aren’t left staring at plausible-but-wrong numbers, wondering what on earth is happening. Debugging web infrastructure isn’t solving a profound mystery; it’s a matter of diligently doing the work. By tracing a request through structured logs and inspecting the state at each boundary, you can almost always find exactly where the data was lost. The answer is in there, waiting to be found.

The Boundary Between Worlds

Navigating the boundary where these two worlds meet is our core business at Sturdy Statistics, and it’s how we convert unstructured data into processable entities.

Our customers hand us raw information, and typically, neither we nor the customer knows exactly what shape that data will take. That is why they hire us. Our infrastructure has to live entirely in the open world: it must be endlessly flexible, capable of automatically parsing, coercing, and structuring messy, unknown inputs.

How do we cross that boundary from open-world, open-ended ingestion to closed-world, tightly structured equation solving? It’s closely related to how we automatically surface structure from unstructured data.

Bayesian models occupy an interesting middle ground between open- and closed-world paradigms. As software, Bayesian models are extremely rigid. You define a graph structure, specify probability distributions, and implement inference algorithms that are not so different from the equation solvers used in physics. (In fact, many of them originated in physics.)

But philosophically, the model is built around a very different idea: we do not know the true structure of the world. Bayesian models are unusual in that they explicitly represent uncertainty within a closed (mathematically certain) system. While the model represents a concrete theory about the data, the Bayesian framework acknowledges that we cannot know the truth about the data. Instead of producing a single deterministic answer, the model integrates over many possible explanations consistent with the data. It is perhaps surprising that this flexibility requires exceptionally rigid software.

This means that Bayesian modeling combines a closed software structure with an open view of reality. The program itself is rigid, but the mathematics explicitly accounts for uncertainty about the world it describes.

Embracing the Open World

When I first started building web infrastructure, I struggled because I was trying to apply the habits I had learned writing closed-world physics simulations. I learned a lot by trial and error, but the shift didn’t “click” for me until I encountered Rich Hickey’s design philosophy.

Hickey’s designs in Clojure and Datomic are remarkably insightful for open-world problems precisely because they acknowledge that the full structure of your data is fundamentally unknowable. When you receive a payload, treating it as an open map rather than trying to coerce it into a rigid object hierarchy allows your system to be resilient to change. You operate only on the keys you care about and either strip the rest or pass it along.

Similarly, this mindset completely changes how you handle the hardest bugs in web infrastructure: concurrency and distributed state. In an open world, state isn’t a single variable to be locked and mutated. Tools like core.async acknowledge that you are often just coordinating independent, asynchronous flows of information. Using channels to manage these flows – rather than relying on shared, mutable memory – has proven amazingly useful in our real-world applications for orchestrating unrelated tasks.

This philosophy extends directly to another major theme of our work: web security. In an open world, we do not control our inputs. We have to handle data that is messy, incomplete, malformed, and frequently actively adversarial. You might assume that securing against these threats requires rigid, closed-world code across the board. In reality, while specific security behaviors – like tenancy boundaries and authentication checks – must have very strict, absolute definitions, the code that handles the data must remain flexible.

Paradoxically, flexible data handling makes those strict security boundaries much easier to enforce. Many common exploits arise from rigid-but-wrong boundaries: forcing a malicious payload into a complex, inflexible object hierarchy often leads to deserialization flaws, parser differentials, or edge-case bypasses. When you treat input simply as open data, you can rigorously validate the exact fields required for authorization and safely discard the rest, without your system implicitly trusting or coercing a malicious structure.

The Takeaway

At the outset of a project, the most valuable question you can ask yourself is simply: Am I building a closed world or an open one? If you are implementing a closed world – a mathematical model, a specific algorithm, a bounded physical simulation – the structure is fixed. Your greatest danger is subtle, silent error. The tools you need are constraints, explicit invariants, and the discipline to tightly couple your code to your theory.

If you are building an open world – a web service, an ingestion pipeline, a distributed system – your greatest dangers are complexity and brittleness. You exist to manage information that is incomplete and constantly evolving. The tools you need are flexibility, immutable data structures, and clear boundaries that isolate state.

The two types of software reward very different instincts: one requires the rigor of a mathematician, the other the adaptability of a systems engineer. By recognizing which world you’re operating in at any given moment, you stop fighting the nature of the problem, and your code naturally becomes both simpler and much more reliable.

The Core Philosophy: Files are State

You might associate make with C projects, where it’s commonly used as a build tool. The standard pattern is to recompile any .o object file that is older than a .c source file using something like this:

SOURCES=$(wildcard *.c)
OBJECTS=$(SOURCES:.c=.o)

MACROFILES=defs.h constants.h

%.o: %.c $(MACROFILES)
    $(CC) $(CFLAGS) -c $< -o $@

If you stop to think about it, this is incredibly powerful. This short script:

1. Finds every .c source file in the directory.
2. Uses Make’s substitution system to build a list of corresponding object files.
3. Constructs a Directed Acyclic Graph (DAG) of all dependencies.
4. Considers an object file outdated if it is older than its source file or any macro files.
5. Resolves them using the provided recipe.

The syntax seems arcane if you’re not used to it, but once you learn it, it lets you write clear, concise dependency rules.

Ops as State Management

I understand Ops to be fundamentally about state management:

• Is the systemd unit file (/etc/.../app.service) older than my configuration template?
• Is the installed JAR file older than the latest release?
• Does the GPG keyring exist?
• Has the signature on the JAR file been verified?

Crucially, when you model your deployment using Make, you get idempotency for free.

A Bash script usually runs every command every time unless you litter it with if [ ! -f ... ] checks. Make solves this natively. If the target exists and is up to date, Make does nothing.

For example, installing a GPG key for signature verification:

$(KEYRING_DIR):
    sudo install -d -m 0755 "$(KEYRING_DIR)"

$(KEYRING_FILE): | $(KEYRING_DIR)
    # make a tempdir for the download
    tmpdir="$$(mktemp -d)"
    trap 'rm -rf "$$tmpdir"' EXIT

    aws s3 cp "$(KEY_URL)" "$$tmpdir/key.asc" --no-progress
    # check fingerprint...
    gpg --dearmor < "$$tmpdir/key.asc" > "$$tmpdir/key.gpg"
    sudo install -m 0644 "$$tmpdir/key.gpg" "$(SIGNING_KEY)"

Here, $(KEYRING_FILE) requires $(KEYRING_DIR) to exist, but changes to the directory’s timestamp won’t trigger a rebuild of the file. If $(KEYRING_FILE) exists, Make skips this entire block. Re-runs become instant.

Make like it’s 1999! (Or, It’s not 1970 anymore)

Standard Make has defaults that are hostile to scripting. It executes every line in a separate sub-shell (meaning variables don’t persist), and it doesn’t fail fast by default.

We can fix that with a few lines of “Modern Make” configuration:

.ONESHELL:
.DELETE_ON_ERROR:
.SHELLFLAGS := -euo pipefail -c
SHELL := /bin/bash

.ONESHELL: Tells Make to run the entire recipe in a single shell instance. You can write standard, multi-line Bash scripts with loops and variables right inside your Makefile. .SHELLFLAGS := -euo pipefail -c: This is the “strict mode” of Bash.

• -e: Exit immediately if any command fails.
• -u: Treat unset variables as errors.
• -o pipefail: If curl | tar fails, the whole command fails (not just the last part).

.DELETE_ON_ERROR: If a rule fails halfway through (e.g., a download gets interrupted), Make deletes the partial file. This prevents you from deploying a corrupt artifact on the next run.

(N.B.: the make that ships with OSX is very old and does not support these features; if you develop on a Mac and would like to follow along, you will need to install a “modern” version.)

Key Patterns for Server Orchestration

Once you have this foundation, you can implement patterns that rival complex orchestration tools.

SUBMAKE := $(MAKE) --no-print-directory -f $(SELF) $(SUBMAKE_VARS)

.PHONY: stage-jar stage-jar-as-app-user

# install the jar file with tight perms if it has changed
stage-jar: $(LOCAL_JAR)
    @echo ">> Staging latest uberjar into $(RELEASES_DIR)"
    sudo install -C -m 0400 -o "$(APP_USER)" -g "$(APP_GROUP)" "$(JAR_FILE)" "$(JAR_DEST)"
    sudo -u $(APP_USER) $(SUBMAKE) stage-jar-as-app-user

# update the STABLE_JAR_LINK symlink; needs to run as APP_USER
stage-jar-as-app-user:
    @echo ">> Updating stable symlink"
    CUR=$$(readlink "$(abspath $(STABLE_JAR_LINK))" 2>/dev/null || echo "")

    if [ "$$CUR" = "$(abspath $(JAR_DEST))" ]; then
      echo "-- Symlink already points to latest."
    else
      ln -sfnv "$(abspath $(JAR_DEST))" "$(abspath $(STABLE_JAR_LINK))"
      echo "-- Updated symlink $(STABLE_JAR_LINK) -> $(JAR_DEST)"
    fi

nginx-https:
    tmp="$$(mktemp)"
    trap 'rm -f "$$tmp"' EXIT

    export DOMAIN="$(DOMAIN)" SERVICE_NAME="$(SERVICE_NAME)"
    envsubst '$${DOMAIN} $${SERVICE_NAME}' < "$(CONF_FULL_HTTPS_SRC)" > "$$tmp"

    sudo install -m 0644 "$$tmp" -- "$(CONF_SITE_DEST)"
    $(SUBMAKE) nginx-test

deploy: install-jar server-restart tail-server-logs

The Secure Supply Chain: Signed Artifacts

The strongest argument for this approach is supply-chain integrity. How do you ensure the code running on the server is exactly what you built in CI, without giving the server dangerous permissions?

We use GPG-signed Uberjars.

In our CI pipeline, we sign the artifact (app.jar.asc). The Makefile on the server acts as a gatekeeper.

deploy: install-jar ...

install-jar: ... verify-signature

verify-signature: $(JAR_FILE) $(ASC_FILE)
    gpg --verify $(ASC_FILE) $(JAR_FILE)

The Make DAG enforces that signature verification must succeed before installation can occur. If verification fails, Make returns a non-zero exit code and the install-jar step never happens. Consequently, the deploy never happens either.

This allows for Credential-less Deploys. The server doesn’t need secrets to pull code. It doesn’t need to authenticate with a container registry or a git repo. It only needs an AWS IAM role with s3:GetObject permissions on the release bucket.

We don’t even grant s3:ListBucket permissions. The Makefile constructs the exact path based on the version ID (constructed using the latest.mk trick described above). This separates the signal (what version should be live) from the payload (the secure artifact).

Conclusion

Make is often overlooked, but its primitives—DAGs, file timestamps, and shell integration—are very useful for deployment orchestration.

Of course, Make is not a cloud provisioning tool. It will not manage hundreds of nodes or reconcile distributed state. But for single-node services, controlled clusters, or high-security environments, its explicitness is an advantage.

Make lets you establish clear dependency rules, and it respects the “Principle of Least Surprise.” It works on your laptop, your CI runner, and your production server with zero setup. It turns your infrastructure into a clean, dependency-driven graph. We find that it’s unreasonably effective.

The Allure of the Universal Adapter

Quantitative analysts and data scientists generally prefer rigorous statistics: they want to run regressions, perform clustering, and look at hard numbers. At the same time, traditional statistical tools choke on unstructured text: you can’t run a SQL query on a pile of PDFs.

Enter the LLM.

For developers, the LLM acts as a “Universal Adapter.” It allows you to plug unstructured text into your analytical pipeline. The intention is usually modest and reasonable: “I just need the LLM to parse this text so I can analyze it.” You view the LLM as a translation layer from unstructured text into data that you can then subject to your rigorous standards.

But once you introduce an LLM into your pipeline, that is rarely where it stops.

The Trap: from Formatting to Judgment

The problem lies in the architecture of the models themselves. LLMs are conditional probability models. They predict the next token based on the sequence that came before. The structure they learn is implicit; it is locked inside the model’s weights and can only be accessed via prompting.

Because the information is implicit, extracting it into a clean, external format is surprisingly high-friction. You might start by asking the LLM to extract keywords for a regression, but you quickly realize it’s easier to just ask the LLM to summarize the trend itself.

Instead of:

“Extract all sentiment scores and topic clusters so I can run a multivariate analysis.”

The prompt becomes:

“Read these 50 emails and tell me why the customer churned.”

This is the Creeping Delegation of Responsibility.

Without explicitly deciding to do so, you have moved from using the LLM as a reader (formatting data) to using it as a judge (analyzing data). You have swapped rigorous, corpus-level statistics for a black-box, un-auditable analysis. Once the LLM becomes the judge, you lose:

• Deterministic outputs
• Dataset-level metrics
• Clear test coverage
• Explicit business logic

This also hurts the accuracy of analytical work. LLMs struggle with the “Lost in the Middle” phenomenon, where information in the center of a long context window is ignored. They can overlook subtle connections and hallucinate false ones. Most importantly, LLMs do not operate over datasets; they operate over sequences. Even when you provide multiple documents in a context window, the model processes them token-by-token, without an explicit representation of cross-document structure.

The Better Way: Sturdy Statistics

To solve this, we need to look at the underlying math. While LLMs rely on conditional probability, Sturdy Statistics is built on a joint probability model.

Joint models have long been the workhorses of science because they provide explicit, inspectable structure. Sturdy Statistics has made this approach operational for text analysis.

Instead of burying structure inside a black box, Sturdy Statistics performs a one-time transformation of your unstructured text into an explicit, queryable format.

Here is how the workflow shifts:

1. Structure: Sturdy Stats processes your raw text (news, emails, transcripts) and converts it into a structured format.
2. Query: An analyst (or even an LLM) queries this structure using explicit business logic expressed in standard SQL.
3. Analyze:
   • Quantitative: If you need hard numbers, the SQL query returns deterministic, audit-ready data. You can see the prevalence of a signal across the entire dataset instantly.
   • Qualitative: If you need to understand the “why,” the system links every data point back to the specific words, sentences, or paragraphs that drove it.

The “Oracle Mode” for RAG

This approach doesn’t just help human analysts; it significantly upgrades your LLM applications as well. When you use Sturdy Stats as the backend for Retrieval-Augmented Generation (RAG), you have far more than a fuzzy embedding search. You can feed the LLM highly specific, relevant data derived from a structured SQL query. This effectively putting the LLM into “Oracle Mode:” it works on a concise prompt containing the answer to the question.

This means shorter prompts, less noise, lower latency, and cheaper inference costs. Most importantly, it gives the LLM a factual grounding that it cannot achieve on its own.

Conclusion

It is understandable why we reach for LLMs to handle unstructured data. They are powerful, accessible tools. But we must be careful not to confuse reading text with analyzing trends.

Don’t let the formatting challenge force you into delegating your analytical judgment to a chat model. By using a joint probability model to structure your data first, you retain the rigor of quantitative analysis while unlocking the full potential of your unstructured text.

If your team is relying on LLMs to interpret entire datasets, it may be time to rethink the architecture. We’d be happy to walk through your current workflow and show you where structure can restore both rigor and control.

Get in touch.

The Broken Promise of AI Margins

Traditional software has an economic model that’s the envy of every other industry: you pay a fixed cost to build the product, but near-zero marginal costs to serve it. Whether you have ten users or ten thousand, the cost of the next request is negligible. This “build once, sell forever” dynamic is the foundation of SaaS.

When the current AI boom began, many companies looked at LLMs as a “cheat code” to bypass the hard part of this equation. The naive hope was to skip the heavy fixed costs of traditional engineering – the development work, complex logic, and specialized talent – and skip straight to the high-margin profits of software.

To their credit, they were half right. LLMs do lower the upfront development cost, sometimes dramatically. You can prototype complex features in an afternoon that used to take months. But the rub comes after you launch: you discover that, rather than removing cost, you’ve delayed it. And in doing so, you’ve converted a fixed cost to a fixed marginal cost. Instead of a one-time fixed cost, you are now paying a constant, unyielding marginal “tax” on every single interaction.

Teams expecting software margins are ending up with utility margins (at best).

Why do so many AI products fail the fundamental test of software economics?

The Token Tax: Why Scale Hurts

Large Language Models (LLMs) behave less like software assets you own and more like metered utilities you rent.

In traditional software, logic is explicit in the code and state is preserved. In LLM architectures, on the other hand, logic is implicit in the model context and can only be retrieved via “prompting.” Thus, you pay a recurring toll – a “Token Tax” – on every single interaction.

Moreover, because models are stateless, they have amnesia; long-term memory must be re-supplied on each call. To answer a follow-up question, you must pay to re-feed the model the original context, plus the new question, plus the history of the conversation.

This is a simple fact about LLM architecture, but its economic implications are punishing:

• Linear (or worse) Cost Scaling: There is no natural point where marginal costs taper off.
• Context Inflation: As features mature, prompts get longer. You pay more for the same user action in month 12 than you did in month 1 because you are stuffing more context, more examples, and more safeguards into the window.
• Zero Amortization: You are billed every time the model re-interprets the same text. You never “own” the processing work.

Agentic Workflows Multiply the Problem

The industry is currently pivoting toward “Agentic Workflows:” multi-step processes where models plan, search, tool-use, and refine. While this may improve a product’s performance, it wreaks havoc on its unit economics. A single user request now triggers a chain reaction of planning loops, self-correction, and tool routing. What looks like one API call to the user may be a dozen or more calls on the backend.

Consequently, COGS (Cost of Goods Sold) increases directly with product sophistication.

“I’ve spoken with 20+ AI founders… They all said the same thing: ‘We’re scaling… but token costs and latency are starting to hit us hard.’… As you scale, agents start re-querying the same info, looping through tools, and sending massive context windows. The result? Cost and latency grow faster than usage.” — [Gleb Gordeev via LinkedIn]

This removes the primary advantage of software. If your costs grow in lockstep with your revenue, you aren’t building a SaaS company; you are building a services firm with robotic employees.

The Solution: Structure Once, Query Cheaply

The mistake lies in treating LLMs as end-to-end data processors. Modern AI workloads are expensive because they repeatedly re-read raw, unstructured text. Sending entire documents to an LLM just to extract a single answer is economically inefficient.

The obvious solution is to shift from a Variable Cost model (re-reading text every time) to a Fixed Cost model (structuring text once).

Conclusion: Restoring Software Economics

With Sturdy Statistics, token usage grows sub-linearly relative to your user base. New data makes the system smarter, but not more expensive to query.

To build a viable business on top of AI, you must reject the Token Tax. Stop paying a model to read the same document a thousand times. Pay to structure it once, and let your margins scale like software again.

If you’re building on LLMs and starting to feel the Token Tax, it’s not simply a pricing problem; it’s architectural. We’d be happy to walk through your current workflow and show you where structure can restore your margins.

Get in touch.

The Heavy Lifting

A single request to our server triggers a surprising amount of work:

The “Sluggish” Server

Given all that complexity, along with the fact that I’m using a high-level dynamic programming language, I wasn’t surprised when my initial benchmarks showed the server responding in 200 microseconds (µs).

I ran a profiler expecting to see SHA-512 or Ed25519 dominating.

Instead, nearly 90% of the runtime was attributed to the most mundane line in the entire library:

(defn encode-rt-message [msg-map]
  (let [sorted-entries (sort-tags msg-map)
        tag-bytes      (mapv #(tag/tag->bytes (key %)) sorted-entries)
        val-bytes      (mapv #(tag/pad4 (val %)) sorted-entries)

        ;; THE BOTTLENECK:
        val-lens       (mapv alength val-bytes)

        ...]

This line is arguably the most trivial part of the entire codebase. It iterates over 5(ish) byte arrays and asks: “How long are you?”

That’s it.

Yet this one line accounted for almost the entire request time.

The Fix

I wrapped alength in an anonymous function, so that I could include a type hint:

;; BEFORE (~31µs)
(mapv alength val-bytes)

;; AFTER (~4µs)
(mapv (fn [^bytes v] (alength v)) val-bytes)

I profiled the code with and without the type hint. The encoding time dropped from 31µs to 4µs.

Why was (mapv alength ...) so Slow?

Clojure emitted no reflection warnings when I ran my tests; the code is perfectly legal.

But mapv is a higher-order function. It receives alength as a generic IFn object and calls invoke() on it for every item. This means that:

• The compiler cannot inline the operation because the function is passed as a value.
• alength itself must perform a runtime check (RT.alength) to ensure the argument is an array.
• Finally, it calls java.lang.reflect.Array.getLength

The overhead of dynamic dispatch, runtime type checking, and reflection adds up in a tight loop!

By contrast, once I wrote:

(fn [^bytes v] (alength v))

the compiler had enough static information to emit a single arraylength bytecode instruction. We replaced a complex chain of method calls with a single JVM bytecode.

End-to-End Benchmark

To verify the impact, I ran a full end-to-end benchmark using Criterium’s quick-benchmark.

Why Did the Speedup Get Larger?

In isolated tests, the improvement was ~8×. Amdahl’s law suggests that, in the real system, we should see a substantially lower improvement. Instead, we saw the improvement grow to ~13×.

I can’t explain this fully, but my working hypothesis is contention in the reflective call path. When multiple workers hit the same reflective, non-inlinable call site, the JVM cannot optimize it effectively. Removing that reflective barrier allows the JIT to inline and parallelize cleanly.

The result: better scaling under load.

The Lesson

I learned that, when optimizing Clojure code, “no reflection warnings” is not always the end of the story. When you pass low-level primitives through higher-order interfaces, you may accidentally force the runtime back onto generic (and slower) paths. The compiler needs enough information to emit primitive bytecode.

In this case, the code I thought was complex – the crypto, Merkle trees, and protocol gymnastics – was fine. It was the “trivial” line that killed performance.

Without a profiler, I would never, ever, have suspected it.

The Problem

I’m working in Clojure, and I need to import a NumPy binary file into a DuckDB table. My .npy file contains a large 2D array. The array is:

• Row-major (C order)
• Little-endian
• float32
• Small enough to easily fit into memory
• Large enough that boxing every element is prohibitive (i.e., I can’t afford to represent each number as an independent heap object.)

(I wasn’t using Python or NumPy itself; I routinely use the NumPy format in our C programs because it’s simple, well-specified, and trivial to map directly into memory.)

My goal was to load this data into DuckDB with the following schema:

• A single column
• Each row contains an array (vector) of float32s
• Eventually converted to a sparse array

Conceptually, this is a very natural representation: in machine learning terms, each row is an observation, and the list is a fixed-length feature vector. While DuckDB is a column-major database, this isn’t an incompatibility: a column of vectors is physically stored in a way that mirrors a row-major file. Because the logical structure of the database matches the physical layout of my NumPy file, I should have been able to bulk-append the data with zero copying, zero transposing, and zero boxing.

This seemed like a straightforward plumbing task.

(...
  (:require
    [sturdy.fs :as sfs])
  (:import
    (java.nio ByteBuffer ByteOrder)))

(let [bs  (sfs/slurp-bytes file)
      bb  (doto (ByteBuffer/wrap bs) (.order ByteOrder/LITTLE_ENDIAN))
      n   (quot (alength bs) 4)
      arr (float-array n)]
  (dotimes [i n]
    (aset-float arr i (.getFloat bb))))

(require '[tech.v3.dataset :as ds])
(require '[tech.v3.datatype :as dtype])

(let [buf      (dtype/->buffer data)
      rowviews (mapv (fn [r]
                         ;; zero-copy view into buf
                         (dtype/sub-buffer buf (* r cols) cols))
                       (range rows))]
  (ds/->dataset {:c1 rowviews}))

(np/npy->dataset-rowlists "test-resources/npy-fixtures/shape_2x3__dtype_f4.npy")
;; => _unnamed [2 1]:
;;    |               :c1 |
;;    |-------------------|
;;    | [-3.0 -1.75 -0.5] |
;;    |   [0.75 2.0 3.25] |

The False Start: Fighting the Formats

That one limitation changed everything. To get the data in, I had to transform it into a format tmducken could accept: a “wide” dataset with hundreds of scalar columns (one per feature). This meant I had to:

1. Transpose the data in Clojure (from row-major list to column-major scalars).
2. Ingest the wide table.
3. Transpose again inside DuckDB (collapsing columns back into a list) using SQL.

(defmacro def-transposer
  "Define a transpose fn for a primitive array type.
   src-tag is one of: bytes, shorts, ints, longs, floats, doubles
   array-ctor is e.g. float-array
   aset-fn is e.g. aset-float"
  [fname src-tag array-ctor aset-fn]
  (let [src  (with-meta (gensym "src")  {:tag src-tag})
        rows (gensym "rows")
        cols (gensym "cols")
        dsts (gensym "dsts")
        dstj (with-meta (gensym "dst") {:tag src-tag})]
    `(defn ~fname
       [~src ~rows ~cols]
       (let [~dsts (vec (repeatedly ~cols #(~array-ctor ~rows)))]
         (loop [i# 0
                base# 0]
           (when (< i# ~rows)
             (loop [j# 0]
               (when (< j# ~cols)
                 (let [~dstj (~dsts j#)]
                   (~aset-fn ~dstj i# (aget ~src (+ base# j#))))
                 (recur (long (inc j#)))))
             (recur (long (inc i#)) (long (+ base# ~cols)))))
         ~dsts))))

(def-transposer transpose-float  floats  float-array  aset-float)
(def-transposer transpose-double doubles double-array aset-double)
(def-transposer transpose-byte   bytes   byte-array   aset-byte)
(def-transposer transpose-short  shorts  short-array  aset-short)
(def-transposer transpose-int    ints    int-array    aset-int)
(def-transposer transpose-long   longs   long-array   aset-long)

(defn wide->array
  [conn src-table-name dst-table-name {:keys [dtype col-names]}]
  (let [sql-type  (duckdb-type dtype)
        num-cols  (count col-names)
        arr_spec  (->> col-names
                       (map (fn [c] (name c)))
                       (string/join ", ")
                       (format "array_value(%s)"))
        sql'  (format "SELECT line_no, %s::%s[%d] AS vals FROM %s ORDER BY line_no"
                      arr_spec
                      sql-type
                      num-cols
                      src-table-name)
        sql   (format "CREATE OR REPLACE TEMPORARY TABLE %s AS %s"
                      dst-table-name
                      sql')]
    (duckdb/run-query! conn sql)))

The Real Solution: Stop Fighting

Stepping back, I realized I was trying to force a square peg into a round hole through brute force and sheer will. The better solution was to abandon the “table” concept entirely during ingestion and treat the data as a sparse coordinate list.

Instead of transposing the array, I can unnest the data as I read it. I decode the stream and accumulate row and column indices for every value. This requires no transpose, doesn’t explode my column count, and naturally handles both C-ordered and Fortran-ordered arrays.

Here’s my new function in action:

(npy->dataset-unnested-nz "test-resources/npy-fixtures/shape_2x3__dtype_u4.npy")
;; => _unnamed [6 3]:
;;    | :row_no | :col_no |       :val |
;;    |--------:|--------:|-----------:|
;;    |       0 |       0 | 4294967291 |
;;    |       0 |       1 |         12 |
;;    |       0 |       2 |         29 |
;;    |       1 |       0 |         46 |
;;    |       1 |       1 |         63 |
;;    |       1 |       2 |         80 |

Unnesting expands the storage somewhat—4 bytes for a single float32 becomes 14 bytes per element (value + row index + col index). However, my data is sparse, so this form allows me to simply skip zero values. This offsets the extra storage cost and, in my case, actually saved memory.

The Lesson

The lesson I’m taking away isn’t “use UNNEST” — it’s that intractable performance problems often come from abstractions meeting at the wrong boundary. (Though “always use UNNEST” isn’t a bad takeaway, either.)

The data started out contiguous, typed, and structured. In my first attempt, as it crossed library boundaries, those guarantees were erased, and most of my effort went into reconstructing them. The “obvious” path of treating the data as a dense table forced me to fight against the physical reality of how the data was stored.

By shifting the perspective to a coordinate list, the friction disappeared.

If you’re interested, I wrapped this work into a small, open-source Clojure library for reading NumPy .npy files and moving them efficiently into columnar systems. You can find the code on GitHub, or browse the documentation on cljdoc.

RoughTime in a Nutshell

RoughTime is built on a very simple idea: a server signs a statement that includes a client-chosen random value (a nonce) and a timestamp. Because the nonce did not exist before the client generated it, any valid signature proves the server produced the response after receiving the request.

The essential process looks like this:

1. A client retrieves the server’s address and public key out of band. (Ours is on our website; Cloudflare maintains an ecosystem file with a list of other participating servers.)
2. A client generates a 32 byte nonce and sends it to the RoughTime server. Verification relies upon the nonce being unpredictable and unique.
3. The RoughTime server responds with a) the client’s nonce, b) a time range indicating the server’s current time, and c) a signature of them both.

Using the server’s public key, the client checks that the signature is valid. This proves two things:

1. Freshness: The response must have been generated after the request, because it incorporates the client’s unpredictable nonce.
2. Authenticity: Only the server holding the matching private key could have produced the signature.

The client now possesses a cryptographically verifiable timestamp from the server, independent of any assumptions about trusted middlemen or synchronized clocks.

There’s a little more to the protocol: for example, servers should delegate signatures to limit exposure of its private key (similar to X.509), the server may batch responses using a Merkle tree, and the IETF version of the protocol introduced version specification. But none of these details change the fundamental idea behind RoughTime.

Decentralized Verification via “Happens-Before” Ordering

If desired, the client can derive a new nonce from a previous response. For example, you might use a hash:

next_nonce = SHA512(response_bytes)[0:32]

This new nonce can be sent to another RoughTime server, providing a chain of responses and replies with guaranteed happened-before ordering.

This allows the client to construct a sequence like:

1. Generate nonce A → send to Server A → receive response A′
2. Derive nonce B = SHA512(A′)[:32] → send to Server B → receive response B′
3. Derive nonce C = SHA512(B′)[:32] → send to Server C → receive response C′
4. And so on…

Each step is cryptographically tied to the previous one. This produces a tamper-evident, cross-server chain with a guaranteed happens-before relationship: A’ happened before B’, B’ happened before C’, etc.

If server B reports an incorrect or inconsistent time, its response will fall outside the time ranges reported by the other servers in the chain. For example:

Here, the responses from servers A and C are consistent with one another: C should come after A, and their ranges overlap once you account for the reported uncertainty. Server B, however, is clearly incompatible with the ordering—its claimed time cannot fit between A′ and C′. If this happens, the client can:

• Produce the signed responses proving that A′ precedes B′ precedes C′,
• Demonstrate that server B’s claimed time is incompatible with that ordering, and
• Publish this as a cryptographic proof of misbehavior.

Crucially, any third party can verify this proof without trusting the client and without recourse to a centralized authority. This is the core “lie-detecting” property of the RoughTime protocol: if a server equivocates, it creates evidence against itself in doing so.

This is also why the ecosystem needs multiple independent organizations hosting servers. Today, the canonical ecosystem lists only two servers supporting the current IETF protocol, with one more supporting a slightly outdated version. This is insufficient for establishing a meaningful chain of trust. We want to help the ecosystem grow, and we hope others will follow.

Bridging the Version Gap

Chaining is an essential part of the RoughTime protocol. However, the ecosystem is currently split between the original Google “v1” implementation and various iterations of the IETF Drafts. In practice, this limits one’s ability to build meaningful response chains.

Because our Clojure implementation supports all protocol versions (from the early Google spec to the latest IETF Drafts), it enables users to chain requests across the entire available ecosystem. Our client can directly query every major server, bridging version differences to cryptographically confirm timestamp consistency:

Furthermore, by acting as a “universal responder,” our server ensures that it can participate in any chain, regardless of which client or protocol version other participants are using. This interoperability transforms otherwise isolated servers into a resilient, global network.

Anti-Amplification and Secure Design

RoughTime requests are required to be at least 1024 bytes, and the request is required to be larger than the response; smaller requests are simply dropped by the server. (In response to a 1024 byte request, our server responds with a 420 byte response.) This prevents RoughTime servers from acting as DDoS amplifiers, and the minimum request size naturally rate-limits clients.

Using UDP also sidesteps the bootstrapping paradox faced by secure NTP-over-TLS approaches. A client with a severely skewed clock may be unable to perform TLS handshakes, because certificate validation itself depends on accurate time. RoughTime requires no TLS, no HTTP, and no session establishment of any kind, allowing clients to retrieve a verifiable timestamp even when their local clock is wildly incorrect.

Finally, UDP keeps the surface area for security holes intentionally small. One request produces one response; there is no retransmission logic to implement, no state machine, and no long-lived connection. This simplicity reduces the attack surface, improves auditability, and makes independent implementations easier to write, verify, and deploy.

Why Clojure?

We wrote our RoughTime server from scratch in Clojure because we think the language (and the JVM platform beneath it) aligns naturally with the security and auditability goals of the protocol. Though RoughTime is a small protocol, it is essential to implement it correctly. Clojure gives us a clear, predictable, and expressive implementation, without sacrificing performance or operational stability.

Explore the Code

If you’re interested in RoughTime (and especially if you’d like to help strengthen the ecosystem) we invite you to explore the implementation and try running a server yourself.

Our website includes a live demonstration showing example requests to, and responses from, multiple servers across the ecosystem. It’s a useful way to see how the protocol works in practice.

You can find our full source code here:

• Core protocol implementation: roughtime-protocol
• Command-line client: roughtime-client
• UDP server: roughtime-server

If you’d like to run your own server, the server README and DEPLOY notes include detailed instructions for deploying on EC2. We provide a pre-built Makefile which encodes the steps, and a full deployment should less than fifteen minutes.

The RoughTime ecosystem is still small enough that each new server measurably improves its trustworthiness. If you want to make a direct, practical contribution to internet security, this is one of the simplest and most impactful ways to do it. We’d love to have you join the network.

Two Philosophies of Learning

The current generation of AI largely depends on scale: harvest the entire internet, build a massive model, and distill general linguistic ability from sheer exposure. (Or, perhaps, memorize the dataset.)

At Sturdy Statistics, we’ve taken a different approach. We train models exclusively on a client’s own data (such as emails, transcripts, reviews, or other documents), and we never use external corpora or pretraining. Instead of learning language by imitation, our models learn structure through mathematically defined priors derived from linguistic theory.

These priors encode formal knowledge about syntax, semantics, and statistical relationships. They come from human scholarship, not internet text, and they give our models a principled, interpretable inductive bias that allows generalization from small datasets. Where LLMs pursue breadth, we pursue fidelity.

Because our systems are Bayesian, every parameter has a meaning and every inference can be traced back to specific source data. If the model misclassifies something, we can identify why: we know which prior contributed, which words updated the posterior, and how the model’s beliefs shifted in response to the data.

This transparency extends to the data itself. When a client trains a Sturdy Statistics model, the full training set is known, finite, and auditable. We don’t hide any pretraining corpus and, consequently, there are no surprise behaviors inherited from internet data.

Anthropic’s findings highlight why this matters. If you know every line of data your model sees, dataset poisoning becomes practically impossible. And when parameters are interpretable, even small behavioral shifts can be diagnosed and corrected rather than guessed at.

The Virtue of Small, Known Data

Anthropic’s paper shows how easily large systems can be nudged by small adversarial perturbations. The same logic applies in reverse: small, well-characterized datasets can produce remarkably stable models.

Our systems can infer meaningful structure from only a few hundred words (a few thousand is better), because the inductive bias comes from the priors — not from massive amounts of raw text. Because of this built-in inductive bias, our AI doesn’t need to learn “how language works” from scratch; it begins with a mathematically grounded model of language and updates that model based on the client’s data alone.

A Complementary Vision of AI

Anthropic’s results show that even the largest AI systems can be swayed by tiny, unseen influences in their training data. If you don’t control your corpus, you don’t control your model. Unfortunately, no amount of scale can compensate for that.

That’s why our approach is different. By training solely on your own data, with interpretable Bayesian models, we give you:

• control over your data pipeline
• transparency into every parameter
• protection from poisoning and drift
• models whose behavior you can explain to auditors, regulators, and customers

Trustworthy AI begins with knowing your data. Sturdy Statistics makes that possible.

Introduction — The Underestimated Classics

HTML and CSS aren’t new. They were built by people who needed to solve the real problem of how to mark up, style, and share documents over a network. After decades of iteration, they remain the best solution to that problem.

Like SQL for data or Clojure for logic, HTML + CSS are declarative, composable, and enduring — elegant tools that reward clarity, not limitations to be abstracted away.

A Precursor: (La)TeX

Before I ever wrote web pages, I wrote scientific papers — in LaTeX. LaTeX separates markup from presentation just as the web does. You write a paper once, describing its content and structure, and select a style file that decides how it looks.

If you submit to The Astrophysical Journal, you load their style file and instantly have an ApJ paper: the typeface, page layout, headers, equations, and citations all match the ApJ style. If you hop the pond and change the style file to Monthly Notices of the Royal Astronomical Society, everything shifts: your centered equations flush left, and the margins, fonts, bibliography format, and paper size all become distinctly British. (It won’t change the spelling for you, however; the editor will make you replace every instance of color with “colour.”)

Later, when you fold that paper into your dissertation, it’s another style file swap.

The content never changes — only its presentation.

That separation is what makes the document portable, maintainable, and semantically clear. The source says what is present — section, caption, equation — while the style determines how it’s displayed. The how may vary by context; the what never does.

The Web’s Separation of Concerns

HTML and CSS inherit this same brilliance. HTML describes structure using semantic units such as headings, lists, quotations, figures, captions, or navigation. CSS describes presentation: the layout, color, motion, and rhythm.

Because those concerns remain distinct, the web can still serve many audiences at once. A screen reader, a phone browser, and a search crawler can all parse the same page and extract exactly what they need. The structure is explicit, and because it’s explicit, it’s accessible.

That separation also makes web pages maintainable and robust. You can redesign a site without touching its content, or rewrite the content without worrying about layout. It’s the same idea that made TeX last.

How Frameworks Try to Hide It

New frameworks have layered abstractions on top of HTML and CSS. The intention is understandable, but the cost is real — it blurs the original separation of concerns that make HTML and CSS so powerful. Markup becomes entangled with logic. Styles live in scripts. Accessibility and semantics become afterthoughts.

This can risk the very properties that made the web universal: transparency, portability, and graceful degradation. HTML and CSS are already abstractions — carefully designed ones. Adding more abstraction isn’t automatically progress — it needs to earn its keep.

Leaning Into the Foundations

At Sturdy Statistics, we’ve found that embracing “plain” HTML + CSS makes our platform simpler and faster.

• Pages load quickly and payloads stay small.
• They render gracefully on old hardware and slow networks.
• We keep full control over semantics and accessibility.

Modern CSS has matured into a genuinely expressive design language. Flexbox, Grid, variables, transitions, and @media queries can achieve most layouts and animations with no JavaScript at all.

It’s astonishing how much you can build with zero frameworks and a few hundred well-written lines of CSS. (If you want to see an example, you can check out our Flashpaper site.)

What We’ve Learned

Whether in Clojure, SQL, LaTeX, or HTML + CSS, we’ve found that the older declarative systems — the ones designed by people who had to get it right — are often the most reliable foundations for building something new.

The modern web runs on layers of abstraction, but sometimes the best layer is still the one closest to the metal: a clean HTML document and a well-written stylesheet. You might be surprised how far that can take you.

Postscript

I need to confess a half-truth in this post: while I do appreciate HTML, I rarely write it raw anymore. Since discovering Hiccup, I’ve written structured web documents directly in Lisp. The Lisp syntax feels like the natural dual of HTML. I like the HTML standard; I just prefer writing it in Lisp rather than XML.

What Does Hacker News Like?

# pip install sturdy-stats-sdk plotly
from sturdystats import Index
from plotly import express as px
index = Index(id="index_f2ffe2f7901845c59c15aab45685fa3c")

df = index.queryMeta("""
WITH t AS (
    SELECT 
        UNNEST(sum_topic_counts_vals) as topic_vals,
        UNNEST(sum_topic_counts_inds) - 1 as topic_id, -- duckdb is 1 indexed
        year(published::DATE) as year,
        score
    FROM doc
)
SELECT 
    topic_id,
    year,
    count(*) as n_posts,
    avg(score) as avg_score,
    avg((score > 10)::int) as p10,
    avg((score > 100)::int) as p100,
FROM t
WHERE topic_vals > 5
GROUP BY topic_id, year
ORDER BY topic_id, year
""", paginate=True)

tmp = index.topicSearch(limit=500).set_index("topic_id").to_dict()
df["topic"] = (df.topic_id).apply(tmp["short_title"].get)
df["topic_group"] = (df.topic_id).apply(tmp["topic_group_short_title"].get)
df = df.dropna().copy()
df.sample(5)

## Add a prior to require a higher standard of evidence 
df["P(score>100)"] = df.apply(lambda x: (x["p100"]*x["n_posts"]+ 2)/ (x["n_posts"]+50), axis=1)

df["title"] = "Show HN<br>Performance"
fig = px.treemap(df, path=["title", "year", "topic_group", "topic"], 
            values="n_posts", color_continuous_scale="greens",
            color="P(score>100)", height=600,  )

If a picture is a thousand words, I like to think this interactive treemap is at least one thousand and one. You can click into a box to dig into the year or topic group. The first thing that jumps out is just how much bigger the 2025 box is than any other year. The second is how much lighter the 2025 box is compared to the previous few years.

I enjoy history, but I am often told that people care much more about what is happening now. So let’s dig into specifically 2025.

Performance Across Years

# pip install sturdy-stats-sdk seaborn 
from sturdystats import Index
import seaborn as sns
from matplotlib import pyplot as plt

index = Index(id="index_f2ffe2f7901845c59c15aab45685fa3c")
df = index.queryMeta("""
    SELECT year(timestamp::TIMESTAMP) as year, 
    score FROM doc 
    ORDER BY doc_id""", 
paginate=True)

plt.figure(figsize=(10, 6))
Xmax = 100
tmpdf = df.loc[df.year > 2020].copy().sort_values("year")
tmpdf["year"] = tmpdf.year.astype(str)
tmpdf["score"] = tmpdf.score.apply(lambda x: min(x, Xmax))
sns.displot(tmpdf, x="score", kind="ecdf", hue="year" )
plt.title('Score Distribution by Year (CDF)', fontsize=14, fontweight='bold')
plt.xlabel('Score', fontsize=12)
plt.ylabel('Density', fontsize=12)
plt.tight_layout()
plt.xlim([10, Xmax])
plt.ylim([.75, 1.0])
plt.show()

There are three distinct groups: 2022 as the heyday of Show HN posts, 2021, 2023, & 2024 as still extremely highly performing, and 2025 with a significant performance reduction. If I were to make a guess at the root cause of this difference, I would have two (not necessarily contradictory) theories.

The first is tied to the software job market. I can easily imagine that the combination of remote work and post-covid tech boom drove much more overall Hacker News engagement and more interesting Show HN projects. The job market and remote work receded a little in ’23 and ’24 but morale seems to have cratered in 2025. Which leads to my second theory.

AI has driven much more content, by both aiding in its creation and creating a cohort of startups and companies chasing a piece of the pie. This AI-driven content tends to be more shallow, and its noise makes it harder to discover genuine, interesting content. There is some evidence for this theory in the expontential growth in the quantity of hacker news posts from 2022 onwards.

These are both guesses. I don’t have an easy avenue to dig into the economic theory, but I do want to dig a bit deeper into the composition of posts in 2025.

Digging into 2025

The first thing I want to dig into is the change in topics of viral posts. Take a look at this slope plot, which shows the change in content of all posts between 2024 and 2025. Each side of the plot lists topics by rank, and a line connects their 2024 position to their 2025 position; topics which decreased in prominence have negative slopes, shown in red. Topics which surged in prominence have positive slopes, shown in blue.

You can see a spike in topics related to AI such as “Agent Connectivity”, “AI Coding”, & “AI Automation”, which lends credence to at least my theory that AI related posts grew more common in Show HNs. However, I’m curious to dig into not just how often topics were posted in 2025, but how successful each topic was.

This plot shows that, outside of the “AI Automation” topic, which saw slightly higher represented in viral posts than it’s overall prevalence, AI related Show HN posts underperform expectations. The majority of the bottom right “quadrant of death” is AI related topics. Topics such as “AI Coding” and “AI Image Generation” see noticable drops as well. It is interesting to me that “Document Ingestion and Retrieval” strongly overperforms expectations as a tool that supports a lot of these other AI topics (something about selling pick-axes?). But standing above all, the deep blue line shooting like Superman across the sky is DIY Hardware IoT Projects. I confess that I am a sucker for these posts, as it appears all of Hacker News is as well. These tends to be extremely authentic, lack any sort of commercial viability and just have a lot of soul put into them.

The last thing I wanted to dig into is the audience segmentation aspect of Hacker News. There’s two distinct audiences viewing posts. The first is the much more hardcore technical readers that view the “New” and “Show” pages. The second is the more mass market crowd that goes on the front page.

So some takeaways I have is:

1. The real heavy-lifting for the performance of “DIY Hardware” projects is really coming from the first wave of Hacker News readers (though it does get a sizeable boost from the front page audience as well).

2. Nearly every AI related topic does worse once it clears the 10 point threshold than any other category. This means that either the people looking through the New and Show sections are disporportionately interested in AI. This is very possible, but from my interaction with this crowd from my posts, these users tend to be more technically minded (think DIY hardware, rather than landing-page builders).

   However, my Spidey-sense is going off a little bit. AI has received a massive wave of funding for startups and enterprises have carved out budget space to invest in solutions. This is obviously not hard evidence, but circumstantially, it looks like there is a evidence that some level of voting rings are on within this AI posts.

   However, either explanation would explain why 2025 is such a challenging year for viral Show HN posts. There exists this set of topics that are more frequent than ever, that quickly garner early points, but do not resonate with the broader audience. Beyond that is speculation on my part.

What’s the best time to post?

Ah the question you really wanted to know. This was actually the first question I asked as well. I did a simple SQL query estimate. However, Mike McCourt saw a chance to apply a Gelman-style hierarchical mixed-effects model to better leverage the data to answer this question. I won’t spoil his analysis but I’ll leave you with a little preview …

Struct Meets Structure

C’s design mirrors the structure of our models almost perfectly. We represent each node in a Bayesian DAG as a struct, and each dependency as a pointer — so the graph in memory looks exactly like the one on paper. There’s no abstraction layer to cross, and no framework needed to translate our intent into a data model.

That simplicity has real advantages. Because our models are acyclic and fixed in shape, our data structures don’t mutate; only their values do. As a result, we can allocate nearly everything at startup and keep it until shutdown. We worked hard to make sure almost nothing ever needs to be reallocated. The result is a codebase that nearly saturates the CPU, is easy to reason about, and performs consistently.

From Graphs to Loops

By packing nodes into contiguous arrays, we replaced pointer-chasing with stride-based memory access, which modern CPUs handle extremely well. Then we realized we could go even further.

No matter how complex the original graph, our proprietary algorithm νmix reduces each inference iteration to a two-dimensional loop: one compact dimension, and one over the number of topics. Careful layout ensures each iteration reads a compact, aligned block that fits entirely in the CPU cache. Those loops are automatically vectorized by the compiler, and the predictability helps out the CPU’s prefetcher.

We also spent time with pen-and-paper, rewriting our core equations until the inner step reduced to a combination of integer operations and fused-multiply-add (FMA) updates.

It’s a good reminder that performance often comes not from clever tricks, but from disciplined data layout and from predictable control flow.

Predictability Over Dynamism

C gives us that predictability. We avoid dynamic memory wherever possible, and we don’t rely on garbage collection, reference counting, or hidden allocators. Every object’s lifetime is explicit: if it’s allocated, we know when, why, and how much.

This constraint forces discipline. By designing our system around stable structures, we eliminated entire classes of bugs. It’s a relief to know exactly what the program is doing, at every moment. This engine runs every training job and on nearly every API query — across multiple servers and clouds, wherever data needs to be processed. Despite the code’s complexity and its central role in everything we do, in practice we don’t worry about it at all; it’s by far the most solid part of our entire stack.

Simple to Teach, Simple to Learn

The language has a reputation for being difficult, but in our experience the opposite is true: we have found C to be very easy to teach. Its syntax is small, its standard library is compact, and its semantics are straightforward once you understand memory. Moreover, everything in the language corresponds to something concrete: there are values, pointers, blocks of memory, and function calls. We’ve found that, if someone has a background in mathematics, he or she can start writing great C code in less than a week.

There’s very little to hide or to abstract away. The same directness that makes C efficient for machines also makes it clear for novice programmers. At the same time, LLVM and Clang make advanced work very pleasant. Their static analyzers and sanitizers give us the same visibility we’d expect in a higher-level language, but without sacrificing control.

The Performance of Clarity

We didn’t choose C out of nostalgia. We chose it because, for our problem, clarity and performance turned out to be the same thing.

C makes our models transparent. It forces us to express every dependency, every allocation, every update explicitly — and it rewards that discipline with speed. C’s direct memory model helped us write good code by keeping cause and effect visible: when something is slow, you can see exactly why, and fix it.

Abstraction often promises efficiency, but in our experience it can deliver complexity instead. We’ve found that the simplest tools — when used judiciously — can be the most efficient. C may be old, but it remains a language of great clarity and power.

The Quiet Miracle of SQL

SQL isn’t new. It was designed in the 1970s by IBM researchers, yet it still runs much of the modern world. Banks, airlines, telecom systems, logistics networks — the infrastructure we all rely on — still depends on it. That longevity isn’t inertia; it’s evidence of a design that got the fundamentals right.

SQL was created by people solving real, critical problems: consistency, concurrency, recovery, and correctness at scale. The result was a language that’s both rigorous and flexible.

At Sturdy Statistics, we’re drawn to tools that are stable, composable, and well-designed — and SQL’s design philosophy echoes that perfectly. The same principles that made SQL powerful in the 1970s — a clear data model, declarative syntax, and a query optimizer that handles complexity for you — still make it powerful today.

A Common Language for Humans and Machines

SQL’s enduring power lies in its declarativity: you describe relationships and constraints, and let the system figure out how to satisfy them. It’s one of the few languages where you can express your intent clearly and still let the system handle optimization for you.

It’s also composable and set-based rather than iterative — closer to algebra than to imperative code. These properties make SQL expressive enough for complex analysis and simple enough for inspection. The same query can run locally, or distributed across a cluster, without any change in syntax or meaning.

This became especially clear when we decided to treat SQL not as glue code, but as a primary language for analysis. When we started moving more of our logic into SQL itself, our data pipelines got smaller, faster, and easier to understand. For us, SQL is a thoughtful, well-designed language for reasoning about our data. It forces structure to be explicit, so both people and programs can see exactly how the pieces fit together.

Why DuckDB

DuckDB is small, embedable, and perfect for analytical workloads. It gives you the performance of columnar storage, the simplicity of local execution, and the reliability of a real database. Its API is compact but expressive, and its support for function chaining and higher-order operations makes it a natural fit for our workloads.

At first, local execution might sound like a limitation; in practice, it’s the opposite. Today’s CPUs are astonishingly fast. We run our analysis on Apple M4 chips, and it would be difficult to saturate them on a real-world data task. Distributed processing would be significantly slower for our use cases, especially once you factor in network latency. DuckDB lets us process our data where it is, and keeping computation close to the data has sped up our entire pipeline dramatically.

DuckDB fits our broader design philosophy: composing self-contained, minimal, transparent systems, which use the right tool for the task at hand.

What We’ve Learned

While we do have a fondness for old tools at Sturdy Statistics, we didn’t choose SQL out of nostalgia. For us, SQL is simply the right tool for certain jobs: it makes reasoning easier, encourages clarity in our code, and executes reliably.

In a field often obsessed with novelty, SQL reminds us that some problems were already solved — beautifully — decades ago. Sometimes the best progress comes not from replacing old tools, but from rediscovering how well they still work.

Why Clojure?

Clojure gave us:

• Real concurrency — threads, futures, and async channels — without the GIL
• Predictable, fast performance — with far fewer surprises than Python
• Rock-solid stability — the JVM’s garbage collector and threading model make it ideal for long-running, 24/7 systems
• REPL-driven development — hot-reloading, tiny feedback loops, and real “live coding”
• Zero dependency drama (so far) — we have yet to hit a single conflict

That last point feels cultural as much as technical. Clojure libraries rarely introduce breaking changes. The ecosystem values stability and composability. Python, in contrast, rewards shipping fast — sometimes at the cost of compatibility.

We expected these improvements. What we didn’t expect was how much better our code would become. Writing pure functions and working with immutable data nudged us toward patterns that were simpler, clearer, and more reliable. The codebase feels smaller, more transparent, and easier to reason about — even though neither of us knew a word of Clojure when we began the rewrite.

The Human Side

There’s also a more personal angle. Around the time we began the rewrite, my first child was born. I found myself working at odd hours — fifteen minutes here, three minutes there, sometimes a quiet stretch from 2 a.m. to 4 a.m. Our Python codebase was very well written, but under that kind of schedule, I couldn’t easily “re-enter” it. It was too stateful, too sprawling, and too dependent on remembering context.

Clojure was different. Knowing that everything is a pure function meant I could open a single file, change one small piece, and be confident it wouldn’t ripple through the rest of the system. The language made the project fit the rhythm (or chaos) of my life. I could still contribute meaningfully, even in short bursts.

Another benefit: I work in Emacs, and tools like paredit, ido, dabbrev-expand, and key-chord-mode — plus a custom keymap — let me code one-handed while holding my baby. For reasons I can’t quite explain, I’ve never found a particularly comfortable way to work with Python… the whitespace delimiters don’t gel with my incremental, interactive development style.

To be clear, we could have built similar habits in Python — but we never did. In Clojure, they came naturally.

Continuous Improvement

The rewrite isn’t finished; we’re taking it one component at a time. But already, the experience has reshaped how we think about our system. We’ve become faster, our infrastructure has become more predictable, and our mental load is lighter.

For a startup building probabilistic models of complex systems with a small team, reliability and simplicity help a great deal.

In rewriting our system, we discovered that language choice shapes not only how software runs, but how teams think and work.

If you’re a small company hitting the scaling limits of your Python stack — and you’re willing to learn something new — we’d encourage you to give Clojure a try. You might find, as we did, that it helps you build not just better systems, but better flexibility when building them.

1. Introduction

In the previous post, we explored how hypernetworks enable a neural network to adapt dynamically to new datasets. By learning dataset embeddings and mapping them to network parameters, hypernetworks allowed us to train a single model across heterogeneous data and to generalize the model to unseen datasets without retraining the entire network.

However, we saw that approach has clear limitations. Our hypernetwork produced a single embedding per dataset, but offered no way to quantify uncertainty in those embeddings. We also saw that a hypernetwork’s flexibility comes at a cost: with no principled way to incorporate prior knowledge or to enforce structure, our model sometimes predicted functional forms that differed dramatically from the true solution. This was especially apparent out of sample, but making out-of-sample predictions is one of the primary reasons we’d want to use a hypernetwork.

This brings us to an alternative approach to machine learning: Bayesian hierarchical models.

In many real-world settings, data is hierarchical. Observations are grouped into datasets, each governed by its own hidden parameters. For example:

• In scientific experiments, results may vary across different labs or instruments.
• In medicine, patient outcomes may depend on hospital-specific conditions.

To make this concrete, we will use the same example problem we used in Part I: this “toy problem” quantifies the essential features of hierarchical data. Each dataset consists of observations (𝝂, y) drawn from Planck’s law:

where:

• 𝝂 is the covariate (frequency),
• y is the response (brightness or flux),
• T is a dataset-specific parameter (temperature), constant within a dataset but varying across datasets, and
• ε is Gaussian noise with scale σ, which is unknown but remains the same across datasets.

The dataset is hierarchical because of its temperature structure: each dataset i has a constant T_i, with multiple i.i.d. observations (𝝂, y) ∈ g(T_i). At the same time, there are multiple datasets, each with different temperatures T, and since g(T_i) ≠ g(T_j), the overall population of samples is not i.i.d.

The key point is that while (𝝂, y) are observed, the dataset-level parameter T is part of the data-generating function, but is unknown to the observer. While the function f(𝝂; T) is constant, since each dataset has a different T, the mapping y(𝝂) varies from one dataset to the next.

Bayesian hierarchical models address this type of structure directly, by treating dataset parameters as random variables drawn from a shared distribution. The distribution is constant, but draws from the distribution are random and will naturally vary from one dataset to the next. This approach naturally mirrors the hierarchical nature of the data.

Compare this to the hypernetwork approach, in which we trained a secondary network to generate 1^st-layer weights from a dataset embedding. Like a statistical distribution, this network is a fixed function which outputs effectively random (via the embedding) variables which are specific to each dataset. A Bayesian hierarchical model accomplishes a similar effect, but it does it in a mathematically principled way that is grounded in statistics. We will see below how this statistical grounding confers several important advantages.

A key advantage of Bayesian modeling is its framework for incorporating prior knowledge. Priors enable you to bake domain expertise, scientific laws, or reasonable constraints right into the model, making the model both more interpretable and more robust. Unlike neural networks, where structure is implicit and hidden among millions of learned weights, Bayesian models provide a transparent, probabilistic framework for combining prior beliefs with observed data.

The result is a modeling approach that shines in exactly the settings where we saw hypernetworks struggle: small, censored, or noisy datasets, and out-of-sample predictions. Moreover, Bayesian hierarchical models give us not just estimates, but full posterior distributions over parameters, allowing us to reason about both values and uncertainties in a principled way.

2. What is a Bayesian Hierarchical Model?

Most machine learning models estimate fixed point values for their parameters. A regression model, for instance, finds a single best-fit coefficient for each predictor; a neural network learns a fixed set of weights to map inputs to outputs.

Bayesian models take a different approach. Instead of committing to a single value, they learn an entire probability distribution over each parameter. This distribution captures both the most likely values and the model’s uncertainty about them. The result is not just an estimate, but a quantified measure of confidence.

A Bayesian hierarchical model extends this idea by introducing multiple layers of uncertainty. Instead of treating dataset-specific parameters independently, it assumes that they are drawn from a shared prior distribution, reflecting the belief that datasets are related. This structure allows the model to pool information across datasets while still preserving dataset-level variation.

To make this concrete, return to our example problem in which each dataset is governed by a temperature parameter T. A naive approach would estimate each T_i in isolation, using only the data from dataset i. This wastes information: small datasets wouldn’t have enough information to infer all the model parameters, and we miss an opportunity to share statistical strength across the population.

A hierarchical model addresses this by assuming that each dataset’s parameter is drawn from a common distribution:

These hyperparameters capture the overall variation of T across datasets, ensuring that dataset-level estimates remain reasonable and consistent with the known population, even when observations are scarce. In this way, hierarchical priors act as a kind of structured regularization: when data are abundant, estimates can diverge as required to fit the data; when data are sparse, the prior pulls estimates toward the population distribution.

This structure offers two major benefits:

1. Pooling information: Datasets with limited observations borrow strength from the global distribution, leading to more stable and reliable estimates.
2. Quantifying uncertainty: Instead of producing a single point estimate for each T_i, the model yields a posterior distribution, capturing a range of plausible values and our confidence in them.

A Bayesian hierarchical model balances flexibility with structure. It allows each dataset to have its own parameters, but constrains those parameters through a shared distribution, providing both stability and uncertainty quantification.

3. Defining the Model Structure

To see Bayesian hierarchical modeling in action, let’s define a concrete generative model for our problem. In order to build a model which can train on heterogeneous data, we need to estimate a dataset-specific parameter T_i for each dataset. We make the model hierarchical by also estimating the population distribution of temperatures.

import pymc as pm
import numpy as np

# Assume we have multiple datasets: (vs[i], ys[i]) for dataset i
num_datasets = len(vs)
coords = {
    "dataset": 1+np.arange(num_datasets)
}

with pm.Model(coords=coords) as hierarchical_model:

    # Hyperpriors for the Gamma distribution controlling T
    ɑ = pm.Exponential('ɑ', scale=1)
    β = pm.Exponential('β', scale=1)

    # Alternative (and weakly-informative) choices:
    # ɑ = pm.HalfNormal('ɑ', sigma=2) or ɑ = pm.Gamma('ɑ', alpha=2, beta=1)
    # β = pm.HalfNormal('β', sigma=2) or β = pm.Gamma('β', alpha=2, beta=1)

    # Dataset-specific T values drawn from a Gamma prior
    T = pm.Gamma('T', alpha=ɑ, beta=β, size=num_datasets, dims="dataset")

    # Shared noise parameter across all datasets
    σ = pm.Exponential('σ', scale=1, initval=5)

    # Model each dataset separately
    for i in range(num_datasets):
        y_1 = pm.Deterministic(f'y_1_{i}',
                               vs[i]**3 / (pm.math.exp(vs[i] / T[i]) - 1))

        # the "observed" y has noise.  model it as Gaussian noise with scale σ
        y_obs = pm.Normal(f'y_obs_{i}', mu=y_1, sigma=σ, observed=ys[i])

     # more advanced: account for the fact that data is truncated
     # y_latent = pm.Normal.dist(mu=y_1, sigma=σ,)
     # obs = pm.Truncated("obs", y_latent, lower=0, upper=50, observed=y)


# Perform inference
with hierarchical_model:
    trace = pm.sample(chains=4, cores=2)

4. Results

With the Bayesian hierarchical model defined, we can now evaluate its performance on the same synthetic datasets used in the hypernetwork experiment. Each dataset has its own hidden temperature parameter T_i, while the noise scale σ is shared across all datasets.

5. Conclusion & Looking Ahead

Bayesian hierarchical models offer a principled way to handle hierarchical data. By combining dataset-specific parameters, shared priors, and explicit uncertainty quantification, they provide models that are both flexible and interpretable. Compared to hypernetworks, Bayesian models shine in small-data and noisy settings, where pooling information and reasoning about uncertainty are critical.

At the same time, Bayesian inference can be computationally demanding, especially for large datasets or complex models. Hypernetworks, by contrast, scale more efficiently, taking advantage of modern hardware and optimization techniques. The choice between these approaches depends on the problem: when robustness and uncertainty matter most, Bayesian models are often the better fit; when scalability and speed are paramount, hypernetworks may be preferable.

Since I used the same function in the Bayesian model that I also used to generate the data, you might suspect that the advantages the Bayesian model shows here result from simple cheating: that the correct solution is somehow baked into the model. In the next post, I will show that this is not the case: I will show that the advantages I have shown are inherent in the technique and that other, less prescient modeling choices can yield equally good results if implemented reasonably.

1. Introduction

In many real-world problems, the data is hierarchical in nature: observations are grouped into related but distinct datasets, each governed by its own hidden properties. For example, consider a clinical trial testing a new drug. The trial spans multiple hospitals and records the dosage administered to each patient along with the patient’s outcome. The drug’s effectiveness is, of course, a primary factor determining outcomes—but hospital-specific conditions also play a role. Patient demographics, procedural differences, and even how results are recorded can all shift the recorded outcomes. If these differences are significant, treating the data as if it came from a single population will lead to flawed conclusions about the drug’s effectiveness.

From a machine learning perspective, this setting presents a challenge. The dataset-level properties—how outcomes vary from one hospital to another—are latent: they exist, but they are not directly observed. A standard neural network learns a single, constant map from inputs to outputs, but that mapping is ambiguous here. Two different hospitals, with different latent conditions, would produce different outcomes, even for identical patient profiles. The function becomes well-defined (i.e. single-valued) only once we condition on the dataset-level factors.

To make this concrete, we can construct a toy example which quantifies the essential features we wish to study. Each dataset consists of observations (𝝂, y) drawn from a simple function with a hierarchical structure, generated using Planck’s law:

where:

• 𝝂 is the covariate (frequency),
• y is the response (brightness or flux),
• T is a dataset-specific parameter (temperature), constant within a dataset but varying across datasets, and
• ε is Gaussian noise with scale σ, which is unknown but remains the same across datasets.

This could represent pixels in a thermal image. Each pixel in the image has a distinct surface temperature T determining its spectrum, while the noise scale σ would be a property of the spectrograph or amplifier, which is consistent across observations.

The key point is that while (𝝂, y) pairs are observed and known to the modeler, the dataset-level parameter T is part of the data-generating function, but is unknown to the observer. While the function f(𝝂; T) is constant (each dataset follows the same general equation), since each dataset has a different T, the mapping y(𝝂) varies from one dataset to the next. This fundamental structure — with hidden dataset-specific variables influencing the observed mapping — is ubiquitous in real-world problems.

Naively training a single model across all datasets would force the network to ignore these latent differences and approximate an “average” function. This approach is fundamentally flawed when the data is heterogeneous. Fitting a separate model for each dataset also fails: small datasets lack enough points for robust learning, and the shared functional structure, along with shared parameters such as the noise scale σ, cannot be estimated reliably without pooling information.

What we need instead is a hierarchical model — one that accounts for dataset-specific variation while still sharing information across datasets. In the neural network setting, this naturally leads us to meta-learning: models that don’t just learn a function, but learn how to learn functions.

2. Why Standard Neural Networks Fail with Hierarchical Data

A standard neural network trained directly on (𝝂, y) pairs struggles in this setting because it assumes that one universal function maps inputs to outputs across all datasets. In our problem, however, each dataset follows a different function y(𝝂), determined by the hidden dataset-specific parameter T. The single-valued function f(𝝂; T) is not available to us because we cannot observe the parameter T. Without explicit access to T, the network cannot know which mapping to use.

3. Dataset-Adaptive Neural Networks

# dataset embeddings (initialized randomly & updated during training)
dataset_embeddings = tf.Variable(
    tf.random.normal([num_datasets, embed_dim]),
    trainable=True
)

# Assign dataset indices for each sample
dataset_indices = np.hstack([np.repeat(i, len(v)) for i, v in enumerate(vs)])
x_train = np.hstack(vs).reshape((-1, 1))
y_train = np.hstack(ys).reshape((-1, 1))

# Retrieve dataset-specific embeddings
E_train = tf.gather(dataset_embeddings, dataset_indices)

def build_hypernetwork(embed_dim):
    """Generates parameters for the first layer of the main network"""

    emb = K.Input(shape=(embed_dim,), name='dataset_embedding_input')

    l = K.layers.Dense(16, activation='mish', name='Hyper_L1')(emb)
    l = K.layers.Dense(32, activation='mish', name='Hyper_L2')(l)

    # Generate layer weights (32 hidden units, 1 input feature)
    W = K.layers.Dense(32, activation=None, name='Hyper_W')(l)
    W = K.layers.Reshape((32, 1))(W)  # Reshape to (32, 1)

    # Generate biases (32 hidden units)
    b = K.layers.Dense(32, activation=None, name='Hyper_b')(l)

    return K.Model(inputs=emb, outputs=[W, b], name="HyperNetwork")

class DatasetSpecificLayer(K.layers.Layer):

    def __init__(self, **kwargs):
        super(DatasetSpecificLayer, self).__init__(**kwargs)

    def call(self, inputs):
        """ Applies the dataset-specific transformation using generated weights """

        x, W, b = inputs  # unpack inputs

        x = tf.expand_dims(x, axis=-1)       # Shape: (batch_size, 1, 1)
        W = tf.transpose(W, perm=[0, 2, 1])  # Transpose W to (batch_size, 1, 32)

        out = tf.matmul(x, W)          # Shape: (batch_size, 1, 32)
        out = tf.squeeze(out, axis=1)  # Shape: (batch_size, 32)

        return out + b  # Add bias, final shape: (batch_size, 32)

embed_dim = 4
hypernet = build_hypernetwork(embed_dim)

def build_base_network(hypernet, embed_dim):
    """ Main network that takes x and dataset embedding as input """

    inp_x = K.Input(shape=(1,), name='input_x')
    inp_E = K.Input(shape=(embed_dim,), name='dataset_embedding')

    # Get dataset-specific weights and biases from the hypernetwork
    W, b = hypernet(inp_E)

    # Define a custom layer using the generated weights
    l = DatasetSpecificLayer(name='DatasetSpecific')([inp_x, W, b])

    # Proceed with the normal dense network
    l = K.layers.Activation(K.activations.mish, name='L1')(l)
    l = K.layers.Dense(32, activation='mish', name='L2')(l)
    l = K.layers.Dense(32, activation='mish', name='L3')(l)
    out = K.layers.Dense(1, activation='exponential', name='output')(l)

    return K.Model(inputs=[inp_x, inp_E], outputs=out, name="BaseNetwork")

4. Training Results

With the embeddings, base network, and hypernetwork now stitched together, we can now evaluate the model on our test problem. To do this, we train on a collection of 20 synthetic datasets generated from Planck’s law as described in Section 1. Each dataset has its own temperature parameter T, while the noise scale σ is shared across all datasets.

The figure below shows the training results. Each panel shows a distinct dataset from the test. In each panel,

• the Blue solid curve shows the true function derived from Planck’s law,
• the Black points shows observed training data,
• the Red dashed curve shows predictions from the hypernetwork-based model, and
• the Gold dotted curve shows predictions from a conventional neural network trained separately on each dataset.

Several key patterns are evident:

1. Comparable overall accuracy: In many cases (such as the 1^st column of the 1^st row), the hypernetwork’s predictions (red) are very similar to those of an isolated neural network (gold). This shows that, despite strongly restricting the model and removing ~95% of its parameters, sharing parameters across datasets does not sacrifice accuracy when sufficient data are available.

2. Improved stability: When training data are sparse (such as in the 1^st column of the 2^nd row, or the last column of the 3^rd row), the hypernetwork over-fits considerably less than the isolated neural network. Its predictions remain smoother and closer to the true functional form, while the isolated neural network sometimes strains to fit individual points.

3. Pooling across datasets: By training on all datasets simultaneously, the hypernetwork learns to separate the shared structure [such as the noise scale σ, or the underlying functional form f(𝝂; T)] from dataset-specific variation (the embedding E). This shared learning stabilizes predictions across the board, but it is especially visible in the panels with particularly noisy data (such as the last column of the 2^nd row, or the 2^nd column of the 3^rd row).

5. Predictions for New Datasets

The training performance of our hypernetwork is encouraging, but the real test is how the model adapts to new datasets it has never seen before. Unlike a conventional neural network — which simply applies its learned weights to any new input — our model is structured to recognize that each dataset follows a distinct intrinsic function. To make predictions, it must first infer the dataset’s embedding.

class DatasetEmbeddingModel(K.Model):

   def __init__(self, base_net, embed_dim):
       super(DatasetEmbeddingModel, self).__init__()
       self.base_net = base_net
       self.E_new = tf.Variable(
           tf.random.normal([1, embed_dim]), trainable=True, dtype=tf.float32
       )
       # for better performance on small datasets, use tensorflow_probability:
       # self.E_new = tfp.distributions.Normal(loc=0, scale=1).sample((1, embed_dim))


   def call(self, x):
       # Tile E_new across batch dimension so it matches x's batch size
       E_tiled = tf.tile(self.E_new, (tf.shape(x)[0], 1))
       return self.base_net([x, E_tiled])

   def loss(self, y_true, y_pred):
       mse_loss = K.losses.MSE(y_true, y_pred)
       reg_loss = 0.05 * tf.reduce_mean(tf.square(self.E_new))  # L2 regularization on E
       return mse_loss + reg_loss

6. Discussion & Next Steps

The hypernetwork approach shows how neural networks can go beyond brute-force memorization and move toward structured adaptation. By introducing dataset embeddings and using a hypernetwork to translate them into model parameters, we designed a network which is able to infer dataset-specific structure, rather than simply averaging across all datasets. This allows the model to generalize from limited data—a hallmark of intelligent systems.

The results highlight both the strengths and limitations of this strategy:

Background

At Sturdy Statistics, we build sparsity-inducing hierarchical mixture models for unsupervised and semi-supervised text organization. These models are extensions of topic models, with the ability to add arbitrary levels of hierarchy or additional branches, and to embed metadata into the structure of the graph. Our models learn a set of high level “topics” from the data, and then annotate the original data with these high level topics. We use DuckDB to store topic annotations at the paragraph and document level.

Because we explicitly model the power-law behavior of text data, our priors adopt a rich-get-richer scheme. Consequently, most topic activations are near zero. That means our data is inherently sparse — and ripe for storage optimization. There are a number of analytical benefits to this sparsity, but for the purpose of this post, the primary benefit is some sweet storage and speed wins.

Once I saw how sparse our data really was, I wanted to see how much space we could actually save in practice.

Sparsity

The next steps were

1. Write arrays into DuckDB in a sparse format
2. Write a set of wrappers that enable me to operate on these sparse arrays as if they are standard DuckDB lists
3. Benchmark performance and storage wins (I’m optimistic)

For (1), I decided to use SciPy style sparse arrays (Dimension: int, indices: list(int8), Values: list(float32)). For example: [7, 0, 0, 0, 23] => (5, [1, 5], [7, 23])

import numpy as np
def numpy_to_sparse(x):
    x = np.where(x < np.max(x)/3e3, 0.0, x).astype(np.uint8)
    inds = np.where(x!=0)[0]
    vals = x[inds]
    dim = len(x)
    vals = (dim, inds.astype(np.uint16), vals.astype(np.float32))
    return vals

def numpy_from_sparse(vals):
    dim = vals[0]
    inds = vals[1]
    data = vals[2]
    arr = np.zeros(dim, dtype=np.float32)
    arr[inds] = data
    return arr

When building wrapper functions, I had to choose between storing the sparse format in one column or splitting it into three. The benefit of storing it in one column is that I could fully abstract away the fact that these arrays are sparse. However, because DuckDB is a columnar data store, I opted to split out the indices and the values so I could flexibly access either component without reading both. Our data is sparse enough that the indices alone provide a mechanism to rapidly filter excerpts that contain a topic to some degree. If I stored the indices in the same column as the values, I would have to read 5× as much data (int8 vs int8+float32). The immediate performance benefit made the choice easy. I also fully removed the dimensionality parameter, opting to store it in the table metadata.

If my goal were to integrate this into DuckDB, I would have opted for the more ergonomic approach.

I put together a few helper functions that enabled me to perform all the operations. We use these functions internally, and we also publicly expose them in our API.

CREATE OR REPLACE TEMPORARY MACRO sparse_list_extract(position, inds, vals) AS  
    COALESCE(vals[list_position(inds, position)], 0);

CREATE OR REPLACE TEMPORARY MACRO sparse_list_select(positions, inds, vals) AS  
    list_transform(positions, x-> sparse_list_extract(x, inds, vals));

CREATE OR REPLACE TEMPORARY MACRO dense_x_sparse_dot_product(arr, inds, vals) AS 
    list_dot_product(list_select(arr, inds)::DOUBLE[], vals::DOUBLE[]);

CREATE OR REPLACE TEMPORARY MACRO sparse_to_dense(K, inds, vals) AS 
    list_transform(
        generate_series(1, K), 
        x-> coalesce(list_extract(vals, list_position(inds, x)), 0)
    );

These helpers replace array indexing, multi-indexing and converting sparse lists to dense. I was actually surprised how rarely I actually needed to perform this conversion: instead, for operations such as Hellinger distance or dot products, I can often restrict computation to the overlap in sparse indices instead (spoiler: performance wins are upcoming).

Benchmarking the Code

My first attempt was to benchmark DuckDB file sizes storing various levels of sparsity. I wanted a clear, granular understanding of how sparsity affects data at various levels of dimensionality and sparsity. I decided to put together some artificial data to benchmark the results…

Hint: this was a bad idea

I used numpy to generate a matrix of K×D where K is the number of topics (embedding dimension in Neural terms) and D is the number of documents. I set the first S rows to be 1/K and the rest were zero. I took these matrices and created two DuckDB files: one with raw storage and the other sparsely encoded.

I was shocked to see that there was virtually no difference in the file size. I was stunned that a 512-dimensional array occupied nearly the same space in DuckDB as two 8-dimensional arrays.

In my eagerness to get an easy 75% reduction in file size, I completely overlooked the fact that DuckDB applies Snappy compression to its files. Compressing long runs of zeros and ones is trivial for a compressor, so my synthetic data didn’t reveal much. Even though the synthetic benchmarks didn’t deliver the expected 75% reduction, they exposed how compression interacts with sparsity — a key insight for real-world data. I re-ran the test with random values in random positions and saw a modest but real improvement: ~20% average reduction.

Note: at this point in time my entire DuckDB file was a single database table consisting of a single list column, stored either raw or sparsely across two columns.

This was a solid win, albeit one much less than I hoped for. But it was enough to see this project through to the next stage.

Real Data (or the only benchmark that mattered)

I downloaded the DuckDB files from our publicly hosted indices to migrate to our sparse format for storage and performance benchmarking. I repurposed my previous script to read the original DuckDB files, convert the relevant list fields to a sparse format, and write a new file. The result made me simultaneously excited, confused — and hesitant:

This is a major reduction in our file sizes, hitting an over 50% reduction for some of our largest files.

What happened?

My initial test only measured a single column at a time. In production, however, we store several sparsely-populated arrays alongside any arbitrary metadata provided by our users or by our integrations. Since this metadata is not sparse, I had expected its presence would reduce the impact of our total file size reduction. But it had the opposite effect: we see a much more pronounced difference on real data. It appears that the additional columns handicapped the Snappy compression, rendering our sparse storage much more effective at the file level, not just the array level.

While we didn’t see the 75% reduction I had initially hoped for, compression still left us with a 52% average reduction across all files. I’ll take it!

Scaling with Dimensionality

As expected, higher-dimensional arrays yielded larger storage reductions. Since our biggest datasets also have the highest topic counts, they benefited most.

While the large dataset performance is a big win, we care most about general performance because we support so many use cases:

• Our public Deep Dive Search creates a few hundred indices per day, split between our fast mode (48 dimensions) and our comprehensive mode (96 dimensions).
• Our production platform integrations have a slightly lower number of analyses trained per day with a default of 196 dimensions.
• Our large-scale models tend to be a more deliberate effort, with maybe 1–2 of these trained per week.

At the moment, the total amount of storage is relatively evenly split between these three categories of models. (And we hope these numbers go up as we begin to announce Sturdy Statistics publicly!)

Performance

I was both optimistic and concerned about the impact of sparse storage on performance. It turned out that we saw no observable changes in our search query performance or our topic retrieval performance. This was likely because I was able to leverage my dense_x_sparse_dot_product SQL function to avoid casting any sparse arrays to dense.

Our TopicSearch and TopicDiff however, initially saw significant performance degradations — about 2–5× slower. These APIs are much more analytical; they perform statistics over topic counts and expect dense arrays. I simply converted the sparse arrays to dense in my first pass, which obviously added overhead.

However, our new sparse storage format opened the door to fully leveraging UNNEST much more efficiently due to the lower dimensionality of the arrays and the separate indices-and-values arrays. Thanks to sparse storage, an UNNEST that previously expanded rows by 500× now expands them by just 3–8×.

This not only sped up DuckDB queries but also let me migrate a major portion of my Python post-processing directly into DuckDB. With some tinkering I was able to see a solid 2–10× performance improvement, with the larger improvements occurring on the larger datasets.

Sparse storage not only saved space—it made some operations up to 10× faster.

Conclusion

My main conclusion — one I’ve learned before and will likely learn again — is that the only benchmark that matters is the one on your own data. Simulations and estimates are helpful thought exercises; real workloads decide.

Second, while I initially tried to make sparse arrays feel “just like” dense arrays, embracing the different data structure is what unlocked most of the gains. UNNEST became powerful because indices and values live in separate columns. The “right” architecture wasn’t what I first expected; it emerged from benchmarking real use cases and looking carefully at what we were actually storing.

These sparse array functions live in a small GitHub repo, with documentation with examples. We plan to reach out to the DuckDB team soon to explore official support for sparse-style arrays. If you’re interested in running with this idea — for their own project or via open — I’d love to chat or help out however I can.

AI Eats the (Early-Stage Funding) World

The release of ChatGPT in 2023 reshaped venture capital overnight. We can see this by inspecting the portfolio of Y Combinator (YC) startups. I collected YC company descriptions across cohorts and used the Sturdy Statistics API to automatically structure the data.

The hierarchical “sunburst plot” below shows the data from 2020–2023. The inner ring shows high-level themes, which branch into more granular themes in the outer ring of the plot. Clicking on a high-level theme expands it, making the 2^nd level topics easier to read.

Before ChatGPT, we can see that YC investments spanned a wide range: YC covered finance, healthcare, logistics, supply chain, and general business tools. And it’s not all software: we see Space Technologies and Autonomous Vehicles in the portfolio. The AI-related fields Data Analysis and Automation appear in the list, but as two categories among many.

After 2023, the landscape changed completely. The following figure shows the themes in YC company descriptions since 2023:

We can see that virtually all prominent categories are now AI-related: we have automation, data analysis, and developer tools… The diversification of the early 2020s vanished. AI really did eat the early-stage software world.

(If you like, you can explore the data yourself in the interactive dashboard.)

Waves of Investment

Thus far, we’ve looked at aggregate data before and after the release ChatGPT. If we zoom in and inspect the year-to-year shifts in investment, a surprising twist in the story emerges.

Take a look at this slope plot, which shows the change in rank for topics between 2023 and 2024. Each side of the plot lists topics by rank, and a line connects their 2023 position to their 2024 position; topics which decreased in prominence have negative slopes, shown in red. Topics which surged in prominence have positive slopes, shown in blue.

Parting Thoughts

The data show a clear trajectory: LLM applications are shifting right, away from analytical fields with hard metrics, to fuzzier domains with ambiguous outcomes, to meta-level tools and moonshots whose impact may only be measurable years into the future.

To be clear, there are valuable near-term uses for LLMs: translation, copyediting, summarizing content, presenting structured data in narrative or conversational form, and writing certain types of code all come to mind.

But the trillion dollar bets being placed on full-scale automation across the economy risk overlooking a sobering truth: the more measurable the field, the faster it has already rejected wholesale LLM automation.

My concern is that this recognition will eventually spread to fuzzier fields too—just on a longer timescale. If so, the scale of investment by that time, along with the attendant costs of failure, could be enormous.

Footnotes

[1] This data was organized by leveraging hierarchical mixture models.

[2] The slope plots are accessible here.

[3] The annotated dataset is publicly accessible at index_c50d731139f74898b4c004937ae77a83. This is most easily accessed using sturdy-stats-sdk. Additional documentation.

1: Citizen’s Monetization Infuriation

I was considering downloading a neighborhood watch application. I’d heard good things about Citizen so I decided to check it out.

At first glance it looks good: protect the world, 4.8 stars, 474 thousand reviews. But let’s dig a little deeper.

I downloaded every app story review that was publicly visible and organized it hierarchically. This does not look like a 4.8 star app. After a double take, I decided to look at the distribution of ratings.

An average rating of 1.5 over this time interval. Probably the thing to start looking at first. Something I want to emphasize: I scraped every single publically available reviewsince July 2023 sorted by recency. So there has been a massive drop in its rating.

2: AirBnB – Hotels just might survive

4.8 stars. 683 thousand ratings. Everyone knows AirBnB.

You know where this is going. Let’s dig in.

Yeesh. Not much good there. Good news for us though is that the timeline is a little more interesting in this analysis.

3: Hinge

To start off with, it actually turns out that Hinge has by far the best reviews of any of the dating apps I analyzed.

Thre is significantly more discussion around Connection and Matchmaking than any other dating app, and much less discussion about Subscription Billing Issues, Technical Issues, and Scams and Security.

Maybe its 4.4 star rating is reflective of it’s current state.

Conditional vs. Joint Probability: The Foundations Matter

Conditional probability is the foundation of nearly all AI systems today — including every large language model. They predict the next word or label based on what’s come before. In other words, they predict outputs conditionally upon the inputs. This works well in many cases, but there’s a catch: the structure of the world — how different variables relate to one another — is never made explicit in this type of model; it is only conditional on input. Structure is learned implicitly, absorbed into model weights during training, and is only accessible via creative “prompting.”

This architecture choice has consequences. Because the structure is hidden, the model has to rediscover it every time you make a query. The results can be powerful, but they’re also stochastic, hard to reproduce, and difficult to trace. And because this approach has become so standard, one might forget to ask whether there’s a better way.

At Sturdy Statistics, we start from a different foundation: joint probability. That means we model the full structure of the data — not just what happens next, but how everything fits together. Joint probability represents a theory about your data, not just an interpolation of it. It doesn’t just fill in missing words; it models how words, topics, and contexts interact. A joint model understands relationships in a way that’s accessible to the model’s author.

These kinds of models are the workhorses of the natural and social sciences. They’re used when you have a theory to test, when you care about uncertainty, and when getting the answer right matters. Until now however, joint models were too slow and too complex to apply to large-scale text. Sturdy Statistics has changed that. Our model runs where you need it to — whether your dataset is tiny or massive.

With joint modeling, Sturdy Statistics can tell you things like:

• how consumer sentiment depends on product category—and why;
• which trends emerged (or disappeared) in a company’s most recent quarterly report;
• how each agent in your contact center handles each question type—and which are most effective;
• what’s present in your data that you didn’t know to ask about;
• how uncertainty shifts as data volume changes; and
• which words or phrases most influence each prediction.

This structure is explicit, auditable, and grounded in statistical theory.

Why does this matter? Because joint models reveal the underlying structure of your data — while conditional models can’t tell you when they don’t know something, or anything you didn’t think to ask for.

Structure Is Power: The Practical Upside

LLMs succeed because they’ve absorbed patterns from enormous datasets — but they do so conditionally, not structurally. They find structure implicitly, by internalizing statistical regularities without ever exposing them to the user.

This makes LLMs brittle on rare or unusual inputs. It also means you can’t ask them why they gave a particular answer, or how confident they are in it.

By contrast, explicit structure allows our models to generalize more effectively from less data. Because the model represents a theory about the world, it doesn’t learn ab initio; it uses prior knowledge. Neither does our model have to re-learn its context every time you use it. Instead, our models use incoming data efficiently, focusing on what’s new rather than what’s already known.

Our approach has major benefits:

• Higher few-shot accuracy: Built-in structure means strong performance even with limited data.
• Quantified uncertainty: Every prediction includes an interpretable confidence estimate.
• Interpretability: You can inspect the assumptions, priors, and data behind every result. If there’s a mistake, you can fix it — directly and deterministically.

This kind of transparency is invaluable in regulated environments, audit settings, or any situation where trust matters.

Zipf’s Law: Why Rare Data Matters

Real-world data has a long tail. A few things happen often, but most things happen rarely. This is formalized in Zipf’s Law, which shows that the most frequent words in a corpus are extremely common—but most words are not. In language, the tail dominates the meaning.

LLMs are inherently biased toward the head of the distribution. They rely on overwhelming amounts of data to learn rare patterns, and even then, performance on long-tail items is inconsistent.

Our models are different. We’re built to expect rarity. We have:

• Zipf-aware priors that model skewed, power-law frequency distributions;
• Robust handling of unseen categories;
• Clean extrapolation even when examples are sparse.

This makes our system excel where others struggle: identifying rare issues, analyzing niche domains, and delivering insights even when data is incomplete.

Practical Technology: A Probabilistic Engine with a Friendly Interface

You don’t need to know Bayesian statistics to use our system. All you need to do is:

1. Upload your documents: reviews, call transcripts, support tickets, or any other unstructured text. Or, if you’d prefer to have us curate the data, use one of our integrations.
2. We process and structure the data into meaningful rows and columns.
3. You query the results using ordinary SQL, or using one of our prebuilt no-code dashboards.

Under the hood, our engine combines:

• Hierarchical Bayesian inference
• Zipf-aware modeling
• Transparent, interpretable computation of the joint probability distribution

The result is a structured dataset that’s ready for analysis, reporting, or automation — no black box, no guesswork.

How We’re Different from LLMs

Large language models are extraordinary tools for open-ended creativity — from writing and brainstorming to coding and summarization. Our focus is different. When the goal is to extract structure, quantify uncertainty, or analyze text with precision, Sturdy Statistics offers a more rigorous and interpretable approach.

The Future of Trustworthy AI

AI doesn’t have to be a black box.

LLMs will often tell you something plausible. We’ll tell you what’s probable — and how probable.

If your decisions depend on the data, you need that data to be sturdy. That’s what we deliver: dependable models for real-world understanding.