~ $ cd #content
back to writing

~ $ cat writing/semantic-llm-wiki-deterministic-cli.md

LLMs are semantic, filesystems are not

·5 min

Every conversation about agents eventually lands on the same place. Not the model. The context.

You can hand the same task to the same model twice and get a useful answer once and a confident invention the other time, and the difference is almost never intelligence. The focus should shift, and is in fact shifting, towards the harnass, and specifically, the context you provide through knowledge.

That is what got me interested in the LLM wiki.

The LLM wiki, briefly

The pattern is Karpathy's, from a gist that is worth ten minutes of your time.

You point an agent at a pile of raw material. Meeting transcripts, articles, papers, whatever you have been meaning to read. It produces a wiki out of it. One summary page per source. Entity pages for the things worth linking to, the people and the products and the companies. Concept pages carrying the thesis that runs across several sources. An overview tying it together.

Plain markdown, YAML frontmatter, [[wikilinks]]. Obsidian reads it natively, and so do you.

The appealing bit is not the summarising. Anything summarises. The appealing bit is that it compounds. Source eleven does not just add a page, it changes what page four means, because now there is a concept page holding a claim that two sources support and one contradicts. Your reading list stops being a pile and starts being a position.

I wanted that. So I built it the obvious way first, which is to say I gave an agent a folder and a conventions file and asked nicely.

Why that falls over

It worked for about a month. Then slowly, I started noticing inconstencies.

It was not dramatic failure, it was a slow, boring, almost insidious kind of failure. A page lands in the wrong folder. Frontmatter grows a key nobody agreed to. A link points at a page that got renamed two sessions ago, and nothing complains, because there is nothing in a folder full of markdown whose job it is to complain.

I spent a while blaming the model for that. The real failure was mine to not remember that a model is semantic, not deterministic. This means that, in my humble opinion, a model is structurally unsuited to maintain all parts of the llm wiki.

LLMs operate on meaning, not on exact strings. To a model, "the homelab review", "that platform evaluation", and homelab-platform-review-summary are three ways of saying one thing, and being able to see that is precisely why it can read a transcript and know that the vendor mentioned on page one and the "they" on page nine are the same company. Nothing else does that as cheaply.

But a wikilink is not a meaning. It is a string that resolves or does not resolve. An ID either matches a file or matches nothing. Close enough is a 404.

So you get a system whose greatest strength, tolerance for near-misses, is the exact property that quietly destroys a link graph. And the model gets no signal that it happened. Writing a broken link feels identical to writing a working one. Nothing throws. You find out three weeks later when you ask a question and get a worse answer than you should have, and you will not know why.

This is in my humble opinion the failing of the LLM Wiki.

Deterministic checks are the missing half

Which is the point I actually want to make, and it generalises well beyond wikis.

A lot of what we ask agents to do is not judgement at all. It is bookkeeping. Assign an ID. Keep the index current. Check that this link resolves. Remember which state this thing was in when the session died. That work is fully deterministic. We have known how to do it correctly for decades, and we do not need a model for any of it.

So stop asking the model to do it. Put it behind a tool, make the tool the only way through, and let the model spend its attention on the part that genuinely needs a model. Bonus point because a CLI also does not inflate the context window, saving you tokens and giving you less context rot.

That is wiki, which lives at github.com/JoranBergfeld/llm-wiki. A single binary. The agent reads anything it likes, unrestricted, and authors every word of the prose. It just cannot write a file. Every mutation goes through the CLI, and the CLI validates before anything lands on disk.

The split is not arbitrary. The agent does the work that needs judgement. The CLI does the work that needs never forgetting.

Link integrity

Take a wikilink pointing at a page that does not exist.

The upsert is rejected. Not flagged, not logged for later, rejected, with a machine-readable error code naming each dangling target. Nothing was written. The agent reads the code, fixes the link, tries again. A failed write is a conversation, not a crash.

Forward references are sometimes legitimate, so there is an escape hatch, and the escape hatch does not just wave things through. Pass --allow-dangling and the link is permitted, but it is filed as an issue at write time, on the spot. Permitted is not the same as forgotten.

Renames are where link graphs usually die, so renaming goes through the CLI too: it moves the file and rewrites every inbound link in the vault. And because I know perfectly well that I am going to rename something in Obsidian at some point like an animal, lint catches that case as well. It spots the mismatch via the frontmatter ID, repairs its map, and files an issue listing every inbound link I just broke.

One deliberate omission, because it is the clearest example of the discipline. External URL checking exists, but it is a separate opt-in command and it is not part of lint. Every lint check is a pure function of the vault's bytes: same vault in, same findings out, offline, instantly. An HTTP probe is none of those things. Wire it into lint and a flaky wifi connection starts filing findings, and the count of how often a problem recurs, which is the one number I use to decide whether something is systemic, quietly turns into a measure of my network. A check you cannot trust is worse than no check.

ID integrity

Every source and every page gets a ULID at creation, written into frontmatter, permanent.

The filename is a slug, and slugs move. Titles get better. Pages get renamed. The ID does not care. Everything that refers to anything refers to it by ID, so the reference survives the rename.

Provenance runs on the same rails. A page cites the source IDs it came from, and an unknown source ID is a rejected write, not a footnote nobody checks. Which means "where did this claim come from" always resolves to an actual file, still sitting in raw/, immutable, exactly as it arrived. That is the question I care most about being able to answer, and it is the one a decayed wiki loses first.

There is a map from ID to path, and it is deliberately worthless. It is a cache. If it gets confused, wiki reindex rebuilds it from the markdown alone. Nothing derived is precious, and I think that is underrated as a design property: the markdown is the truth, and every clever index I build on top of it has to be disposable.

Small thing I got wrong and had to go back and fix, because it is a nice illustration of how sneaky this class of bug is. Sources are deduplicated by content hash. The hash was computed over the raw file, which means the same document picked up on Windows and on Linux hashed differently, because CRLF. Content-addressed dedup that is sensitive to invisible whitespace is not content-addressed. Deterministic is a thing you have to actually check, not a thing you get by writing the word in a spec.

Why any of this matters

Because your knowledge is your context, and your context is the ceiling on every agent you run.

We spend an enormous amount of effort on the top of that stack. Better prompts, better tools, better orchestration. All fine. But it sits on top of what the agent can retrieve, and if that layer is quietly rotting then everything above it is worse than you think and you have no instrument that tells you so.

A knowledge base that decays is genuinely worse than not having one. If I have no wiki, I go and read the source. If I have a wiki I trust and it has been slowly losing links and dropping provenance for a month, I stop reading the source, and I get confidently wrong answers with a citation attached. The failure is silent, which is the worst property a failure can have.

Deterministic checks are how you make it loud.

What is still rough

Quite a lot, and I would rather say so.

The checks verify shape, not truth. A vault full of beautifully linked, perfectly structured nonsense passes every one of them. There is work in there aimed at the semantic side, scoring retrieval against questions you write yourself and having the agent re-read a page against its sources cold, but that is asking a model to grade a model and I do not trust it the way I trust a link check.

Obsidian is a read-only viewer by convention, which is a polite way of saying nothing actually stops me editing a page in it and breaking the invariant the whole design rests on. Lint catches some of that afterwards. Catching it afterwards is not the same as preventing it.

The thing I am happy about is that the specific failure I built it to stop has stopped, and I can now tell when something is broken instead of finding out from a bad answer.

I did not want the answer to a model problem to be a CLI. Writing argument parsers is not why I got into this. But the model was never the thing that was broken. We just had too high expectations from that semantic powerhouse when it comes to determinism.