What we're building: a local RAG service

Step 1 of 8, and it runs nothing. This is the map: what the finished service does, the pipeline it is built from, and the seven steps between here and there. Every step starts from the state the previous one left behind, and each ends tagged in the examples repository, so the tags form one project growing a piece at a time.

RAG — retrieval-augmented generation — is the first AI feature most products actually need. A model only knows what was in its training data, and its context window is a few thousand words at best; anything else has to be fetched and handed to it. That fetch is the whole product. If retrieval brings back the right paragraph, a small model answers well; if it brings back three adjacent ones, no model saves you. The generation side is one API call. The retrieval side is chunking, embedding, storage and ranking — ordinary backend work, which is exactly where Rust is already strong.

The finished service

One binary, ragd, started with a documents directory. It ingests what it finds, then serves two routes on localhost: POST /ingest adds a document to the store, and POST /ask takes a question and streams an answer back over Server-Sent Events, the retrieved chunks named first. Inference runs on your own CPU. No API key, no rate limit, no per-token bill — the words retrieve, then answer, no cloud are the track’s contract, and the hosted-API option in step 7 sits behind a trait you can ignore.

Three decisions are fixed for the whole track, because too much of the later work depends on them:

• candle for inference. A pure-Rust tensor library from Hugging Face: no Python, no runtime to install, weights loaded straight from safetensors files. Step 2 picks the model and does the memory arithmetic.
• all-MiniLM-L6-v2 for embeddings. A 384-dimension sentence embedding model, about 22 million parameters, comfortably fast on CPU. Small enough that embedding a whole document library is minutes, not hours.
• SQLite for storage. Chunks and their vectors live in a rusqlite table; step 5 scans it brute-force first, then adds an index. A separate vector database is a dependency this track does not need until the numbers say so.

Everything above the model — chunking, storing, ranking, prompting — is plain Rust with no AI in it. That is not a limitation; it is the point.

The pipeline

The service has two halves, and the step order is the order they have to be built in.

     ingest (steps 3–5)                          answer (steps 6–7)
     ─────────────────────                       ────────────────────────────────────
document ──► chunk ──► embed ──► SQLite          question ──► embed ──► k nearest
            (4)      (3)      (5)                              │
                                                               ▼
                                            prompt ◄── context window ──► model ──► SSE
                                              (6)                            (2, 7)

Ingest happens once per document: split it into overlapping pieces (step 4), run each piece through the embedding model (step 3), and store text and vector together (step 5). Answer happens per question: embed the question with the same model, find the chunks whose vectors are closest by cosine distance, and build a prompt that contains them (step 6). The model then writes the answer into a stream (step 7), one token at a time, so the first word arrives in well under a second even when the full answer takes ten.

The single load-bearing fact in the whole design: both halves go through the same embedding model. A question and a document chunk are comparable only because they end up in the same 384-dimensional space. Change one side’s model without the other and every search silently returns nonsense — which is why the store records which model embedded it.

The route

#	Article	What lands
2	Running a model locally	candle, the weights, the memory it needs; the first tokens on CPU
3	Embeddings and vector similarity	cosine distance computed by hand on real vectors
4	Chunking documents for retrieval	size, overlap, and what each choice loses
5	A vector store in SQLite	chunks plus vectors, brute-force search, then an index
6	Retrieving and prompting	the context window, and the prompt template around it
7	An HTTP API for questions	axum, POST /ask, streaming over SSE
8	Evaluating retrieval quality	a small labelled set, and honest numbers for it

Step 2 is the only step that is not Rust code: it runs a model from the shell first, because the memory math and the token-by-token output are easier to see before anything wraps them.

What this service is not

• Not an agent. One retrieval pass, one answer. No tool calls, no self-correction loops, no planning. If the right chunk is not in the store, the answer is wrong, and step 8 measures exactly how often.
• Not a fine-tuning job. Nothing about the model changes at any step. All the domain knowledge sits in the store, which is why re-ingesting a document is instant and updating a model is never necessary.
• Not GPU-bound. Every step runs on CPU. The models are chosen small on purpose: step 2 works out the arithmetic for a 1–2B-parameter model in 4-bit, and the answer fits in ordinary laptop memory.
• Not multi-tenant. One process, one SQLite file, no auth, no queues. It is a service you run next to your editor, not one you put a load balancer in front of — yet.

What you need first

• Concurrency and async. The HTTP half is Tokio all the way down. Concurrency and async and Practical crates cover the working set.
• HTTP servers. Step 7 is an axum application with a streaming endpoint. Serving HTTP is the prerequisite.
• Result and ?. A model that fails to load and a document that will not parse are both errors to handle, not crashes. Handling errors covers the pattern.

Every step is written and verified against rustc 1.99.0 with edition 2024, the compiler and edition the rest of this site uses.

Where the code lives

Step 2 starts the project at llm-apps/ in the root of the examples repository and tags it llm-ch02. Every step after that commits its end state and tags it (llm-ch03, llm-ch04, and so on), so git checkout llm-ch05 is a service that builds and runs. This article commits nothing; the first tag lands with step 2.

Next

Step 2 gets a model running on your own machine: the candle runtime, where the weights come from, and the memory arithmetic that decides which model fits. The track overview keeps the full list current as articles land.