Split the Pipeline by Workload Shape

Our knowledge base product runs on two languages. Ingestion is a Go binary that wakes up, chews through a batch of documents, and exits. Question answering is a Python service that never sleeps. Every time someone new sees that on a diagram they ask the same question, and it is the wrong question. They ask why two languages. The interesting split is not the language. It is that ingestion and question answering are two genuinely different kinds of work wearing the same product, and the language difference is mostly an accident of history that I will own up to before the end.

01One product, two workloads

The product is easy to describe in two sentences. You upload documents, and it builds a wiki out of them: entity pages, concept pages, and summaries, each carrying inline citations back to the exact source chunk it came from. Then you ask it questions, and it answers using only what it ingested, citing as it goes.

Two sentences, two completely different machines. I did not appreciate how different until I stopped looking at what they do and started looking at where their time and their failures actually go.

Ingestion is wide and shallow. Query is narrow and deep. Almost every infrastructure decision downstream falls out of that one asymmetry.

02The pattern is older than the product

What we built is not novel, and that is a compliment to it. It is the async worker pattern: a synchronous API accepts work, validates it, hands it to a queue, and returns a receipt. Workers drain the queue out of band. Every mature stack has a version of this. Rails grew Sidekiq, Python grew Celery, and if you are on a cloud you already have a managed queue and a batch runtime sitting there waiting for you.

Concretely: the Python service takes the upload, parses the file or URL down to markdown, and then stops. It does not chunk, it does not embed, it does not synthesize anything. It publishes a job and hands back a job id. The job travels through a queue, lands on an authenticated dispatch endpoint, and triggers a batch job. The Go worker starts, reads its instructions out of environment variables, does the heavy work, writes to Postgres, and exits.

The one rule that makes any of this work is that the seam is a queue and not a function call. Everything good in this post falls out of that. So does everything painful.

03What the split actually buys you

These are the real wins, and they are worth stating precisely, because microservices are good is not an argument.

That last number is not a typo, and it is the single best piece of evidence in this post. Every question in production is served by one worker, because the query path spends its life waiting on network calls rather than burning CPU. If that path were throughput bound, one worker would be an outage. It is not throughput bound, so it is not an outage.

04What the split actually costs you

Now the part architecture posts skip. Every item below is something we are actively living with, not a hypothetical.

05Why Go, honestly

I promised to own up to the language, so here it is. Go owns ingestion because the engine we forked was already written in Go. We started from an existing open source retrieval engine, kept its chunkers, its retriever, and its embedding layer, and bolted on a job mode entrypoint, our own page synthesis passes, and a different database backend. Nobody sat in a room and weighed the merits.

What ingestion actually needs is unglamorous. A batch is five documents. Each gets chunked, every chunk gets an embedding call, then two model passes decide which entities and concepts each chunk supports, and a reduce phase runs one page generation call per page touched. That is several hundred outbound HTTP calls, and essentially the whole job is spent waiting on them. What you need is to hold many slow network calls in flight at once and put a hard ceiling on exactly how many. Go makes that ceiling a two line concern.

// the map phase: documents in parallel, but never more than N
eg, ctx := errgroup.WithContext(ctx)
eg.SetLimit(mapParallel) // 10

for _, op := range pendingOps { // a batch is 5 documents
    op := op
    eg.Go(func() error {
        // chunk, embed every chunk, then two model passes
        return s.mapOneDocument(ctx, op)
    })
}
if err := eg.Wait(); err != nil {
    return err
}

There are four such limiters in the ingest path and their numbers are the entire performance story: five documents to a batch, ten parallel page generations in the reduce phase, four parallel citation batches inside a document, five embedding calls in flight. It is genuinely pleasant. Bounded concurrency in Go is two lines and it is hard to get wrong.

And it is also not load bearing, which is where most polyglot posts quietly start lying. Ingestion is not CPU bound. The chunker does not contain a single goroutine; it is plain sequential string processing. The job is nothing but waiting on HTTP. Python's asyncio with a semaphore would have handled this workload perfectly well, and the honest counterfactual is that if we had started from an empty directory, both halves of this system would be Python and I would have one fewer toolchain to patch on a Friday.

So the polyglot part is not a triumph. It is a cost we accepted, because the alternative was rewriting a working retrieval engine from scratch and that was obviously the worse trade. Inheriting a good codebase in a second language is a fine reason to have a second language. Wanting a second language is not a reason at all.

06Why Python kept the question

The query side, by contrast, I will defend on the merits. It never had a throughput problem to solve. It has a branching problem. A question arrives, a small fast model sorts it into one of six types, and each type gets a genuinely different retrieval strategy.

Six strategies, a router in front of them, prompt changes that depend on the type, and a fallback ladder under every single step. That is orchestration surface area, not throughput. What you want from a language there is not concurrency primitives. You want to constrain what the model is allowed to say, and to degrade gracefully when it says something else anyway.

So the decisions are structured and the answer is not.

class SubQuestions(BaseModel):
    sub_questions: list[str] = Field(min_length=1, max_length=3)

# the "at most 3" is a decoding constraint, not a polite request in a prompt
model = ChatModel(model=CLASSIFIER_MODEL, temperature=0.0)
result = model.with_structured_output(SubQuestions).invoke(prompt)

# ...but the answer call is deliberately unstructured: free prose,
# with inline [c1] labels the frontend resolves back to source chunks.
answer = chat_model.invoke(grounded_prompt)

The router cannot invent a seventh query type, because the enum will not let it. The decomposer cannot return nine sub questions, because the schema caps it at three. Those invariants live in the type system instead of in prompt text and hope. Meanwhile the call that actually writes prose for a human is left completely free, because the last thing you want to do to an answer is force it into a shape.

Structured for decisions, prose for the answer. If you take one implementation detail from this post, take that one.

07So should you do this

Here is the checklist I would actually use now, having lived with the thing.

08The rule

The split by workload shape is the part I would do again tomorrow, and it is the part worth copying. Ingestion is a wide, bounded fan out over slow network calls that nobody is watching. Query is a narrow, deep decision tree in front of a person who is waiting. Those want different runtimes, different timeouts, different failure semantics, and different error budgets. Putting them in one process makes every one of those decisions a compromise.

The split by language is the part I would not go looking for. We have it because we inherited a working engine, and that was a good trade. It is not a design achievement. It is a bill we agreed to pay, and we pay it every time we patch a dependency twice.

And the thing actually holding the system together is neither of those. It is that the seam is a queue and a schema rather than a shared codebase. That is also exactly why the seam is where our bugs live. You do not get the isolation for free. You buy it, and the currency is contracts nobody will enforce for you.