Passing Data Between Microservices

01The problem statement

Our product has a feature that turns an uploaded PDF into a browsable wiki. Under the hood it's a small distributed system with two services. A Python API service handles the request: it downloads the PDF from Supabase storage, parses it into Markdown, and creates a wiki job. A Go worker on Cloud Run does the generation. It takes the parsed Markdown and builds the wiki. Two languages, two runtimes, and one handoff in the middle.

That handoff was the weak point. We passed the parsed Markdown to the worker through an environment variable. It was the simplest thing that could possibly work, and it did work. Every local run, every staging deploy, every demo went through fine. Then a real user uploaded a document that parsed to roughly 47 KB of Markdown. Cloud Run caps environment variables at about 32 KB, so the worker failed before it executed a single line of our code.

Two services need to share a payload whose size we don't control, and we were moving it over a channel with a hard cap.

— the problem, in one sentence

Once you write the problem down like that, the incident stops being about one oversized PDF. Any input larger than the limit was going to fail, on any day, for any user. The 47 KB document didn't break the system. It exposed a system that was already broken for a whole class of inputs.

02Why testing never caught it

Every document we used during development parsed to well under 32 KB. Not because we chose them carefully. Small, convenient fixtures are simply what you reach for when you're iterating fast. Production data has a long tail, and boundaries live in the tail. The failure was never hiding; we just never pushed an input past a limit we didn't know we had.

The uncomfortable lesson: "works in testing" is a statement about your fixtures, not about your system. If a feature accepts user-sized input, at least one test should cross the size you're silently assuming is the maximum. A boundary that isn't in your test data is a boundary you'll discover in production.

03What we were actually solving

Before comparing solutions, it's worth being precise about why the original design was wrong. Environment variables are a configuration channel. They exist to carry API keys, database URLs, Redis hosts, and feature flags: small, static values that describe how a process should run. We used that channel as a data plane, and inherited a platform constraint that had nothing to do with our data.

There's a second, quieter issue I only appreciated while fixing the first one: environment variables are visible. They show up in deploy descriptions, cloud consoles, and crash tooling. Routing user document content through service configuration isn't just a size problem. It puts customer data in places that were built for operators, not payloads.

04The architecture we started with

With the requirements written down, I mapped out the realistic ways to move the Markdown from Python to Go. There are four worth taking seriously. All of them work, and that's exactly what makes this an engineering decision rather than a puzzle with one correct answer.

05Way 1: Redis as a claim check

The Python service stores the Markdown in Redis under a unique key and passes only the job ID to the worker. The worker starts, asks Redis for the payload, and gets on with its actual job. When processing finishes, the key is deleted, or it simply expires via a TTL, which doubles as garbage collection for jobs that crash midway.

06Way 2: Object storage

Same claim-check shape, different store. The service uploads the Markdown as a file to a GCS bucket or Supabase storage and hands the worker a path. This is the right call when payloads get genuinely large. Whether the file is 50 KB or 100 MB, the architecture stays identical. Files also persist until you delete them, so a failed job can be retried hours later, and an engineer can download the exact artifact a job saw and debug with it.

07Way 3: The database

Store the Markdown in a table keyed by job ID; the worker queries it on start. You get persistence, easy retries, and an audit trail of every job essentially for free. The catch is that you're now using your relational database as a scratch pad. Large text blobs bloat tables, indexes, and backups, and reads are slower than Redis. It's the right choice when job history and auditability are product requirements, and a lazy default when they're not.

08Way 4: The worker owns the whole pipeline

The structural alternative: stop sharing parsed content entirely. Send the worker the PDF's location and let it download and parse the document itself. The worker becomes fully self-contained, which is genuinely attractive. There's no shared storage and no handoff protocol at all. But the worker now needs Supabase credentials, a storage SDK, and a PDF parser. The Docker image grows, and the parsing logic either lives in two languages or migrates to Go. You haven't removed coupling; you've traded a data dependency for a dependency pile inside the worker. It's worth it only if the worker is deliberately meant to own the complete workflow.

09Why we went with Redis

The decision fell out of the requirements, not out of preference. Our data is temporary; the Markdown is worthless minutes after the wiki is generated. The worker consumes it within seconds, on the hot path of a user-facing request. The payloads are tens of kilobytes, not megabytes. That profile of short-lived, consumed-immediately, latency-sensitive data is exactly the shape Redis is built for.

There was also a practical reason that doesn't show up in architecture diagrams: we were already running Redis. Other microservices in our system use it for queues and caching, so the infrastructure, the monitoring, and the operational muscle memory were already in place. Adding a few claim-check keys cost us nothing new. It also lined up with where the system is heading: we had already planned to use Redis as a cache for vector embeddings in our AI features. Choosing it here was an investment in a tool we knew we'd lean on more, not a one-off dependency.

It also asked the least of the codebase. The Python service swaps an env-var write for a SET with a TTL; the worker swaps a read for a GET. The PDF is still parsed exactly once, the worker stays a pure wiki generator, and the transport between services now carries a few dozen bytes no matter what a user uploads. As a bonus, user document content no longer sits in service configuration where consoles and deploy logs can see it.

The strongest runner-up was object storage, and it's worth saying why it lost: durability. It offers persistence we don't need, at the price of latency we would actually feel. If our payloads grow from kilobytes to megabytes, say full textbooks instead of documents, the balance flips and object storage becomes the right answer. That boundary is written down in our docs, so future-us knows when to switch.

10Conclusion: engineering is choosing, not solving

Any of these four ways would have closed this incident. That's the real takeaway. Engineering decisions are rarely about discovering the one correct answer. They're about how your product actually works: how long its data lives, how fast it has to move, what happens when a step dies halfway, and what your team can operate confidently. Different designs reach the same result. The engineer's job is to understand the trade-offs and choose what fits their system, deliberately.

For this system, at this scale, with this data lifetime, and with Redis already humming in our stack, the answer was Redis. For a batch pipeline crunching gigabyte archives, the same reasoning would land on object storage. Neither answer is more correct in the abstract; each is correct against its own constraints. The decision isn't permanent either. The discipline of making it explicitly, with the alternatives written down, is the part that compounds.

Ways can differ and still arrive at the same result. Choosing between them, deliberately and against your own constraints, is the actual job.

11What I took away