Every PR Gets Its Own World: Building Pavo’s Infrastructure for SDLC 2.0

An animated per-PR routing diagram showing a request moving through forked services while unchanged services come from the shared baseline.

From faster coding to faster shipping—Pavo’s parallel validation layer lets humans and agents develop, test, and ship changes concurrently across a shared distributed system.

TL;DR

Our shared staging environment had become a queue. A developer could either wait for the current test to finish or clone the entire stack and pay roughly five times as much for infrastructure.

We replaced those full-stack clones with request-routed per-PR environments. Each sandbox runs only the services changed by a pull request and borrows everything else from one shared baseline. The monthly staging bill returned from roughly 5X to ~X, while a sandbox becomes ready in about 100 seconds after its images are built.

The routing layer was the easy part. The real engineering work was making the routing key survive every HTTP call, queue, and thread handoff—and doing so without asking every service team to reinvent propagation.

5X → ~X
staging infrastructure cost: ~100s
sandbox startup: 50
concurrent sandboxes: 3 vs 12
deployments for three changes

X denotes the monthly cost of one full environment. Total PR-to-ready time is usually around ten minutes because image builds remain the dominant step.

Staging had become a queue

At 7:45 one evening, a teammate asked when staging would be free. They needed to load-test a GitHub connector, but another test already occupied the environment. The answer was: after midnight.

That was not unusual. We had one shared staging environment, which meant one branch could be validated at a time. To create some parallelism, we began cloning the entire stack into fresh GCP projects. Five or six copies relieved the queue, but they also multiplied our infrastructure bill.

We were left with two poor choices: queue behind the shared environment, or duplicate every service and datastore for every active test. Coding agents made the problem more urgent by increasing the rate at which changes reached validation.

The idea we borrowed from Uber

Uber's SLATE architecture describes short-lived test environments selected by a routing key. Instead of reproducing the complete system for each pull request, it runs the changed service and routes everything else to an already-running baseline.

A comparison of request routing and infrastructure cloning for three pull requests across services A, B, C, and a database. — FIG. 01 — With three services and a database, three independent changes require twelve deployments when every environment is cloned. Routing needs only three forks and borrows the rest from one shared baseline.

Clone everything and the cost scales with the size of your system. Route requests and it scales with the size of your change.

We bought the request-routing layer from Signadot and built the surrounding application work ourselves: context propagation, queue handling, lifecycle automation, and observability. The same model could be assembled on Envoy or Istio; the important architectural choice is routing rather than cloning.

What a developer sees

A developer opens a pull request. CI identifies the services touched by the change and builds only those Docker images. When the build finishes, the sandbox is typically ready in about 100 seconds. End to end, the wait is closer to ten minutes because building the images still takes most of the time.

A pull request comment showing that a sandbox was created, the services forked, and the routing key for the environment. — FIG. 02 — The PR bot reports the sandbox, its forked services, and the routing key used to enter it.

Only the changed services receive forked pods. Databases, object storage, and unchanged services continue to come from the shared baseline. Requests carrying the PR's routing key pass through the forks; a request without the key follows the baseline as usual.

An animated request path crossing forked and baseline services based on a per-PR routing key. — FIG. 03 — One request crosses both worlds: forked services where the PR changed code, and baseline services everywhere else.

New commits update the sandbox, and an idle or closed pull request removes it. Cross-service work also stays simple: branches with the same name join the same routing group, so related changes in different repositories can be tested as one environment.

Context propagation was the real work

A request-routed sandbox can fail silently. If one service drops the routing header, the request falls back to baseline, succeeds on the wrong pods, and gives the developer a convincing but false result. Our central guarantee therefore became: the routing key must survive every hop.

We already had a shared platform SDK that acted as the network door for our services. We extended it to carry the routing key as OpenTelemetry baggage across HTTP, Google Pub/Sub, and thread handoffs. Application services do not need to read or write the key themselves.

Centralize the routing key in one SDK, or you'll re-litigate it in every service.

This propagation work took most of the implementation effort, but it produced a second benefit. The same instrumentation gave us distributed traces across both synchronous calls and queues. When a key disappears, a trace shows exactly which boundary dropped it.

A Grafana Tempo trace following one routed request across five forked and baseline services. — FIG. 04 — The same routed request in Grafana Tempo, visible as one trace across five services.

Async is where routing schemes go to die

Sidecars can route HTTP, but they cannot see a message after it enters a queue. If baseline and sandbox workers consume the same subscription, they compete for work with no guarantee that the fork receives the message generated by its request.

Sidecars route HTTP. They can't see your queues, so async is a separate problem you have to solve yourself.

For sandbox requests, publishers attach the routing key as a message attribute. We create a per-sandbox subscription on the existing Pub/Sub topic and use a broker-side filter so only matching messages reach the forked worker. One shared topic can therefore serve every sandbox without creating shadow topics.

The baseline subscription is deliberately unfiltered, so it still sees tagged messages. Before processing one, the baseline worker asks Signadot's RouteServer whether the routing key belongs to an active sandbox. If it does, baseline skips the message and lets the fork own it.

A Pub/Sub routing diagram with one filtered sandbox subscription and an unfiltered baseline subscription that checks the route server before processing. — FIG. 05 — Pub/Sub filters tagged messages into the sandbox; the baseline worker checks the route server and skips work owned by a live fork.

Kafka needs a different implementation because the broker does not provide equivalent attribute filters. A fan-out consumer must perform the selection itself. Signadot describes that pattern in Using Sandboxes in Kafka-based Asynchronous Applications.

Forks share the baseline's Postgres, Redis, Elasticsearch, and object storage. That is the source of the cost advantage, but it also means one pull request can read or write data used by another. We do not run schema migrations inside a sandbox; migrations happen only after merge.

We considered per-PR schemas and isolated datastores, but deliberately left them out. For our workload, request-level isolation produced most of the value without rebuilding the expensive part of every environment.

One exception exposed the difference between stateless compute and persistent infrastructure. Our GitHub connector used a single-writer persistent disk, which could not attach to a fork while baseline held it. The fork now uses ephemeral storage and re-clones repositories. Initial synchronization is slower, but the sandbox remains disposable and reliable.

Keeping the bill flat

Request routing only controls unit cost. Lifecycle automation keeps abandoned environments from accumulating. Each sandbox has a 12-hour idle TTL that resets on every push. A new commit can recreate a sandbox that has already expired.

We cap the fleet at 50 concurrent sandboxes. If the cap is reached, the least-recently-updated sandbox is evicted first. The result is a predictable ceiling rather than a staging estate that grows with every open pull request.

A cost comparison showing five full cloned environments reduced to one shared baseline with lightweight forks. — FIG. 06 — One baseline plus lightweight forks brought staging from roughly 5X back to ~X while expanding capacity from about five environments to fifty sandboxes.

Isolate at the request level, not the infrastructure level.

The larger lesson was not that every team should adopt the same vendor or topology. It was that the economics of test environments change when isolation follows a request instead of a cluster. The routing key, propagation path, queue semantics, and lifecycle policy then become first-class parts of the platform—not details left to each service.