Microservices Pattern

Shadow / Mirror Deployment

Copy real traffic to the new version and compare its answers - no user ever sees them.

deploy

Lesson

Shadow deployment: test on real traffic, risk no users

Send a copy of every real request to the new version, compare its answers, and throw them away. This lesson explains when that helps and the two rules that keep it safe - then you build a mirror that finds a bug no user sees.

The idea in short

Copy real traffic to the new version - and throw its answers away.

Shadow deployment (also called traffic mirroring or a dark launch) runs the new version beside the old one and sends it a COPY of real production requests. Users only ever get the old version's answers. The new version's answers are compared, measured, and thrown away.

It tests the new version with real inputs and real volume, while no user depends on it. Remember it as: copy, compare, discard.

At a glance

Users affectedNone - they never see the shadow's answers.

What it provesCorrect answers and speed on real traffic.

What it does not proveWhether users like the new version.

CostEvery request is processed twice.

Main dangerSide effects - payments, emails, writes - done twice.

An everyday picture: a trainee pilot

Same flight, same instruments - but the captain is flying.

A trainee sits beside the captain and makes every decision as if flying, but their controls are not connected. Afterwards, their decisions are compared with the captain's. The passengers were never in the trainee's hands, yet the trainee was tested on a real flight.

v2 is the trainee: it sees every real request and decides what it would answer, but only v1 is connected to the user.

Words you need

Four words used in this lesson.

You will see these words in proxy and service-mesh documentation.

Small dictionary

PrimaryThe current version. Its answer goes to the user.

ShadowThe new version. It gets a copy; its answer is only compared.

MirroringSending a copy of each request to the shadow.

Side effectAnything a request changes outside itself: a payment, an email, a database row, a message.

How it works, step by step

Use Next to walk through a real run.

Step through the diagram. It is a real run of the lab: 200 price lookups and 20 orders through a mirror, with v2 being a rewrite that should give identical answers.

request flowShadow traffic: v2 sees everything, users see only v1step 1 / 4

Users

price lookups and orders

Mirror

mirror.js on port 4000

v1 - primary

answers the user

v2 - shadow

answer compared, then dropped

React Flow

1 - Every request goes to both

The mirror sends each request to v1 and a copy to v2, marked with the header x-shadow: true. The user gets v1's answer.

user gets

v1's answer

v2 gets

a copy of every request

user latency

1.9 ms (direct: 1.6 ms)

risk to users

none

A real run: 200 price lookups through the mirror. v2 is a rewrite that must give the same answers.

The steps

1. DeployRun v2 beside v1, with no users.

2. MirrorCopy every request (or a sample) to v2, marked as a shadow request.

3. AnswerSend the user v1's answer - never wait for v2.

4. CompareRecord whether v2's answer and speed match v1's.

5. DecideFix what differs; when it matches, release v2 with another pattern.

Why teams use it - and what it costs

The safest way to test with real traffic.

No test environment has your real users' inputs. Shadowing gives the new version all of them, with no risk to users. In the lab, it found a wrong price - 40 of 200 answers differed, all for one course - and no user ever saw it.

It costs double processing, and it needs care: the shadow must not slow users down, and must not repeat side effects.

Benefits and costs

BenefitReal traffic, zero user impact.

BenefitFinds wrong answers and slow paths before release.

BenefitGood for load-testing with real request shapes.

CostEvery request is handled twice.

CostSide effects must be blocked.

CostSays nothing about user behaviour.

Detail 1: best for rewrites with the same answers

A difference should mean a bug.

Shadowing is simplest when the new version should answer exactly like the old one: a rewrite in a new language, a new database, a refactored pricing engine, a faster algorithm. Then every difference is a bug, and the comparison is just "same or not". The lab's v2 keeps prices in paise instead of rupees; one price was typed wrongly, and the shadow found it.

When the new version is meant to answer differently - a new discount rule, a new ML model - you must compare against the EXPECTED new answer, or compare statistics (for example, how often two models agree), not just equality.

Detail 2: rule one - never make users wait for the shadow

The copy goes in the background.

The user must get v1's answer as soon as v1 answers, whatever v2 is doing. In the lab, v2 was made 500 ms slower. With the copy sent in the background, users stayed at about 2 ms - the same as with no mirror (1.6 ms). When the mirror waited for v2 first, every user waited about 505 ms. If v2 crashes or hangs, the user must not notice either.

Detail 3: rule two - the shadow must not act

Payments, emails and writes happen twice unless you stop them.

A copied request is a real request. If it says "create an order" and v2 is connected to the real payment provider, the customer is charged twice. In the lab, 20 mirrored orders produced 20 extra charges. When v2 checked the x-shadow header and skipped the charge, it made 0.

Use several layers of protection: mark every copy (x-shadow: true) and skip side effects for marked requests; give the shadow its own database and queues; point it at test versions of payment, email and SMS providers; and start by mirroring only read requests (GET).

Watch out: Charging a customer, sending an email or publishing a message cannot be undone. Before you mirror any request that changes something, prove that the shadow cannot reach real payment, email and messaging systems.

In the real world

NGINX's mirror module and service meshes.

NGINX has a mirror directive that sends a copy of each request to another location; NGINX ignores the mirror's response. Service meshes such as Istio can mirror a percentage of traffic to another version; Istio describes mirrored traffic as "fire and forget" - responses from the mirror are discarded.

These configuration examples show the shape. They were not run in this lesson's lab, which uses a small Node.js mirror.

NGINX - send a copy of every request to the shadow

upstream primary { server 127.0.0.1:4001; }
upstream shadow  { server 127.0.0.1:4002; }

server {
  listen 80;

  location / {
    mirror /mirror;              # also send a copy here
    proxy_pass http://primary;   # the user gets this answer
  }

  location = /mirror {
    internal;
    proxy_pass http://shadow$request_uri;
    proxy_set_header X-Shadow "true";
  }
}

Istio - mirror 10% of traffic to v2

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: price
spec:
  hosts: [price]
  http:
    - route:
        - destination: { host: price, subset: v1 }
          weight: 100
      mirror: { host: price, subset: v2 }
      mirrorPercentage: { value: 10.0 }

When to use it, and when not

High confidence before risky changes.

Use it when wrong answers are expensive and the old version is a good reference.

Decide

Good fitRewrites, database migrations, performance work.

Good fitNew search, recommendation or ML models (compare their answers).

Poor fitMostly write traffic that is hard to isolate.

Poor fitDoubling the processing cost is too expensive.

Poor fitYou want to know whether users like it - use an A/B test.

Common mistakes

And how to avoid each one.

The first two can hurt real customers.

Mistake -> fix

Shadow connected to real payments or emailMark copies, skip side effects, use test providers (lab: 20 extra charges vs 0).

Waiting for the shadow before answeringSend the copy in the background (lab: 2 ms vs 505 ms).

Comparing when answers are meant to changeCompare with the expected new answer, or compare statistics.

Shadow logs mixed with real metricsLabel shadow traffic and keep its metrics separate.

Shadowing everything at onceStart with reads, or a sample, then grow.

Compared with the other patterns

Shadow serves nobody; the others serve someone.

Shadow is usually a step before another pattern, not a release on its own.

Shadow and its neighbours

ShadowReal traffic, no user gets the new answer.

CanaryA few real users get the new version.

Blue-greenEveryone gets the new version after the switch.

A/B testTwo groups of users, to compare business results.

Interview questions

Short answers you can give in your own words.

What is shadow deployment? Mirroring a copy of production traffic to a new version whose responses are compared and discarded, so it is tested on real traffic with no user impact.

What are its risks? Side effects executed twice - payments, emails, writes - so shadow traffic must be marked and isolated; and added latency if the proxy waits for the shadow, so mirroring must be asynchronous.

How is it different from canary? A canary serves real users with the new version. A shadow serves nobody; it only observes.

Remember

Copy real requests to the new version; users only get the old version's answers.
Best for rewrites where the answers should not change - a difference means a bug.
Never make users wait for the shadow (lab: 2 ms vs 505 ms).
Never let the shadow act: mark copies and skip side effects (lab: 20 extra charges vs 0).
Shadow proves correctness and speed, not whether users like it.
After shadowing, release with another pattern - canary or blue-green.

Check yourself

Answer in your head first, then open the answer.

Whose answer does the user get in a shadow deployment?

The old version's (the primary's).

In the lab, how was the Redux bug found without any user seeing it?

The mirror compared answers: 40 of 200 differed, all for Redux (799 vs 800).

What happened to user latency when the mirror waited for a slow shadow?

It rose from about 2 ms to about 505 ms.

Why must the shadow skip payments?

A copied order is a real order - the customer would be charged twice.

Shadow or A/B test: which tells you whether users prefer v2?

An A/B test. Shadowing never shows v2 to users.

Hands-on lab

Build it: a mirror that finds a bug no user sees

A price service in two versions and a mirror, in plain Node.js - no Docker, nothing to install. The mirror finds a bug in the rewrite using real requests, and you measure what a slow shadow and an unguarded shadow cost. Every output is from a real run (Node 22).

Set up the project

Nothing to install - only Node.js.

The lab has a price service that comes in two versions, a mirror on port 4000 that users talk to, and a test script. v1 is the current code. v2 is a rewrite that keeps prices in paise (1/100 of a rupee) - it should give exactly the same answers, but one price was typed wrongly.

Terminal

mkdir deploy-lab && cd deploy-lab
npm init -y
npm pkg set type=module
# no packages to install - only Node.js itself

The service and the mirror

Two versions of the service; a mirror that copies every request.

price-service.js answers GET /price?course=... and takes orders with POST /orders - each order "charges" the customer (a counter). It can be made slow (SLOW_MS) and can be told to skip side effects for shadow traffic (GUARD_SHADOW=1).

mirror.js sends each request to v1 and a copy, marked x-shadow: true, to v2. The user gets v1's answer. When both answers are in, it compares them in the background and keeps statistics at /admin/shadow. With AWAIT_SHADOW=1 it does the wrong thing: it waits for v2 before answering the user.

price-service.js

// price-service.js - prices courses and takes orders.
// v1: the current code.  v2: a rewrite that must give the SAME answers (it has one bug).
// Run:  VERSION=v1 PORT=4001 node price-service.js
import http from "node:http";

const VERSION = process.env.VERSION ?? "v1";
const PORT = Number(process.env.PORT ?? 4001);
const SLOW_MS = Number(process.env.SLOW_MS ?? 0);                  // make this version slow
const GUARD_SHADOW = process.env.GUARD_SHADOW === "1";             // skip side effects for shadow traffic

const BASE = { python: 499, redux: 799, docker: 599, sql: 399, aws: 999 };
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
let charges = 0;                                                   // payments really taken

function price(course) {
  if (VERSION === "v1") return BASE[course];
  // v2: the rewrite - prices now kept in paise (1/100 rupee) and converted back
  const paise = { python: 49900, redux: 79950, docker: 59900, sql: 39900, aws: 99900 };
  return Math.round(paise[course] / 100);                          // the bug: redux was typed as 79950
}

const server = http.createServer(async (req, res) => {
  const url = new URL(req.url, "http://x");
  await wait(SLOW_MS);
  const isShadow = req.headers["x-shadow"] === "true";

  if (url.pathname === "/price") {
    const course = url.searchParams.get("course");
    return res.end(JSON.stringify({ course, price: price(course) }));
  }
  if (url.pathname === "/orders" && req.method === "POST") {
    if (!(GUARD_SHADOW && isShadow)) charges++;                   // the side effect: charge the customer
    return res.end(JSON.stringify({ ok: true, charged: !(GUARD_SHADOW && isShadow) }));
  }
  if (url.pathname === "/admin/stats") return res.end(JSON.stringify({ version: VERSION, charges }));
  res.writeHead(404);
  res.end("{}");
});

server.listen(PORT, () => console.log(VERSION, "price-service on port", PORT));

mirror.js

// mirror.js - port 4000. Users get v1's answer; a copy of every request also goes to v2.
// Run:  node mirror.js            (copy is sent in the background - users never wait for v2)
//       AWAIT_SHADOW=1 node mirror.js   (the wrong way: wait for v2 before answering)
import http from "node:http";

const PRIMARY = "http://localhost:4001";   // v1 - its answer goes to the user
const SHADOW = "http://localhost:4002";    // v2 - its answer is only compared, never returned
const AWAIT_SHADOW = process.env.AWAIT_SHADOW === "1";
const stats = { compared: 0, same: 0, different: 0, examples: [] };

function send(base, req, body, extraHeaders = {}) {
  return new Promise((resolve) => {
    const r = http.request(base + req.url, { method: req.method, headers: { ...req.headers, ...extraHeaders } }, (answer) => {
      let text = "";
      answer.on("data", (chunk) => (text += chunk));
      answer.on("end", () => resolve({ status: answer.statusCode, text }));
    });
    r.on("error", () => resolve({ status: 0, text: "" }));
    r.end(body);
  });
}

const server = http.createServer(async (req, res) => {
  if (req.url === "/admin/shadow") return res.end(JSON.stringify(stats));
  let body = "";
  for await (const chunk of req) body += chunk;

  const primary = send(PRIMARY, req, body);
  const shadow = send(SHADOW, req, body, { "x-shadow": "true" });   // mark the copy

  // compare in the background when both answers are in
  Promise.all([primary, shadow]).then(([p, s]) => {
    if (req.method !== "GET") return;                              // only compare reads
    stats.compared++;
    if (p.text === s.text) stats.same++;
    else {
      stats.different++;
      if (stats.examples.length < 3 && !stats.examples.some((e) => e.url === req.url)) {
        stats.examples.push({ url: req.url, v1: p.text, v2: s.text });       // one example per URL
      }
    }
  });

  if (AWAIT_SHADOW) await shadow;                                  // the wrong way
  const answer = await primary;
  res.writeHead(answer.status, { "content-type": "application/json" });
  res.end(answer.text);
});

server.listen(4000, () => console.log("mirror on http://localhost:4000", AWAIT_SHADOW ? "(waits for shadow)" : ""));

shadow-test.js

// shadow-test.js - 200 price lookups and 20 orders through the mirror, then the results
const COURSES = ["python", "redux", "docker", "sql", "aws"];
const times = [];

for (let i = 0; i < 200; i++) {
  const t = Date.now();
  await fetch(`http://localhost:4000/price?course=${COURSES[i % COURSES.length]}`);
  times.push(Date.now() - t);
}
for (let i = 0; i < 20; i++) await fetch("http://localhost:4000/orders", { method: "POST", body: "{}" });
await new Promise((resolve) => setTimeout(resolve, 1500));        // let the last shadow calls finish

const average = times.reduce((a, b) => a + b, 0) / times.length;
console.log(`users: 200 price lookups, average ${average.toFixed(1)} ms, slowest ${Math.max(...times)} ms`);
const shadow = await (await fetch("http://localhost:4000/admin/shadow")).json();
console.log(`compared ${shadow.compared}: ${shadow.same} same, ${shadow.different} different`);
for (const e of shadow.examples) console.log("  e.g.", e.url, "v1", e.v1, "v2", e.v2);
const v1 = await (await fetch("http://localhost:4001/admin/stats")).json();
const v2 = await (await fetch("http://localhost:4002/admin/stats")).json();
console.log(`20 orders: v1 charged ${v1.charges} times, v2 (shadow) charged ${v2.charges} times`);

Find the bug with real traffic

40 of 200 answers differ - and no user saw them.

Start both versions and the mirror, then run the test: 200 price lookups and 20 orders. Users got their answers in about 2 ms. The mirror compared 200 answers: 160 the same, 40 different - every Redux lookup, 799 from v1 against 800 from v2.

Terminals

VERSION=v1 PORT=4001 node price-service.js     # terminal 1
VERSION=v2 PORT=4002 node price-service.js     # terminal 2
node mirror.js                                 # terminal 3
node shadow-test.js                            # terminal 4

Output - measured

users: 200 price lookups, average 1.9 ms, slowest 35 ms
compared 200: 160 same, 40 different
  e.g. /price?course=redux v1 {"course":"redux","price":799} v2 {"course":"redux","price":800}
20 orders: v1 charged 20 times, v2 (shadow) charged 20 times     <- see step 5

A slow shadow, sent the right and the wrong way

2 ms for users, or 505 ms.

Restart v2 with SLOW_MS=500. With the normal mirror, users stayed at 2.0 ms on average. Restart the mirror with AWAIT_SHADOW=1 and run again: every lookup took about 505 ms, because the mirror waited for the slow shadow before answering. For reference, going straight to v1 with no mirror took 1.6 ms.

Terminals 2 and 3

VERSION=v2 PORT=4002 SLOW_MS=500 node price-service.js   # terminal 2
node mirror.js                                           # terminal 3 - then run the test
AWAIT_SHADOW=1 node mirror.js                            # terminal 3 - the wrong way

Output - measured

copy sent in the background:
  users: 200 price lookups, average 2.0 ms, slowest 35 ms

mirror waits for the shadow:
  users: 200 price lookups, average 505.3 ms, slowest 547 ms

Stop the shadow from charging customers

20 extra charges, then 0.

Look again at step 3's last line: 20 orders, and the shadow charged 20 times too. In a real system that is 20 customers charged twice. Restart v2 with GUARD_SHADOW=1: it now checks the x-shadow header and skips the charge for copied requests. The shadow charged 0 times, and the comparisons still worked.

Terminal 2

VERSION=v2 PORT=4002 GUARD_SHADOW=1 node price-service.js

Output - measured

compared 200: 160 same, 40 different
20 orders: v1 charged 20 times, v2 (shadow) charged 0 times

Shadow deployment, measured

Comparing answers40 of 200 differed - the Redux bug - and 0 users saw it.

Slow shadow, background copyUsers ~2 ms, the same as without a mirror.

Slow shadow, mirror waitsUsers ~505 ms.

Orders without a guard20 extra charges.

Orders with the x-shadow guard0 extra charges.

Practice on your own

1.
Fix the Redux price in v2 and run shadow-test.js again. What should the comparison show now?
Hint
200 same, 0 different.
2.
Make v2 crash on one course (throw an error for "sql"). What do users see? What does the mirror record?
Hint
Users only get v1's answer; the mirror sees status 500 or no answer from v2.
3.
Change mirror.js to mirror only 10% of requests. Why would a team do that?
Hint
Shadowing doubles the work - sampling keeps the cost down.
4.
Make the mirror also record how much slower or faster v2 is than v1 on average.
Hint
Time both requests; keep a running total of the difference.

Comments

Loading comments...

Shadow / Mirror Deployment

Shadow deployment: test on real traffic, risk no users

The idea in short

An everyday picture: a trainee pilot

Words you need

How it works, step by step

1 - Every request goes to both

Why teams use it - and what it costs

Detail 1: best for rewrites with the same answers

Detail 2: rule one - never make users wait for the shadow

Detail 3: rule two - the shadow must not act

In the real world

When to use it, and when not

Common mistakes

Compared with the other patterns

Interview questions

Remember

Check yourself

Build it: a mirror that finds a bug no user sees

Set up the project

The service and the mirror

Find the bug with real traffic

A slow shadow, sent the right and the wrong way

Stop the shadow from charging customers

Practice on your own

Comments

AI

System Design

Backend

Frontend

Fundamentals