Follow a single request through both credentials, defend the check that looks redundant, measure what short-lived identity actually costs, and see where you get all of this in production without writing it.
Learning objectives
Trace one request through both proofs and name the step that fuses them.
Justify keeping a control that appears redundant today.
State the availability cost that short-lived credentials introduce.
Map each part of this system to the production technology that provides it.
Follow one request all the way
The Go service wants GET /orders/1001 on the TypeScript service, on behalf of .
user-8f3a
1. Get an identity — once at startup, then every twenty seconds:
stage 1
Go service → workload API over a unix socket, sending nothing
← certificate for spiffe://demo.local/service/go, valid 60s
2. Get a token — cached for about fifty seconds:
stage 2
Go service → auth server, over mTLS with that certificate
"swap sess-go-8f3a for a token aimed at .../typescript"
← JWT: sub=user-8f3a, act=.../go, aud=.../typescript, exp=60s
3. Make the call:
stage 3
Go service → TypeScript service, mTLS + Authorization: Bearer <token>
4. The receiver checks both, and checks that they agree:
stage 4
layer 1: certificate chain valid? yes
exactly one URI SAN? yes
SPIFFE ID is .../go, which we allow? yes ← the caller
layer 2: alg, signature, iss, exp, claim shape yes
aud == our own SPIFFE ID? yes
act.sub == the caller from layer 1? yes ← the fusion
authz: token scope includes orders:read? yes
order 1001 belongs to sub user-8f3a? yes ← the decision
The fusion line is the point of the whole identity story: two independent proofs, and a check that they describe the same caller.
The two lines below it are the point of the whole system. Identity gets you to the question. It never answers it.
What each layer alone would allow
Certificate only: a genuine service fetching the wrong user's data. Token only: a stolen token replayed from anywhere. Both, unfused: a token minted for one service, replayed by another. Only the comparison closes all three.
Why keep a check that looks redundant
You may have noticed that check 6 looks pointless now. The TypeScript service already refuses anyone who is not spiffe://demo.local/service/go at the TLS layer. So why compare the actor again?
Keep it. This is defense in depth: two independent controls, so that one mistake does not become a breach.
And realistically, allow lists widen. Today one service may call you. In a year, five may. The moment that list has two entries, check 6 stops being redundant. It starts stopping a real attack: a token minted for service A, replayed by service B, with both on the allow list.
A rule worth carrying
Security controls that look unnecessary today are often the ones that save you after a configuration change nobody thought about. The cost of keeping this one is a single string comparison.
A real difference between Go and Node
Both languages block a caller that is not allowed. They block it at different layers, and it is worth knowing which is which.
Point one service's client at the wrong peer identity and watch what comes back:
Target
Result
TypeScript service
403 Forbidden {"error":"forbidden"}
Go service
tls: ... alert bad certificate — the connection never completes
The Go server's own log says why:
go output
http: TLS handshake error from 127.0.0.1:49586: peer is [spiffe://demo.local/service/typescript], expected one of [spiffe://demo.local/service/auth]
Go checks the SPIFFE ID inside VerifyPeerCertificate, during the handshake. The HTTP request never exists.
Node validates the chain during the handshake, then checks the SPIFFE ID in the request handler, so it replies 403.
Both are safe: the request never reaches business logic either way. Go's is stricter, because Node has already parsed an HTTP request before saying no. It is worth knowing when somebody asks why the two logs look different.
Things to try
Watch a rotation. Wait twenty seconds. Every process logs rotated svid while traffic keeps flowing — and remember from Lesson 13 that this only demonstrates rotation if connection reuse is off. With pooling on, the traffic you are watching may still be riding the original handshake.
Restart the auth server. Its signing key was in memory, so it is gone, and a new one takes its place under a new kid. Watch what happens next, and be careful about what you conclude.
New tokens verify fine: the services see an unrecognized kid, refetch /jwks, and carry on. But tokens minted before the restart are now unverifiable by anyone who had not already cached the old key, because that key no longer exists anywhere. Whether a given service notices depends on whether it happened to cache first. The window is short — one token lifetime — but the behaviour is inconsistent, and inconsistent is the hardest kind of failure to diagnose.
That is loss, not rotation. Real rotation keeps the old key and publishes it alongside the new one until every token it signed has expired, which is why Lesson 9 built jwks to return a list. Production auth servers hold signing keys in a durable store or a key management service precisely so a restart is not a key change.
Kill the workload API. Everything keeps running for up to a minute on already-issued certificates, logging rotation failed, keeping old svid. Then calls start failing.
Bring it back and watch carefully, because "it heals" is too generous. The agent creates its CA in memory at startup, so a restart produces a new trust root. Each workload refreshes on its own timer, so between the first refresh and the last one, some hold certificates from the old root and some from the new — and neither trusts the other.
You may not see this by accident
If you started every process at once, their twenty-second timers are in lockstep and they all move to the new root together, so the window is close to zero and nothing appears to break. To observe it, start the TypeScript service about ten seconds after the Go service, then restart the agent. Real fleets are always skewed; your laptop is only accidentally synchronised.
With the timers skewed, the failure is unmistakable, and it goes both ways at once:
go service
12:49:47 rotated svid, now valid until 12:50:47
12:49:57 --> ts unreachable: untrusted certificate: x509: certificate signed
by unknown authority ... "workload-api-ca"
typescript service
12:49:58 --> go unreachable: Error: unable to verify the first certificate
Both messages say the same thing in two dialects: the peer presented a certificate from a root I do not have. Traffic resumes only once the last workload has picked up the new bundle.
That gap is avoidable, and real SPIRE avoids it with the same overlap rule you met for signing keys, applied to trust instead of signatures. The root is durable rather than per-process. Rotation happens at an intermediate CA. The trust bundle carries the old and new anchors together throughout the changeover; only then does issuance switch; and only after that is the old anchor removed.
The pattern, one more time
Every credential rotation in a distributed system is a three-phase move: publish both, switch, retire. Skip the overlap and you convert a routine rotation into an outage whose length depends on how skewed your refresh timers happen to be.
The trade-off, honestly
That last experiment is not a bug. It is the price.
Short-lived credentials make the issuer critical infrastructure. If it is down for longer than a certificate lifetime, everything stops.
That is why real SPIRE runs an agent on every node rather than one central service everyone dials. A node's agent failing takes down that node, not the fleet.
You are trading "a leaked credential lasts a decade" for "the issuer must be highly available". That is almost always worth it, because the first failure is silent and the second one is loud. But know that you are making the trade, and size the issuer accordingly.
What this project is not
workload-api/ is not SPIRE. It is about 150 lines doing the same four things so that you can read all of it in one sitting. Real SPIRE adds:
two components — a central server holding the CA, plus an agent on every node;
node attestation — the agent proves which machine it is on using a cloud instance identity document or a TPM;
a gRPC API with streaming, so rotation is pushed rather than polled;
selector sets — several selectors that must all match, instead of one substring;
a rotating CA, with trust bundles carrying old and new keys during the overlap;
federation between trust domains, so two organizations can verify each other.
Your application code barely changes across that gap. You still fetch an SVID and still verify peers by SPIFFE ID.
Some other honest limits of what you built:
Concern
Status here
Authorization
Scope granted at issuance and enforced per route, plus per-record ownership. Deliberately small: no roles, no hierarchy, no delegation chains
Revoking a user session before its token expires
Not possible; the token is valid for its full minute
Audit trail
Log lines only
The browser-to-BFF hop
Entirely out of scope
Attestation strength
One command-line substring, deliberately weak
Signing-key durability
In memory, so a restart makes outstanding tokens unverifiable
Trust-root rotation
No overlap, so an agent restart partitions the mesh until every workload refreshes
Naming those gaps is part of the work. A design you cannot criticize is a design you do not understand.
This is not a toy architecture
It is worth knowing that the shape you just built is the shape the largest production systems converged on independently.
Google publishes its internal model as BeyondProd. Strip the product names away and the correspondence is close to exact:
What you built
What Google calls it
Mutual TLS between every pair of services
ALTS — "a mutual authentication and transport encryption system for services in Google infrastructure"
A SPIFFE ID that names a service, not a host
Identities "bound to services instead of to a specific server name or host"
The workload API attesting a process before issuing it a certificate
The host integrity system provisions a machine credential, then Borg Prime grants per-microservice credentials based on the microservice's identity
A short-lived token naming the end user, issued centrally
End-user context tickets — "integrity-protected, centrally-issued, forwardable credentials that attest to the identity of an end user who made a request"
Trusting identity rather than network position
The founding premise: no service is trusted because of where it sits
Read the third row again, because it answers a question Lesson 11 raised and did not fully settle. Google does attestation in two stages: the machine proves itself first — its credential can only be decrypted if secure boot verified successfully — and only then does the scheduler hand out workload credentials on that machine. Your SO_PEERCRED check is the second stage in miniature. The first stage is the hardware root of trust the course does not build.
One difference worth understanding
Lesson 8 told you never to forward the user's token downstream. Google's end-user context tickets are explicitly forwardable. That looks like a flat contradiction. It is not, and the reason is worth your time, because it is the difference between copying a design and understanding it.
The confusion comes from one word doing three jobs. "Token" can mean any of these, and they have completely different risk profiles:
What it is
Who issued it
Who accepts it
Lifetime
The user's login credential
Your identity provider, for the user
Your edge, and often everything behind it
Hours to weeks
An audience-pinned delegation token
A trusted third party, per hop
Exactly one named service
Seconds
A forwardable user assertion
A trusted third party, once
Any service inside a controlled fabric
Seconds
Forwarding the first one is always wrong. That is what Lesson 8 forbids. A session token or login JWT is broad, long-lived, and accepted by your edge, so every service that touches it can impersonate that user everywhere, indefinitely. One over-logged header and someone owns the account.
Rows two and three are both fine. They are different engineering trades, and you should be able to argue for either.
Design A: audience-pinned, what you built
Each hop asks the auth server for a token aimed at exactly one callee, and aud names that callee. Check 4 rejects it anywhere else.
What it costs. One token exchange per destination. A five-hop call chain means five exchanges, so the auth server sits on the critical path of every request and its latency and availability become yours. The cache hides most of that, but not all of it.
What it buys. When every resource server enforces aud, a stolen token is useful against exactly one service, for under a minute. The restriction is inside the signed token, so an intermediary cannot widen it without invalidating the signature. A verifier can still be misconfigured to ignore aud; signed policy only works when the receiver enforces it.
Design B: forwardable, what Google does
The edge mints one end-user context ticket when the request arrives, and that same ticket travels down the whole call chain. Google's description is precise: "integrity-protected, centrally-issued, forwardable credentials that attest to the identity of an end user who made a request of the service."
What it costs. The ticket is accepted by many services, so it is only as safe as the environment it travels through. If any hop can leak it to somewhere outside that environment, it is usable there too.
What it buys. One mint per request instead of one per hop. At a fan-out of thousands of services, that difference is not a micro-optimization; it is the difference between the design working and not working.
What makes Design B safe there
It is not the ticket alone. Every hop it crosses is already ALTS-authenticated in both directions, the ticket is integrity-protected so no service can alter the user it names, it is short-lived, and it never leaves that fabric. Take the forwardable ticket and drop it into a system without those properties and you have built something strictly weaker than Design A.
Why this course uses Design A
Three reasons, and only one of them is about security.
It makes the audience check real. Check 4 is one of the six, and with a forwardable token there is no single audience to compare against — the check either disappears or becomes a list membership test. You would have implemented five checks and read about a sixth. Building the version where aud genuinely restricts something is what lets you attack it in Lesson 10 and watch it refuse.
Its cost is invisible at this scale. The argument for forwarding is amortizing one mint across a deep call chain. This system has two services and one hop. Paying for a property you get to keep, when the cost is a cached call every fifty seconds, is an easy trade.
It fails safe for a reader who deploys it. Design B is safe inside a fabric that took Google years to build. Someone who finishes this course and ships it to two containers behind a load balancer does not have that fabric. The stricter default is the responsible one to teach.
The question behind both designs
Not "may this be forwarded?" but "if this leaks, what can the holder do, to whom, and for how long?" Design A answers: one service, one minute. Design B answers: any service in the fabric, one minute — which is a fine answer once you can describe the fabric precisely. If you cannot describe it, you do not have one.
When you would switch
Move to a forwardable assertion when the exchange traffic genuinely hurts: deep call chains, wide fan-out, or an auth server that has become a latency or availability problem you can measure rather than imagine.
If you do, you owe the design three things that Design A gave you for free:
authenticated transport on every hop the ticket crosses;
integrity protection, so no intermediate service can rewrite the user it names;
an egress boundary the ticket cannot cross.
Ship the forwardable token without those and you have recreated the Lesson 4 problem with better cryptography.
A service mesh such as Istio or Linkerd — sidecars handle mTLS and rotation, and your application code goes back to plain HTTP;
SPIRE directly — for mixed virtual machines, containers, and bare metal;
Cloud workload identity — the managed equivalent on the major cloud providers;
An identity provider for the token half — the same products that already issue your user logins.
The value of having built it by hand is that when you configure one of those, you know what each setting actually does. You will recognize requestCert without rejectUnauthorized. You will notice a system that reads alg from the token. And you will recognize the day somebody proposes forwarding a raw user token downstream.
The whole course in one table
Lesson
Credential
Proves
Lifetime
Stored
3
API key
Possession of a string
Forever
Source or config
6
Plus TLS
The server is genuine
10 years
Disk
7
mTLS certificate
Which service
10 years
Disk
10
Plus exchanged token
Which user
60s token, 10y certificate
Disk
12
SPIFFE SVID
Which service
60 seconds
Nothing
13
SVID plus token
Service and user
60 seconds each
Nothing
Read the last column from top to bottom. Credentials went from a string in a configuration file to something with no persistent existence at all.
The decisions behind the design
Prove possession without sending the private key
Why: In a TLS handshake, the caller's certificate supplies a name and its private key signs a fresh transcript. The private key is never transmitted, and a copied handshake signature is not reusable on a different connection.
Alternative: The one-key-for-everyone bearer design from Lesson 3. Per-caller API keys, TLS transport, expiry, and overlapping rotation can repair several of its weaknesses, but any copied bearer value remains usable like the original until it expires or is revoked.
Alternative:
Sign tokens with ES256, not HS256
Why: Verifiers hold only a public key, so compromising a verifier does not let an attacker mint tokens.
Alternative: HMAC. One key, less machinery, and every verifier becomes a potential forger.
Alternative:
Read the actor from the certificate, never from the body
Why: A service must not be able to name itself, or it can name anyone.
Alternative: An actor_token field in the request. Requires validating yet another credential to learn something the connection already proved.
Alternative:
Make certificates expire in sixty seconds
Why: Expiry removes the common-case need for per-certificate revocation, which almost nobody operates correctly. Emergency controls — dropping a trust anchor, stopping issuance, draining connections — are still required.
Alternative: Ten-year certificates plus a revocation list. Cheaper to issue, and silently useless when it matters.
Alternative:
Compare the token's actor with the connection identity
Why: Two valid credentials that describe different callers are not a proof of anything.
Alternative: Trust the allow list alone. Correct today, wrong the moment the list has two entries.
Alternative:
You finished
You started with two programs that answered anybody.
You ended with two that prove which service is calling and which user it is acting for. Neither credential existed a minute ago, and neither can be used to open a new connection a minute from now.
Along the way you built a private certificate authority, a token exchange service, a JWKS endpoint, and a workload attestation agent — every one of them from the standard library, with nothing installed.
If you want to keep going, there are three natural next steps. Replace the substring selector with a real attestation policy. Add roles or groups on top of the scope check, so permissions do not have to be listed one by one. Then put the whole thing behind a service mesh, so you can compare what it does with what you now know it must be doing.
Course checkpoint
You have finished the course when you can explain all of these without rereading:
the four stages of one authenticated request, and which stage fuses the layers;
why the actor comparison is worth keeping even when it looks redundant;
why Go rejects during the handshake while Node rejects in the handler;
what short-lived credentials cost, and why real deployments run an agent per node;
five things this project does that a real system also does, and the limits it does not cover;
how you would recognize each of the mistakes in Lessons 4, 7, and 10 in someone else's codebase.