Microservice Security: Mutual TLS & cert-manager
Standard web HTTPS connections use one-way TLS: the client browser verifies the X.509 certificate presented by the server to confirm server identity, but the server does not cryptographically authenticate the client's identity during the TLS handshake phase.
Inside a zero-trust Kubernetes microservice cluster, assuming that network traffic originating from internal pod IP addresses is trustworthy creates severe vulnerability. If an attacker breaches a single frontend container, they can make unauthenticated HTTP calls directly to internal payment or user database microservices.
Mutual TLS (mTLS) enforces two-way cryptographic authentication: both client and server present X.509 certificates during the TLS handshake, verifying each other's identities before transmitting data. cert-manager automates the issuance, renewal, and management of these internal X.509 certificates in Kubernetes. This guide details mTLS handshake mechanics, cert-manager Issuer configurations, SPIFFE workload identities, and zero-downtime certificate rotation.
Mental Model: One-Way TLS Encryption vs Mutual TLS (mTLS) Peer Authentication
One-way TLS only validates server identity via a trusted Certificate Authority (CA), leaving server-to-server microservice calls vulnerable to spoofing if IP access controls fail.
Mutual TLS (mTLS) Architecture requires bidirectional identity validation:
1. Server Verification: Client verifies the Server's X.509 certificate against the internal cluster CA. 2. Client Verification: Server requests, receives, and verifies the Client's X.509 certificate against the internal cluster CA. 3. Encrypted Tunnel: Data is encrypted end-to-end using TLS 1.3 symmetric keys. For secrets and mesh security, review securing cloud infrastructure hashicorp vault and implementing service mesh traffic management envoy proxy.
Quick reference
- Enforces bidirectional cryptographic identity verification for all internal service-to-service calls.
- Prevents man-in-the-middle (MITM) attacks and unauthorized internal microservice impersonation.
- TLS 1.3 handshake negotiates ephemeral Diffie-Hellman keys for forward secrecy.
- Integrates with Kubernetes service meshes (Istio, Linkerd) for transparent mTLS encryption.
- Powers zero-trust network architectures at Netflix, Google, Cloudflare, and CoreConcept.
Remember this
Deploy Mutual TLS (mTLS) to enforce two-way cryptographic identity verification across internal microservices.
Automated X.509 Certificate Provisioning via cert-manager & Let's Encrypt / Vault
cert-manager automates certificate lifecycles in Kubernetes using declarative ClusterIssuer and Certificate CRDs:
1apiVersion: cert-manager.io/v12kind: Certificate3metadata:4 name: payment-api-cert5 namespace: production6spec:7 secretName: payment-api-tls-secret8 duration: 2160h # 90 days9 renewBefore: 360h # Renew 15 days prior to expiration10 subject:11 organizations: ["CoreConcept"]12 commonName: payment-api.production.svc.cluster.local13 dnsNames:14 - payment-api.production.svc.cluster.local15 issuerRef:16 name: internal-ca-issuer17 kind: ClusterIssuerQuick reference
- cert-manager CRDs (Certificate, ClusterIssuer) automate X.509 issuance and auto-renewal.
- Supports multiple CA backends including HashiCorp Vault, Let's Encrypt, and internal Root CAs.
- Stores issued X.509 TLS keypairs directly in Kubernetes Secrets mounted to pod volumes.
- Automatic renewal (renewBefore) prevents catastrophic outages caused by expired certificates.
- Monitors certificate status metrics via Prometheus integration for proactive alert routing.
Remember this
Configure cert-manager Certificates with automated renewBefore windows to eliminate certificate outages.
SPIFFE / SPIRE Workload Identity & Cryptographic SVID Attestation
Using generic IP addresses or DNS names for mTLS authentication is brittle in dynamic Kubernetes environments where pods spin up and down constantly.
SPIFFE (Secure Production Identity Framework for Everyone) standardizes workload identity format:
spiffe://cluster.local/ns/production/sa/payment-api-service-account
SPIRE (SPIFFE Runtime Environment) attests pod identity during startup by inspecting Linux kernel cgroups and Kubernetes service accounts, issuing short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) directly to container memory.
Quick reference
- SPIFFE IDs define universal, platform-agnostic URI identity strings for cloud workloads.
- SPIRE attestation inspects container cgroups and Kubernetes service accounts cryptographically.
- Short-lived SVID certificates (1-hour lifespan) minimize exposure windows if keys are leaked.
- Eliminates hardcoded credentials and static Secret tokens from application container images.
- Enforces strict microservice identity attestation across multi-cloud Kubernetes clusters.
Remember this
Adopt SPIFFE/SPIRE workload identities to issue cryptographically attested SVID certificates to pods.
Automated Zero-Downtime Certificate Rotation & Expiration Monitoring
Short-lived certificates (e.g. 24-hour expiration) require resilient Zero-Downtime Rotation:
1. Dual Trust Bundles: When rotating the internal Root CA, servers must trust both the old Root CA and new Root CA concurrently during the transition phase.
2. Hot-Reloading TLS Keypairs: Applications must dynamically reload updated TLS certificates from disk volumes when cert-manager updates the mounted Secret, without restarting the main container process:
1// Go Hot TLS Certificate Reloading2config := &tls.Config{3 GetCertificate: func(info *tls.ClientHelloInfo) (*tls.Certificate, error) {4 return loadLatestCertFromDisk("/etc/tls/tls.crt", "/etc/tls/tls.key")5 },6}Quick reference
- Dual trust bundles enable zero-downtime Root CA rotations across microservice fleets.
- Dynamic TLS keypair hot-reloading updates certificates in RAM without restarting application containers.
- Prometheus cert-manager alerts trigger when certificates reach 7 days remaining lifespan.
- Automated rotation tests verify that short-lived (24h) certificates renew reliably in CI/CD.
- Guarantees unbroken zero-trust security compliance across production Kubernetes deployments.
Remember this
Implement TLS certificate hot-reloading and dual trust bundles for zero-downtime CA rotation.
Key takeaway
To test cert-manager locally, install cert-manager via Helm (helm install cert-manager jetstack/cert-manager). Deploy a self-signed ClusterIssuer and verify Secret creation.
Related Articles
Explore this topic