A cross-border payments platform on Kubernetes, with card data sealed by hardware security modules
A cross-border payments company moving money between the United States, Europe and Latin America, a leader in its segment.
- Scale
- 3 Kubernetes clusters in 2 regions, about 140 services, 2,400 transactions per second at peak, network-attached HSM cluster
- Engagement
- Design, build and migration over 11 months
The challenge
Card and account data moved through a platform that had grown from a few services into more than a hundred, all in one cluster with shared credentials. Every new audit meant proving which workloads could touch card data, and the honest answer was "most of them." The company needed a platform where the card-data environment was small, provable and sealed by keys that never leave hardware.
Payments companies face two pressures at once: ship features weekly, and prove to an assessor that card data is safe. The platform had grown from a handful of services into more than a hundred sharing one Kubernetes cluster and a handful of long-lived credentials. Asking which workloads could reach card data produced the answer "most of them," which made every audit a manual search through manifests. The goal was not just a safer cluster but a smaller one: shrink the part of the platform where card data exists until it can be described in a page.
The architecture separates concerns by cluster. A dedicated cardholder-data cluster runs only the tokenization service and the few components that must handle a full card number. Cilium enforces default-deny network policy with eBPF, and services authenticate each other with mutual TLS 1.3 using short-lived SPIFFE identities, so a stolen credential is useful for minutes, not months. Key material sits in a clustered pair of network-attached hardware security modules per region, reached over PKCS#11. The modules hold the key-encryption keys; data keys (AES-256‑GCM) are generated per record class, wrapped by the HSM and unwrapped only inside the tokenization service. Kubernetes Secrets in etcd are protected by KMS v2 envelope encryption against the same modules, so a copy of an etcd snapshot is unreadable without the hardware.
Migration followed the data. The tokenization service went live first, in shadow mode, issuing tokens alongside the legacy path while a reconciliation job compared results. Services then moved to tokens one at a time, each release removing a copy of card data from its database. Historic data was re-encrypted in batches under the new key hierarchy during low-traffic windows, with an automatic stop if error rates rose. Delivery moved to GitOps with Argo CD, and the build pipeline began signing images with cosign; admission policy refuses anything unsigned from entering the cardholder cluster. Traffic was shifted between regions during the cutover so that neither region carried a risky change alone.
Testing combined engineering and attack. Load tests reproduced the 2,400 transactions per second peak with HSM latency measured at the 99th percentile, and the tokenization service was sized so that losing one HSM in a pair cost headroom, not availability. Failure drills removed a node, a zone and an HSM during a test run. Our penetration testers then attacked from the position of a compromised pod: they tried lateral movement, service-account abuse, reading Secrets from etcd and extracting keys. Findings were fixed and retested, and the reports were written so an assessor could reuse them as evidence. The platform's controls are designed to support PCI DSS 4.0.1 requirements for protecting stored account data, strong cryptography and logging; the assessment itself remains the company's, and the assessor's, to complete.
The result is a card-data environment reduced to one cluster and one service, keys that cannot be exported from the hardware, and logs written to write-once storage with synchronized time so that an investigator can reconstruct what happened. The engineering teams ship to the other clusters as quickly as before, because only the card-data path carries the extra controls. When customers in the European Union ask where their personal data is processed, the answer is a diagram and a policy rather than a search.
What our team delivered
- A dedicated cardholder-data cluster, separate from the general-purpose clusters, so the PCI DSS scope covers one well-defined platform instead of the whole company
- Default-deny network policy enforced by Cilium (eBPF), with mutual TLS 1.3 between services using short-lived SPIFFE workload identities
- Network-attached hardware security modules in a clustered pair per region, reached over PKCS#11 and mutually authenticated TLS, holding the key-encryption keys
- Kubernetes KMS v2 envelope encryption of Secrets in etcd, and AES-256-GCM data keys for stored account data, wrapped by HSM-held keys and rotated on a schedule
- A tokenization service as the only component that sees a full card number; every other service holds only tokens
- Pod Security Standards at the restricted level, admission control with policy-as-code, and signed container images (Sigstore cosign) with a software bill of materials for every build
- GitOps delivery with Argo CD, two-person review on production changes, and runtime detection with eBPF-based syscall monitoring
- Audit and access logs shipped in real time to write-once storage, with synchronized time
Result
A card-data environment reduced to one cluster and one tokenization service, keys that cannot be exported from the HSMs, and evidence that is generated by the platform rather than assembled by hand before each assessment. In the modeled migration, the 140 services moved in waves over eleven months, with the tokenization service live first so that every wave reduced the card-data scope instead of growing it. HSM failover and key ceremonies were rehearsed before cutover, the second region carried production load in a game day, and policy-as-code checks blocked a non-compliant deployment before it reached the cluster.
Separate by region and by business unit
Each business unit lives in its own VRF and VLAN in every region, so a problem in retail cannot reach finance. Traffic between units has to pass a firewall that denies by default and allows only what is written down.
Every sign-in, verified
Staff sign in once, but behind the scenes the device, the identity, a phone approval and your access policy are all checked before any app opens, and strangers are turned away.
Super Intelligence data flow
Every request is classified by sensitivity, so confidential data stays on models inside your walls while routine work can use the best frontier AI.
Incidents resolved five times faster
Mean time to resolve (MTTR) for incidents, by quarter, for a 250‑seat organisation. The first four quarters are an in-house on-call rota; the last four are under a managed service with a 24/7 operations centre, runbooks and monitoring. Illustrative figures.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.