Envelope Encryption for Secrets: Why One KMS Key Isn't Enough

The most common secrets-at-rest design is also the worst one: store every secret encrypted under the same key. A new engineer writes it in an afternoon, ships it to production, and nothing visibly goes wrong for a year. Then the one key leaks — through a backup, a disk image, a rogue admin, a misconfigured IAM policy — and every secret your system has ever stored is compromised in a single event.

Envelope encryption is the pattern that makes that event survivable. Every secret gets its own data encryption key (DEK). The DEKs are encrypted by a key encryption key (KEK) that lives in a KMS or HSM. The plaintext secret exists only in memory when it's being served. A DEK leak compromises exactly one secret; a KEK leak compromises nothing if the KMS protects the raw key material (which is the whole point of a KMS).

This post walks through the shape of envelope encryption, why naive single-key encryption isn't enough, and what a production-grade multi-KMS implementation looks like in a secrets management platform.

The naive design (and why it fails)

The simplest "encrypt secrets at rest" implementation is:

1. Generate one AES-256 key at setup.
2. Store it somewhere — environment variable, config file, a "secrets table" column in your own database.
3. Every time a secret is stored: ciphertext = AES_GCM(secret, master_key, nonce).
4. Every time a secret is read: secret = AES_GCM_decrypt(ciphertext, master_key, nonce).

This works. It's also fragile in ways that become load-bearing at enterprise scale:

• Key rotation is a migration. To rotate the master key you must read every ciphertext, decrypt with the old key, re-encrypt with the new one, and write it back. For a million-secret vault, that's an operational incident.
• Key leakage is total compromise. One compromised master means every secret since the beginning of time is exposed.
• Per-secret access control is grafted on. The crypto layer doesn't know who's allowed to see what; authorization is entirely in the application layer.
• Hardware protection is all-or-nothing. Either the master key lives on an HSM (and you can never touch ciphertext outside the HSM network) or it lives in memory (and it's stealable).

The envelope pattern

Envelope encryption splits the single key into two layers:

• Data Encryption Key (DEK) — a fresh AES-256 key generated per secret (or per secret-version). Encrypts the actual plaintext.
• Key Encryption Key (KEK) — a long-lived key that encrypts the DEK. Lives in a KMS or HSM and never leaves.

Write path:

1. Generate a random 32-byte DEK.
2. Encrypt the plaintext secret: ciphertext = AES_GCM(plaintext, DEK, nonce).
3. Send the DEK to KMS for encryption: encrypted_dek = KMS.Encrypt(KEK, DEK).
4. Store (ciphertext, encrypted_dek, nonce). Zero the DEK from memory.

Read path:

1. Load (ciphertext, encrypted_dek, nonce).
2. Send encrypted_dek to KMS: DEK = KMS.Decrypt(KEK, encrypted_dek).
3. Decrypt the secret: plaintext = AES_GCM_decrypt(ciphertext, DEK, nonce).
4. Return the plaintext. Zero the DEK.

The database or object store holds only ciphertext and encrypted DEKs. Without access to the KMS/HSM that holds the KEK, a full database dump is useless. A DEK leak affects one secret. A KEK rotation is a KMS operation that re-wraps the DEKs, not the ciphertext — constant-time per secret regardless of secret size.

Why "any KMS" isn't the right answer

Vendors often ship envelope encryption with a single supported KMS ("we integrate with AWS KMS"). That's a great demo and a terrible production constraint. Real enterprises need:

• Multi-cloud. A customer running primarily in Azure doesn't want AWS KMS as a hard dependency.
• BYOK (Bring Your Own Key). Regulated verticals require the KEK to live in a customer-controlled KMS, not the vendor's infrastructure.
• HSM-backed. FIPS 140-2/3 Level 3 and some compliance regimes require hardware key storage.
• Air-gapped deployments. Some customers can't talk to a cloud KMS at all; they need a local HSM or a local software key provider for dev/CI.

CoreLink's crypto layer abstracts the KMS behind a simple interface: Encrypt(plaintext []byte) ([]byte, error) and Decrypt(ciphertext []byte) ([]byte, error). Under the hood, the supported backends are:

• AWS KMS — symmetric CMK with kms:Encrypt / kms:Decrypt.
• Azure Key Vault — RSA-OAEP or AES-KW on Premium SKU keys.
• GCP KMS — cryptoKey encrypt/decrypt on regional keys.
• HashiCorp Vault Transit — transit/encrypt / transit/decrypt, useful when the customer already runs Vault for other workloads.
• PKCS#11 HSM — direct integration with SafeNet Luna, Thales Luna, or any generic PKCS#11 v2.40 HSM via CGo bindings.
• Local key file — AES-256 key in a local file, for dev and CI only. Rejected at startup when the environment is production.

Each backend implements the same Go interface; the choice is a config change, not a code change. A customer can run one tenant on AWS KMS and a co-tenant on Azure Key Vault in the same deployment.

Per-tenant KEKs, per-secret DEKs

The other dimension most implementations get wrong is KEK scoping. If every tenant in a multi-tenant SaaS shares a single KEK, a KMS policy change that grants a contractor access to the KEK grants them cryptographic access to every tenant's secrets — regardless of application-layer authorization.

The right shape is a KEK per tenant (minimum) and ideally a KEK per workspace within a tenant. CoreLink stores the KEK key ID on the tenant record and resolves it at encrypt/decrypt time. Key rotation is a KMS-side operation: when a new key version rolls, KMS automatically uses the new version for encrypts, and decrypts for old DEKs continue to work because KMS still holds the old version. The application never tracks key versions explicitly.

What rotation actually means

"Rotate the encryption key" is vague. In an envelope design, three things rotate, and they're rotating at different rates:

1. DEKs — new per secret-version. Every time you create a new version of a secret, you get a new DEK. Automatic, no operator action.
2. KEK versions — managed by the KMS. AWS KMS automatic key rotation rolls the KEK every year; Azure Key Vault lets you set a rotation policy. When a new KEK version exists, new DEK encryptions use it; old DEKs keep working because the KMS retains the old version for decryption.
3. KEK material (full re-key) — rare. Requires decrypting every DEK with the old KEK and re-encrypting with the new one. Only needed if you suspect the old KEK was compromised, or you're migrating KMSes. CoreLink supports this via a migration service that iterates the secrets table, but it's an explicit operator action, not an automatic background job.

Distinguishing these three is how crypto conversations with auditors stay productive. "We rotate our encryption key every 90 days" is an ambiguous answer; the honest answer specifies which of these three is rotating, on what cadence, and by what mechanism.

Failure modes worth stress-testing

• KMS outage during write. If the KMS is unreachable, the write fails. Good behavior. The alternative — writing unencrypted or using a cached DEK — is worse.
• KMS outage during read. Same: read fails. Caching decrypted DEKs in memory is a performance optimization with security trade-offs; if you do it, cap TTL aggressively (60 seconds) and scope the cache per request.
• KEK deletion. If the KEK is deleted (rare, but AWS KMS does support scheduled deletion), every secret under it becomes permanently unrecoverable. Every production KMS should have deletion protection enabled.
• KEK region outage. For cloud KMSes, multi-region replication is a KMS feature; enable it. For single-region deployments, accept that a region outage means the secrets are unavailable until the region recovers.

Evaluating a secrets platform's crypto

Questions to ask a vendor:

• What's the DEK lifecycle? Per secret? Per version? Per tenant?
• Which KMSes does the KEK backend support, and is the selection per-tenant or global?
• Can customers BYOK?
• Is there an HSM option (PKCS#11) for regulated deployments?
• What happens during a KMS outage — degraded read, degraded write, or hard fail?
• What's the cost model at volume? Cloud KMSes charge per decrypt; at a million secrets * 10 reads/day, the KMS bill can matter.

CoreLink is built around envelope encryption from day one, with five KMS backends and per-tenant key scoping. The platform services page lists the full capability set; the use-case pages for rotation and compliance cover the operational surface.

Encryption at rest is the floor, not the ceiling. But the floor is where most implementations fall through — and envelope is the only shape that holds your weight when it matters.