Cryptography & Key Store

This section covers the PGP encryption/decryption engine used by the pgp.encrypt and pgp.decrypt job steps, the file signing and verification system, and the centralized Key Store that manages all cryptographic key material.

7.1 Unified Crypto Interface

All cryptographic operations go through a common provider interface, consistent with the plugin pattern used by the Job Step registry and Compression providers.

public interface ICryptoProvider
{
    Task<CryptoResult> EncryptAsync(
        EncryptRequest request,
        IProgress<CryptoProgress> progress,
        CancellationToken cancellationToken);
    Task<CryptoResult> DecryptAsync(
        DecryptRequest request,
        IProgress<CryptoProgress> progress,
        CancellationToken cancellationToken);
    Task<CryptoResult> SignAsync(
        SignRequest request,
        IProgress<CryptoProgress> progress,
        CancellationToken cancellationToken);
    Task<VerifyResult> VerifyAsync(
        VerifyRequest request,
        IProgress<CryptoProgress> progress,
        CancellationToken cancellationToken);
}

public record EncryptRequest(
    string InputPath,
    string OutputPath,
    IReadOnlyList<Guid> RecipientKeyIds,           // One or more public keys
    Guid? SigningKeyId,                             // Optional: sign-then-encrypt
    OutputFormat Format);                           // Armored (.asc) or Binary (.pgp)

public record DecryptRequest(
    string InputPath,
    string OutputPath,
    Guid PrivateKeyId,
    bool VerifySignature);                         // Also verify if signed

public record SignRequest(
    string InputPath,
    string OutputPath,
    Guid SigningKeyId,
    SignatureMode Mode);                           // Detached, Inline, or Clearsign

public record VerifyRequest(
    string InputPath,
    string? DetachedSignaturePath,                 // null = inline/clearsigned
    Guid? ExpectedSignerKeyId);                    // null = accept any known key

public record VerifyResult(
    bool IsValid,
    VerifyStatus Status,                           // Valid, Invalid, UnknownSigner, ExpiredKey, RevokedKey
    string? SignerFingerprint,
    DateTime? SignatureTimestamp);

public enum VerifyStatus
{
    Valid,
    Invalid,
    UnknownSigner,
    ExpiredKey,
    RevokedKey
}

The pgp.encrypt, pgp.decrypt, pgp.sign, and pgp.verify job steps delegate entirely to this interface. Step configuration references keys by their Key Store ID, which the engine resolves at runtime.

7.2 Library Selection

Courier uses BouncyCastle (Portable.BouncyCastle NuGet package) as its PGP engine. BouncyCastle is the de facto .NET library for OpenPGP operations — it is mature, actively maintained, and widely deployed in enterprise environments.

Key capabilities used:

• RSA key pair generation (2048, 3072, 4096 bit)
• OpenPGP encryption and decryption with support for multiple recipients
• Detached, inline, and clearsigned signatures
• ASCII-armored and binary format output
• Passphrase-protected private key handling
• Streaming API for processing large files without full memory load

BouncyCastle also supports ECC algorithms (Curve25519, NIST P-256/P-384), which positions Courier for the planned ECC key support without changing libraries.

7.3 Key Store

The Key Store is a centralized, encrypted vault for all PGP key material. It is a first-class entity in Courier with its own management API and UI surface.

7.3.1 Key Entity

Each key in the store has the following metadata:

The algorithm enum is designed with room for expansion. Adding ECC support in the future requires adding new enum values and implementing the BouncyCastle ECC key generation — no schema migration needed.

7.3.2 Key Status Lifecycle

Keys follow a defined lifecycle with five states:

    ┌────────┐
    │ Active │
    └──┬─┬─┬─┘
       │ │ │
       │ │ └──────────────┐ (manual revoke)
       │ │                │
       │ │ (approaching   │
       │ │  expiration)   │
       │ │           ┌────▼────┐
       │ │           │ Revoked │  ── cannot be used by any job
       │ │           └─────────┘
       │ │
       │ ┌────▼─────┐
       │ │ Expiring │  ── still usable, emits warning events
       │ └──┬──┬────┘
       │    │  │
       │    │  └── (expires) ──► Retired
       │    │
       │    └── (manual revoke) ──► Revoked
       │
       │ (manual retire)
  ┌────▼────┐
  │ Retired │  ── cannot be used by new jobs, existing references warn
  └─────────┘
       │
       │ (manual delete)
  ┌────▼────┐
  │ Deleted │  ── soft delete, key material purged, metadata retained for audit
  └─────────┘

Active: Key is available for use by any job. Default state after generation or import.

Expiring: Key's expiration date is within the configured warning window (default: 30 days). The key is still fully functional but domain events are emitted (KeyExpiringSoon) for V2 alerting. A background service checks expiration dates daily and transitions keys automatically.

Retired: Key has been manually retired or has passed its expiration date. Jobs cannot use a retired key for new operations. If an existing job references a retired key, the step fails with a descriptive error indicating the key must be updated. Decryption and signature verification with retired keys remain possible to handle legacy files.

Revoked: Key has been explicitly revoked, indicating it should no longer be trusted under any circumstances. All operations (including decryption and verification) are blocked. Verification against a revoked key returns VerifyStatus.RevokedKey. This is a terminal state — a revoked key cannot be re-activated.

Deleted: Soft delete. Private key material is securely purged (overwritten with zeros before removal). Public key data and metadata are retained for audit trail purposes. The key no longer appears in normal queries but is visible in audit history.

7.3.3 Key Generation

Courier can generate RSA key pairs internally:

• Supported sizes: 2048, 3072, 4096 bit (default: 4096)
• Key format: OpenPGP-compatible key pair with configurable user ID (name + email)
• Expiration: Optional expiration date set at generation time
• Passphrase: Optional passphrase protection on the private key

Generation uses BouncyCastle's RsaKeyPairGenerator with a secure random source. The key pair is immediately encrypted and stored in the Key Store. The public key is made available for export.

7.3.4 Key Import

Keys can be imported from external sources in the following formats:

On import, Courier:

1. Parses the key material and extracts metadata (fingerprint, algorithm, expiration, user IDs)
2. Validates key integrity (verifies self-signatures)
3. Checks for fingerprint collisions with existing keys in the store
4. If importing a private key, prompts for the passphrase and verifies it can unlock the key
5. Encrypts private key material via envelope encryption (random DEK + Key Vault wrap) before storage
6. Creates the Key Store record with status Active

If the imported key is already expired, it is imported with status Retired and a warning is returned.

7.3.5 Key Export

Public key export (authenticated, all roles):

• Formats: ASCII-armored (.asc) or binary (.pgp)
• Access: Available via API (GET /api/v1/pgp-keys/\{id\}/export/public?format=armored) and the frontend UI
• Authentication required — all public key exports go through the standard Entra ID auth pipeline. Even public keys reveal identity, partner relationships, and operational metadata. All exports are logged as audit events.

Shareable public key links (optional, Admin-only):

For scenarios where a partner needs to download a public key without Entra ID credentials, Admins can generate a time-limited, single-purpose shareable link:

• Endpoint: POST /api/v1/pgp-keys/\{id\}/share → returns a URL with a cryptographic token
• URL format: /api/v1/pgp-keys/shared/\{token\} — no authentication required on this endpoint
• Token: 256-bit random, stored hashed (SHA-256) in the database alongside key ID, expiration, and creator
• Default expiration: 72 hours (configurable per link, maximum 30 days, set via security.max_share_link_days system setting)
• Single key, read-only: The token grants access to one specific public key export only, nothing else
• Audit: Link creation, every download via the link, and link expiration are all logged with the requesting IP address
• Revocation: Links can be revoked before expiration via DELETE /api/v1/pgp-keys/\{id\}/share/\{token\}
• Disabled by default: The security.public_key_share_links_enabled system setting defaults to false. Admins must explicitly enable the feature before any shareable links can be generated

Private key export is a sensitive operation:

• Requires explicit user confirmation
• Exported private key is optionally re-encrypted with a user-provided passphrase
• The export event is prominently logged in the key audit trail
• A future enhancement could require multi-user approval for private key export

7.3.6 Private Key Encryption at Rest

V1 Implementation Note — Local KEK (No Azure Key Vault)

The design below describes the target architecture using Azure Key Vault for KEK management. In V1, we opted not to use Azure Key Vault. Instead, credential encryption uses a local AES-256-GCM envelope encryption scheme with a KEK sourced from an environment variable (COURIER_ENCRYPTION_KEY, base64-encoded 256-bit key) or appsettings.json configuration (Encryption:KeyEncryptionKey).

V1 implementation (AesGcmCredentialEncryptor):

• KEK is loaded from configuration at startup (validated: must be exactly 32 bytes)
• DEK wrapping uses AES-256-GCM (KEK wraps DEK locally) instead of Key Vault WrapKey/UnwrapKey
• Blob format: [1B version][12B DEK-wrap-nonce][16B DEK-wrap-tag][32B wrapped-DEK][12B data-nonce][16B data-tag][N bytes ciphertext] (89 + N bytes)
• DEK is zeroed via CryptographicOperations.ZeroMemory() in finally blocks
• The KEK does exist in process memory (unlike Key Vault where it never leaves the HSM)
• A pre-generated dev-only key is provided in appsettings.Development.json; production deployments must supply their own key

Migration path to Key Vault (V2): Replace AesGcmCredentialEncryptor with the EnvelopeEncryptionService described below. Since both schemes use AES-256-GCM for data encryption, migration requires only re-wrapping DEKs (unwrap with local KEK, re-wrap via Key Vault) — the ciphertext payload is unchanged. The ICredentialEncryptor interface abstracts this swap.

Security trade-off: The local KEK approach is simpler to deploy (no Azure dependency) but the KEK resides in process memory. An attacker with a memory dump could extract it. Key Vault eliminates this risk. For V1's deployment model (single-tenant, controlled infrastructure), this trade-off is acceptable.

All private key material is encrypted before storage in PostgreSQL using AES-256-GCM envelope encryption with Azure Key Vault wrap/unwrap operations. The master key (KEK) never leaves Key Vault.

Encryption flow (on key generation or import):

1. Generate a random 256-bit Data Encryption Key (DEK) using RandomNumberGenerator
2. Generate a random 96-bit IV
3. Encrypt the private key material with the DEK using AES-256-GCM, producing ciphertext + authentication tag
4. Call Azure Key Vault WrapKey to wrap the DEK with the KEK (RSA-OAEP-256). Key Vault returns the wrapped DEK and the KEK version used
5. Store the following as a single binary blob in private_key_data:

┌────────────┬──────────────┬─────────┬──────────┬─────────────┬────────────┐
│ KEK Version│ Wrapped DEK  │ IV      │ Auth Tag │ Ciphertext  │ Algorithm  │
│ (36 bytes) │ (256 bytes)  │(12 bytes)│(16 bytes)│ (variable)  │ (2 bytes)  │
└────────────┴──────────────┴─────────┴──────────┴─────────────┴────────────┘

Decryption flow (on private key use):

1. Parse the stored blob to extract KEK version, wrapped DEK, IV, auth tag, and ciphertext
2. Call Azure Key Vault UnwrapKey with the wrapped DEK and KEK version. Key Vault returns the plaintext DEK
3. Decrypt the ciphertext with the DEK using AES-256-GCM, verifying the authentication tag
4. Use the plaintext private key for the requested operation
5. Zero the plaintext DEK and private key from memory when done (CryptographicOperations.ZeroMemory)

Key properties:

• Each entity gets its own random DEK — no DEK is ever reused across entities
• The KEK (RSA 2048-bit, Key Vault key, not a secret) is never exported or held in application memory
• Key Vault operations are remote HTTPS calls — latency is ~20ms per wrap/unwrap
• The wrapped DEK is opaque to the application — only Key Vault can unwrap it
• Storing the KEK version enables seamless rotation: new encryptions use the latest KEK version, old data references the version it was encrypted with

Master key rotation:

When the KEK is rotated in Key Vault (new version created), a background task iterates all encrypted entities and re-wraps their DEKs:

1. Unwrap the DEK using the old KEK version (stored in the blob)
2. Re-wrap the DEK using the new KEK version
3. Update the stored blob with the new wrapped DEK and new KEK version
4. The actual ciphertext is untouched — only the DEK wrapper changes

This means rotation does not require re-encrypting any private key material, only re-wrapping the DEKs.

What works in V1 without any code changes: When a new KEK version is created in Key Vault, all new encryptions automatically use the latest version (the CryptographyClient defaults to the latest version for WrapKeyAsync). All existing encrypted blobs continue to decrypt correctly because the blob stores the specific KEK version URI, and UnwrapKeyAsync uses that version. Key Vault retains all previous versions unless explicitly disabled. This means KEK rotation is immediately safe — the transition period where old and new versions coexist is handled natively by Key Vault's versioning.

What's planned for V2: An automated background task that iterates all encrypted entities and re-wraps their DEKs with the current KEK version. This is a key hygiene measure (ensures old KEK versions can eventually be disabled) but is not required for security — old versions remain usable until explicitly disabled in Key Vault.

If Azure Key Vault is unavailable at startup, Courier refuses to start rather than operating with unencrypted keys.

7.3.7 EnvelopeEncryptionService Implementation

The EnvelopeEncryptionService is the single code path for all at-rest encryption in Courier. Its design enforces a critical invariant: no key material — neither the KEK nor any unwrapped DEK — is ever cached in process memory beyond a single encrypt or decrypt operation.

// Courier.Infrastructure/Encryption/EnvelopeEncryptionService.cs
public sealed class EnvelopeEncryptionService : IEnvelopeEncryptionService
{
    private readonly CryptographyClient _cryptoClient;  // Azure SDK — calls Key Vault REST API
    private readonly string _kekKeyName;

    // NOTE: CryptographyClient does NOT download the key. It sends wrap/unwrap
    // requests to Key Vault over HTTPS. The KEK plaintext never exists in this process.
    public EnvelopeEncryptionService(CryptographyClient cryptoClient, string kekKeyName)
    {
        _cryptoClient = cryptoClient;
        _kekKeyName = kekKeyName;
    }

    public async Task<EncryptedBlob> EncryptAsync(
        ReadOnlyMemory<byte> plaintext, CancellationToken ct)
    {
        // 1. Generate a random DEK — unique per call, never reused
        var dek = new byte[32]; // AES-256
        RandomNumberGenerator.Fill(dek);

        // 2. Generate a random IV
        var iv = new byte[12]; // AES-GCM 96-bit
        RandomNumberGenerator.Fill(iv);

        // 3. Encrypt plaintext with the DEK
        var ciphertext = new byte[plaintext.Length];
        var tag = new byte[16]; // AES-GCM 128-bit auth tag
        using (var aes = new AesGcm(dek, tagSizeInBytes: 16))
        {
            aes.Encrypt(iv, plaintext.Span, ciphertext, tag);
        }

        // 4. Wrap the DEK via Key Vault (remote HTTPS call)
        //    The KEK never leaves Key Vault. We send the DEK *to* Key Vault,
        //    Key Vault encrypts it with the KEK, and returns the wrapped blob.
        var wrapResult = await _cryptoClient.WrapKeyAsync(
            KeyWrapAlgorithm.RsaOaep256, dek, ct);

        // 5. Zero the plaintext DEK *immediately* — it must not survive this method
        CryptographicOperations.ZeroMemory(dek);

        // 6. Return the composite blob (no secret material remains in memory)
        return new EncryptedBlob(
            KekVersion: wrapResult.KeyId,     // Key Vault key version URI
            WrappedDek: wrapResult.EncryptedKey,
            Iv: iv,
            AuthTag: tag,
            Ciphertext: ciphertext);
    }

    public async Task<byte[]> DecryptAsync(EncryptedBlob blob, CancellationToken ct)
    {
        // 1. Unwrap the DEK via Key Vault (remote HTTPS call)
        //    Key Vault decrypts the wrapped DEK using the KEK version recorded in the blob.
        var unwrapResult = await _cryptoClient.UnwrapKeyAsync(
            KeyWrapAlgorithm.RsaOaep256, blob.WrappedDek, ct);
        var dek = unwrapResult.Key;

        try
        {
            // 2. Decrypt the ciphertext with the unwrapped DEK
            var plaintext = new byte[blob.Ciphertext.Length];
            using (var aes = new AesGcm(dek, tagSizeInBytes: 16))
            {
                aes.Decrypt(blob.Iv, blob.Ciphertext, blob.AuthTag, plaintext);
            }
            return plaintext;
        }
        finally
        {
            // 3. Zero the plaintext DEK — even if decryption fails
            CryptographicOperations.ZeroMemory(dek);
        }
    }
}

public record EncryptedBlob(
    string KekVersion,          // Key Vault key version URI (e.g., "https://vault.azure.net/keys/courier-kek/abc123")
    byte[] WrappedDek,          // DEK encrypted by KEK (opaque — only Key Vault can unwrap)
    byte[] Iv,                  // AES-GCM initialization vector (unique per operation)
    byte[] AuthTag,             // AES-GCM authentication tag (integrity verification)
    byte[] Ciphertext);         // Encrypted payload

What this code guarantees:

• The CryptographyClient is an Azure SDK client that sends HTTPS requests to Key Vault. It never downloads the KEK. The WrapKeyAsync / UnwrapKeyAsync methods send the DEK to Key Vault for server-side encryption/decryption using the KEK that exists only inside Key Vault's HSM or software boundary.
• The plaintext DEK exists in process memory only for the duration of a single EncryptAsync or DecryptAsync call, and is zeroed via CryptographicOperations.ZeroMemory in a finally block before the method returns.
• No DEK is ever cached, pooled, or stored in a field. Every encrypt operation generates a fresh DEK. Every decrypt operation unwraps the DEK, uses it, and zeros it.
• The service is registered as a singleton (the CryptographyClient is thread-safe and connection-pooled by the Azure SDK), but it holds no mutable state — no keys, no caches, no session data.

What this code explicitly does NOT do (anti-patterns that would compromise the security model):

Latency trade-off: Every encrypt or decrypt operation incurs a Key Vault round-trip (~20ms). For Courier's workload (decrypt a credential or private key at job execution time, not on every API request), this is acceptable. A job that decrypts 3 credentials and 1 PGP key adds ~80ms of Key Vault overhead to its startup — negligible against a multi-minute file transfer. The NFR target (Section 2.11) accounts for this overhead.

DI registration:

// Courier.Infrastructure/DependencyInjection.cs
services.AddSingleton(sp =>
{
    var vaultUri = sp.GetRequiredService<IConfiguration>()["KeyVault:Uri"]!;
    var keyName = sp.GetRequiredService<IConfiguration>()["KeyVault:KekKeyName"]!;
    var credential = new DefaultAzureCredential();

    // CryptographyClient — performs remote wrap/unwrap. Does NOT download the key.
    var cryptoClient = new CryptographyClient(
        new Uri($"{vaultUri}/keys/{keyName}"), credential);

    return new EnvelopeEncryptionService(cryptoClient, keyName);
});

Note on CryptographyClient vs KeyClient: The Azure SDK has two clients for Key Vault keys. KeyClient manages key lifecycle (create, rotate, delete, get metadata). CryptographyClient performs cryptographic operations (wrap, unwrap, sign, verify) without ever exposing the private key material. Courier uses CryptographyClient for envelope encryption and KeyClient only for the KEK rotation background task (to create new key versions and read metadata). Neither client can export an HSM-backed private key.

7.4 Encryption Operations

7.4.1 Single-Recipient Encryption

The standard flow for encrypting a file:

1. Resolve the recipient's public key from the Key Store by ID
2. Validate the key is in Active or Expiring status
3. Open a streaming pipeline: FileStream (input) → PGP EncryptionStream → FileStream (output)
4. Write output in the configured format (armored or binary)
5. Report progress via IProgress<CryptoProgress> at regular intervals

7.4.2 Multi-Recipient Encryption

When multiple recipient key IDs are provided, the file is encrypted such that any recipient can decrypt it with their own private key. BouncyCastle handles this natively by adding multiple public key encrypted session keys to the PGP message.

All recipient keys must be Active or Expiring. If any key is Retired, Revoked, or Deleted, the operation fails with a descriptive error listing the invalid keys.

7.4.3 Sign-Then-Encrypt

A common workflow is to sign a file with the sender's private key and then encrypt it for the recipient. This is supported as a single step configuration:

{
    "step_type": "pgp.encrypt",
    "config": {
        "recipient_key_ids": ["<recipient-uuid>"],
        "signing_key_id": "<sender-private-key-uuid>",
        "output_format": "armored"
    }
}

The engine signs the data first (using the private key), then encrypts the signed payload. The recipient decrypts first, then verifies the signature — both operations are automatic when verify_signature: true is set on the decrypt step.

7.5 Decryption Operations

Decryption resolves the private key from the Key Store, handles passphrase unlocking transparently, and streams the decrypted output:

1. Load the encrypted file as a stream
2. Resolve the private key and decrypt it via envelope decryption (Key Vault unwraps the DEK, DEK decrypts the key material)
3. If the private key has a passphrase, unlock it
4. Stream decrypt: FileStream (encrypted) → PGP DecryptionStream → FileStream (output)
5. If verify_signature is enabled and the message was signed, verify the signature against known public keys in the store
6. Write the VerifyResult to JobContext for downstream decision-making

If decryption fails (wrong key, corrupted data), the step fails with a specific error type that the job's failure policy can act on.

7.6 Signing & Verification

7.6.1 Signature Modes

The signature mode is configured per step. Detached is the default and most common for file transfer workflows.

7.6.2 Verification

Signature verification can be performed as a standalone step (pgp.verify) or as part of decryption. The result is a typed VerifyResult written to the JobContext:

• Valid: Signature matches, signer key is Active or Expiring
• Invalid: Signature does not match the data (file may be tampered)
• UnknownSigner: Signature is cryptographically valid but the signer's public key is not in the Key Store
• ExpiredKey: Signer's key has expired (signature may still be trustworthy depending on policy)
• RevokedKey: Signer's key has been revoked (signature should not be trusted)

Downstream steps can reference the verification status in their configuration to branch behavior — for example, a file.move step could be configured to only execute if verify_status == Valid.

7.7 Streaming Crypto for Large Files

All cryptographic operations use BouncyCastle's streaming API. The pipeline for a large file encryption:

FileStream (source, 80KB buffer)
  → [Optional] SignatureGenerationStream
    → PgpEncryptedDataGenerator (streaming)
      → [Optional] ArmoredOutputStream
        → FileStream (output, 80KB buffer)

Memory usage is bounded to approximately 2× the buffer size regardless of file size. For 6–10 GB files, the bottleneck is disk I/O, not memory. Progress is reported every 10MB processed, consistent with the compression system.

7.8 Key Rotation Management

A background service (KeyExpirationService) runs daily and performs the following:

1. Scan all keys with Active status and an expires_at date
2. Transition keys within the warning window (default: 30 days) to Expiring status
3. Emit KeyExpiringSoon domain events for each newly transitioned key
4. Transition keys past their expiration date to Retired status
5. Emit KeyExpired domain events for each newly retired key

In V1, these domain events are recorded in the key audit log. In V2, event subscribers will deliver notifications via email, Slack, or UI alerts. The event infrastructure mirrors the Job Engine's notification hooks (Section 5.17), ensuring consistent patterns across the system.

Optional auto-generation: A future enhancement could automatically generate a replacement key pair when a key enters Expiring status. The new key would be created with the same parameters (algorithm, size, user ID) and linked to the expiring key as its successor. This is not in V1 scope but the schema accommodates it with a nullable successor_key_id column.

7.9 Key Audit Log

All key operations are recorded in the key_audit_log table:

This log is append-only and never modified. It provides full traceability for compliance requirements — you can answer questions like "which jobs used this key in the last 90 days" or "who exported this private key and when."

7.10 ECC Future-Proofing

While V1 supports RSA only, the system is designed for ECC support without breaking changes:

• The algorithm enum on the key entity includes placeholder values for ECC_CURVE25519, ECC_P256, and ECC_P384
• The ICryptoProvider interface is algorithm-agnostic — it receives key IDs and resolves the correct BouncyCastle implementation at runtime
• BouncyCastle already supports ECC key generation and OpenPGP operations with elliptic curves
• Adding ECC requires: implementing ECC key generation, adding ECC-specific validation rules, and testing interoperability with common PGP clients (GPG, Kleopatra)

No database migration, interface changes, or job configuration changes will be needed.

---