Evidence integrity and signing

When an auditor pulls your RBAC drift history six months after a run, the only question that matters is whether the record set in front of them is the record set your pipeline actually produced — or something edited afterward to look clean. This section covers the three cryptographic layers that make that question answerable: content hashing so any single record is self-verifying, hash-chaining so the sequence of records cannot be re-ordered or silently truncated, and cryptographic signatures so provenance is bound to a key an auditor can check without trusting your storage.

The failure scenario if you skip this is quiet but fatal to a compliance narrative. A drift log stored as ordinary rows in a table — or as JSON files in a bucket — is trivially mutable by anyone with write access, including the very operator whose access the log is supposed to police. “The database says no unauthorized grants occurred” is worth nothing if the database row could have been updated after the fact. Integrity engineering converts your evidence from an assertion you make into a record an auditor can independently verify, which is the difference between a passed control and a finding.

Prerequisites and scope

This section assumes you already emit a stable evidence record per drift run — the normalized diff, the applied weights, and the routing decision — from a stage upstream of storage. If you do not yet produce those records deterministically, start with the evidence shapes in audit report artifact formats first, because integrity guarantees are only as good as the canonicalization underneath them: two byte-equal serializations of the same logical record must hash identically, every time.

Concretely you need Python 3.11+, the standard-library hashlib (no dependency) for hashing and chaining, and the cryptography library version 42 or newer for Ed25519 (cryptography.hazmat.primitives.asymmetric.ed25519). Record models are pydantic v2. Storage is deliberately out of scope for the crypto layer — the guarantees hold whether the log lands in PostgreSQL, S3 with object lock, or an append-only file — but the write path must be genuinely append-only, which is where this section meets audit trail retention and query.

Core implementation walkthrough

The three layers compose bottom-up: canonicalize, then hash, then chain, then sign. Each step below is independently runnable.

Step 1 — Canonicalize, then hash each record

A content hash is only meaningful if the bytes fed to it are canonical. Serialize with sorted keys and fixed separators so logically identical records always produce identical bytes, then take the SHA-256 hex digest.

import hashlib
import json
from pydantic import BaseModel

class DriftRecord(BaseModel):
    run_id: str
    environment: str
    principal: str
    privilege: str
    object_scope: str
    decision: str          # "alert" | "ledger" | "log"

def canonical_bytes(model: BaseModel) -> bytes:
    """Deterministic bytes: sorted keys, no incidental whitespace."""
    payload = model.model_dump(mode="json")
    return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")

def content_hash(model: BaseModel) -> str:
    return hashlib.sha256(canonical_bytes(model)).hexdigest()

Verify that canonicalization is order-insensitive — the same logical record hashes identically regardless of field construction order:

a = DriftRecord(run_id="r1", environment="prod", principal="svc",
                privilege="SELECT", object_scope="cust.cards", decision="alert")
b = DriftRecord(decision="alert", object_scope="cust.cards", privilege="SELECT",
                principal="svc", environment="prod", run_id="r1")
assert content_hash(a) == content_hash(b)   # canonical form is identical

Step 2 — Chain the records so order is tamper-evident

A per-record hash proves a single record is intact but says nothing about whether records were re-ordered or deleted. Bind each entry to its predecessor by embedding the prior entry’s hash; the first entry links to a fixed genesis hash. Now any deletion, insertion, or re-order breaks the chain at a detectable position. The full build-and-verify walkthrough lives in hash-chaining drift audit logs for tamper evidence; the essential shape is:

GENESIS = "0" * 64

def link(seq: int, prev_hash: str, record: DriftRecord) -> dict:
    body = {"seq": seq, "prev_hash": prev_hash, "record": record.model_dump(mode="json")}
    entry_hash = hashlib.sha256(
        json.dumps(body, sort_keys=True, separators=(",", ":")).encode("utf-8")
    ).hexdigest()
    return {**body, "entry_hash": entry_hash}

chain: list[dict] = []
prev = GENESIS
for i, rec in enumerate([a, b]):
    entry = link(i, prev, rec)
    chain.append(entry)
    prev = entry["entry_hash"]

For large runs, chaining every record individually is fine, but if you need to prove membership of one record in a batch without releasing the whole batch, hash the records into a Merkle tree and chain only the root — same link discipline, cheaper inclusion proofs.

Step 3 — Sign the chain head with Ed25519

Hashing and chaining protect integrity but not provenance: anyone who can rewrite the whole log can also recompute every hash and produce a self-consistent forgery. A signature closes that gap by binding the chain head to a private key the forger does not hold. Sign the final entry hash; publish a detached signature and the public key. The end-to-end key handling and ingest verification are covered in signing compliance manifests with Ed25519.

from cryptography.hazmat.primitives.asymmetric.ed25519 import (
    Ed25519PrivateKey, Ed25519PublicKey,
)
from cryptography.exceptions import InvalidSignature

signing_key = Ed25519PrivateKey.generate()          # persist the private key securely
head = chain[-1]["entry_hash"].encode("utf-8")
signature = signing_key.sign(head)                  # 64-byte detached signature

# Distribute the public key; an auditor verifies without trusting your storage.
pub: Ed25519PublicKey = signing_key.public_key()
try:
    pub.verify(signature, head)                     # returns None on success
    verdict = "PASS"
except InvalidSignature:
    verdict = "FAIL"
assert verdict == "PASS"

The same signing discipline is what routes an approved exception past the drift alarm without a hole in the audit trail — see cryptographic manifest signing for access exceptions.

Idempotency and safety contract

Integrity operations must be pure functions of their inputs. Re-hashing an unchanged record yields the same digest, so re-running the hash or chain-build pass over an existing log is a no-op that either reproduces identical entry hashes or surfaces a mismatch — never a silent rewrite. Treat verification as strictly read-only: it recomputes and compares, and it must never “repair” a broken chain, because a repair is indistinguishable from a cover-up. If verification fails, the correct action is to freeze the log, alert, and investigate, not to re-sign.

Two hard rules keep the contract sound. First, the private signing key never touches the same trust boundary as the log storage; if an attacker who can edit records can also sign, the signature proves nothing. Second, canonicalization is frozen — changing serialization rules (key order, number formatting, timezone encoding) after records are signed silently invalidates every prior signature, so version the canonicalization scheme and record its version inside each entry.

Compliance alignment

The three layers map cleanly onto the audit-integrity language that assessors already use. Content hashing and chaining satisfy the “protect audit information from unauthorized modification” intent of NIST SP 800-53 AU-9 and the audit-log integrity expectations behind SOC 2 CC7.x; the Ed25519 signature adds the non-repudiation and provenance that PCI-DSS Requirement 10.5 asks for when it requires audit trails be protected so they “cannot be altered.” The evidence artifact this section produces is compact and portable: for each run, the canonical record bytes, its chain entry (seq, prev_hash, entry_hash), and a detached signature over the chain head, plus the published public key. An auditor needs nothing from you but those files and the key to reach an independent PASS or FAIL — which is exactly the property that turns a monitored control into an attestable one under the criteria mapped in continuous control monitoring.

Troubleshooting matrix

Related

• Compliance Evidence and Audit Reporting — the parent area this integrity work feeds evidence into
• Audit Trail Retention and Query — the append-only storage and query layer these guarantees sit on top of
• Audit Report Artifact Formats — the canonical evidence shapes that get hashed and signed
• Hash-Chaining Drift Audit Logs for Tamper Evidence — building and verifying the SHA-256 chain in Python
• Signing Compliance Manifests with Ed25519 — key handling, detached signatures, and verification at ingest

Up: Compliance Evidence and Audit Reporting