Test Coverage and Compliance#

This flake includes an SCF-driven compliance layer that lets you:

• Apply Secure Controls Framework (SCF) controls as NixOS modules
• Attest each control against typed assessment objectives (the SCF PPTDF model)
• Run the runtime-verifiable assessment objectives inside a NixOS VM using the NixOS test framework
• Emit a structured evidence trail and render it to a Markdown attestation report

This document explains:

• What the compliance layer provides
• How to run it
• How it is wired together in the flake
• How to add or extend controls

The module lives under modules/compliance/:

• default.nix – flake wiring: control registry, flake-level exports, the NixOS test, the attestation-report package, and the compliance NixOS module
• options.nix – the tsunaminoai.compliance.* option surface and the read-only compliance.evidence output
• lib.nix – helpers (mkControl, mkAssessmentObjective, filterControlsByFrameworks)
• scf-data.nix – the authoritative SCF crosswalk: authoritativeSources (FDIs) and controls (per-control framework references and assessment objectives)
• controls/<SCF-ID>.nix – one file per implemented control

1. What You Get#

At a high level, this flake provides:

• Compliance data at flake level

Exposed under flake.compliance:

• scfData – the imported scf-data.nix crosswalk (authoritativeSources + controls)
• controls – all 8 implemented control attribute sets (allControlImpls)
• testFrameworks – the FDIs the flake-level NixOS test attests against

• testApplicable – the subset of controls required by testFrameworks

• A compliance NixOS module

Exposed as flake.nixosModules.compliance, which:

• Merges every control’s configToApply into the host configuration (gated behind tsunaminoai.compliance.enable)
• Declares the tsunaminoai.compliance.* options

• Exposes a read-only compliance.evidence attestation trail

• A compliance VM test

Exposed via perSystem.checks.compliance, which:

• Boots a NixOS VM with the compliance module enabled
• Runs the Technology-tier assessment-objective check snippets from the framework-applicable controls

• Is runnable with nix flake check or nix build .#checks.<system>.compliance

• An attestation report package

Exposed via perSystem.packages.attestation-report, which renders the evidence trail to Markdown via modules/compliance/scripts/render-report.py.

2. Quick Start#

2.1. Run all flake checks#

From the flake root:

# Runs every check under perSystem.checks.*, including the compliance VM test
nix flake check

The compliance check is the only compliance-related VM test wired into nix flake check. (The cache and graphical checks are currently commented out in modules/flake/checks/default.nix, pending issues #165 and #162.)

2.2. Run only the compliance VM test#

# Replace x86_64-linux with your system if different
nix build '.#checks.x86_64-linux.compliance' -L

This:

• Builds a NixOS VM with tsunaminoai.compliance.enable = true
• Applies the merged controls from the compliance NixOS module
• Runs the Technology-tier assessment-objective check snippets for the framework-applicable controls
• Fails if any assertion fails

2.3. Render the attestation report#

nix build .#attestation-report && cat result

This renders the SCF evidence trail (control IDs, framework crosswalks, per-objective status and evidence prose) to Markdown.

2.4. Inspect a host’s evidence trail#

For any NixOS host that imports the compliance module with enable = true:

nix eval .#nixosConfigurations.<host>.config.compliance.evidence --json | jq .

3. Architecture#

3.1. Controls live one-per-file under controls/#

Each control is a separate file under modules/compliance/controls/, built with the mkControl helper from lib.nix. The implemented controls are:

default.nix imports these files directly into allControlImpls:

allControlImpls = [
  (import ./controls/CFG-02.nix {lib = l;})
  (import ./controls/CFG-07.nix {lib = l;})
  (import ./controls/NET-02.nix {lib = l;})
  (import ./controls/LGO-02.nix {lib = l;})
  (import ./controls/LGO-06.nix {lib = l;})
  (import ./controls/VPM-03.nix {lib = l;})
  (import ./controls/IAC-02.nix {lib = l;})
  (import ./controls/IAC-06.nix {lib = l;})
];

nix eval --file modules/compliance/example.nix \
  --apply 'ctrls: map (c: c.scf_id) ctrls'

Control schema (mkControl)#

Each control is created with mkControl (lib.nix) and has:

• name – human-readable control name
• scf_id – the real SCF control identifier (e.g. CFG-02)
• configToApply – a NixOS module fragment (attrset) that enforces the control
• assessmentObjectives – a list of typed assessment-objective records (see below)
• alerts – optional metadata for alerting systems
• monitors – optional metadata for external monitoring
• metadata – control metadata, including a frameworks crosswalk (FDI → reference string)

Assessment objectives (mkAssessmentObjective)#

Each entry models one SCF assessment objective in the PPTDF (Process / People / Technology / Data / Facilities) framework:

• id – objective id (e.g. CFG-02_A03)
• text – the SCF assessment-objective text
• pptdf – the tier ("Process", "People", "Technology", "Data", "Facilities")
• rigor – an integer rigor level
• check (optional) – a Python snippet (run against machine in the NixOS test). Used for Technology/Data-tier objectives that are runtime-verifiable.
• evidence (optional) – attestation prose for Process/People-tier objectives.
• status (optional) – overridden at build time ("automatable" when a check is present, otherwise "configured").

Example (the real CFG-02 control, abridged):

{lib}:
lib.mkControl {
  name = "Secure Baseline Configurations — SSH Hardening";
  scf_id = "CFG-02";

  configToApply = {
    services.openssh = {
      enable = true;
      settings = {
        PasswordAuthentication = false;
        PubkeyAuthentication = true;
        PermitRootLogin = "prohibit-password";
        StrictModes = true;
        X11Forwarding = false;
        PermitEmptyPasswords = false;
      };
    };
    environment.etc."ssh/sshd_config" = {
      mode = "0600";
      user = "root";
      group = "root";
    };
  };

  assessmentObjectives = [
    # Process tier: build-time attestation (no live check)
    (lib.mkAssessmentObjective {
      id = "CFG-02_A01";
      text = "a current baseline configuration ... is developed and documented.";
      pptdf = "Process";
      rigor = 1;
      status = "configured";
      evidence = "SSH baseline declared in NixOS module CFG-02.nix (pure, reproducible, git-tracked)";
    })

    # Technology tier: runtime NixOS test assertion
    (lib.mkAssessmentObjective {
      id = "CFG-02_A03";
      text = "security configuration settings ... are enforced.";
      pptdf = "Technology";
      rigor = 1;
      evidence = "sshd_config enforces PasswordAuthentication no, PubkeyAuthentication yes, ...";
      check = ''
        machine.succeed("systemctl is-active sshd")
        print("PASS CFG-02_A03: sshd service is active")
        machine.succeed("grep -E '^PasswordAuthentication no$' /etc/ssh/sshd_config")
        print("PASS CFG-02_A03: PasswordAuthentication is disabled")
      '';
    })
  ];

  metadata = {
    scf_domain = "Configuration Management";
    control_type = "Preventive";
    implementation_status = "Implemented";
    frameworks = {
      "general-nist-800-53-r5" = "CM-2, CM-6, CM-7";
      "general-pci-dss-4-0-1" = "1.1, 1.2.1, 2.2, 2.2.1, 8.3.2, 8.5";
      "us-federal-cmmc-2-l2" = "CM.L2-3.4.1, CM.L2-3.4.2";
      "general-cis-csc-8-1" = "4.1, 4.2";
      "general-iso-27001-2022" = "A.8.9";
    };
  };
}

3.2. Framework selection and aggregation#

Selection is driven by the SCF crosswalk in scf-data.nix, not by any per-control .applicable flag. The filterControlsByFrameworks helper returns the controls whose frameworks.<FDI> reference is non-empty for any selected framework:

# Controls required by the test frameworks (used to build the check test).
testFrameworks = ["general-pci-dss-4-0-1" "general-nist-800-53-r5"];
testRequiredIds = builtins.attrNames (l.filterControlsByFrameworks scfData.controls testFrameworks);
testApplicable = builtins.filter (c: builtins.elem c.scf_id testRequiredIds) allControlImpls;

The flake-level NixOS test only runs the framework-applicable controls’ Technology-tier checks:

testChecks =
  lib.concatMap
    (ctrl: map (ao: ao.check) (builtins.filter (ao: ao ? check) ctrl.assessmentObjectives))
    testApplicable;

mergedNixosConfig = lib.foldl' lib.recursiveUpdate {} (map (c: c.configToApply) allControlImpls);

3.3. Flake-level exports#

flake.compliance = {
  scfData = scfData;          # imported scf-data.nix crosswalk
  controls = allControlImpls; # all 8 control implementations
  testFrameworks = testFrameworks;
  testApplicable = testApplicable;
};

3.4. The compliance NixOS module#

flake.nixosModules.compliance declares the options (via ./options.nix), merges all controls, and emits the evidence trail — all gated behind lib.mkIf cfg.enable:

flake.nixosModules.compliance = { config, lib, ... }: let
  cfg = config.tsunaminoai.compliance;
  mergedNixosConfig = lib.foldl' lib.recursiveUpdate {} (map (c: c.configToApply) allControlImpls);
  evidenceTrail = map (ctrl: {
    scf_id = ctrl.scf_id;
    name = ctrl.name;
    domain = ctrl.metadata.scf_domain or "Unknown";
    frameworks_covered = /* FDIs with a non-empty crosswalk in scf-data.nix */;
    framework_references = ctrl.metadata.frameworks or {};
    config_applied = ctrl.configToApply;
    assessment_objectives = /* AOs with derived status */;
  }) allControlImpls;
in {
  imports = [./options.nix];
  config = lib.mkIf cfg.enable (mergedNixosConfig // {
    compliance.evidence = evidenceTrail;
  });
};

3.5. Options#

The module declares the following options (see options.nix):

Enabling compliance on a host looks like:

{
  imports = [
    # ...
    self.nixosModules.compliance
  ];

  tsunaminoai.compliance = {
    enable = true;
    frameworks = ["general-nist-800-53-r5" "general-pci-dss-4-0-1"];
  };
}

4. The Compliance VM Test#

The compliance test is provided via perSystem.checks.compliance and is defined using pkgs.testers.runNixOSTest:

perSystem = {pkgs, ...}: {
  checks.compliance = pkgs.testers.runNixOSTest {
    name = "SCF Compliance Check — CFG-02, CFG-07, NET-02, LGO-02, LGO-06, VPM-03, IAC-02, IAC-06";
    nodes.machine = {
      imports = [self.nixosModules.compliance];
      tsunaminoai.compliance = {
        enable = true;
        frameworks = testFrameworks;
      };
      system.stateVersion = "26.05";
    };
    testScript = ''
      machine.wait_for_unit("default.target")
      print("System booted with SCF compliance module (frameworks: ...)")
      ${builtins.concatStringsSep "\n" testChecks}
      print("All SCF assessment objective checks passed")
    '';
  };
};

Key points:

• The single VM node is named machine.
• The node sets tsunaminoai.compliance.enable = true (required — see §3.4) and frameworks = testFrameworks.
• testScript boots the VM, then runs every framework-applicable Technology-tier assessment-objective check snippet (testChecks) against machine.

There is no longer any flat per-control checks list, and there are no separate security-vm-* checks — SSH, firewall, file-permission, audit, and similar assertions are assessment-objective check snippets inside the relevant controls (CFG-02, NET-02, LGO-02, etc.) and are all executed by this single compliance test.

5. The Attestation Report#

perSystem.packages.attestation-report renders the evidence trail for the framework-applicable controls to a self-contained Markdown document:

packages.attestation-report =
  pkgs.runCommand "attestation-report.md" {
    buildInputs = [pkgs.python3];
    passAsFile = ["evidenceJson"];
    evidenceJson = reportEvidenceJson;
  } ''
    python3 ${./scripts/render-report.py} "$evidenceJsonPath" > $out
  '';

Build and view it with:

nix build .#attestation-report && cat result

The renderer (modules/compliance/scripts/render-report.py) walks each control’s assessment objectives, deriving status = "automatable" for objectives that carry a check and status = "configured" for Process/People-tier objectives attested by their evidence prose.

6. How to Add a New Control#

To do it by hand:

1. Create modules/compliance/controls/<SCF-ID>.nix. Return a single mkControl record:

{lib}:
lib.mkControl {
  name = "...";
  scf_id = "<SCF-ID>";
  configToApply = { /* NixOS fragment that enforces the control */ };
  assessmentObjectives = [
    (lib.mkAssessmentObjective {
      id = "<SCF-ID>_A01";
      text = "...";
      pptdf = "Technology";       # or Process / People / Data / Facilities
      rigor = 1;
      check = ''machine.succeed("...")'';  # Technology/Data tier
      # evidence = "...";                   # Process/People tier
    })
  ];
  metadata = {
    scf_domain = "...";
    frameworks = {
      "general-nist-800-53-r5" = "...";
      # other FDIs the control maps to
    };
  };
}

2. Register it in allControlImpls in modules/compliance/default.nix:

allControlImpls = [
  # ...
  (import ./controls/<SCF-ID>.nix {lib = l;})
];

3. Add the crosswalk entry to scf-data.nix. Add the control under controls (with its frameworks references and assessment objectives) so filterControlsByFrameworks and the frameworks_covered evidence field pick it up.

4. Rebuild and run the test:

nix build '.#checks.x86_64-linux.compliance' -L

5. Inspect output – you should see PASS <SCF-ID>_... lines for the new control’s Technology-tier objectives (when the control is referenced by testFrameworks).

7. CI / Automation#

Because the compliance test is wired through perSystem.checks.compliance, it works naturally with:

• nix flake check locally
• CI systems that run nix flake check or build .#checks.<system>.compliance directly

A typical CI setup might:

• Run nix flake check on every push / PR
• Build .#checks.x86_64-linux.compliance -L for better error isolation
• Build .#attestation-report and publish the rendered Markdown as an artifact
• Fail the pipeline if any compliance assertion fails

8. Summary#

This flake’s compliance layer gives you:

• Declarative SCF controls as Nix expressions, one file per control under controls/
• Typed assessment objectives (PPTDF) with runtime check snippets or evidence prose
• Framework-driven selection via the scf-data.nix crosswalk and filterControlsByFrameworks
• VM-based validation of the runtime-verifiable objectives via a single NixOS test
• A structured evidence trail (compliance.evidence) and a rendered attestation report

Use:

• nix flake check for the full suite
• nix build '.#checks.<system>.compliance' -L for targeted compliance testing
• nix build .#attestation-report && cat result for the rendered evidence report
• The scf-add-control skill (or §6) to add a new control

^1 ^1 ^1 ^1 ^1 ^1 ^1 ^1 ^1