Test Coverage and Compliance#

This flake includes an integration‑testing and compliance layer that lets you:

• Apply Secure Controls Framework (SCF)-style controls as NixOS modules
• Run those controls in NixOS VMs using the NixOS test framework
• Aggregate all applicable controls into a single compliance module
• Execute the resulting checks via nix flake check or targeted nix build calls

This document explains:

• What the test/compliance layer provides
• How to run it
• How it is wired together in the flake
• How to add or extend controls and tests

1. What You Get#

At a high level, this flake provides:

• Compliance data at flake level

Exposed under flake.compliance:

• controls – all defined SCF‑style controls
• applicable – controls filtered by their .applicable flag
• configs – per‑control configToApply fragments

• merged – a single merged NixOS config from all applicable controls

• A compliance NixOS module

Exposed as flake.nixosModules.compliance, which:

• Imports the merged configuration

• Exposes metadata about which controls are applied via options under tsunaminoai.compliance.*

• A compliance VM test

Exposed via perSystem.checks.compliance, which:

• Boots a NixOS VM with the compliance module applied
• Runs all control‑defined checks inside the VM

• Is runnable with nix flake check or nix build .#checks.<system>.compliance

• Additional security VM tests (on this branch)

There are separate NixOS tests (e.g. SSH, firewall, permissions) that validate host security posture using pkgs.testers.runNixOSTest.

2. Quick Start#

2.1. Run all flake checks#

From the flake root:

# Run all checks, including compliance and VM security tests
nix flake check

This will:

• Evaluate the flake
• Build and run each check under perSystem.checks.*
• Boot the compliance VM test
• Boot other VM tests (SSH, firewall, permissions, etc.) if defined

2.2. Run only the compliance VM test#

# Replace x86_64-linux with your system if different
nix build '.#checks.x86_64-linux.compliance' -L

This:

• Builds a NixOS VM
• Applies the compliance NixOS module (all merged controls)
• Runs all defined SCF control checks inside the VM
• Fails if any check fails

2.3. Run a specific security VM test#

If you have additional security tests (e.g. SSH hardening):

nix build '.#checks.x86_64-linux.security-vm-ssh' -L
nix build '.#checks.x86_64-linux.security-vm-firewall' -L
nix build '.#checks.x86_64-linux.security-vm-permissions' -L

Each of these:

• Boots a minimal or host‑like NixOS VM
• Runs focused assertions (e.g. SSH config, firewall rules, file permissions)
• Produces human‑readable ✓ output for each assertion

3. Architecture#

3.1. Controls and example.nix#

Controls are defined in a separate file (for example example.nix) which returns a list of control attribute sets.

Each control is created with a helper such as mkControl and typically has:

• name – human‑readable control name
• external_id – SCF‑style identifier (e.g. SCF-LOC-SSH-01)
• applicable – boolean; if false, the control is ignored in this environment
• group – grouping/category
• configToApply – a NixOS module fragment (attrset) that enforces the control
• alerts – metadata for alerting systems
• monitors – metadata for external monitoring
• checks – list of strings, each string is a Python snippet to run in the VM test
• metadata – additional SCF/compliance metadata

Example (simplified):

{ lib }:
with lib; [
  (mkControl {
    name = "Local SSH Configuration - Key-Based Authentication";
    external_id = "SCF-LOC-SSH-01";
    applicable = true;
    group = "Local System Hardening";

    configToApply = {
      services.openssh = {
        enable = true;
        settings = {
          PasswordAuthentication = false;
          PubkeyAuthentication   = true;
          PermitRootLogin        = "prohibit-password";
        };
      };
    };

    checks = [
      ''
        machine.succeed("systemctl is-active sshd")
        print("✓ SCF-LOC-SSH-01: SSH service is active")
      ''
      ''
        machine.succeed("grep 'PasswordAuthentication no' /etc/ssh/sshd_config")
        print("✓ SCF-LOC-SSH-01: Password authentication is disabled")
      ''
    ];

    metadata = {
      description = "Verify SSH is configured for key-based authentication only";
      control_type = "Preventive";
      implementation_status = "Implemented";
    };
  })
]

3.2. Compliance aggregation module#

There is a flake module (call it the “compliance module”) that:

1. Loads all controls

l = lib // import ./lib.nix { inherit lib; };

allControls = import ./example.nix { lib = l; };

2. Filters to applicable controls

applicableControls = builtins.filter (c: c.applicable) allControls;

3. Collects configs and merges them

configsToApply = map (c: c.configToApply) applicableControls;

mergedConfig = lib.foldl' lib.recursiveUpdate {} configsToApply;

4. Flake‑level exports

flake.compliance = {
  controls   = allControls;
  applicable = applicableControls;
  configs    = configsToApply;
  merged     = mergedConfig;
};

5. NixOS module for compliance

flake.nixosModules.compliance = { lib, ... }: {
  imports = [
    mergedConfig
  ];

  options.tsunaminoai.compliance.controlsApplied = lib.mkOption {
    type = lib.types.listOf lib.types.attrs;
    default = applicableControls;
    description = "List of compliance controls applied to this system";
    readOnly = true;
  };
};

This lets any NixOS configuration use the compliance module:

{
  imports = [
    # ...
    self.nixosModules.compliance
  ];
}

4. The Compliance VM Test#

The compliance test is provided via perSystem.checks.compliance.

Conceptually, it:

• Boots a NixOS VM
• Applies the compliance module (which includes mergedConfig)
• Runs every control’s checks snippets

4.1. Flattening checks#

Each control has a checks list (list of strings). They are combined like this:

# Extract and flatten all .checks from applicableControls
allComplianceChecks =
  lib.concatMap (c: c.checks) applicableControls;

Now allComplianceChecks is a list of Python snippets ready to be appended to the testScript.

4.2. Test definition#

The test is defined using pkgs.testers.runNixOSTest:

perSystem = { system, pkgs, config, ... }: {
  checks.compliance = pkgs.testers.runNixOSTest {
    name = "SCF Compliance Check";

    nodes.machine1 = { config, pkgs, ... }: {
      imports = [
        self.nixosModules.compliance
      ];
      system.stateVersion = "25.11";
    };

    testScript = { nodes, ... }:
      builtins.concatStringsSep "\n" (
        [
          ''
            machine.wait_for_unit("default.target")
            print("✓ System booted with compliance module")
          ''
        ]
        ++ allComplianceChecks
      );
  };
};

Key points:

• nodes.machine1 defines the VM’s NixOS config.
• imports = [ self.nixosModules.compliance ] applies all merged controls.
• testScript is built by concatenating a bootstrap snippet and all control checks with \n.

5. Additional Security VM Tests#

On this branch (e.g. 133-test-coverage) you may also see additional checks under perSystem.checks.*, such as:

• security-vm-ssh – SSH hardening tests
• security-vm-firewall – firewall tests
• security-vm-permissions – critical file permission tests
• And potentially host‑specific tests (e.g. for mokou)

These use the same underlying mechanism:

security-vm-ssh = pkgs.testers.runNixOSTest {
  name = "security-ssh-hardening";

  nodes.machine = { config, pkgs, ... }: {
    # Minimal NixOS or host‑like modules
    services.openssh.enable = true;
    # ...
  };

  testScript = ''
    machine.wait_for_unit("multi-user.target")
    machine.wait_for_unit("sshd.service")

    machine.succeed("grep 'PasswordAuthentication no' /etc/ssh/sshd_config")
    print("✓ SSH password auth disabled")
    # more assertions…
  '';
};

You run them exactly like the compliance test:

nix build '.#checks.x86_64-linux.security-vm-ssh' -L

6. How to Add a New Control#

1. Open your controls file (e.g. example.nix).
2. Add a new mkControl block:

• Set applicable = true to include it.
• Define configToApply with your NixOS fragment.
• Define checks as a list of Python snippets operating on machine.

3. Rebuild and run tests:

nix flake check
# or
nix build '.#checks.x86_64-linux.compliance' -L

4. Inspect output – you should see ✓ lines for the new control’s checks.

If you want a control to be present in the flake metadata but not actually enforced/check‑run in a given environment, set applicable = false.

7. How to Add a New VM Security Test#

1. Create or update a checks file (e.g. under modules/flake/checks/).

2. Define a new test under perSystem.checks:

perSystem = { pkgs, ... }: {
  checks.security-vm-myservice = pkgs.testers.runNixOSTest {
    name = "security-myservice";

    nodes.machine = { config, pkgs, ... }: {
      # Import base host modules or minimal config
      # imports = [ ... ];

      services.myservice.enable = true;
    };

    testScript = ''
      machine.wait_for_unit("multi-user.target")
      machine.wait_for_unit("myservice.service")
      machine.succeed("systemctl is-active myservice")
      print("✓ myservice is active")
    '';
  };
};

3. Run it:

nix build '.#checks.x86_64-linux.security-vm-myservice' -L

8. CI / Automation#

Because all of this is wired through perSystem.checks, it works naturally with:

• nix flake check locally
• CI systems that run nix flake check or build specific checks.<system>.* attributes

A typical CI setup might:

• Run nix flake check on every push / PR
• Optionally build each VM test individually for better error isolation
• Fail the pipeline if any compliance/security test fails

9. Summary#

This flake’s test‑coverage and compliance layer gives you:

• Declarative SCF‑style controls as Nix expressions
• Automatic aggregation into a single compliance module
• VM‑based validation of those controls via NixOS tests
• Additional host/security tests for SSH, firewall, permissions, etc.
• A clean path to integrate all of this into nix flake check and CI

Use:

• nix flake check for the full suite
• nix build '.#checks.<system>.compliance' -L for targeted compliance testing
• Add or modify controls in example.nix and they will automatically be included in the compliance VM test.

^1 ^1 ^1 ^1 ^1 ^1 ^1 ^1 ^1