Skip to content

The table format

TL;DR

qmstrace ledger table writes one JSON document per table, format qmstrace.table/1: what was read, then the columns, then the rows. Columns are named after the project's own item types and found by name; a new key or column keeps /1, so a reader ignores what it does not know. Identical inputs give identical bytes.

The document

Key Holds
format qmstrace.table/1
table matrix, requirements or coverage
project the project's name, from its trace schema
draft a reminder that the table is generated, never an issued revision
schema the trace schema's digest — the rules the table was read under
ledger the ledger's commit and whether its working tree was clean; null outside Git
read each repo read: its name, its location from the ledger, its commit and clean
pinned false: every repo was read from its working tree, not at a pin
violations, gaps the counts ledger check would report
notices everything that could not be verified or was not pinned, and what the table leaves out
reports with --report only: each report read — its repo, label, sha256, entry count and format
columns the columns, in order
rows one object per row, keyed by column name

There is no timestamp and no path of the machine: the commits say what was read. A repo with uncommitted work is marked "clean": false — the table shows its working tree, which no commit holds.

Columns

Each column names itself, the kind of value its cells hold — id, ids (a list, sorted with numbers read as numbers), text, count or verdict — the item type those values are, and the link via which they hang off the row.

Columns follow the project's trace schema. Two names are fixed: the requirement type every table starts from, and the verifies link. An item type that is both a source and a target of verifies is a test case; one that is only a source, an automated test. The other links that count toward coverage, walked from the requirement, give the rest:

  • what the requirement points at — its origin clauses;
  • what points at the requirement — its design elements, the first level of the matrix's rows;
  • what points at those — the components that implement a design element;
  • the requirement fields the schema names under [tables.matrix] fields.

A link that counts toward no coverage — Assumes — is in no table.

The matrix

One row per requirement, design element and test case, requirement-major:

  • a test case's row lists the automated tests that run it, through another test case too;
  • an automated test that verifies directly, with no test case between, has a row of its own;
  • a test case nothing runs still has its row, with no automated test;
  • a test case that verifies the requirement itself repeats on each of its design rows;
  • a requirement with nothing below it still has one row;
  • withdrawn items are in no row, and a notice counts them, as it counts items that reach no requirement.

The cases fixture project's matrix:

{
  "format": "qmstrace.table/1",
  "table": "matrix",
  "project": "CASES",
  "draft": "generated from the ledger — not an issued revision",
  "schema": "sha256:aca979f608c4631fb064128ebd0cb9ea00a9dca3c66d08f0769de2d880373f43",
  "ledger": {
    "commit": "88b8304a05f44a9a1eb25bc7bbe6f8a146e6a67b",
    "clean": true
  },
  "read": [
    {
      "repo": "agent",
      "location": "../agent",
      "commit": "24f6279d9825fe7f941517035047d8a6b98c5562",
      "clean": true
    },
    {
      "repo": "ledger",
      "location": ".",
      "commit": "88b8304a05f44a9a1eb25bc7bbe6f8a146e6a67b",
      "clean": true
    },
    {
      "repo": "service",
      "location": "../service",
      "commit": "e07096c462636cbc981538a88e669fb2ec6467ca",
      "clean": true
    }
  ],
  "pinned": false,
  "violations": 0,
  "gaps": 1,
  "notices": [
    "baselines: judged from each repo's reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches",
    "baselines in ledger: no matching reachable local tag — nothing frozen",
    "change requests: change-requests (1 readable; 0 malformed) — working tree, not pinned"
  ],
  "columns": [
    {
      "name": "origin_clause",
      "cell": "ids",
      "type": "origin_clause",
      "via": "Origin"
    },
    {
      "name": "requirement",
      "cell": "id",
      "type": "requirement",
      "via": null
    },
    {
      "name": "title",
      "cell": "text",
      "type": null,
      "via": null
    },
    {
      "name": "design_element",
      "cell": "id",
      "type": "design_element",
      "via": "Satisfies"
    },
    {
      "name": "component",
      "cell": "ids",
      "type": "component",
      "via": "Implements"
    },
    {
      "name": "test_case",
      "cell": "id",
      "type": "test_case",
      "via": "verifies"
    },
    {
      "name": "automated_test",
      "cell": "ids",
      "type": "automated_test",
      "via": "verifies"
    }
  ],
  "rows": [
    {
      "origin_clause": [
        "AGR-1"
      ],
      "requirement": "REQ-SVC-01",
      "title": "A call carries a key",
      "design_element": "DSN-SVC-01",
      "component": [
        "agent:src/Agent/KeyCheck.cs",
        "service:src/service/keys.py"
      ],
      "test_case": "TC-SVC-01.1",
      "automated_test": [
        "agent:tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_NoKey_IsFalse",
        "service:tests/unit/test_api.py::test_a_call_with_a_wrong_key_is_refused",
        "service:tests/unit/test_api.py::test_a_call_without_a_key_is_refused"
      ]
    },
    {
      "origin_clause": [
        "AGR-1"
      ],
      "requirement": "REQ-SVC-01",
      "title": "A call carries a key",
      "design_element": "DSN-SVC-01",
      "component": [
        "agent:src/Agent/KeyCheck.cs",
        "service:src/service/keys.py"
      ],
      "test_case": "TC-SVC-01.2",
      "automated_test": [
        "agent:tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_AnyKey_DecidesAlone",
        "service:tests/unit/test_api.py::test_the_key_check_runs_before_any_handler"
      ]
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": "TC-SVC-02.1",
      "automated_test": [
        "service:tests/integration/test_store.py::test_a_restart_keeps_every_field"
      ]
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": "TC-SVC-02.2",
      "automated_test": []
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": null,
      "automated_test": [
        "service:tests/integration/test_store.py::test_a_record_survives_a_restart"
      ]
    }
  ]
}

Verdicts

Given --report REPO=FILE, each naming the repo whose tests it ran, the matrix gains one column per level, verdict_<level>, and an overall verdict. A verdict cell holds the worst verdict of the row's automated tests there — fail, no-result, skip, pass — and every report their runs came from, by repo, label and sha256. A level the row has no test at is null; a row no automated test runs is untested.

The levels are the automated tests' levels from the schema, then those of a test-case field that [tables.matrix] level names. A test's level is where it sits when its type has levels; only a test whose type has none takes its test case's level. Where the two differ, where the test sits wins, and a notice counts it.

The reports are read as one, whatever their order. A report inside the repo it names is labelled by its path from that repo's root; one outside, by its file name alone. Reports that match no automated test of the repo they name are refused, exit 2. The cases project's matrix with the agent's TRX file and a JUnit report of the service:

{
  "format": "qmstrace.table/1",
  "table": "matrix",
  "project": "CASES",
  "draft": "generated from the ledger — not an issued revision",
  "schema": "sha256:aca979f608c4631fb064128ebd0cb9ea00a9dca3c66d08f0769de2d880373f43",
  "ledger": {
    "commit": "88b8304a05f44a9a1eb25bc7bbe6f8a146e6a67b",
    "clean": true
  },
  "read": [
    {
      "repo": "agent",
      "location": "../agent",
      "commit": "24f6279d9825fe7f941517035047d8a6b98c5562",
      "clean": false
    },
    {
      "repo": "ledger",
      "location": ".",
      "commit": "88b8304a05f44a9a1eb25bc7bbe6f8a146e6a67b",
      "clean": true
    },
    {
      "repo": "service",
      "location": "../service",
      "commit": "e07096c462636cbc981538a88e669fb2ec6467ca",
      "clean": false
    }
  ],
  "pinned": false,
  "violations": 0,
  "gaps": 1,
  "notices": [
    "baselines: judged from each repo's reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches",
    "baselines in ledger: no matching reachable local tag — nothing frozen",
    "change requests: change-requests (1 readable; 0 malformed) — working tree, not pinned",
    "Verdicts are matched against the working trees — each report's commit NOT VERIFIED."
  ],
  "reports": [
    {
      "repo": "agent",
      "label": "TestResults/unit.trx",
      "sha256": "70d39941e7178b376f562b191659112a90841df4ca48873144d64a3e4df7c052",
      "entries": 3,
      "format": "trx"
    },
    {
      "repo": "service",
      "label": "report.xml",
      "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297",
      "entries": 5,
      "format": "junit"
    }
  ],
  "columns": [
    {
      "name": "origin_clause",
      "cell": "ids",
      "type": "origin_clause",
      "via": "Origin"
    },
    {
      "name": "requirement",
      "cell": "id",
      "type": "requirement",
      "via": null
    },
    {
      "name": "title",
      "cell": "text",
      "type": null,
      "via": null
    },
    {
      "name": "design_element",
      "cell": "id",
      "type": "design_element",
      "via": "Satisfies"
    },
    {
      "name": "component",
      "cell": "ids",
      "type": "component",
      "via": "Implements"
    },
    {
      "name": "test_case",
      "cell": "id",
      "type": "test_case",
      "via": "verifies"
    },
    {
      "name": "automated_test",
      "cell": "ids",
      "type": "automated_test",
      "via": "verifies"
    },
    {
      "name": "verdict_unit",
      "cell": "verdict",
      "type": null,
      "via": null
    },
    {
      "name": "verdict_integration",
      "cell": "verdict",
      "type": null,
      "via": null
    },
    {
      "name": "verdict_system",
      "cell": "verdict",
      "type": null,
      "via": null
    },
    {
      "name": "verdict",
      "cell": "verdict",
      "type": null,
      "via": null
    }
  ],
  "rows": [
    {
      "origin_clause": [
        "AGR-1"
      ],
      "requirement": "REQ-SVC-01",
      "title": "A call carries a key",
      "design_element": "DSN-SVC-01",
      "component": [
        "agent:src/Agent/KeyCheck.cs",
        "service:src/service/keys.py"
      ],
      "test_case": "TC-SVC-01.1",
      "automated_test": [
        "agent:tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_NoKey_IsFalse",
        "service:tests/unit/test_api.py::test_a_call_with_a_wrong_key_is_refused",
        "service:tests/unit/test_api.py::test_a_call_without_a_key_is_refused"
      ],
      "verdict_unit": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "agent",
            "label": "TestResults/unit.trx",
            "sha256": "70d39941e7178b376f562b191659112a90841df4ca48873144d64a3e4df7c052"
          },
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      },
      "verdict_integration": null,
      "verdict_system": null,
      "verdict": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "agent",
            "label": "TestResults/unit.trx",
            "sha256": "70d39941e7178b376f562b191659112a90841df4ca48873144d64a3e4df7c052"
          },
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      }
    },
    {
      "origin_clause": [
        "AGR-1"
      ],
      "requirement": "REQ-SVC-01",
      "title": "A call carries a key",
      "design_element": "DSN-SVC-01",
      "component": [
        "agent:src/Agent/KeyCheck.cs",
        "service:src/service/keys.py"
      ],
      "test_case": "TC-SVC-01.2",
      "automated_test": [
        "agent:tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_AnyKey_DecidesAlone",
        "service:tests/unit/test_api.py::test_the_key_check_runs_before_any_handler"
      ],
      "verdict_unit": {
        "verdict": "fail",
        "reports": [
          {
            "repo": "agent",
            "label": "TestResults/unit.trx",
            "sha256": "70d39941e7178b376f562b191659112a90841df4ca48873144d64a3e4df7c052"
          },
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      },
      "verdict_integration": null,
      "verdict_system": null,
      "verdict": {
        "verdict": "fail",
        "reports": [
          {
            "repo": "agent",
            "label": "TestResults/unit.trx",
            "sha256": "70d39941e7178b376f562b191659112a90841df4ca48873144d64a3e4df7c052"
          },
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      }
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": "TC-SVC-02.1",
      "automated_test": [
        "service:tests/integration/test_store.py::test_a_restart_keeps_every_field"
      ],
      "verdict_unit": null,
      "verdict_integration": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      },
      "verdict_system": null,
      "verdict": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      }
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": "TC-SVC-02.2",
      "automated_test": [],
      "verdict_unit": null,
      "verdict_integration": null,
      "verdict_system": null,
      "verdict": {
        "verdict": "untested",
        "reports": []
      }
    },
    {
      "origin_clause": [
        "AGR-2"
      ],
      "requirement": "REQ-SVC-02",
      "title": "Records survive a restart",
      "design_element": "DSN-SVC-02",
      "component": [
        "service:src/service/store.py"
      ],
      "test_case": null,
      "automated_test": [
        "service:tests/integration/test_store.py::test_a_record_survives_a_restart"
      ],
      "verdict_unit": null,
      "verdict_integration": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      },
      "verdict_system": null,
      "verdict": {
        "verdict": "pass",
        "reports": [
          {
            "repo": "service",
            "label": "report.xml",
            "sha256": "12188e631ee19a5ef3fee6d5708580a06aaca3c1846675b8dfeb1ac5fa02f297"
          }
        ]
      }
    }
  ]
}

The requirements register and the coverage summary

The register has one row per requirement, withdrawn ones included: its title, its origins, the fields its type declares, its Change and Status, its repo, and where it is declared.

The coverage summary has one row per tier that reaches a requirement — its design, its test cases, its automated tests. Each row names the rules that ask for the tier, how many requirements those rules select (of — a rule's where narrows it), how many are covered, and which are not. An automated test reaches a requirement directly or through test cases, never through a design element. Coverage counts links, not passing tests.

Versioning

A new key, column, table, cell kind or notice keeps qmstrace.table/1: find columns by name, and ignore what you do not know. Removing or renaming anything, changing a cell's JSON type, the row grain or the order, or what clean, pinned or draft mean, makes qmstrace.table/2.

The document is UTF-8. On a console that cannot print UTF-8, write it with --out instead.