qmstrace results¶

TL;DR

qmstrace results REPORT... matches pytest JUnit reports or TRX files, read as one, to scanned automated tests, and follows their verifies links to requirements — directly, or through the test cases they run. It shows requirement verdicts, the test cases on the way, missing results, shared tests and unmatched entries. Exit 0 means a results view was produced, even if tests failed. The report's commit is not verified.

Synopsis¶

Usage: qmstrace results [OPTIONS] {REPORT...}

  Match reports to scanned tests and their explicit verifies links.

  Each report is pytest JUnit XML or a TRX file (dotnet test --logger trx); its
  root element decides which. Several reports are read as one, in any order: a
  run in two of them counts once, with its worst verdict. Informational: failing
  tests and partial matches still exit 0. Unmatched entries are listed; reports
  with no matches exit 2. Relative report paths start at -C or the current
  directory, not the discovered repo root. A report outside the repo is shown by
  its file name. The reports' commit is not verified.

Arguments:
  REPORT...  Pytest JUnit XML reports or TRX files, read as one —
             TestResults/*.trx.  [required]

Options:
  -C PATH  Run as if qmstrace had been started in PATH, as with git -C. Before
           or after the command.
  --help   Show this message and exit.

A relative report path starts at the invocation directory, or the directory supplied with -C, even when qmstrace discovers the repository above it. -C works before or after the command. An explicitly supplied report is read even when Git ignores it or it is outside the repository. A report is shown by its path from the repository root, or, outside the repository, by its file name alone; two reports that would be shown alike are refused, as is a report given twice.

Several reports¶

Several reports are read as one, so qmstrace results TestResults/*.trx reads each test project's file in one run. A run reported in two files is one run: it keeps the worst verdict and counts once as a duplicate, as a repeated entry within one file does. The order the reports are given in changes nothing. With more than one report, the header lists each one with its entry count, so a missing or truncated file is visible, and every run that did not pass names the reports it was in. Each run also keeps the sha256 of its reports' bytes. One report prints the same view as before.

Matching and counts¶

The report's root element decides how it is read: testsuite or testsuites is pytest JUnit XML, and TeamTest's TestRun is a TRX file. Any other root is refused, as are DTDs, entities and external references.

JUnit. The reader accepts pytest's default output in either xunit1 or xunit2 format, including nested suites and XML namespaces. Matching is exact: the report's classname and test name without parameters must identify one scanned pytest test. Class methods retain their class names. Parameter IDs may contain brackets.

TRX, as dotnet test --logger trx writes it. Matching is exact on the class and method the result's test definition names: the class namespace-qualified, a nested class joined to its outer class with +. The DisplayName plays no part in matching, so rewording a test does not unmatch it; renaming or moving its method does. A result's name is its testName, so each row of a [Theory] is a run of one automated test. Only the class, the method, the name, the outcome and the first line of an error message are read: a TRX file's paths, machine and user names, output, stack traces and attachments are never shown. With several test projects, dotnet test writes a TRX for each: give results every one of them.

Custom class-name prefixes, identity collisions and entries with missing identity attributes are reported; qmstrace does not guess a match.

An automated test is a scanned test function or test method. A reported run is a distinct full (classname, name) identity, including parameters. Duplicate XML entries for the same full identity are combined using the worst verdict and counted explicitly. This includes pytest's separate entries for a body failure and teardown error. Duplicate entries do not establish retry history. Global counts count each automated test once, even when requirement rows overlap.

A partial set of parameter variants does not prove every possible variant ran.

Verdicts¶

Subject	Rule
Automated test with reported runs	Worst of `fail`, `skip`, `pass`, in that order
Automated test absent from the report	`no-result`
Test case, or requirement, that automated tests reach	Worst of their verdicts: `fail`, `no-result`, `skip`, `pass`
Test case, or requirement, that no automated test reaches	`untested`

XML error and failure both mean fail. Expected failures and unexpected passes follow pytest's XML representation: qmstrace adds no separate xfail policy. In a TRX file, Passed means pass and NotExecuted means skip; any other outcome — Failed, Timeout, Aborted, Inconclusive — or none means fail, and the outcome is shown beside the run.

A verdict follows verifies links: from an automated test to what it verifies, and on through a test case to what the test case verifies. A requirement's row names the test cases its verdict came through. A test case no automated test runs adds nothing: it neither passes a requirement nor holds one back. A design element carries Satisfies, not verifies, so a test of a design element does not imply a verdict for its requirements; such tests and test cases are listed apart. Nonpassing test details name the affected runs and show outcome messages. A shared test names the other items it verifies.

Inspection and analysis requirements appear separately. Their linked test results, if any, do not change their verification method or establish approval.

Partial report example¶

This miniature service report contains a skipped design test, a shared failing test with a teardown error, an absent test, and an entry that no longer matches a scanned test:

report: report.xml
  6 XML entries · 5 reported runs (distinct full identities)
  matched: 4 runs of 3 automated tests · scanned: 4 automated tests · unmatched: 1 run identities
  matched against the current working tree — report commit NOT VERIFIED
  a partial parameter set does not prove every possible variant ran
  baselines: judged from reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches
  baselines: no matching reachable local tag — nothing frozen locally
  change requests: NOT VERIFIED — ../ledger/change-requests is unavailable; fresh citations are verified at the ledger's import
  1 duplicate XML entries combined by full identity; retry attempts are not inferred

requirements — direct verifies links only:
  REQ-SVC-01  pass      1 automated tests · 2 runs — A call carries a key · requirement · docs/req/service-srs.md:12
  REQ-SVC-02  fail      2 automated tests · 1 runs — Records survive a restart · requirement · docs/req/service-srs.md:21
    fail      tests/integration/test_store.py::test_a_record_survives_a_restart · 1 runs · shared: also verifies DSN-SVC-02
      fail: test_a_record_survives_a_restart (error, failure)
        error: teardown could not close store
        failure: record missing after restart
    no-result tests/integration/test_store.py::test_a_restart_loses_no_accepted_record · 0 runs

inspection / analysis — no verification or approval is established by this report:
  REQ-SVC-03 — The record format is written down · Verify-by: inspection · requirement · docs/req/service-srs.md:28
    no linked automated tests

tests without a direct requirement target — no requirement verdict inferred:
    skip      tests/unit/test_keys.py::TestTheKeyCheck::test_it_runs_before_any_handler · 1 runs · verifies DSN-SVC-01
      skip: test_it_runs_before_any_handler (skipped)
        skipped: handler fixture unavailable

unmatched report entries:
  pass      ('tests.removed.test_old', 'test_old') · 1 XML entries · no scanned automated test matches this identity
  Fix: check report provenance and scan patterns; use default pytest JUnit names.

result: 3 matched automated tests · 4 matched runs · 1 unmatched run identities · exit 0 (informational) — report commit NOT VERIFIED; this repo's tests only

A partial report retains recognized results and lists unmatched identities. A requirement with a known failure remains fail even when another directly linked test has no result.

Through test cases¶

In a project with a Test Specification, the automated tests name the test case they run, and the test case names what it verifies. Run from the product repo, the view reads the ledger's test cases with their own links — read for verdicts only, never counted as this repo's links:

report: report.xml
  5 XML entries · 5 reported runs (distinct full identities)
  matched: 5 runs of 5 automated tests · scanned: 5 automated tests · unmatched: 0 run identities
  matched against the current working tree — report commit NOT VERIFIED
  a partial parameter set does not prove every possible variant ran
  baselines: judged from reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches
  baselines: no matching reachable local tag — nothing frozen locally
  change requests: ../ledger/change-requests (1 readable; 0 malformed) — current working tree, not pinned

requirements — verdicts along verifies links, from automated tests directly or through test cases:
  REQ-SVC-01  pass      via TC-SVC-01.1 · 2 automated tests · 2 runs — A call carries a key · requirement, in sibling ledger — not pinned · ../ledger/docs/req/service-srs.md:5
    via TC-SVC-01.1  pass · 2 automated tests
  REQ-SVC-02  pass      1 automated tests · 1 runs — Records survive a restart · requirement, in sibling ledger — not pinned · ../ledger/docs/req/service-srs.md:12

test cases that reach no requirement — no requirement verdict inferred:
  TC-SVC-01.2  pass · 1 automated tests · verifies DSN-SVC-01 — The key check runs before any handler
    pass      tests/unit/test_api.py::test_the_key_check_runs_before_any_handler · 1 runs
  TC-SVC-02.1  pass · 1 automated tests · verifies DSN-SVC-02 — A restart keeps every field of a record
    pass      tests/integration/test_store.py::test_a_restart_keeps_every_field · 1 runs

result: 5 matched automated tests · 5 matched runs · 0 unmatched run identities · exit 0 (informational) — report commit NOT VERIFIED; this repo's tests only

From a TRX file¶

The fixture project's C# agent runs the same test cases from xUnit. In this TRX file one [Fact] was skipped and one row of a [Theory] failed; each run is shown by its testName, with its outcome and the first line of its message:

report: report.trx
  3 XML entries · 3 reported runs (distinct full identities)
  matched: 3 runs of 2 automated tests · scanned: 2 automated tests · unmatched: 0 run identities
  matched against the current working tree — report commit NOT VERIFIED
  a partial parameter set does not prove every possible variant ran
  baselines: judged from reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches
  baselines: no matching reachable local tag — nothing frozen locally
  change requests: ../ledger/change-requests (1 readable; 0 malformed) — current working tree, not pinned

requirements — verdicts along verifies links, from automated tests directly or through test cases:
  REQ-SVC-01  skip      via TC-SVC-01.1 · 1 automated tests · 1 runs — A call carries a key · requirement, in sibling ledger — not pinned · ../ledger/docs/req/service-srs.md:5
    via TC-SVC-01.1  skip · 1 automated tests · verifies REQ-SVC-01 — A call without a valid key is refused
      skip      tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_NoKey_IsFalse · 1 runs
        skip: TC-SVC-01.1 a call without a key is refused (NotExecuted)
          NotExecuted: the key store is offline

test cases that reach no requirement — no requirement verdict inferred:
  TC-SVC-01.2  fail · 1 automated tests · verifies DSN-SVC-01 — The key check runs before any handler
    fail      tests/Agent.Tests/Unit/KeyCheckTests.cs::KeyCheckTests::Admits_AnyKey_DecidesAlone · 2 runs
      fail: TC-SVC-01.2 the key check decides before any handler(key: "a-wrong-key", admitted: True) (Failed)
        Failed: Expected KeyCheck.Admits(key, "the-key") to be True, but found False.

result: 2 matched automated tests · 3 matched runs · 0 unmatched run identities · exit 0 (informational) — report commit NOT VERIFIED; this repo's tests only

Repository boundary¶

Local tests can verify a sibling's requirement. The output labels its owner and the unpinned working-tree source; it does not claim to include the sibling's own tests:

report: report.xml
  1 XML entries · 1 reported runs (distinct full identities)
  matched: 1 runs of 1 automated tests · scanned: 2 automated tests · unmatched: 0 run identities
  matched against the current working tree — report commit NOT VERIFIED
  a partial parameter set does not prove every possible variant ran
  baselines: judged from reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches
  baselines: no matching reachable local tag — nothing frozen locally
  change requests: NOT VERIFIED — ../ledger/change-requests is unavailable; fresh citations are verified at the ledger's import

requirements — direct verifies links only:
  REQ-CLT-01  no-result 1 automated tests · 0 runs — The client sends its key on every call · requirement · docs/req/client-srs.md:3
    no-result tests/unit/test_api.py::test_every_call_carries_the_key · 0 runs
  REQ-SVC-01  pass      1 automated tests · 1 runs — A call carries a key · requirement, in sibling service — not pinned · ../service/docs/req/service-srs.md:12

result: 1 matched automated tests · 1 matched runs · 0 unmatched run identities · exit 0 (informational) — report commit NOT VERIFIED; this repo's tests only

When the sibling cannot be read, a cited ID has an unknown type. Its linked test results are shown separately and are never presented as requirement verdicts:

report: report.xml
  1 XML entries · 1 reported runs (distinct full identities)
  matched: 1 runs of 1 automated tests · scanned: 2 automated tests · unmatched: 0 run identities
  matched against the current working tree — report commit NOT VERIFIED
  a partial parameter set does not prove every possible variant ran
  external AGR-*: NOT VERIFIED — ../ledger/agreement.md is not reachable; each cited AGR-N is judged at the ledger's import
  sibling service: NOT REACHABLE at ../service
  baselines: judged from reachable local tags — remote-tag completeness NOT VERIFIED; qmstrace never fetches
  baselines: no matching reachable local tag — nothing frozen locally
  change requests: NOT VERIFIED — ../ledger/change-requests is unavailable; fresh citations are verified at the ledger's import

requirements — direct verifies links only:
  REQ-CLT-01  no-result 1 automated tests · 0 runs — The client sends its key on every call · requirement · docs/req/client-srs.md:3
    no-result tests/unit/test_api.py::test_every_call_carries_the_key · 0 runs

unverified targets — type unknown, not requirement verdicts:
  REQ-SVC-01  pass      1 automated tests · 1 runs — REQ-SVC-01 · not verified here — judged at the ledger's import

result: 1 matched automated tests · 1 matched runs · 0 unmatched run identities · exit 0 (informational) — report commit NOT VERIFIED; this repo's tests only

Unread sources and graph violations remain visible. Matching uses the current working tree; qmstrace does not verify which commit produced a report. Results are not persisted.

Exit codes¶

0: at least one scanned test matched and the informational view was produced. Failed tests, partial matches and graph findings do not change this code.
2: usage or configuration prevented reading, or the report was unreadable, malformed, unsafe, empty, unsupported, or matched no scanned tests. The diagnostic suggests a next step.
3: an internal fault.

Use the test runner's exit code — pytest's, or dotnet test's — to gate test success and check to gate traceability. See Generate and interpret a pytest report and Trace a C# repo for the workflows.