Specifications, Requirements and Test Plans

Topics We'll Cover

• Why a Simulator Needs a Specification
• What Makes a Good Requirement
• EARS: Five Patterns for Requirements
• Interactive: Check a Requirement
• Functional and Non-Functional Requirements for Simulators
• Requirements Capture for Mission-Mode Software
• The V-Model
• Traceability, Generated From the Test Run
• Template 1: a Simulator Specification
• Template 2: a Test Plan
• Template 3: a Performance Report
• Reviews, Baselines and Sign-Off
• What to Take Away

Why a Simulator Needs a Specification

A simulator's answers feed decisions: how many lanes, how much memory bandwidth, whether a design meets its target before silicon exists. Without a written statement of what it must do, and how well, nobody can say whether it is right, only whether it runs. A specification makes "right" checkable:

• For its users: what questions it can answer, at what fidelity, with what error against which reference.
• For its developers: what to build and what not to build. "Not modelled" is a requirement too.
• For verification: every requirement has a test or other evidence, and the gaps are visible (slide 08).

This deck uses one complete worked example throughout: the specification, test plan and traceability matrix of Torch_Sim_Frontend (deck 10), written for this purpose and kept in its repository. The test-level detail is in deck 06.

What Makes a Good Requirement

INCOSE's Guide to Writing Requirements and ISO/IEC/IEEE 29148 (requirements engineering) describe the same core characteristics. For an individual requirement these include being necessary, appropriate to its level, unambiguous, complete, singular (one thing), feasible, verifiable, correct and conforming to the agreed style. A set of requirements must also be complete and consistent as a whole.

Verifiable is the one engineers break most: if no test, analysis, inspection or demonstration could fail it, it is a wish, not a requirement. Each requirement names its verification method (Test, Analysis, Inspection, Demonstration).

EARS: Five Patterns for Requirements

The Easy Approach to Requirements Syntax (Mavin, Wilkinson, Harwood and Novak, IEEE RE 2009, doi:10.1109/RE.2009.9) constrains natural language to a few templates. The keyword at the start says what kind of requirement it is, which removes a whole class of ambiguity (the patterns):

Unwanted-behaviour requirements are the ones teams forget: they come from asking "what if it goes wrong?" of every event. In simfront that question produced SF-06 (unknown operators) and SF-15 (a library upgrade that changes the arithmetic).

Interactive: Check a Requirement

Type or pick a requirement. The checker names its EARS pattern and flags the problems the writing rules warn about: vague and unmeasurable words, escape and open-ended clauses, superfluous infinitives, passive voice, non-binding verbs and more than one "shall". It is a set of simple patterns, not a judge: a clean result means only that nothing obvious is wrong.

Functional and Non-Functional Requirements for Simulators

Functional requirements say what the simulator does: which components and behaviours it models, which inputs it accepts, which outputs it produces. Non-functional requirements say how well, and for a simulator they carry most of the value:

An accuracy requirement without a named reference and tolerance is unverifiable. Writing it forces the important conversation early: "accurate against what, and how close is close enough for the decision this supports?"

Requirements Capture for Mission-Mode Software

Usage varies between organisations, so define the term before using it. Here, mission-mode software means the software that runs on the product in its operational mode, doing the job the product exists for. That is as opposed to the test, bring-up, calibration, diagnostic and manufacturing software that runs on the same hardware at other times. Its requirements are captured differently because the stakes, the environment and the evidence expected are different:

• Capture techniques: operational scenarios (a concept of operations), a mode and state diagram, interface control documents for every boundary, and a hazard or failure analysis whose results become unwanted-behaviour requirements.
• Where a simulator helps: mission-mode requirements on timing and throughput can be checked on a model (a virtual platform or a performance model) long before silicon. The simulator then needs requirements of its own, at the fidelity those checks depend on.
• Regulated domains add standards that specify the process (for example DO-178C in airborne software, ISO 26262 in road vehicles, IEC 61508 for functional safety generally). Which one applies, if any, depends on the product.

The V-Model

• Each level on the left is verified by the level opposite it on the right. Requirements are verified by system tests; needs are validated by users with real workloads. Writing the right-hand plan when the left-hand document is written is the point: a requirement nobody can test shows up immediately.
• Verification asks "did we build it right?" (against the spec); validation asks "did we build the right thing?" (against the need). A simulator can pass every test and still answer the wrong question.
• Iterative teams still use the V, once per increment: each release's spec, tests and matrix move together.

Traceability, Generated From the Test Run

simfront's tests name the requirements they verify (@pytest.mark.req("SF-04")). ci/trace_matrix.py runs the suite in-process, collects each test's markers and outcome, joins them to the requirement table in docs/spec.md and writes docs/traceability.md. It checks both directions: every requirement has a passing test, and every test traces to a requirement or is listed as untraced.

for r in reqs:
    tests = by_req.get(r["id"], [])
    if r["method"].startswith("T"):
        res = [col.outcome.get(t, "not run") for t in tests]
        ok = bool(tests) and all(x == "passed" for x in res)
        gaps += not ok
        names = "<br>".join(f"`{t.split('/')[-1]}`" for t in tests) or "**no test**"
        verdict = f"{sum(x == 'passed' for x in res)}/{len(tests)} passed" if tests else "**GAP**"
    else:
        names, verdict = demonstration_evidence(r), "demonstrated"
    lines.append(f"| **{r['id']}** {r['text']} | {r['pattern']} | {r['method']} | {names} | {verdict} |")

Source: docs/traceability.md in Torch_Sim_Frontend

Template 1: a Simulator Specification

The requirements table, as written (excerpt):

| ID | Type | Pattern | Requirement | Verification |
|---|---|---|---|---|
| SF-01 | Functional | Ubiquitous | The front end shall record, for every captured operator, the shape, element size and identity of each input and output tensor and whether it is a model parameter. | T |
| SF-02 | Functional | Event-driven | When a model is traced on the meta device, the front end shall allocate no parameter storage. | T |
| SF-03 | Functional | Ubiquitous | For SwiGLU decoder configurations, the traced weight-matmul FLOPs of a prefill shall equal 2 × matmul parameters × tokens exactly. | T |
| SF-04 | Functional | Ubiquitous | The four front ends shall report identical matmul FLOPs and weight bytes for the same model and input, apart from constant folding that is documented per case. | T |
| SF-05 | Functional | Event-driven | When a decode step is traced, the traced matmul FLOPs shall equal the closed form plus the rotary-frequency matmul, and the traced weight bytes shall equal the closed form's per-step weight traffic plus the RMSNorm weights. | T |
| SF-06 | Functional | Unwanted behaviour | If an operator has no cost rule, then the front end shall cost it at zero and name it in the coverage report. | T |

A row per requirement, in a file under version control next to the code, reviewed in the same pull request as the change it describes.

Template 2: a Test Plan

A test plan says what will be tested, how a result is judged right, and when testing is finished. Its sections follow the usual contents (as in ISO/IEC/IEEE 29119-3), cut to what a small tool needs: test items and scope; approach; pass/fail criteria; entry and exit criteria; environment; deliverables; risks. The approach is the heart of it, because a test is only as good as its oracle, and each level here uses one that shares no code with what it checks:

Source: docs/test_plan.md in Torch_Sim_Frontend

• Exit criteria are measurable: all tests pass on two Python versions; the gate passes; the traceability matrix has no gaps.
• Risks name their mitigation: for example, "a PyTorch upgrade changes the decomposition" is mitigated by a gate that separates arithmetic (fails) from operator-count drift (reported).

Full plan: docs/test_plan.md.

Template 3: a Performance Report

Every code repository in this series has one: examples/results.md, written by examples/results.py, which is the only source of the numbers in its README and decks. The structure generalises to any simulation study:

The environment section from simfront's report, generated, never typed:

Source: examples/results.md in Torch_Sim_Frontend

A report generated by a script can be regenerated after any model change, which is what keeps the numbers in decks and READMEs true. Timing results also need repeat runs and a statement of spread (deck 11).

Reviews, Baselines and Sign-Off

• Review requirements like code: in a pull request, against a checklist (the characteristics on slide 02, the rules the checker on slide 04 approximates), by someone who will have to verify them.
• Baseline the spec at a version once agreed; change it only through the same review, recording why in its change history. Tag the spec, the tests and the matrix together at each release.
• Sign-off is a decision on evidence: the matrix with no gaps, the test and gate results from the release build, the accuracy against references, the open issues each with a stated impact (deck 08).
• Who signs depends on what the simulator feeds: a study that informs a tape-out decision deserves the same rigour as the RTL sign-off it shortcuts.

What to Take Away

• Write requirements to be verified: singular, measurable, with a method. EARS's five patterns make the trigger and the state explicit.
• For simulators the non-functional requirements carry the value: accuracy against a named reference and tolerance, determinism, speed, capacity, and the fidelity boundaries.
• Mission-mode software (defined carefully) needs modes, fault handling and budgets captured as requirements, and evidence for each.
• Generate the traceability matrix from the test run. Its first run here found a real gap.
• Plans name their oracles; reports are generated by scripts.

• Torch_Sim_Frontend/docsThe specification, the test plan and the generated traceability matrix used throughout.
• EARSThe patterns, with examples, from one of their authors.
• LLM Inference Simulators 06Validation against references, in depth.
• Interview_Software_TestingTesting interview questions, including test planning.