Jenkins for Hardware and Simulation Teams

Topics We'll Cover

• Why Simulators Need CI More Than Most Code
• Declarative and Scripted Pipelines
• Agents, Labels and the Tools on Them
• Matrix Builds: a Parameter Sweep in CI
• Shared Libraries
• Test and Coverage Reports
• Nightly Regressions and Parameters
• Gating on Performance and Accuracy
• Credentials
• Interactive: Every Build, and Why Some Failed
• The Real Runs
• Jenkins and GitHub Actions
• What to Take Away

New to Jenkins? Start with Introduction to Jenkins: controller and agents, plugins, a first pipeline, credentials and test reports, from the same local Jenkins.

Why Simulators Need CI More Than Most Code

A simulator's product is its answers. A change can leave every unit test green and still move a bootstrap time by 3% or slow the nightly sweep to half speed. CI for a simulator therefore checks four things on every change: that it builds and passes its tests, that its answers agree with references, that its performance has not regressed, and that it produces the results the decks and reports quote.

Each code repository in this series has a Jenkinsfile, and every one of them was run on a local Jenkins (2.580.1, in user space). These are the pipelines:

The Jenkins here runs on one machine, is reachable only from it, and has no separate agents. Slide 12 compares it with the GitHub Actions workflows that run the same tests on every push.

Declarative and Scripted Pipelines

A declarative pipeline is a fixed structure (pipeline, agent, parameters, stages, post) that Jenkins can validate before it runs and draw as a stage graph. A scripted pipeline is Groovy code inside node { }, with loops, maps and try/catch. Here is the same sweep as a scripted pipeline that also handles a failure on purpose:

node('sim') {
    stage('Checkout') {
        git url: 'https://github.com/BrendanJamesLynskey/Memory_System_Sim.git', branch: 'main'
    }
    stage('Sweep') {
        def cells = [:]
        for (preset in ['ddr4', 'hbm']) {
            for (pattern in ['stream', 'random']) {
                def p = preset, q = pattern            // capture loop variables for the closure
                cells["${p}/${q}"] = {
                    sh "PYTHONPATH=src python3 -m memsim.cli --preset ${p} --pattern ${q} --n 3000"
                }
            }
        }
        parallel cells
    }
    stage('A failure that is handled') {
        try {
            sh 'PYTHONPATH=src python3 -m memsim.cli --preset no-such-preset'
        } catch (err) {
            echo "expected failure, handled: ${err}"
            currentBuild.description = 'sweep ok; bad preset rejected'
        }
    }
}

Its log shows the four parallel branches and the handled failure: expected failure, handled: hudson.AbortException: script returned exit code 2.

Agents, Labels and the Tools on Them

The controller schedules; agents run steps. A stage asks for an agent by label (agent { label 'sim' }), so a pipeline says what it needs, not which machine to use. For hardware and simulation teams, labels usually encode tools and licences:

• Tools: verilator, vivado-2025.2, systemc, gpu. A simulator that needs a commercial RTL simulator runs only where its licence and install are.
• Capacity: long regressions on big machines; quick checks anywhere. Executors per agent bound how many builds share a machine; locks (for example with a lockable-resources plugin) protect scarce things like a hardware board.
• Reproducibility: pin tool versions per label, or run in containers, so a failure is the change, not the machine.

Here there is one built-in node labelled sim, with one executor, run under systemd-run -p MemoryMax=6G: on an 8-core, 15 GB machine, two heavy builds at once have crashed it before.

environment {
    // Empty unless the VERILATOR_ROOT parameter is set. Never default a parameter to "${env.HOME}/...":
    // parameters {} is evaluated before the agent exists, so it becomes "null/..." (builds #1-#3 failed so).
    VR = "${params.VERILATOR_ROOT ?: ''}"
    MAKEFLAGS = '-j2'
    // HOME is resolved by each step's shell, so each step that needs Verilator sets it up itself
    VPATH = 'export VERILATOR_ROOT="${VR:-$HOME/.local/opt/verilator-deb/root/usr/share/verilator}"; [ -d "$VERILATOR_ROOT" ] && export PATH="$VERILATOR_ROOT/bin:$PATH" || unset VERILATOR_ROOT; '
}

And name such helpers carefully: Rust_DES_Kernel's first version of this fix called its helper CARGO, which cargo and maturin read as the path of the cargo binary, and its build failed until it was renamed WITH_CARGO.

Matrix Builds: a Parameter Sweep in CI

A declarative matrix runs the same stages for every combination of its axes, as parallel branches, minus the excludes. For a simulator, the axes are configurations: memory presets, access patterns, model sizes, simulator back ends.

stage('Sweep') {
    matrix {
        agent { label 'sim' }
        axes {
            axis { name 'PRESET';  values 'ddr4', 'hbm' }
            axis { name 'PATTERN'; values 'stream', 'random', 'stride' }
        }
        excludes {
            exclude {                       // not a meaningful cell for this demo
                axis { name 'PRESET';  values 'hbm' }
                axis { name 'PATTERN'; values 'stride' }
            }
        }
        stages {
            stage('Simulate') {
                steps {
                    unstash 'src'
                    sh 'PYTHONPATH=src python3 -m memsim.cli --preset $PRESET --pattern $PATTERN --n 3000 | tee cell.txt'
                    sh 'mv cell.txt "cell-$PRESET-$PATTERN.txt"'
                    archiveArtifacts artifacts: "cell-${PRESET}-${PATTERN}.txt"
                }
            }
        }
    }
}

Demo_matrix #1 on the local Jenkins: two presets × three patterns, minus one excluded cell, so five cells.

Shared Libraries

When five repositories need the same stages, copy-paste drifts. A shared library is a Git repository of Groovy that pipelines load by name. A file in vars/ becomes a step. This one gives any simulator the house regression stages:

def call(Map cfg) {
    String py = cfg.get('python', 'python3')
    stage('Setup') {
        sh "${py} -m venv .venv && .venv/bin/pip install -q -e '.[test]'"
    }
    stage('Tests') {
        sh ".venv/bin/pytest -p no:logging --junitxml=junit.xml ${cfg.get('pytestArgs', '')}"
        junit 'junit.xml'
    }
    if (cfg.gate) {
        stage('Gate') {
            sh ".venv/bin/python ${cfg.gate}"
        }
    }
    if (cfg.results) {
        stage('Results') {
            sh ".venv/bin/python ${cfg.results} > /dev/null"
            archiveArtifacts artifacts: cfg.resultsFile
        }
    }
}

@Library('simeng-ci') _

node('sim') {
    git url: 'https://github.com/BrendanJamesLynskey/Memory_System_Sim.git', branch: 'main'
    simRegression(gate: 'ci/perf_gate.py --margin 0.5', results: 'examples/results.py',
                  resultsFile: 'examples/results.md')
    stage('Publish (credential demo)') {
        withCredentials([string(credentialsId: 'results-upload-token', variable: 'TOKEN')]) {
            // The token is masked wherever it would be printed.
            sh 'echo "would upload results.md with token $TOKEN"'
        }
    }
}

The library is configured once on the controller, with its repository and default version. The log opens with Loading library simeng-ci@main, and the stages it defined ran Memory_System_Sim's 31 tests and its gate. Pin a library to a tag in production pipelines, so a library change cannot break every build at once.

Test and Coverage Reports

Every test framework in this series writes JUnit XML (pytest, cargo-nextest, GoogleTest, cocotb's runner) and every coverage tool Cobertura XML (pytest-cov, cargo-llvm-cov). Jenkins reads both, keeps their history and draws trends (deck 06, slide 13):

stage('Coverage') {
    steps {
        sh(env.WITH_CARGO + 'cargo llvm-cov --release --cobertura --output-path coverage.xml')
        // cargo-llvm-cov lists each generic instantiation as its own method, and the
        // Coverage plugin's Cobertura parser rejects duplicate names: skip them.
        recordCoverage(tools: [[parser: 'COBERTURA', pattern: 'coverage.xml']],
                       sourceCodeRetention: 'LAST_BUILD', ignoreParsingErrors: true)
    }
}

// The workspace is reused between builds (it keeps the virtualenv and build caches), so
// delete the previous build's reports first. Without this a build that fails before its
// tests run publishes the last build's JUnit results as its own (Rust_DES_Kernel #4 did).
stage('Clean reports') {
    steps {
        sh 'rm -f target/nextest/ci/junit.xml pytest-junit.xml coverage.xml perf_report.md sweep.csv'
    }
}

• junit in post { always { } }, so failing tests are still recorded: the reports matter most on a red build.
• recordCoverage (the Coverage plugin). Rust_DES_Kernel's first build failed here: cargo-llvm-cov lists each generic instantiation as its own method, and the Cobertura parser rejected the duplicates. ignoreParsingErrors: true was the fix.
• Delete old reports at the start of a build. Rust_DES_Kernel's build #4 failed before its Python tests ran, yet Jenkins recorded all 46 tests: the workspace is reused, and the post step found build #3's pytest-junit.xml (its suite is timestamped 13 hours earlier). A failed build that reports passing tests is worse than one that reports none. Fixed on 2026-10-03: all five Jenkinsfiles now start by deleting their report files (below). cleanWs() would also work, but it would throw away the cached virtualenv and build directories. The next failure, #7, reported its own 15 failed tests out of 47.
• A reused workspace can also hide stale dependencies. #7 and #8 failed because the workspace's virtualenv still held the Python reference simulator from GitHub at an older commit. pip treats an installed git dependency with an unchanged version number as satisfied. GitHub Actions starts from scratch and passed the same commit. Force-reinstalling that dependency on every build fixed it, and #9 and #10 passed. Cache the expensive parts of a workspace, not the parts under test.
• Archive what the reports are made of (results.md, perf_report.md, sweep.csv), so a number in a report can be traced to the build that produced it.

Rust_DES_Kernel #10: 47 tests from two frameworks, one report.

Nightly Regressions and Parameters

Fast checks run on every change; the expensive ones (seed sweeps, load-latency curves, quantum sweeps, every model at every length) run nightly. One Jenkinsfile can serve both through a parameter:

parameters {
    booleanParam(name: 'NIGHTLY', defaultValue: false, description: 'Also run the model x length sweep')
    string(name: 'PERF_MARGIN', defaultValue: '0.5',
           description: 'Allowed slow-down of trace capture against ci/perf_baseline.json')
}

stage('Nightly sweep') {
    when { expression { params.NIGHTLY } }
    steps {
        sh '.venv/bin/python ci/sweep.py'
        archiveArtifacts artifacts: 'sweep.csv'
    }
}

• A nightly job (or a triggers { cron('H 2 * * *') } block; H spreads jobs across the hour) sets NIGHTLY=true. The jobs here were triggered through the REST API.
• Parameter gotchas, both met here: a parameter default computed from env becomes the text null (resolve paths in the step's shell instead), and once a job has parameters a plain /build request is rejected with HTTP 400; use /buildWithParameters.
• Nightly results are artefacts too: each repository's sweep writes a CSV that the next morning's comparison can diff.

Gating on Performance and Accuracy

A gate is a stage that compares this build with a stored baseline and fails the build if it got worse. For a simulator there are three kinds, and these repositories use all three:

def compare(base: dict, now: dict, margin: float, strict: bool = False) -> tuple[list[str], int]:
    """The gate's verdict: (report lines, number of failures)."""
    lines = ["# simfront gate", "", "| check | baseline | now | verdict |", "|---|---|---|---|"]
    bad = 0
    for k, b in base["traces"].items():
        n = now["traces"][k]
        for f in ("flops", "weight_bytes"):
            ok = n[f] == b[f]
            bad += not ok
            lines.append(f"| {k}: {f} | {b[f]:,} | {n[f]:,} | {'ok' if ok else 'CHANGED'} |")
        for f in ("ops", "bytes"):
            ok = n[f] == b[f]
            bad += strict and not ok
            lines.append(f"| {k}: {f} | {b[f]:,} | {n[f]:,} | {'ok' if ok else 'drift (framework)'} |")
    b, n = base["capture_s"], now["capture_s"]
    ok = n <= (1 + margin) * b
    bad += not ok
    lines.append(f"| capture llama3-8b prefill 2048 (s) | {b:.3f} | {n:.3f} | {'ok' if ok else 'SLOWER'} "
                 f"(margin {margin:.0%}) |")
    return lines, bad

The gate is itself tested: a test changes one FLOP in a copy of the baseline and checks the gate fails (deck 09). A speed gate on a shared machine needs a generous margin and best-of-N timing, or it will be flaky (deck 11).

Credentials

Pipelines need secrets: a token to publish results, a licence server, an SSH key for a lab machine. Jenkins stores them encrypted on the controller, and a pipeline refers to them by ID. withCredentials binds one to an environment variable for a block, and masks its value in the log:

stage('Publish (credential demo)') {
    withCredentials([string(credentialsId: 'results-upload-token', variable: 'TOKEN')]) {
        // The token is masked wherever it would be printed.
        sh 'echo "would upload results.md with token $TOKEN"'
    }
}

• The credential here is a demo string, not a real token. The log shows **** where its value would be.
• Use single quotes in sh so the shell expands $TOKEN. Groovy interpolation ("$TOKEN" in double quotes) puts the secret into the command line itself, and Jenkins warns about it.
• Masking is a safety net, not a guarantee: a secret that is transformed (base64, split, hashed) is no longer recognised. Scope credentials to the folders that need them.

Interactive: Every Build, and Why Some Failed

Every build on the local Jenkins, from its API. Choose a job to see its builds; the failures are where the lessons were.

Durations and results from the Jenkins API, recorded in snippets/RESULTS.md.

The Real Runs

Each job's page: the last artefacts, the coverage and test-result trends, and the stage view, one row per build. Choose a repository:

Repository Rust_DES_Kernel RTL_CoSim_NTT Memory_System_Sim SystemC_Accelerator_Model Torch_Sim_Frontend

Before any of these ran, each Jenkinsfile was checked by the declarative linter (POST /pipeline-model-converter/validate), which also catches typos that would otherwise surface only at run time:

Jenkins and GitHub Actions

Every repository here uses both: GitHub Actions on every push, Jenkins for the full pipeline with its gates and nightly sweeps. They suit different jobs:

• The GitHub Actions workflows here run the tests on Python 3.10 and 3.12, the Verilator and Icarus matrix and the sanitizer builds, and every one passes on GitHub.
• The Jenkins here is local only: its pages are not reachable from outside, and it has no separate agents. The screenshots on slides 04, 05, 06 and 11 are of its real runs.
• See the Pipeline syntax reference, shared libraries and GitHub's workflow syntax. Plugin details change between releases; check them against your versions.

What to Take Away

• Declarative by default, linted before it runs; scripted Groovy where logic needs it, ideally inside a shared library.
• Labels say what a stage needs: tools, licences, capacity. Resolve tool paths inside each step's shell.
• Matrix builds are parameter sweeps; nightly parameters keep one pipeline for both fast and deep checks.
• Gate on exact behaviour, on speed with a margin, and report drift. Test the gate itself.
• Record JUnit and Cobertura even when red, and archive every result a report quotes.
• Credentials by ID, masked, single-quoted.

• snippets/t07The matrix, scripted and shared-library pipelines shown here, and the library itself.
• JUnit, Coverage, Pipeline Graph View and Stage View pluginsThe plugins behind the reports and graphs.
• Interview_CI_CDCI/CD interview questions.
• Deck 08: Jira and Engineering MetricsLinking builds to work items, and the metrics they feed.