forked from 1b-agent/fieldcraft-upstream
de8c0e2700
* docs: verification and validation against theory, experiments and published benchmarks; fixtures Owner decision 2026-10-13: test against experimental and theoretical data; results published by vendors (e.g. Abaqus benchmark and verification manuals) may be used as cited reference values for our own models of the same problems; once a case matches, its result becomes a fixture so fidelity cannot regress. Commercial products are still never run and their files and text never copied. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm * docs: V&V policy in AGENTS.md, ADR 0001 and the roadmap Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm --------- Co-authored-by: jyviko <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
30 lines
2.2 KiB
Markdown
30 lines
2.2 KiB
Markdown
# Verification and validation
|
|
|
|
Every solver capability is checked against three kinds of reference, and every accepted result is
|
|
frozen so it cannot drift.
|
|
|
|
| Kind | What it checks | Sources |
|
|
|---|---|---|
|
|
| Theory | The code solves the equations right | Closed-form solutions (beams, plates, Lamé, Kirsch, waves, shocks, Poiseuille…), patch tests, convergence rates, mesh invariants |
|
|
| Experiment | The model predicts reality | Measurements published in papers and reports (Taylor bar, cavity and cylinder flows, FSI experiments…) |
|
|
| Published benchmarks | Agreement with established codes on standard problems | NAFEMS benchmarks, vendor benchmark and verification manuals (e.g. Abaqus Benchmarks and Verification manuals), other published reference solutions |
|
|
|
|
Rules:
|
|
- Each case is **our own model** of the problem, built from the published description (geometry,
|
|
material, loads, boundary conditions). We never run a commercial product and never copy its
|
|
input files, manual text or figures; only the published numbers are used, with a citation.
|
|
- Each case lives in `tests/oracle/<case>/`:
|
|
- `recipe.json` — the model as ModelCraft commands (or a SolidCraft recipe for geometry);
|
|
- `expected.json` — the reference values with `kind` (theory | experiment | benchmark),
|
|
`source` (citation: document, table or figure, page), `quantity`, `value`, `tolerance` and
|
|
the reason for that tolerance (mesh, measurement scatter, model differences);
|
|
- `fixture.json` — ModelCraft's accepted result, written by `cargo xtask oracle --bless <case>`
|
|
only after the case passes against `expected.json`.
|
|
- `cargo xtask oracle` runs every case, compares with `expected.json` (pass/fail with error) and
|
|
with `fixture.json` (any change beyond the fixture tolerance fails CI, even if the reference
|
|
still passes), and writes `docs/oracle.md` with the table of cases, sources, errors and status.
|
|
- A fixture is updated only deliberately (`--bless`), in a PR that says why the result moved
|
|
(a better element, a finer default mesh), never to hide a regression.
|
|
- The theory suite is the gate for every PR; experiment and benchmark cases run in CI too, with
|
|
their cited tolerances.
|