docs: verification and validation; published benchmarks; fixtures (#31)

* docs: verification and validation against theory, experiments and published benchmarks; fixtures

Owner decision 2026-10-13: test against experimental and theoretical data; results published by
vendors (e.g. Abaqus benchmark and verification manuals) may be used as cited reference values for
our own models of the same problems; once a case matches, its result becomes a fixture so fidelity
cannot regress. Commercial products are still never run and their files and text never copied.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm

* docs: V&V policy in AGENTS.md, ADR 0001 and the roadmap

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm

---------

Co-authored-by: jyviko <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
authored and GitHub committed 2026-10-13 18:43:36 -04:00
1 parent aa68bbbcdf
commit de8c0e2700
4 files changed
+45 -8

No files matched your search

+5 -2
View File
@@ -89,8 +89,11 @@ dependency of `modelcraft-geometry` only behind its non-default `scripts` featur
tooling (`modelcraft-testkit`) enables; the shipped library opens designs without it.
**Commercial products** (Abaqus, Nastran, LS-DYNA, LS-PrePost, HyperMesh, ANSA, Femap, Patran,
ANSYS) are never run, scripted or used as a source of data. Deck formats come from the vendors'
public reference manuals, described in our own words with our own examples.
ANSYS) are never run or scripted, and their input files, text and figures are never copied. Deck
formats come from the vendors' public reference manuals, described in our own words with our own
examples. Published numerical results (benchmark and verification manuals) may serve as cited
reference values for our own models of the same problems (owner decision 2026-10-13;
docs/verification.md).
## Consequences
+29
View File
@@ -0,0 +1,29 @@
# Verification and validation
Every solver capability is checked against three kinds of reference, and every accepted result is
frozen so it cannot drift.
| Kind | What it checks | Sources |
|---|---|---|
| Theory | The code solves the equations right | Closed-form solutions (beams, plates, Lamé, Kirsch, waves, shocks, Poiseuille…), patch tests, convergence rates, mesh invariants |
| Experiment | The model predicts reality | Measurements published in papers and reports (Taylor bar, cavity and cylinder flows, FSI experiments…) |
| Published benchmarks | Agreement with established codes on standard problems | NAFEMS benchmarks, vendor benchmark and verification manuals (e.g. Abaqus Benchmarks and Verification manuals), other published reference solutions |
Rules:
- Each case is **our own model** of the problem, built from the published description (geometry,
material, loads, boundary conditions). We never run a commercial product and never copy its
input files, manual text or figures; only the published numbers are used, with a citation.
- Each case lives in `tests/oracle/<case>/`:
- `recipe.json` — the model as ModelCraft commands (or a SolidCraft recipe for geometry);
- `expected.json` — the reference values with `kind` (theory | experiment | benchmark),
`source` (citation: document, table or figure, page), `quantity`, `value`, `tolerance` and
the reason for that tolerance (mesh, measurement scatter, model differences);
- `fixture.json` — ModelCraft's accepted result, written by `cargo xtask oracle --bless <case>`
only after the case passes against `expected.json`.
- `cargo xtask oracle` runs every case, compares with `expected.json` (pass/fail with error) and
with `fixture.json` (any change beyond the fixture tolerance fails CI, even if the reference
still passes), and writes `docs/oracle.md` with the table of cases, sources, errors and status.
- A fixture is updated only deliberately (`--bless`), in a PR that says why the result moved
(a better element, a finer default mesh), never to hide a regression.
- The theory suite is the gate for every PR; experiment and benchmark cases run in CI too, with
their cited tolerances.