test: seeded goldens for the robustness seeds (#826)
tests/robustness/goldens.json records what each of seeds 0-95 builds: its steps, how it ended (ok, or the kind and the start of the error it was refused with) and, for the design it left, the bodies' total volume and area (within a relative 1e-6) and their face, edge and shell counts. tests::robustness_goldens fails on any change; ROBUSTNESS_GOLDENS_BLESS="why" records one, with the reason and the seeds it moved. Pull requests that touch crates/kernel run every 8th seed; pushes to main and the nightly run every seed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
200a83a629
commit
534393ed5a
9 files changed
+338
-5
No files matched your search
@@ -76,7 +76,8 @@ On GitHub, one job builds the tests into a
|
||||
Pull requests skip a few heavy tests (each takes one to four minutes); they run on every push to
|
||||
`main`, nightly, and in `cargo xtask test`:
|
||||
|
||||
- `solidcraft-engine`: `tests::hostile_params_never_panic`, everything in `tests::robustness`,
|
||||
- `solidcraft-engine`: `tests::hostile_params_never_panic`, everything in `tests::robustness` and
|
||||
`tests::robustness_goldens` (pull requests that touch `crates/kernel` run its `sample_*` part),
|
||||
`pattern_suppress_tests::enclosure_arc_extinction_keeps_a_real_sharp_inner_lip_after_resize_and_reopen`,
|
||||
`pattern_suppress_tests::enclosure_datums_follow_wall_height_and_board_depth_with_material_proofs`,
|
||||
`cmd::plastic::tests::enclosure_snap_fit_follows_resized_front_wall`
|
||||
|
||||
@@ -36,6 +36,24 @@ rejected before any jobs or output directories are created. The maximum seed
|
||||
- `cargo test` runs a fixed set of seeds that must stay clean
|
||||
(`crates/engine/src/tests/robustness.rs`), next to the minimised cases fixed so far.
|
||||
|
||||
## Seeded goldens
|
||||
|
||||
`tests/robustness/goldens.json` records what each of seeds 0–95 builds (eight steps at most):
|
||||
its steps, how it ended (ok, or the kind and the start of the error it was refused with), and
|
||||
for the design it left the bodies' total volume and area (within a relative 1e-6) and their
|
||||
face, edge and shell counts (exact). `tests::robustness_goldens` fails on any change, better or
|
||||
worse, so a kernel change shows every seed it moves. Pull requests that touch `crates/kernel`
|
||||
run every 8th seed (`sample_0` … `sample_3`, about two minutes); pushes to `main` and the
|
||||
nightly run them all (`all_0` … `all_3`).
|
||||
|
||||
To record an intended change, give the reason:
|
||||
|
||||
```
|
||||
ROBUSTNESS_GOLDENS_BLESS="why it changed" cargo test --profile ci -p solidcraft-engine --lib robustness_goldens::all
|
||||
```
|
||||
|
||||
The changed seeds and the reason are added to the file's `notes`; say why in the pull request too.
|
||||
|
||||
## Successful Boolean retries
|
||||
|
||||
When an accepted Boolean result needs a numerical retry, the feature shows a
|
||||
|
||||
Reference in new issue
Block a user