1b-agent/OneCAD
1
0 Fork 0
forked from 1b-agent/solidcraft-upstream
Code Pull Requests Activity

Action matrix: nightly shards cut by measured cost, a fresh baseline, and a determinism check (#1256)

Browse Source 
* Action matrix: nightly shards cut by measured cost, heavy designs in slices, and a determinism check

The full matrix ran as four fixed quarters of the corpus and two of them hit the 60-minute
job limit. Now action_matrix_shard runs shard k of n (ACTION_MATRIX_SHARD=k/n), planned from
tests/action_matrix/costs.json: designs that take more than five minutes are split into slices,
and the pieces go largest first to the least loaded shard. The nightly runs 12 shards and uploads
each one's outcomes and timings; cargo xtask action-matrix --from DIR --bless records them in the
baseline and costs.json without running the matrix again. action_matrix_determinism runs a sample
twice and lists any action whose outcome changed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Action matrix: costs measured on runners, 10 shards, determinism on the lighter designs

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Action matrix: baseline and costs from a full run on main, 16 shards, timeouts listed

The baseline is regenerated from the 10 shards' results of a full run on 01b1b705
(cargo xtask action-matrix --from DIR --bless): 8 new designs, 302 actions that pass now,
names that changed, and 40 actions that passed before and fail now, each listed in #1255
(22 fillets refused since #949, 4 hinge blends since #1101, 9 on the loft cut, 5 single).
costs.json now holds every design's measured time, and the nightly runs 16 shards of about
17 minutes. An action that passes in the baseline and times out is listed but not judged,
and stays passing when blessed: two runs of the same build differed only on 10 actions that
took 50-84 s against the 90 s limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Action matrix: slice heavy designs by a hash of each action's name

Slicing by index gave some slices only fillets on oracle 41 (28 slices, a multiple of the
actions per edge), three times the planned time; two shards ran past their limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
bherbruckandClaude Opus 5.5 authored and GitHub committed 2026-10-14 21:18:11 -04:00
1 parent 16c49c3e86
commit c159a3e277
6 files changed
+2487 -1936
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/solidcraft-cli/tests/action_matrix.rs
+303 -49
View File
@@ -13,8 +13,10 @@
//! fails the test (a regression). Newly passing actions are reported; record them with
//! `cargo xtask action-matrix --bless` (or `ACTION_MATRIX_BLESS=1` on [`action_matrix_all`]).
//! Every CI run checks a fixed sample ([`action_matrix_sample`], under a minute); the full matrix
//! is four ignored tests (`action_matrix_full_N`) that the nightly "Action matrix" job runs side by
//! side. See docs/action-matrix.md.
//! is the ignored [`action_matrix_shard`], which the nightly "Action matrix" jobs run one shard
//! each (`ACTION_MATRIX_SHARD=k/n`), cut by what each design costs (`tests/action_matrix/costs.json`).
//! [`action_matrix_determinism`] runs a sample twice and lists any action whose outcome changed.
//! See docs/action-matrix.md.
//!
//! Each design runs in a child process (this test binary, [`action_matrix_worker`]) so an action
//! that hangs or aborts is recorded as such and the rest of the design still runs.
@@ -33,6 +35,10 @@ use solidcraft_engine::geom::Vec3;
/// One in this many actions runs on pull requests (by a stable hash of its key).
const PR_SAMPLE: u64 = 40;
/// One in this many actions runs twice in the nightly determinism check, on the designs whose
/// whole matrix takes at most [`DETERMINISM_MAX_SECS`] on a runner (most of the corpus).
const DETERMINISM_SAMPLE: u64 = 8;
const DETERMINISM_MAX_SECS: f64 = 300.0;
/// Designs that take seconds to build: the full matrix only.
const SLOW: [&str; 7] = [
"oracle/41-tube-radial-holes",
@@ -623,17 +629,25 @@ fn action_matrix_worker() {
out.flush().unwrap();
};
let Some((_, path)) = designs().into_iter().find(|(n, _)| *n == design) else { panic!("no design {design}") };
// A slice of a heavy design (`j/n`): the actions whose name hashes to `j` modulo `n` (by
// index, slices would take whole kinds of action when n shares a factor with the number of
// actions per entity, and a slice of only fillets runs three times as long); the first
// slice records the load.
let slice: Option<(usize, usize)> = std::env::var("ACTION_MATRIX_SLICE").ok().and_then(|v| parse_part(&v));
let first = slice.is_none_or(|(j, _)| j == 0);
let t0 = Instant::now();
let s = match load(&path) {
Ok(s) => s,
Err(e) => {
emit(outcome(&design, "design", "load", &Err(("load".into(), e)), t0.elapsed().as_millis()));
if first {
emit(outcome(&design, "design", "load", &Err(("load".into(), e)), t0.elapsed().as_millis()));
}
emit(json!({"done": true}));
return;
}
};
let before = errors(&s);
if from == 0 {
if from == 0 && first {
let r = if before.is_empty() { Ok(()) } else { Err(("timeline_error".into(), before.values().cloned().collect::<Vec<_>>().join("; "))) };
emit(outcome(&design, "design", "load", &r, t0.elapsed().as_millis()));
}
@@ -647,6 +661,9 @@ fn action_matrix_worker() {
if only.as_ref().is_some_and(|o| !o.iter().any(|x| x == a.action)) {
continue;
}
if slice.is_some_and(|(j, n)| stable_hash(&format!("{} {}", a.entity, a.action)) % n as u64 != j as u64) {
continue;
}
// Whole families of pieces are sampled together (see [`family`]).
let k = key(&design, &family(&a.entity), a.action);
if !stable_hash(&k).is_multiple_of(sample) {
@@ -708,46 +725,155 @@ fn action_matrix_worker() {
emit(json!({"done": true, "actions": list.len()}));
}
/// Run the actions (one in `sample`) of every design in part `part` of `parts`, in child
/// processes, `jobs` at a time.
fn run_matrix(sample: u64, part: usize, parts: usize) -> Vec<Value> {
/// `"k/n"` (1-based `k`) as `(k - 1, n)`.
fn parse_part(v: &str) -> Option<(usize, usize)> {
let (k, n) = v.split_once('/')?;
let (k, n): (usize, usize) = (k.trim().parse().ok()?, n.trim().parse().ok()?);
(k >= 1 && k <= n).then_some((k - 1, n))
}
/// What one design costs on a CI runner (`tests/action_matrix/costs.json`, measured by the
/// nightly shards): the whole run and the load before its first action, in seconds.
#[derive(Clone, Copy, Debug)]
struct Cost {
secs: f64,
load: f64,
}
fn costs_path() -> PathBuf {
root().join("tests/action_matrix/costs.json")
}
fn read_costs() -> BTreeMap<String, Cost> {
let Some(v) = std::fs::read(costs_path()).ok().and_then(|b| serde_json::from_slice::<Value>(&b).ok()) else { return BTreeMap::new() };
v["designs"]
.as_object()
.into_iter()
.flatten()
.map(|(d, c)| (d.clone(), Cost { secs: c["secs"].as_f64().unwrap_or(0.0), load: c["load"].as_f64().unwrap_or(0.0) }))
.collect()
}
/// A design new to the corpus (not in costs.json yet) is planned as this.
const UNKNOWN_COST: Cost = Cost { secs: 120.0, load: 5.0 };
/// A design that takes longer than this on a runner is split into slices of about this much, or
/// five loads if that is more (each slice loads the design again), so no shard waits on one design.
const SLICE_SECS: f64 = 600.0;
/// Designs that take longer than this to load on a runner stay out of the pull request sample.
const SAMPLE_MAX_LOAD_SECS: f64 = 8.0;
/// A piece of work: one design, or a slice `(j, n)` of a heavy one, and what it should cost.
#[derive(Clone, Debug)]
struct Unit {
design: String,
slice: Option<(usize, usize)>,
cost: f64,
}
impl Unit {
fn name(&self) -> String {
match self.slice {
Some((j, n)) => format!("{} [{}/{n}]", self.design, j + 1),
None => self.design.clone(),
}
}
}
/// The whole matrix split into `parts` shards of about equal cost: heavy designs in slices, then
/// the largest pieces first, each to the shard with the least so far. The same corpus and
/// costs.json always give the same shards.
fn plan(designs: &[String], costs: &BTreeMap<String, Cost>, parts: usize) -> Vec<Vec<Unit>> {
let mut units = Vec::new();
for d in designs {
let c = costs.get(d).copied().unwrap_or(UNKNOWN_COST);
let n = ((c.secs / SLICE_SECS.max(5.0 * c.load)).ceil() as usize).max(1);
if n == 1 {
units.push(Unit { design: d.clone(), slice: None, cost: c.secs });
} else {
let each = c.load + (c.secs - c.load).max(0.0) / n as f64;
units.extend((0..n).map(|j| Unit { design: d.clone(), slice: Some((j, n)), cost: each }));
}
}
units.sort_by(|a, b| b.cost.total_cmp(&a.cost).then_with(|| a.name().cmp(&b.name())));
let mut shards: Vec<(f64, Vec<Unit>)> = vec![(0.0, Vec::new()); parts];
for u in units {
let i = (0..parts).min_by(|&a, &b| shards[a].0.total_cmp(&shards[b].0)).unwrap_or(0);
shards[i].0 += u.cost;
shards[i].1.push(u);
}
shards.into_iter().map(|(_, v)| v).collect()
}
/// Every design's name, with `ACTION_MATRIX_FILTER` applied.
fn design_names() -> Vec<String> {
let designs = designs();
assert!(designs.len() > 80, "only {} designs", designs.len());
let dir = std::env::temp_dir().join(format!("action-matrix-{}-{part}", std::process::id()));
let filter = std::env::var("ACTION_MATRIX_FILTER").ok();
designs.into_iter().map(|(n, _)| n).filter(|n| filter.as_deref().is_none_or(|f| n.contains(f))).collect()
}
/// Run the actions (one in `sample`) of these units in child processes, `jobs` at a time.
///
/// `ACTION_MATRIX_WORKDIR` keeps each design's raw outcomes there (one JSON line each, written as
/// they come, so a run cut short still shows what it did) and `ACTION_MATRIX_TIMINGS` appends
/// each unit's time to that file (JSON lines).
fn run_units(sample: u64, queue: &[Unit]) -> Vec<Value> {
let keep = std::env::var_os("ACTION_MATRIX_WORKDIR").map(PathBuf::from);
static RUN: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
let run = RUN.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let dir = keep.clone().unwrap_or_else(|| std::env::temp_dir().join(format!("action-matrix-{}-{run}", std::process::id())));
std::fs::create_dir_all(&dir).unwrap();
let timings = std::env::var_os("ACTION_MATRIX_TIMINGS").map(PathBuf::from);
let jobs: usize = std::env::var("ACTION_MATRIX_JOBS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or_else(|| std::thread::available_parallelism().map_or(2, |n| n.get()))
.max(1);
let filter = std::env::var("ACTION_MATRIX_FILTER").ok();
let queue: Vec<(String, PathBuf)> = designs
.into_iter()
.enumerate()
.filter(|(i, (n, _))| i % parts == part && filter.as_deref().is_none_or(|f| n.contains(f)) && (sample == 1 || !SLOW.contains(&n.as_str())))
.map(|(_, d)| d)
.collect();
let next = std::sync::atomic::AtomicUsize::new(0);
let results = std::sync::Mutex::new(Vec::new());
std::thread::scope(|scope| {
for _ in 0..jobs {
scope.spawn(|| {
while let Some((name, _)) = queue.get(next.fetch_add(1, std::sync::atomic::Ordering::Relaxed)) {
let out = dir.join(format!("{}.jsonl", name.replace('/', "__")));
let lines = run_design(name, &out, sample);
while let Some(u) = queue.get(next.fetch_add(1, std::sync::atomic::Ordering::Relaxed)) {
let file = u.name().replace('/', "__").replace([' ', '[', ']'], "_");
let out = dir.join(format!("{file}.r{run}.jsonl"));
let t = Instant::now();
let lines = run_design(&u.design, &out, sample, u.slice);
let secs = t.elapsed().as_secs_f64();
let load = lines.iter().find(|o| o["entity"] == "design" && o["action"] == "load").and_then(|o| o["ms"].as_f64()).unwrap_or(0.0) / 1000.0;
if let Some(p) = &timings {
let line = json!({"design": u.design, "slice": u.slice.map(|(j, n)| format!("{}/{n}", j + 1)), "secs": round3(secs), "load": round3(load), "actions": lines.len()});
let mut f = std::fs::OpenOptions::new().create(true).append(true).open(p).unwrap();
writeln!(f, "{line}").unwrap();
}
eprintln!("action matrix: {} took {secs:.1} s ({} outcomes, load {load:.1} s)", u.name(), lines.len());
results.lock().unwrap().extend(lines);
}
});
}
});
let _ = std::fs::remove_dir_all(&dir);
if keep.is_none() {
let _ = std::fs::remove_dir_all(&dir);
}
let mut v = results.into_inner().unwrap();
v.sort_by_key(|o| key(o["design"].as_str().unwrap_or(""), o["entity"].as_str().unwrap_or(""), o["action"].as_str().unwrap_or("")));
v
}
/// Run the actions (one in `sample`) of every design (the pull request sample leaves out the
/// slow-loading ones).
fn run_matrix(sample: u64) -> Vec<Value> {
let costs = read_costs();
let queue: Vec<Unit> = design_names()
.into_iter()
.filter(|n| sample == 1 || !(SLOW.contains(&n.as_str()) || costs.get(n).is_some_and(|c| c.load > SAMPLE_MAX_LOAD_SECS)))
.map(|design| Unit { design, slice: None, cost: 0.0 })
.collect();
run_units(sample, &queue)
}
/// One design in child processes, restarting after an action that hangs or aborts.
fn run_design(design: &str, out: &Path, sample: u64) -> Vec<Value> {
fn run_design(design: &str, out: &Path, sample: u64, slice: Option<(usize, usize)>) -> Vec<Value> {
let _ = std::fs::remove_file(out);
let mut from = 0usize;
let mut results = Vec::new();
@@ -759,6 +885,7 @@ fn run_design(design: &str, out: &Path, sample: u64) -> Vec<Value> {
.env("ACTION_MATRIX_OUT", out)
.env("ACTION_MATRIX_FROM", from.to_string())
.env("ACTION_MATRIX_SAMPLE", sample.to_string())
.env("ACTION_MATRIX_SLICE", slice.map(|(j, n)| format!("{}/{n}", j + 1)).unwrap_or_default())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
@@ -796,6 +923,9 @@ fn run_design(design: &str, out: &Path, sample: u64) -> Vec<Value> {
}
// It stopped inside action `last_start`: record it and go on after it.
let Some(i) = last_start else {
if slice.is_some_and(|(j, _)| j > 0) {
break;
}
extra.push(
json!({"design": design, "entity": "design", "action": "load", "ok": false, "kind": if timed_out { "timeout" } else { "crash" },
"message": format!("the design did not load ({status:?})")}),
@@ -863,9 +993,12 @@ fn write_baseline(results: &[Value]) {
m.entry(design).or_default().entry(entity).or_default().extend(actions.as_str().unwrap_or("").split_whitespace().map(str::to_string));
}
}
// A timeout of an action that passed keeps it passing (see [`ratchet`]).
let base = read_baseline();
for o in results {
let (design, entity, action) = (o["design"].as_str().unwrap_or(""), o["entity"].as_str().unwrap_or(""), o["action"].as_str().unwrap_or(""));
let mark = if o["ok"].as_bool() == Some(true) { "" } else { "!" };
let kept = o["kind"] == "timeout" && base.get(&key(design, entity, action)) == Some(&true);
let mark = if o["ok"].as_bool() == Some(true) || kept { "" } else { "!" };
m.entry(design).or_default().entry(entity).or_default().push(format!("{mark}{action}"));
}
// One entity per line, sorted: small diffs when a few actions change.
@@ -894,10 +1027,18 @@ fn ratchet(results: &[Value], full: bool) {
// baseline, over the same names this run tried, and in this run.
let mut fam: BTreeMap<String, (usize, usize, Vec<String>, Vec<String>)> = BTreeMap::new();
let mut fresh = 0usize;
// An action that passed in the baseline and now runs past ACTION_TIMEOUT is listed but not
// judged: on a busy runner the slowest actions (40-80 s on oracle 41's tube) land either side
// of the limit from one night to the next. An error or a crash is judged as usual.
let mut slow = Vec::new();
for o in results {
let (design, entity, action) = (o["design"].as_str().unwrap_or(""), o["entity"].as_str().unwrap_or(""), o["action"].as_str().unwrap_or(""));
let k = key(design, entity, action);
let ok = o["ok"].as_bool() == Some(true);
if o["kind"] == "timeout" && base.get(&k) == Some(&true) {
slow.push(k);
continue;
}
let f = fam.entry(key(design, &family(entity), action)).or_default();
match base.get(&k) {
Some(was) => {
@@ -934,6 +1075,14 @@ fn ratchet(results: &[Value], full: bool) {
eprintln!("baseline written: {}", baseline_path().display());
return;
}
if !slow.is_empty() {
eprintln!(
"{} actions that pass in the baseline ran past the {} s limit this time (not judged):\n{}",
slow.len(),
ACTION_TIMEOUT.as_secs(),
slow.join("\n")
);
}
if !fixed.is_empty() || fresh > 0 {
eprintln!(
"{} actions pass now that failed in the baseline, and {fresh} are not in it: record them with `cargo xtask action-matrix --bless`:\n{}",
@@ -952,51 +1101,156 @@ fn ratchet(results: &[Value], full: bool) {
/// Pull requests: a fixed sample of the matrix (one action in [`PR_SAMPLE`]) against the ratchet.
#[test]
fn action_matrix_sample() {
let results = run_matrix(PR_SAMPLE, 0, 1);
let results = run_matrix(PR_SAMPLE);
assert!(results.len() > 300, "only {} actions ran", results.len());
ratchet(&results, false);
}
/// The whole matrix against the ratchet, in four parts that the nightly CI job runs side by side
/// (`--run-ignored only`).
fn full_part(part: usize) {
let results = run_matrix(1, part, 4);
/// One shard of the whole matrix against the ratchet: `ACTION_MATRIX_SHARD=k/n` (the nightly
/// "Action matrix" CI jobs run the n shards side by side, `--run-ignored only`). Shards are cut by
/// what each design costs (tests/action_matrix/costs.json), see [`plan`].
/// `ACTION_MATRIX_SHARD_LIST=1` only prints the plan.
#[test]
#[ignore = "heavy: the nightly Action matrix CI jobs run it, one shard each"]
fn action_matrix_shard() {
let Some((k, n)) = std::env::var("ACTION_MATRIX_SHARD").ok().and_then(|v| parse_part(&v)) else { panic!("set ACTION_MATRIX_SHARD=k/n") };
let shards = plan(&design_names(), &read_costs(), n);
for (i, sh) in shards.iter().enumerate() {
let total: f64 = sh.iter().map(|u| u.cost).sum();
eprintln!("shard {}/{n}: {} pieces, about {:.0} s of work{}", i + 1, sh.len(), total, if i == k { " (this one)" } else { "" });
}
eprintln!("this shard: {}", shards[k].iter().map(Unit::name).collect::<Vec<_>>().join(", "));
if std::env::var_os("ACTION_MATRIX_SHARD_LIST").is_some() {
return;
}
let results = run_units(1, &shards[k]);
ratchet(&results, false);
}
/// Determinism: a sample of the matrix (one family in [`DETERMINISM_SAMPLE`], the heaviest
/// designs left out) run twice must give the same outcome for every action. Lists each action
/// whose outcome changed between the runs.
#[test]
#[ignore = "heavy: the nightly Action matrix CI job runs it"]
fn action_matrix_full_1() {
full_part(0);
}
#[test]
#[ignore = "heavy: the nightly Action matrix CI job runs it"]
fn action_matrix_full_2() {
full_part(1);
}
#[test]
#[ignore = "heavy: the nightly Action matrix CI job runs it"]
fn action_matrix_full_3() {
full_part(2);
}
#[test]
#[ignore = "heavy: the nightly Action matrix CI job runs it"]
fn action_matrix_full_4() {
full_part(3);
#[ignore = "heavy: the nightly Action matrix determinism CI job runs it"]
fn action_matrix_determinism() {
let outcomes = |v: Vec<Value>| -> BTreeMap<String, String> {
v.into_iter()
.map(|o| {
let k = key(o["design"].as_str().unwrap_or(""), o["entity"].as_str().unwrap_or(""), o["action"].as_str().unwrap_or(""));
let r = if o["ok"].as_bool() == Some(true) {
"pass".to_string()
} else {
format!("{}: {}", o["kind"].as_str().unwrap_or(""), o["message"].as_str().unwrap_or(""))
};
(k, r)
})
.collect()
};
let costs = read_costs();
let queue: Vec<Unit> = design_names()
.into_iter()
.filter(|n| costs.get(n).copied().unwrap_or(UNKNOWN_COST).secs <= DETERMINISM_MAX_SECS)
.map(|design| Unit { design, slice: None, cost: 0.0 })
.collect();
let a = outcomes(run_units(DETERMINISM_SAMPLE, &queue));
let b = outcomes(run_units(DETERMINISM_SAMPLE, &queue));
let keys: BTreeSet<&String> = a.keys().chain(b.keys()).collect();
let mut changed = Vec::new();
// Pieces that share a name can swap numbers between runs (#569): compare passes per family
// too, and report a family whose pass count differs.
let mut fam: BTreeMap<String, (usize, usize)> = BTreeMap::new();
for k in &keys {
let (x, y) = (a.get(*k).map_or("missing", String::as_str), b.get(*k).map_or("missing", String::as_str));
let mut parts = k.splitn(3, " :: ");
let (d, e, act) = (parts.next().unwrap_or(""), parts.next().unwrap_or(""), parts.next().unwrap_or(""));
let f = fam.entry(key(d, &family(e), act)).or_default();
f.0 += usize::from(x == "pass");
f.1 += usize::from(y == "pass");
if x != y {
changed.push(format!("{k}\n first: {x}\n second: {y}"));
}
}
let families: Vec<String> = fam.into_iter().filter(|(_, (x, y))| x != y).map(|(k, (x, y))| format!("{k}: {x} then {y} pass")).collect();
eprintln!(
"action matrix determinism: {} actions, {} changed between two runs, {} families pass a different number of times",
keys.len(),
changed.len(),
families.len()
);
for c in &changed {
eprintln!(" {c}");
}
assert!(
families.is_empty(),
"{} families of actions pass a different number of times in two runs of the same build:\n{}",
families.len(),
families.join("\n")
);
}
/// The whole matrix at once (`cargo xtask action-matrix`): `ACTION_MATRIX_BLESS=1` rewrites the
/// baseline; `ACTION_MATRIX_RESULTS=file.json` keeps every outcome with its message.
/// `ACTION_MATRIX_MERGE=DIR` runs nothing and takes the outcomes from the `results-*.json` files
/// in DIR instead (the nightly shards' artifacts), and the timings from its `timings-*.jsonl`
/// files, which `ACTION_MATRIX_BLESS` then also writes to tests/action_matrix/costs.json.
#[test]
#[ignore = "cargo xtask action-matrix runs it (CI runs it as the four action_matrix_full parts)"]
#[ignore = "cargo xtask action-matrix runs it (CI runs it as the action_matrix_shard jobs)"]
fn action_matrix_all() {
let results = run_matrix(1, 0, 1);
let Some(dir) = std::env::var_os("ACTION_MATRIX_MERGE").map(PathBuf::from) else {
let results = run_matrix(1);
ratchet(&results, true);
return;
};
let mut results: Vec<Value> = Vec::new();
let mut timings: Vec<Value> = Vec::new();
let mut files: Vec<PathBuf> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.path()).collect();
files.sort();
for f in files {
let name = f.file_name().unwrap_or_default().to_string_lossy().to_string();
let text = std::fs::read_to_string(&f).unwrap_or_default();
if name.starts_with("results-") && name.ends_with(".json") {
results.extend(serde_json::from_str::<Vec<Value>>(&text).unwrap_or_else(|e| panic!("{name}: {e}")));
} else if name.starts_with("timings-") {
timings.extend(text.lines().filter_map(|l| serde_json::from_str(l).ok()));
}
}
let ran: BTreeSet<&str> = results.iter().filter_map(|o| o["design"].as_str()).collect();
let missing: Vec<String> = design_names().into_iter().filter(|d| !ran.contains(d.as_str())).collect();
assert!(missing.is_empty(), "no results for {}", missing.join(", "));
results.sort_by_key(|o| key(o["design"].as_str().unwrap_or(""), o["entity"].as_str().unwrap_or(""), o["action"].as_str().unwrap_or("")));
if std::env::var_os("ACTION_MATRIX_BLESS").is_some() && !timings.is_empty() {
write_costs(&timings);
}
ratchet(&results, true);
}
/// tests/action_matrix/costs.json from the shards' timings: per design, its whole time (a sliced
/// design's slices added up, less the loads all but one of them repeat) and its load.
fn write_costs(timings: &[Value]) {
let mut m: BTreeMap<String, (f64, f64, usize)> = BTreeMap::new();
for t in timings {
let e = m.entry(t["design"].as_str().unwrap_or("").to_string()).or_default();
e.0 += t["secs"].as_f64().unwrap_or(0.0);
e.1 = e.1.max(t["load"].as_f64().unwrap_or(0.0));
e.2 += 1;
}
// Designs these timings don't cover keep their old costs.
for (d, c) in read_costs() {
m.entry(d).or_insert((c.secs, c.load, 1));
}
let mut s = String::from(
"{\n \"about\": \"What each action matrix design costs on a CI runner, in seconds (secs: the whole design, load: before its first action), for cutting the nightly shards (apps/solidcraft-cli/tests/action_matrix.rs, plan). Written with the baseline by cargo xtask action-matrix --bless --from DIR.\",\n \"designs\": {\n",
);
let n = m.len();
for (i, (d, (secs, load, pieces))) in m.into_iter().enumerate() {
let secs = (secs - load * pieces.saturating_sub(1) as f64).max(load);
s += &format!(" {}: {{\"secs\": {:.1}, \"load\": {:.1}}}{}\n", json!(d), secs, load, if i + 1 < n { "," } else { "" });
}
s += " }\n}\n";
std::fs::write(costs_path(), s).unwrap();
eprintln!("costs written: {}", costs_path().display());
}
// ---------------------------------------------------------------------------------------------
// The matrix's causes: one small reproduction per open issue, ignored until it is fixed (the
// issue says how many matrix actions it accounts for). Each runs commands as the desktop sends