fix(gpu): a crash the device caused still costs the backend, however late

#1519 stopped the sentinel blaming a backend for a run that had already
drawn for SENTINEL_PROOF_DELAY, on the reasoning that a backend which
kept the application alive that long had proved itself. That reasoning
does not hold for the failure this repository sees most on integrated
hardware.

Measured here on an Intel Iris Xe: with `--backend dx12` a large drawing
aborts the process 110 s in, long past the proof delay, so the sentinel
carried `survived=1` and the next launch went straight back to dx12 and
crashed again. The same binary on `--backend vulkan` opens the same
drawings with no out-of-memory report and no uncaptured errors at all,
so the backend really is the difference and moving off it really is the
cure. D3D12 is fine while the interface draws and only fails once the
whole scene uploads, which is why the clock is the wrong question.

The crash report from #1521 answers the right one. It records the
backend that was live and what the panic was, and it is named for the
process that wrote it, so it joins exactly to the `pid` the sentinel
already stores — no guessing from whichever report is newest, and no
new state to keep. A sentinel marked survived is now excused only when
that run left no report, or left one that has nothing to do with the
graphics device.

The test that matters is the pair, and both were run end to end:

  killed from Task Manager after 40 s, no report  -> dx12 kept
  device crash after 110 s, report names wgpu     -> dx12 skipped, Vulkan

`is_device_failure` is deliberately narrow: an indexing bug in drawing
code that happens to run on a frame must not cost the user a working
backend.

Co-Authored-By: Claude Opus 5 <[email protected]>

src/gpu_backend.rs

@@ -319,17 +319,46 @@ pub fn crashed_backends() -> Vec<String> {
         return Vec::new();
     };
     let text = std::fs::read_to_string(p).unwrap_or_default();
+    let named = || -> Vec<String> {
+        text.lines()
+            .find_map(|l| l.strip_prefix("backend="))
+            .unwrap_or("")
+            .split(',')
+            .map(str::trim)
+            .filter(|s| !s.is_empty())
+            .map(str::to_string)
+            .collect()
+    };
     if text.lines().any(|l| l.trim() == SURVIVED_MARKER) {
-        return Vec::new();
+        // Surviving the proof delay is good evidence, not proof. A backend
+        // can carry the interface for minutes and still be unable to hold a
+        // large drawing: D3D12 on an integrated GPU runs out of room only
+        // once the whole scene uploads, well past the mark, and a run killed
+        // then used to be excused. The crash report says what actually
+        // happened, so ask it before excusing anything.
+        return match crash_report_blames_the_device(&text) {
+            true => named(),
+            false => Vec::new(),
+        };
     }
-    text.lines()
-        .find_map(|l| l.strip_prefix("backend="))
-        .unwrap_or("")
-        .split(',')
-        .map(str::trim)
-        .filter(|s| !s.is_empty())
-        .map(str::to_string)
-        .collect()
+    named()
+}
+
+/// Whether the run this sentinel belongs to died inside the graphics device.
+///
+/// The sentinel records the process id it armed for, and crash reports are
+/// named for the process that wrote them, so the two join exactly: this asks
+/// about that run and not whichever report happens to be newest.
+fn crash_report_blames_the_device(sentinel: &str) -> bool {
+    let Some(pid) = sentinel
+        .lines()
+        .find_map(|line| line.strip_prefix("pid="))
+        .and_then(|value| value.trim().parse::<u32>().ok())
+    else {
+        return false;
+    };
+    crate::sys::crash_log::report_for_pid(pid)
+        .is_some_and(|report| crate::sys::crash_log::is_device_failure(&report))
 }
 
 /// Record that the armed backend has kept the app alive long enough to be
@@ -1086,6 +1115,67 @@ mod tests {
         let _ = std::fs::remove_dir_all(&dir);
     }
 
+    /// Surviving the proof delay excuses a kill, but not a crash the
+    /// graphics device itself caused. D3D12 on an integrated GPU carries the
+    /// interface for minutes and only fails once a large drawing uploads, so
+    /// a run that died then has to keep costing the backend.
+    #[test]
+    fn a_device_crash_still_blames_the_backend_however_late_it_came() {
+        let _guard = SENTINEL_TESTS.lock().unwrap_or_else(|e| e.into_inner());
+        let dir = std::env::temp_dir().join(format!("ocs-gpu-late-{}", std::process::id()));
+        let _ = std::fs::create_dir_all(&dir);
+        // SAFETY: single-threaded test mutating a process-local test hook.
+        unsafe { std::env::set_var(SENTINEL_DIR_ENV, &dir) };
+
+        let reports = crate::sys::crash_log::directory().expect("a config directory");
+        let _ = std::fs::create_dir_all(&reports);
+        let report = reports.join(format!("crash-1-{}.log", std::process::id()));
+
+        arm_sentinel("dx12");
+        mark_sentinel_survived();
+        let _ = std::fs::remove_file(&report);
+        assert!(
+            crashed_backends().is_empty(),
+            "with no crash report, a survived run blames nobody"
+        );
+
+        // The run left a report, and it is the device's own failure.
+        std::fs::write(
+            &report,
+            crate::sys::crash_log::report_from(
+                "Error in Buffer::get_mapped_range: Validation Error",
+                "wgpu-29.0.4/src/backend/wgpu_core.rs:2253:18",
+                1,
+            ),
+        )
+        .unwrap();
+        assert_eq!(
+            crashed_backends(),
+            vec!["dx12".to_string()],
+            "a device failure must cost the backend even past the proof delay"
+        );
+
+        // A crash that has nothing to do with graphics still costs nothing.
+        std::fs::write(
+            &report,
+            crate::sys::crash_log::report_from(
+                "index out of bounds: the len is 3 but the index is 7",
+                "src/app/commands/draw.rs:120:5",
+                1,
+            ),
+        )
+        .unwrap();
+        assert!(
+            crashed_backends().is_empty(),
+            "an unrelated panic must not cost the user a working backend"
+        );
+
+        let _ = std::fs::remove_file(&report);
+        disarm_sentinel();
+        unsafe { std::env::remove_var(SENTINEL_DIR_ENV) };
+        let _ = std::fs::remove_dir_all(&dir);
+    }
+
     #[test]
     fn force_hooks_drive_probes_without_spawning() {
         // SAFETY: single-threaded test mutating process-local test hooks.