Add design containers with deduplicated blobs and JSON compatibility (#327)

* Add fast design containers with lossless deduplicated payloads

* Keep display caching opt-in to avoid default save overhead

docs/book/src/guide/files.md

@@ -2,7 +2,8 @@
 
 | Format | Read | Write |
 |---|---|---|
-| `.solidcraft` (JSON design, versioned) | yes | yes |
+| `.solidcraft` (versioned container or JSON design) | yes | yes |
+| `.json` (self-contained plain JSON design) | yes | yes |
 | STEP (AP203, AP214, AP242) | solids, assemblies, units, names, colours | AP242 with names, colours and the component tree |
 | IGES 5.3 | manifold solids, or trimmed surfaces sewn into solids | manifold solids with names and colours |
 | 3MF, STL, OBJ | mesh bodies | yes |
@@ -32,7 +33,26 @@ replaced version is kept as `<file>.bak`. Unsaved changes are autosaved to a rec
 and the next launch offers to recover them. Older design files are upgraded on open, and files
 from a newer version are refused with a message.
 
-Current designs use format 12, including persistent analytic offset boundary roles and
+**Save** writes a container by default; files named `.json` keep JSON on subsequent saves.
+**Save As** offers a container or plain JSON on
+desktop; **Save As JSON** is also available in the File menu on desktop and web. Choose
+the `.json` extension for plain JSON. **Export Design as JSON** writes an independent,
+self-contained copy without changing the current file. Scripts can explicitly pass
+`"format": "json"` to `file.save` or `file.save_as` even with a `.solidcraft` name.
+Opening detects the format from the contents, so existing JSON designs keep working.
+Containers favor fast reading and saving by default. Scripts can request
+`"compression": "compact"` when file size matters more than CPU time.
+
+Containers deduplicate identical STEP, mesh and image payloads. Mesh coordinates retain
+their original double precision; this is not a conversion to STL. Geometry still rebuilds
+from the timeline when opening. When a saved display preview is available, desktop Open
+shows it while checking geometry; editing starts after the complete rebuild. Cancelling
+keeps the current design. Previews cannot be edited or measured. This experimental option
+requires `preview: true` in an engine Save/Save As call; ordinary saves omit display caches.
+The repository's `docs/design-container.md` describes limits and encoding details.
+
+Current designs use format 14, including persistent names from exact face history,
+exact planar shell-face anchors, persistent analytic offset boundary roles and
 association profile references, durable sketch face anchors, periodic fit splines and versioned
 revolve conventions. It also retains recorded body transfers, explicit 3D curve coordinate
 spaces and certified face bindings, alongside semantic configuration styles, associative

docs/compatibility.md

@@ -127,3 +127,10 @@ replace any Dock shortcut with the old name (open `SolidCraft.app`, then
 
 `flatpak remote-modify --user --url=https://bherbruck.github.io/solidcraft/flatpak/repo/ solvecraft`
 (use your remote's name), then `flatpak update --user`.
+
+## Draft design containers
+
+The #303 draft used `solvecraft-container/1`. `crates/io/src/design.rs` accepts that
+marker alongside `solidcraft-container/1`, and writes only the new marker. Optional
+old previews are discarded; required document/blob validation is unchanged. The
+`draft_container_marker_remains_readable` regression covers this shim.

docs/design-container.md

@@ -0,0 +1,91 @@
+# Design containers, version 1
+
+New designs use `.solidcraft`; `.solvecraft` remains a supported legacy extension. Opening detects the ZIP
+local-header signature (`50 4b 03 04`) independently of the name; otherwise it uses
+the existing JSON design reader and migrations. Plain JSON remains a fully supported,
+self-contained input/output format. `io::write_design` keeps its JSON contract;
+`io::write_design_container` is the fallible container writer.
+
+Packages have `container.json` with `{"format":"solidcraft-container/1"}` and
+`document.json` with the versioned parametric document (currently `solidcraft/14`).
+The independent versions let document migrations evolve without changing blob encodings.
+Existing serialized document fields are retained automatically, without a second copy
+of the document schema or a deep clone of embedded payloads.
+
+Within `document.json`:
+
+- Import features have empty `step` and a `step_blob` reference.
+- Mesh imports retain each mesh's name and optional colour, with empty `positions`
+  and `triangles` plus a `mesh_blob` reference.
+- Canvases retain placement, format and pixel metadata, with empty `data` plus an
+  `image_blob` reference. Raw image bytes replace base64 in the package.
+
+References are `blobs/<lowercase SHA-256>.<step|mesh|png|jpeg>`, hashed over the exact
+uncompressed payload. Identical content with the same encoding is written once.
+Required parts, malformed references, ambiguous embedded payloads, incorrect hashes,
+invalid mesh data and unsupported versions produce errors before session adoption.
+
+## Mesh encoding
+
+Mesh blobs begin with the eight ASCII bytes `SCMESH01`, followed by little-endian
+`u64` coordinate count and `u64` index count. These count scalar values, not vertices
+or triangles. Next come that many IEEE-754 `f64` coordinates, then `u32` indices.
+Both counts must be divisible by three; the byte length must match exactly. Coordinates
+must be finite and at most 1e9 mm in magnitude; indices must address an existing vertex.
+Names and colours remain in JSON. Signed zero and full coordinate precision survive.
+
+The default `fast` policy stores STEP and binary meshes without Deflate for faster
+saving and reading. Small document metadata uses Deflate level one; PNG/JPEG are
+stored directly. An explicit `compact` policy compresses STEP and mesh payloads at
+level one. Readers support both policies. This deliberately favors interactive file
+operations over maximum compression; compressed STEP was slower in the storage benchmark.
+Use `"compression":"compact"` with `file.save` or `file.save_as` to favor file size,
+or `io::write_design_container_with_compression` from Rust.
+
+## Bounds and recovery
+
+Input and individual expanded parts are capped at 256 MiB. Total declared expanded
+parts and resolved payloads are each capped at 512 MiB, including repeated references
+in the latter budget. There are at most 10,000 entries. Duplicate/ambiguous entry names,
+path traversal, ZIP64 and multi-disk packages are refused. Entries are read in memory;
+no archive paths are extracted to the filesystem. Exact part lengths and ZIP CRCs are
+checked, along with blob hashes. Hashes detect correspondence and corruption, not trust.
+
+Saving retains the existing temporary-file/flush/rename protocol and `.bak` backup.
+Recovery reads both older JSON and new containers. Browser OPFS, downloads and recovery
+use the same container reader/writer; JSON export and Save As remain available.
+
+## Provisional saved display
+
+Experimental engine saves with `preview: true` can include `cache/preview.json` and
+`cache/preview.bin` when the evaluated
+world display meshes are already warm. Ordinary saves omit this redundant display data
+to avoid extra save work; it is not the proposed evaluated B-rep cache. Saving never triangulates a cold body for this
+cache. Pending occurrence moves, active sketch edits, feature errors and oversized
+previews omit it. The cache is bounded to 8 MiB, 1,024 bodies and 100,000 triangles;
+its manifest is at most 64 KiB. Body names/colours and contiguous binary mesh slices
+use the same lossless encoding as imported meshes. Extra cache size is optional;
+it is omitted if it would put the container near the file-size cap.
+
+The manifest includes SHA-256 of the exact document metadata (whose blob references
+contain content hashes), the preview payload, `solidcraft-preview/1` and the application
+package version. A version mismatch invalidates the display cache. Development builds
+with the same package version may briefly show an older picture before the authoritative
+rebuild replaces it; previews never supply model state. Earlier draft containers using
+`solvecraft-container/1` remain readable, with their obsolete previews treated as misses.
+Missing, obsolete, oversized or malformed preview data is a cache miss, not a failed
+design open. All archive-wide safety budgets still apply.
+
+Desktop Open can render these meshes on its existing file worker and show a clearly
+labelled saved picture in the opening dialog before timeline reconstruction finishes.
+It uses only the normal command's file snapshot, without a second file read. The picture
+is approximate shading without face paint or picking. A conservative 16-million-pixel
+raster budget rejects excessive overdraw before rendering, including repeated large
+triangles in a small hostile cache. The picture is never adopted as editable
+geometry, used for measurements, or used to seed evaluated model state. The original
+design remains intact until the complete rebuilt session is ready. Cancellation or
+any change of active document/revision discards both preview and completion.
+
+Normal design reading, CLI and browser opening still rebuild authoritatively; no
+evaluated B-rep is deserialized or adopted. Portable evaluated B-rep/state restoration
+and interactive cached viewport navigation remain further work in #303.

docs/perf.md

@@ -52,6 +52,105 @@ different worker counts: exact display meshes, measures, body/face source identi
 names, colours/opacity, units, rotated assembly placements, duplicates and partial failures
 across the batch boundary. The public analytic/FreeCAD reference suite also checks STEP
 export/import; its current results are reported in docs/oracle.md.
+## Design storage: container and binary payloads (#303, phase one)
+
+Design containers avoid formatting/parsing imported mesh coordinates as JSON numbers.
+The metadata serializer replaces embedded fields before creating JSON values, without
+cloning the full document or duplicating its schema. Binary coordinates retain `f64`
+precision, indices retain `u32`, and SHA-256 deduplicates identical payloads. The reader
+checks archive budgets, CRCs, hashes, exact lengths, finite coordinates and index bounds.
+
+Reproduce the storage measurements with an optimized `ci` IO test executable:
+
+```sh
+cargo test -p solidcraft-io --profile ci benchmark_ -- --ignored --nocapture --test-threads=1
+```
+
+The benchmarks generate an original 100,000-vertex/100,000-triangle mesh and a STEP
+containing 512 disjoint 10 mm boxes. Fixtures are written under `CARGO_TARGET_DIR`
+(or the crate's `target` directory). Each compares nine encoding/decoding samples;
+atomic-save samples include encoding, temporary-file flush, replacement and `.bak`
+creation and alternate JSON/container ordering. Documents are compared after decoding.
+These are storage measurements, separate from timeline evaluation or first-frame latency.
+
+On Windows x64, based on upstream `e6e0a7e` before STEP parallelism (#312), three
+consecutive optimized runs without concurrent compilation gave
+the following medians of the three nine-sample medians (milliseconds, fast mode):
+
+| Fixture / operation | JSON | Container |
+|---|---:|---:|
+| Mesh size, bytes | 15,139,115 | 3,600,828 |
+| Mesh encode | 23.336 | 6.094 |
+| Mesh decode | 57.175 | 2.962 |
+| Mesh atomic save | 52.924 | 14.856 |
+| STEP size, bytes | 1,998,793 | 1,950,056 |
+| STEP encode | 1.507 | 1.473 |
+| STEP decode | 1.479 | 1.318 |
+| STEP atomic save | 7.213 | 7.087 |
+
+Three alternating fresh-process `solidcraft-cli bench-open` runs of each STEP encoding
+returned the same 512 bodies and 6,144 triangles. Median read-plus-parse was 2.274 ms
+for JSON and 2.151 ms for the container; median evaluation was 218.559/236.915 ms and
+display meshing 203.160/197.790 ms respectively. Geometry dominates and these variable
+results do not establish an end-to-end STEP opening improvement. Fresh processes do
+not imply a cold operating-system file cache, and these are not desktop first-frame times.
+
+Fast mode stores STEP and mesh payloads directly. Explicit `"compression":"compact"`
+compresses them at Deflate level one; it is a space/CPU tradeoff. In the initial
+512-box STEP experiment, compression reduced 1,998,793 JSON bytes to 578,659 package
+bytes but increased median encoding from 1.68 to 5.02 ms and decoding from 1.76 to
+4.18 ms. That regression is why compression is not the default. PNG/JPEG are stored
+directly in either policy; small parametric metadata uses Deflate level one.
+
+Plain JSON remains a supported Save As/export format; `.json` saves keep JSON and
+scripts may explicitly select either format. The CLI detects packages before attempting
+JSON parsing. Desktop/browser recovery and browser downloads also use the container.
+See [the versioned container format](design-container.md) for layout and bounds.
+
+This phase does not restore evaluated B-reps, skip STEP import, or make cold geometry
+rebuilds instant. A matching cache key is not enough to trust unvalidated geometry.
+Evaluated B-rep/state restoration remains further work in #303. The provisional saved
+display described below is separate from #287's import parallelism.
+
+## Provisional saved display (#303)
+
+Desktop Open can show an experimental saved picture while the existing file worker rebuilds geometry.
+It requires an explicit `preview: true` engine Save/Save As call. Ordinary saves omit
+display caches to avoid the measured overhead; native evaluated B-rep caching remains open.
+The cache contains only already warm evaluated display meshes, capped at 8 MiB, 1,024
+bodies and 100,000 triangles. Larger or cold models save without it. Cached geometry is
+never used for editing, measurements or picking. Source/document fingerprints and
+payload validation reject stale or malformed caches; cancellation and document/revision
+checks prevent a background result from replacing newer work.
+
+The cached picture is prepared on the file worker from the same file snapshot as the
+normal Open command. It adds work before reconstruction, rather than bypassing it, so
+measure preview readiness and full editing readiness separately. Cached display also
+adds file size and save work; the storage-only results above exclude that optional data.
+
+For an engine-level save benchmark, first generate the original 512-box STEP fixture
+with the ignored IO benchmark, then run the warmed-session benchmark:
+
+```sh
+cargo test --workspace --profile ci benchmark_step_storage -- --ignored --nocapture --test-threads=1
+cargo test --workspace --profile ci benchmark_warm_design_save_with_preview -- --ignored --nocapture --test-threads=1
+```
+
+Set `CARGO_TARGET_DIR` to the workspace's absolute target directory in both commands.
+The engine benchmark alternates nine JSON/container atomic saves, includes the optional
+display cache, and verifies both preview decoding and authoritative document identity.
+Three runs on Windows, with no concurrent build or test suite, measured median atomic
+JSON saves of 7.679/7.610/7.630 ms and container-with-preview saves of
+10.319/10.633/10.541 ms. Preview decoding took 0.964/0.961/1.130 ms; the container
+was 2,510,490 bytes. This optional preview costs save time on this STEP fixture.
+An actual desktop sensor-enclosure opening on e8a8e0f prepared its saved picture in
+24.320 ms; geometry reconstruction, display warmup and thumbnail preparation took
+8,913.212 ms. The same preview experiment is now explicitly opt-in; full geometry
+readiness still requires reconstruction.
+For actual desktop opening phases, launch with `SOLIDCRAFT_PROFILE_OPEN=1`: `ready_ms`
+reports saved-picture preparation, while the normal phase log reports reconstruction,
+display warmup and thumbnail completion. Neither log measures the exact GPU first frame.
+
 ## Analytic extrusion source preparation
 
 New-body extrusion certificates now prepare the sketch's bounded analytic segments once