forked from 1b-agent/opencadstudio-upstream
Add audited, versioned MCP save workflow (#1452)
- `--export ... --target-version`, and explicit target_format / target_version on automation and protocol saves; a conversion that would drop unsupported records is refused unless allow_lossy is set. - `audit` read op: dangling layers/blocks, duplicate handles, non-finite bounds, serialization, lossy records and the source DXF handle graph. - `save_verified`: writes the requested version from the same snapshot a normal save writes, reopens it, checks the version and the entity manifest, audits the ASCII DXF handle graph and returns a SHA-256. - ZOOM EXTENTS includes SDF text glyphs. - Live MCP acceptance script and an optional reference-console audit runner under docs/automation. Squashed from the PR with review fixes: merged with the DWT save path, the save snapshot, the tool lists added since, and vendor-neutral harness names; the dated verification record is left out. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0f99b77004
commit
dddf60d79e
16 files changed
+1102
-22
No files matched your search
@@ -20,6 +20,19 @@ The server provides four tools:
|
||||
- `ocs_execute` performs one operation, an atomic record update, or a sequential batch against the real editor.
|
||||
- `ocs_capture` returns a bounded PNG of the drawing viewport or complete window.
|
||||
|
||||
Run `python docs/automation/mcp_acceptance.py target/debug/OpenCADStudio.exe`
|
||||
for a repeatable end-to-end acceptance. It creates visible geometry through
|
||||
MCP, audits and verified-saves DWG 2000/2013/2018 plus DXF 2000, and records a
|
||||
viewport PNG and JSON report below `target/acceptance/`.
|
||||
|
||||
On a Windows workstation with a reference CAD application's command-line
|
||||
console, run
|
||||
`docs/automation/reference_acceptance.ps1 -Directory target/acceptance/TIMESTAMP -ConsolePath <console.exe>`.
|
||||
It opens the three generated DWGs read-only in isolated console profiles, runs
|
||||
AUDIT without repairs, and keeps one log per version. A console that stays
|
||||
alive after `QUIT` is terminated after the timeout (only the process the runner
|
||||
launched), and `ForcedTermination` is recorded separately from the audit result.
|
||||
|
||||
Native builds also accept a second, headless entry point (`OpenCADStudio --serve`, one
|
||||
JSON request per line) for scripts and CI; the two native channels, their startup dialogs
|
||||
and their limits are described in [native.md](native.md).
|
||||
@@ -40,6 +53,42 @@ The normal flow is to call `ocs_sessions`, pass its returned `session_id` as `oc
|
||||
|
||||
Call `ocs_read` with `op: "capabilities"` before unfamiliar work. It reports the command, geometry, transaction, capture, and database facilities supported by the running build. The `records.collections` list gives every available database collection, its record count, and whether it can be edited.
|
||||
|
||||
Before delivery, call `ocs_read` with `op: "audit"` and the intended
|
||||
`target_format` / `target_version`. The report includes entity manifests,
|
||||
layers, blocks, finite bounds, duplicate entity handles, DXF dangling handle
|
||||
references, source hash, and `dropped_on_save`. A clean in-memory document can
|
||||
still be lossy for a different format, so target selection is part of the
|
||||
audit:
|
||||
|
||||
```json
|
||||
{
|
||||
"ocs_session_id": "SESSION_FROM_OCS_SESSIONS",
|
||||
"op": "audit",
|
||||
"parameters": {"target_format": "dwg", "target_version": "2000"}
|
||||
}
|
||||
```
|
||||
|
||||
Use `save_verified` for the delivery write. It requires an explicit absolute
|
||||
path, refuses overwrite and lossy conversion unless separately acknowledged,
|
||||
writes the requested version, reopens it, checks the actual version, compares
|
||||
the semantic entity manifest, audits raw DXF handle references, and returns a
|
||||
SHA-256 hash. A verification failure preserves the output for diagnosis and
|
||||
reports it as failed instead of silently declaring success.
|
||||
|
||||
```json
|
||||
{
|
||||
"ocs_session_id": "SESSION_FROM_OCS_SESSIONS",
|
||||
"request": {
|
||||
"op": "save_verified",
|
||||
"request_id": "deliver-2026-09-22-1",
|
||||
"path": "C:\\drawings\\issued\\plan.dwg",
|
||||
"target_format": "dwg",
|
||||
"target_version": "2000",
|
||||
"overwrite": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Call `ocs_read` with `op: "record_schema"` before editing an unfamiliar record. With no parameters it lists the complete generated type registry. A collection returns the record types accepted by that collection even when the current drawing has no instance of a type. Supplying both `collection` and `type` returns the type's complete dependency graph, flattened property paths, JSON types, optional and sequence markers, enum variants, integer bounds, unambiguous unit annotations, identity fields, and write rules:
|
||||
|
||||
```json
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
"""End-to-end GUI/MCP acceptance with versioned, verified CAD outputs."""
|
||||
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from mcp_eval import Client, META
|
||||
|
||||
|
||||
def request_id(prefix: str) -> str:
|
||||
return f"accept-{prefix}-{uuid.uuid4().hex}"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
server = Path(sys.argv[1] if len(sys.argv) > 1 else "target/debug/OpenCADStudio.exe").resolve()
|
||||
root = Path(sys.argv[2] if len(sys.argv) > 2 else "target/acceptance").resolve()
|
||||
output = root / time.strftime("%Y%m%d-%H%M%S")
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
client = Client(server)
|
||||
session = None
|
||||
report: dict = {"ok": False, "server": str(server), "output": str(output)}
|
||||
try:
|
||||
sessions = client.tool("ocs_sessions", {"launch_if_none": True})["result"]
|
||||
session = sessions[0]["session_id"]
|
||||
while sessions[0].get("modal"):
|
||||
client.tool(
|
||||
"ocs_execute",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"request": {
|
||||
"op": "action",
|
||||
"request_id": request_id("close-modal"),
|
||||
"name": "close_modal",
|
||||
},
|
||||
},
|
||||
)
|
||||
sessions = client.tool("ocs_sessions", {"launch_if_none": False})["result"]
|
||||
|
||||
client.tool(
|
||||
"ocs_execute",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"request": {"op": "new", "request_id": request_id("new")},
|
||||
},
|
||||
)
|
||||
drawing = client.tool(
|
||||
"ocs_execute",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"response_detail": "changed_entities",
|
||||
"request": {
|
||||
"op": "batch",
|
||||
"request_id": request_id("draw"),
|
||||
"steps": [
|
||||
{"op": "run", "cmd": "PLINE 0,0 200,0 200,120 0,120 C"},
|
||||
{"op": "run", "cmd": "CIRCLE 50,60 15"},
|
||||
{"op": "run", "cmd": "CIRCLE 150,60 15"},
|
||||
{"op": "run", "cmd": "LINE 0,60 200,60"},
|
||||
{"op": "run", "cmd": "TEXT 100,135 8 0 OPEN CAD MCP ACCEPTANCE"},
|
||||
{"op": "action", "name": "zoom_extents"},
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
assert drawing["completed_steps"] == 6, drawing
|
||||
|
||||
audits = {}
|
||||
saves = {}
|
||||
targets = [("dwg", "2000"), ("dwg", "2013"), ("dwg", "2018"), ("dxf", "2000")]
|
||||
for target_format, target_version in targets:
|
||||
key = f"{target_format}-{target_version}"
|
||||
audit = client.tool(
|
||||
"ocs_read",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"op": "audit",
|
||||
"parameters": {
|
||||
"target_format": target_format,
|
||||
"target_version": target_version,
|
||||
},
|
||||
},
|
||||
)
|
||||
assert audit["ok"] is True, audit
|
||||
audits[key] = audit
|
||||
destination = output / f"mcp-acceptance-{target_version}.{target_format}"
|
||||
saved = client.tool(
|
||||
"ocs_execute",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"response_detail": "full",
|
||||
"request": {
|
||||
"op": "save_verified",
|
||||
"request_id": request_id(f"save-{key}"),
|
||||
"path": str(destination),
|
||||
"target_format": target_format,
|
||||
"target_version": target_version,
|
||||
},
|
||||
},
|
||||
)
|
||||
verified = saved.get("result", saved)
|
||||
assert verified["verified"] is True, saved
|
||||
assert destination.is_file(), destination
|
||||
saves[key] = verified
|
||||
|
||||
capture = client.rpc(
|
||||
"tools/call",
|
||||
{
|
||||
"name": "ocs_capture",
|
||||
"arguments": {
|
||||
"ocs_session_id": session,
|
||||
"scope": "viewport",
|
||||
"max_dimension": 1600,
|
||||
},
|
||||
"_meta": META,
|
||||
},
|
||||
)
|
||||
image = capture["content"][0]
|
||||
assert image["type"] == "image" and image["mimeType"] == "image/png", capture
|
||||
png = base64.b64decode(image["data"], validate=True)
|
||||
assert png[:8] == b"\x89PNG\r\n\x1a\n", "capture is not a PNG"
|
||||
width, height = struct.unpack(">II", png[16:24])
|
||||
capture_path = output / "viewport.png"
|
||||
capture_path.write_bytes(png)
|
||||
|
||||
report.update(
|
||||
{
|
||||
"ok": True,
|
||||
"session_id": session,
|
||||
"drawn_entities": drawing["changed_entities"],
|
||||
"audits": audits,
|
||||
"saves": saves,
|
||||
"capture": {"path": str(capture_path), "width": width, "height": height},
|
||||
"metrics": {
|
||||
"tool_calls": client.tool_calls,
|
||||
"tasks": client.tasks,
|
||||
"rpc_calls": client.rpc_calls,
|
||||
},
|
||||
}
|
||||
)
|
||||
finally:
|
||||
client.close()
|
||||
report_path = output / "report.json"
|
||||
report_path.write_text(json.dumps(report, indent=2), encoding="utf-8")
|
||||
print(json.dumps(report, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,6 +5,7 @@ from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
MODERN = "2026-07-28"
|
||||
@@ -91,6 +92,19 @@ def main() -> None:
|
||||
assert "io.modelcontextprotocol/tasks" in discovery["capabilities"]["extensions"]
|
||||
sessions = client.tool("ocs_sessions", {"launch_if_none": True})["result"]
|
||||
session = sessions[0]["session_id"]
|
||||
while sessions[0].get("modal"):
|
||||
client.tool(
|
||||
"ocs_execute",
|
||||
{
|
||||
"ocs_session_id": session,
|
||||
"request": {
|
||||
"op": "action",
|
||||
"request_id": f"eval-close-modal-{uuid.uuid4().hex}",
|
||||
"name": "close_modal",
|
||||
},
|
||||
},
|
||||
)
|
||||
sessions = client.tool("ocs_sessions", {"launch_if_none": False})["result"]
|
||||
active = next(
|
||||
document for document in sessions[0]["documents"]
|
||||
if document["id"] == sessions[0]["document_id"]
|
||||
@@ -98,7 +112,7 @@ def main() -> None:
|
||||
if active.get("start"):
|
||||
client.tool(
|
||||
"ocs_execute",
|
||||
{"ocs_session_id": session, "request": {"op": "new", "request_id": "eval-new"}},
|
||||
{"ocs_session_id": session, "request": {"op": "new", "request_id": f"eval-new-{uuid.uuid4().hex}"}},
|
||||
)
|
||||
|
||||
base = 1_000_000 + int(time.time()) % 100_000
|
||||
|
||||
@@ -52,7 +52,7 @@ def legacy(server: Path) -> None:
|
||||
names = set(definitions)
|
||||
assert names == TOOLS, names
|
||||
execute_request = definitions["ocs_execute"]["inputSchema"]["properties"]["request"]
|
||||
assert len(execute_request["oneOf"]) == 35
|
||||
assert len(execute_request["oneOf"]) == 38
|
||||
op_enum = execute_request["properties"]["op"]["enum"]
|
||||
for shipped in ("entities_create", "entities_delete", "entities_transform",
|
||||
"block_define", "block_delete", "file_identity", "xdata_set",
|
||||
@@ -65,9 +65,12 @@ def legacy(server: Path) -> None:
|
||||
assert execute_request["properties"]["steps"]["maxItems"] == 64
|
||||
assert execute_request["properties"]["cmd"]["examples"][0] == "LINE 0,0 10,10"
|
||||
assert "set_properties" in execute_request["properties"]["op"]["enum"]
|
||||
assert "save_verified" in execute_request["properties"]["op"]["enum"]
|
||||
assert "record_schema" in definitions["ocs_read"]["inputSchema"]["properties"]["op"]["enum"]
|
||||
assert "audit" in definitions["ocs_read"]["inputSchema"]["properties"]["op"]["enum"]
|
||||
read_parameters = definitions["ocs_read"]["inputSchema"]["properties"]["parameters"]["properties"]
|
||||
assert {"collection", "where", "paths"} <= set(read_parameters)
|
||||
assert {"collection", "where", "paths", "target_format", "target_version"} <= set(read_parameters)
|
||||
assert execute_request["properties"]["target_version"]["enum"][0] == "R14"
|
||||
assert execute_request["properties"]["kind"]["enum"] == [
|
||||
"text", "token", "point", "entity", "structure", "selection", "enter"
|
||||
]
|
||||
|
||||
@@ -33,8 +33,8 @@ Two request styles share the channel, told apart by the `protocol` field:
|
||||
|
||||
| Style | Shape | Operations |
|
||||
|---|---|---|
|
||||
| protocol 1 | `{"protocol":1,"request_id":"…","document_id":N,"op":"…"}` — the same requests and replies as `ocs_execute` / `ocs_read`, described in the README | `new`, `open`, `save`, `activate`, `select`, `run`, `start`, `input`, `cancel`, `stop`, `action`, `property`, `set_properties`, `embed_image`, `undo`, `redo`; reads `state`, `capabilities`, `query`, `entities`, `records`, `record_schema`, `layers`, `header`, `properties`, `measure`, `history`, `commands`, `events`, `operation` |
|
||||
| legacy | `{"op":"…"}` without `protocol` | `new`, `open`, `run`, `entities`, `query`, `records`, `record_schema`, `capabilities`, `layers`, `header`, `select`, `save`, `undo`, `redo` |
|
||||
| protocol 1 | `{"protocol":1,"request_id":"…","document_id":N,"op":"…"}` — the same requests and replies as `ocs_execute` / `ocs_read`, described in the README | `new`, `open`, `save`, `save_verified`, `activate`, `select`, `run`, `start`, `input`, `cancel`, `stop`, `action`, `property`, `set_properties`, `embed_image`, `undo`, `redo`; reads `state`, `capabilities`, `audit`, `query`, `entities`, `records`, `record_schema`, `layers`, `header`, `properties`, `measure`, `history`, `commands`, `events`, `operation` |
|
||||
| legacy | `{"op":"…"}` without `protocol` | `new`, `open`, `run`, `entities`, `audit`, `query`, `records`, `record_schema`, `capabilities`, `layers`, `header`, `select`, `save`, `undo`, `redo` |
|
||||
|
||||
The two styles differ in how a waiting command is treated. A protocol-1 `run`
|
||||
or `start` while a command is still active is refused with `command_busy`:
|
||||
@@ -104,8 +104,19 @@ README describes.
|
||||
|
||||
`OpenCADStudio --export IN OUT` loads `IN` and writes `OUT` without a window or
|
||||
a session. The output format follows the extension: `.dxf` writes DXF, any other
|
||||
extension writes DWG at the document's own version. The exit code is 0 on
|
||||
success and 1 when the input cannot be read or the output cannot be written.
|
||||
extension writes DWG at the document's own version. Add `--target-version R14`
|
||||
(or `2000`, `2004`, `2007`, `2010`, `2013`, `2018`) to choose explicitly. The
|
||||
exit code is 0 on success, 1 when the input/output fails, and 2 for an invalid
|
||||
target version.
|
||||
|
||||
The verified interoperability lane uses version 2000 for a DWG -> DXF
|
||||
round-trip. R14 DWG files can be written and reopened, but the current R14
|
||||
DWG -> DXF path can emit object dictionaries that are orphaned from the root,
|
||||
which leaves layout and placeholder ownership invalid. `audit` and
|
||||
`save_verified` reject that ASCII DXF instead of reporting a false green. Do
|
||||
not deliver that conversion unless `save_verified` passes. Raw
|
||||
group-code handle validation is not applied to binary DXF and is reported as
|
||||
skipped in the audit result.
|
||||
|
||||
## Interactive steps
|
||||
|
||||
@@ -129,3 +140,5 @@ For commands whose answers cannot be written on one line, use `start` and then
|
||||
| measure | `{"protocol":1,"op":"measure","handles":["2A"]}` — kernel-computed length, area, bounds and mass properties; no window needed | `ocs_read` with `op: "measure"` and `parameters.handles` |
|
||||
| capture | `{"protocol":1,"op":"capture","path":"out.png"}` — needs the editor window, so it answers `gui_required` here | `ocs_capture` with `scope` (`viewport` or `window`) and `max_dimension` |
|
||||
| `set_properties` | requires `collection`; colours are the serialised enum — `{"Index": 1}` or `"ByLayer"`; a colour name such as `"Red"` is rejected with `invalid_value` | same request through `ocs_execute` |
|
||||
| audit | `{"protocol":1,"op":"audit","target_format":"dwg","target_version":"2000"}` | `ocs_read` with `op: "audit"` and the same target parameters |
|
||||
| save verified | protocol-1 `save_verified` with absolute `path`, unique `request_id`, and target fields | `ocs_execute` with `op: "save_verified"`; returns hash, reopened version and semantic manifest |
|
||||
@@ -0,0 +1,67 @@
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$Directory,
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ConsolePath,
|
||||
[int]$TimeoutSeconds = 25
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$drawingDirectory = (Resolve-Path -LiteralPath $Directory).Path
|
||||
$scriptPath = Join-Path $PSScriptRoot 'reference_audit.scr'
|
||||
$targets = @(
|
||||
@{ Version = '2000'; File = 'mcp-acceptance-2000.dwg' },
|
||||
@{ Version = '2013'; File = 'mcp-acceptance-2013.dwg' },
|
||||
@{ Version = '2018'; File = 'mcp-acceptance-2018.dwg' }
|
||||
)
|
||||
$results = foreach ($target in $targets) {
|
||||
$drawing = Join-Path $drawingDirectory $target.File
|
||||
if (-not (Test-Path -LiteralPath $drawing -PathType Leaf)) {
|
||||
throw "Missing acceptance drawing: $drawing"
|
||||
}
|
||||
$isolation = Join-Path $env:TEMP "ocs-reference-acceptance-$($target.Version)"
|
||||
New-Item -ItemType Directory -Force -Path $isolation | Out-Null
|
||||
|
||||
$start = [Diagnostics.ProcessStartInfo]::new()
|
||||
$start.FileName = $ConsolePath
|
||||
$start.UseShellExecute = $false
|
||||
$start.CreateNoWindow = $true
|
||||
$start.RedirectStandardOutput = $true
|
||||
$start.RedirectStandardError = $true
|
||||
$start.StandardOutputEncoding = [Text.Encoding]::Unicode
|
||||
$start.StandardErrorEncoding = [Text.Encoding]::Unicode
|
||||
@(
|
||||
'/i', $drawing,
|
||||
'/s', $scriptPath,
|
||||
'/l', 'en-US',
|
||||
'/isolate', "ocsaccept$($target.Version)", $isolation,
|
||||
'/readonly'
|
||||
) | ForEach-Object { [void]$start.ArgumentList.Add($_) }
|
||||
|
||||
$process = [Diagnostics.Process]::new()
|
||||
$process.StartInfo = $start
|
||||
[void]$process.Start()
|
||||
$stdout = $process.StandardOutput.ReadToEndAsync()
|
||||
$stderr = $process.StandardError.ReadToEndAsync()
|
||||
$forcedTermination = -not $process.WaitForExit($TimeoutSeconds * 1000)
|
||||
if ($forcedTermination) {
|
||||
$process.Kill($true)
|
||||
$process.WaitForExit()
|
||||
}
|
||||
$text = $stdout.Result + $stderr.Result
|
||||
$log = Join-Path $drawingDirectory "reference-audit-$($target.Version).log"
|
||||
[IO.File]::WriteAllText($log, $text, [Text.UTF8Encoding]::new($false))
|
||||
$passed = [regex]::IsMatch($text, 'Total errors found\s+0\s+fixed\s+0')
|
||||
[pscustomobject]@{
|
||||
Version = $target.Version
|
||||
Passed = $passed
|
||||
ForcedTermination = $forcedTermination
|
||||
ExitCode = $process.ExitCode
|
||||
Log = $log
|
||||
}
|
||||
}
|
||||
|
||||
$results | Format-Table -AutoSize
|
||||
if ($results.Passed -contains $false) {
|
||||
exit 1
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
_.AUDIT
|
||||
_N
|
||||
_.QUIT
|
||||
Reference in new issue
Block a user