feat(stacks): live compose output and deploy history (#1499)
* feat(security): add exact-value redaction for compose output lines
* feat(stacks): emit compose output line by line via optional callback
* test(stacks): isolate collectProcess test from the DB init side effect
stacks.ts transitively imports db.ts and db/drizzle.ts, which run a
top-level seedDatabase() against better-sqlite3 -- unsupported under
Bun (oven-sh/bun#4290). Importing stacks.ts for a plain unit test
crashed at import time and, worse, leaked into unrelated test files
sharing the same bun test process (reproduced against
tests/semver-generic-v2.test.ts). Mock both modules' full export
surface before importing, mirroring tests/backups/route-guards.test.ts's
established mock.module(./db) pattern.
* refactor(stacks): extract collectProcess into a dependency-free core module
Testing it required importing stacks.ts, which transitively pulls in
db.ts and db/drizzle.ts -- both run a top-level seedDatabase() against
better-sqlite3, which does not load under Bun. That crash was cached
process-wide and broke an unrelated, previously-green test file
(tests/semver-generic-v2.test.ts) whenever the full suite ran.
Following the repo's established -core module pattern (nav-preferences-core,
ws-exec-core, etc.), collectProcess now lives in its own dependency-free
file and is re-exported from stacks.ts for its existing callers.
* feat(security): bridge line callbacks through redaction
makeLineForwarder wraps a raw line consumer with redactLine, dropping
withheld lines silently. This is the connector collectProcess-emitted
lines will pass through before reaching the job store.
* feat(stacks): thread onLine through the local compose execution path
executeLocalCompose, executeComposeViaHawser, and executeComposeCommand
all accept an optional onLine callback now, forwarded to whichever
execution path is chosen. The local path (executeLocalCompose) wires it
into collectProcess via makeLineForwarder, redacting against envVars and
secretVars -- deliberately not spawnEnv, which also carries PATH, HOME,
DOCKER_CONFIG and DOCKER_API_VERSION and would over-redact (HOME often
resolves to "/root", five characters, below the safe-replace threshold,
which would withhold every line containing it).
executeComposeViaHawser accepts the parameter for signature parity but
does not use it yet: its /_hawser/compose call is a single
request/response, not a stream, so it cannot report lines as they
happen. That lands in a follow-up task.
startStack, stopStack, restartStack, downStack, and deployStack gained
an onLine parameter (deployStack via a new DeployStackOptions field) and
pass it through to every executeComposeCommand call in their body.
redeployStackFromDir, removeStack, pullStackImages, pullStackService,
and updateStackService are deliberately left unchanged -- they don't run
behind the streaming job endpoints.
* feat(stacks): forward redacted compose output lines to the job store
Each of the five job endpoints (deploy/start/stop/restart/down) now
passes send as the onLine callback, wrapping each line in a
{type: 'line', line} progress event. The 'line' type distinguishes
these from the existing static {status} progress events the endpoints
already send.
* chore(tests): use English throughout, matching the repository language
* refactor(tests): rename identifiers to English
* feat(ui): readJobResponse forwards line messages to an optional callback
* feat(ui): show compose output while stack operations run
* feat(stacks): surface the final output block when an agent sends no lines
* feat(stacks): thread onLine through the create and compose endpoints
POST /api/stacks (create & start) and PUT /api/stacks/{name}/compose
(save & redeploy) both run deployStack inside createJobResponse but never
passed onLine, so no 'line' progress messages were emitted and any live
output view stayed empty until the job settled.
Both now forward lines exactly like the deploy endpoint does. A source-level
guard asserts every deployStack call under src/routes/api carries onLine —
the omission is invisible at runtime, so nothing else would catch the next one.
* feat(ui): show compose output for Create & Start and Save & redeploy
The five stack-row actions opened the output window; the stack modal's two
deploying paths did not, leaving the most common route after an edit —
Save & redeploy — with no visible progress.
The modal takes three optional callbacks and forwards to them. It holds no
output state and renders no window of its own: the loop stays on the page, so
closing the modal cannot tear down a running deploy. Callers that don't pass
them are unaffected.
* feat(hawser): route agent line messages to non-streaming compose requests
* feat(stacks): consume streamed compose lines from the agent
* refactor(logs): make LogViewer's title optional and path-safe
* fix(logs): strip the escape sequences compose actually emits
The first pattern accepted only digits and semicolons as CSI parameter bytes,
so it left private-mode sequences untouched -- including ESC[?25l and ESC[?25h,
which compose's progress display writes on every redraw. A stripper that misses
the sequences of the output it exists to clean does not do its job.
Match the CSI grammar instead: ESC [ , parameter bytes (0x30-0x3F, which is
where '?' lives), intermediate bytes (0x20-0x2F), one final byte (0x40-0x7E).
Also cover the backslash branch of the path-separator test, which the previous
case did not exercise.
* feat(stacks): show compose output in the full log viewer
* feat(stacks): summarise compose output into deploy facts
* feat(stacks): add a status line with duration to the compose output window
Measure wall-clock duration client-side (Date.now() at start, diff on
result) and format it with formatRunStatus() in ComposeOutputModal's
description: "Running…", "Succeeded · 6.2s", or "Failed · exit 1 · 0.9s".
Wires all five +page.svelte stack actions (start/stop/restart/redeploy/
down) plus StackModal's two own action paths (create+start, redeploy on
save) through the same startComposeOutput/finishComposeOutput state, so
every caller of the modal reports success/failure consistently. Without
the StackModal fix, its two actions would have reported every success as
"Failed" (ok defaulting to undefined).
exitCode is read defensively from the job result but StackOperationResult
carries no such field today, so it stays undefined until the server adds
one.
* feat(deploy-logs): add filesystem-backed log store for deploy runs
Provides appendRunLog/readRunLog/deleteRunLog/listRunLogIds/runLogPath
for per-run log files under DATA_DIR/deploy-logs. The run id is
validated (not sanitised) into a filename, so a route-parameter run id
can never resolve outside the log directory. usedBytes/budgetExceeded
compute directory usage against a 2 GiB budget for a later task to
enforce.
* fix(security): keep deploy logs out of the file browser
* test(fs-guard): stop the deploy-log test from leaking DATA_DIR
Bun runs every test file in one process. Setting DATA_DIR at module scope leaked
it into unrelated suites: selfhst-icons builds and creates its cache directory
under DATA_DIR, which fails when that points at a path the machine does not
have. Both selfhst tests went red once this file was added, while each file
passed on its own.
protectedPaths() reads the variable on every call rather than at import time, so
a plain import plus beforeAll/afterAll is enough -- the dynamic import was never
needed, and it was what made the leak module-scoped in the first place.
* fix(logs): let auto-scroll pause while the user reads earlier output
LogViewer's auto-scroll effect jumped to the bottom on every logs update
whenever autoScroll was true, with no way to opt out. In the compose
deploy output window autoScroll stays true for the whole run, so
scrolling up to read an earlier line got immediately overridden by the
next line arriving.
Restore the pause/resume behaviour the removed ComposeOutputModal
output pane used to have: track how close the viewport is to the
bottom on scroll, and skip the auto-scroll effect while the user is
more than 50px away from it. Scrolling back within range resumes it.
The auto-scroll effect's own scrollTop write triggers a 'scroll' event
too, so a flag suppresses handleLogScroll for the duration of that
write -- otherwise a fast enough stream of new lines could grow
scrollHeight again between the write and the event firing, misreading
our own jump as the user having scrolled up.
The distance check is extracted into isScrolledToBottom(), a pure
function, so it has a real test (including the exact-threshold
boundary case).
Also hide the Clear/Refresh buttons when no onClear/onRefresh handler
is supplied -- LogViewer's only current caller (ComposeOutputModal)
passes neither, so both buttons were rendered as inert dead clicks.
* feat(stacks): record only redacted output lines, never the result block
* fix(logs): lift a scroll-up pause when a fresh run starts
userScrolledUp was only ever set, never reset -- so scrolling up once
during a deploy paused auto-scroll for every later run of the same
page session, measured live: heights grew (392->1130px, 446->1130px)
but scrollTop stayed frozen once a prior scroll-up had set the flag,
even on a run nobody had scrolled during.
autoScroll going from false to true is the "a fresh run is starting"
signal (ComposeOutputModal's autoScroll={running} flips false->true at
the exact moment its parent clears composeOutputLines and starts a new
compose run) -- so that edge lifts the pause. Deliberately an edge, not
a level: reacting to autoScroll simply being true would re-lift the
pause on every single update while a run is already in progress,
defeating it entirely. shouldResetScrollPause() is a pure function
again, with a test for exactly that distinction (true -> true must stay
false) alongside the three other transitions.
* fix(logs): avoid state_referenced_locally on previousAutoScroll
npm run check flagged the previous run's seeding line (`let
previousAutoScroll = autoScroll`) as reading a prop in a plain
top-level initializer, which only captures its value once and usually
signals a bug. Leave it unset instead and let the effect seed it on
its first run, where reading autoScroll is tracked -- no behaviour
change, just where the initial read happens.
* feat(sse): hook an optional RunRecorder into both createJobResponse send paths
createJobResponse has two independent send() implementations: the JSON path
(Accept: application/json) and the streaming/job-poll path. A recorder wired
into only one of them would silently miss every deploy driven through the
other -- the UI keeps working, so nobody notices.
Add an optional third parameter, RunRecorder, with line() and end(). Both
send() closures now forward shouldRecord-qualifying lines to recorder.line();
end() is called on every path out of the operation (JSON: in a finally,
awaited before the response is enqueued; streaming: from the .then()/.catch()
that already drive completeJob/failJob, unawaited since the {jobId} response
already went out by the time the operation settles).
The interface is declared here, not built here: constructing a real recorder
needs db.ts, and importing that into sse.ts would make it unimportable under
Bun. Task 10 builds the recorder in the route and imports RunRecorder from
here. Until then both callers omit the third argument, which is exactly the
prior behaviour.
* feat(db): index schedule_executions by entity and environment
Powers per-stack/container deploy-run lookups without a full table scan.
* feat(schedules): register stack_deploy and deploy_log_reconcile types
Widen every schedule-type union that the run record (stack_deploy) and
the reconcile job (deploy_log_reconcile, upcoming) will need: the two
independent unions in src/lib/server/db.ts (ScheduleType) and
src/routes/api/schedules/+server.ts (ScheduleInfo.type, duplicated in
+page.svelte), plus SystemScheduleInfo.type in scheduler/index.ts so the
reconcile job can register as a system schedule alongside system_cleanup.
Give both types real labels in the schedules page instead of falling
into the generic system-job branches: the single-selected filter label,
the two new filter dropdown entries, and the execution detail dialog's
title chain. stack_deploy has no schedule until M3, so its entries are
inert until then -- added now so they aren't forgotten later.
Also cover the two places whose correct behaviour depends on treating
deploy_log_reconcile like the other system schedules: hiding it from
the filter list when "hide system jobs" is toggled, and rejecting an
attempt to pause/resume it through the per-item toggle endpoint the
same way system_cleanup already is (it's a global, settings-driven
schedule, not a per-item cron).
The manual-run endpoint's two type switches are deliberately left
untouched: stack_deploy has no schedule to run, and deploy_log_reconcile
has no trigger function yet (that lands with its own scheduler task).
Wiring a case now would either call nothing or reference a module that
doesn't exist.
npm run check: same 687 pre-existing errors/152 warnings before and
after (confirmed via git stash), only shifted line numbers where lines
were inserted -- no new problems introduced.
* fix(stacks): size the compose output window for build logs
The window was max-w-3xl (48rem) at 60vh. Compose and BuildKit lines are wide
-- step lines, image digests, container names -- so most of them wrapped or
needed horizontal scrolling, which defeats the point of showing them.
Widened to 90rem and 85vh, clamped with min() against the viewport so it still
fits on a laptop screen with a margin instead of overflowing.
* feat(schedules): record every stack_deploy run as a schedule_executions row
Add the run record that the rest of milestone 2 depends on: without it, deploy
logs land on disk with nothing pointing at them, execution history stays empty,
and the upcoming reconciler would treat every log file as orphaned and delete it.
deploy-run-record-core.ts (buildRunDetails, importfree) assembles the details
JSON -- options, summary, exit code, compose/env hash, log file name -- and
refuses to build it without a log file name, since that name is what ties the
record to its file. deploy-run-record.ts (createRunRecorder) is the
database-touching half: creates the schedule_type='stack_deploy',
schedule_id=0 row up front, chains line() calls onto appendRunLog so
concurrent writes can't scramble the log's order, and closes the row in
end() -- which must work even when no line ever arrived, since a deploy can
fail before compose produces any output.
Also switches on deploy-log-store's total-size budget (defined in an earlier
task, never called anywhere until now): once a run's log crosses it, further
lines are dropped, the file gets a line saying so, and details.truncated is
set.
Wires the recorder into both places that call deployStack from an HTTP
handler: the dedicated deploy endpoint and stack-create-with-immediate-start.
Both now read/compute the compose and effective env up front so a compose
hash, env hash, and the schedule_executions row all exist before the SSE job
stream opens -- createJobResponse takes the recorder as an already-resolved
value, not something it can await for itself.
stacks.ts gains an exitCode field on StackOperationResult, populated by the
local/direct compose path (which already had the number, just wasn't
returning it) and left unset on a timeout or over Hawser (protocol doesn't
carry one) -- the recorder falls back to a value consistent with success/
failure when it's missing.
bun test tests/: 2828 pass, 0 fail (baseline 2826 + this task's 2 new tests).
* fix(schedules): keep the deploy summary in sync with what actually got written
line() pushed its argument onto the in-memory buffer synchronously, but only
queued the file write onto the async chain. Called several times before the
chain gets to run any of them (the ordinary case -- compose output arrives in
bursts), a line already sitting in the buffer could still get skipped by the
file write once truncation kicks in partway through the queue, leaving the
eventual summary counting a line that was never actually persisted.
Move the push into the same guarded step as the write so both share one
fate: a line only ever ends up in the summary if it also ended up in the
file. Caught by re-reading the chaining logic after writing it, not by a
test -- verified instead with a live run against a real SQLite db (tsx,
temp DATA_DIR) confirming the three-line/no-line lifecycles still land
correctly. bun test tests/: still 2828 pass, 0 fail.
* feat(stacks): add deploy-run endpoints (list, single, log, delete)
Three routes under api/stacks/[name]/deploys/ each check stacks:view/edit for
themselves -- they do not inherit /api/jobs/{id}'s no-auth model. A shared
loadOwnedDeployRun helper confirms a run actually belongs to the requested
stack and checks environment access against the run's OWN environmentId
(never a caller-supplied env query param), closing the same cross-env bypass
class backups/route-guards.ts already closed for snapshot/config lookups.
getScheduleExecutions gained an entityName filter, the actual consumer of the
(entity_name, environment_id) index from the previous commit.
Also: extracted tests/helpers/db-fake.ts, a shared mock.module() registration
point for $lib/server/db, and moved tests/backups/route-guards.test.ts onto
it. Two test files each wholesale-mocking that module collided (confirmed:
whichever wins clobbers the other's needed export, deterministically) -- the
new helper avoids that by pre-declaring every export name any test needs as a
stable dispatcher, so later-registered implementations are still reachable
regardless of which file's import resolves the specifier first.
* feat(schedules): reconcile deploy-log files against stack_deploy records
Adds a dedicated deploy_log_reconcile schedule, modelled on system-cleanup.ts's
static jobs (own cron + enabled setting, own schedule_executions row) rather
than tacking it onto system_cleanup: the operator can see and disable it
independently.
planReconcile() is a pure decision core: a file with no matching record is
deleted (an orphan with nothing left to explain it); a record with no file is
only marked via details.logMissing, never deleted, since the run metadata
(when, who, what was built) stays valuable once the log text is gone. The
asymmetry is the point -- over-eager deletion here would destroy exactly the
record this job exists to protect.
run/+server.ts gained a deploy_log_reconcile case in both switches (env
resolution and dispatch) -- without it every manual trigger 400s, since both
switches already have a default that rejects unknown types.
ASSUMPTION flagged in deploy-log-reconcile.ts: the record side reads
schedule_executions rows with scheduleType 'stack_deploy' and treats
String(id) as the deploy-log run id. Nothing in this branch creates such a
row yet -- that is Task 10's createRunRecorder, not landed here (confirmed:
no t10/done ref exists). This is read-only against the existing, already
scheduled ScheduleType union, not a substitute for Task 10's write path, but
it is unverified against Task 10's actual output and needs a look once that
lands.
triggerDeployLogReconcile() is a separate function rather than a branch in
triggerSystemJob(): that function dispatches by a flat, type-less jobId
string, and DEPLOY_LOG_RECONCILE_ID (1) would otherwise collide with
SYSTEM_SCHEDULE_CLEANUP_ID (also 1, different scheduleType).
* fix(schedules): don't mark an in-progress deploy's record as logMissing
Read deploy-run-record.ts directly (Task 10's real code, in a sibling clone
not yet merged onto this branch) to check the assumption flagged in the
previous commit -- and it confirmed the record-id linkage is exactly right
(scheduleType 'stack_deploy', String(schedule_executions.id) as the deploy-log
run id), but also surfaced a race this job didn't guard against: the row is
created with status 'running' before the deploy has written a single line,
and the log file only comes into existence on the first line() call. A
reconcile firing in that window would mark an actively-running deploy as
logMissing, and nothing here ever re-checks or clears that flag once the file
does appear -- so the record would carry a false "log missing" permanently.
getScheduleExecutionIdsByType() now also returns status; records still
'queued'/'running' are excluded from markMissing candidacy (a later run will
catch them correctly once the deploy has actually finished) while remaining
in recordIds, so an already-written file for them stays protected from
deletion either way.
* test(schedules): extract the in-progress guard into a tested pure function
isEligibleForMissingMark() replaces the inline NON_TERMINAL_STATUSES array
check in deploy-log-reconcile.ts -- same decision, now importfree and covered
by its own tests (both non-terminal statuses excluded, every terminal status
included), with a mutation counter-test confirmed to go red for the right
reason before being reverted.
* fix(stacks): keep compose output visible below the editor on deploy
Save & redeploy and Create & Start used to hand live output off to a
separate window and close the stack modal immediately, so the editor
and the output were never visible at the same time -- forcing the user
to close the output window just to get back to the compose they were
editing.
The modal now keeps its own output state (lines, running, ok, ms, exit
code) and renders it inline below the editor instead of forwarding to
the page's shared ComposeOutputModal. Whether the modal closes after a
save is now a small pure function, shouldCloseAfterSave(deployed): a
plain save still closes, a deploy stays open so the output stays
visible.
Also fixes a pre-existing bug in the same code path: the error branch
called finishOutput(undefined, false) unconditionally, clobbering the
real ok/exitCode a prior successful finishOutput call had already
recorded when the job itself reported failure. Now guarded on
outputRunning so a real result from the server is never overwritten.
GitStackModal.svelte does not use the onOutput* props and needed no
changes.
* fix(stacks): clear docked deploy output when the modal reopens
StackModal is persistently mounted -- one instance is reused for every
stack. The new output* state introduced for the docked deploy output
was only ever assigned in startOutput(), so it survived close/reopen
just like the pre-existing compose-validate state does. Reopening the
modal for a different stack could show the previous stack's title,
lines and "Succeeded"/"Failed" status under the new stack's editor,
which reads as if something just ran for it.
Reset all seven output* fields in the same open-effect block that
already clears validate panel state on each open, for the same reason.
A reopen of the *same* stack while its own deploy is still running
loses the lines gathered so far, but the poll loop in sse-fetch.ts
keeps running regardless of the modal's open state (same as
ComposeOutputModal), so new lines resume arriving within one poll
interval; and if the job finishes before any new line does,
finishOutput's existing "outputLines.length === 0 && output" fallback
refills the whole transcript from the job's final output. That brief,
self-correcting gap was judged preferable to a stale result silently
misattributed to the wrong stack.
* fix(deploys): require env on the list endpoint to stop cross-environment leaks
GET /api/stacks/[name]/deploys treated `env` as optional. Omitting it left
envIdNum undefined, which skipped the enterprise environment-access check,
made auth.can() evaluate the caller's globally-merged permission instead of
their permission for one specific environment, and dropped the environment
filter from getScheduleExecutions() entirely -- returning stack_deploy runs
(timestamps, status, error text, details.userId) from every environment that
happens to have a same-named stack, regardless of what the caller's role
actually grants.
A stack name alone doesn't identify one environment, so there's no correct
behavior for a missing `env` beyond rejecting the request: parse and validate
it before the permission check runs, so auth.can() and canAccessEnvironment()
always see a definite id. Sibling routes under .../deploys already close this
class via loadOwnedDeployRun (env derived from the loaded run); this was the
one route deriving it from an optional query parameter instead.
* fix(schedules): isolate per-element failures in the reconcile job
Reproduced independently: one throwing deleteRunLog/updateScheduleExecution
call aborted the whole run -- and since the failing element is never
removed/marked, every LATER run failed at the exact same element again,
permanently. A harmless orphan file ordered before the failing one was still
removed; everything after it was not; the failed run's own `details` came
back null, so the partial progress wasn't even visible.
Both loops now wrap each element in its own try/catch (same shape as
scheduler/index.ts's scannerCleanupAllEnvs, which does this per-environment
for the same reason). Failures are counted and logged individually instead
of aborting.
Counts now reflect what actually happened, not what was planned:
deletedFiles/markedRecords only increment on success, plus a new `failed`
count. Previously markedCount was derived from plan.markMissing.length minus
skippedInProgress -- correct only by coincidence, since nothing before this
commit could fail mid-loop; a failing element would have inflated it to the
planned size.
A run with per-element failures is recorded as 'warning' rather than
'success' -- this repo's existing convention for "completed with a caveat"
(see the ScheduleStatus doc comment in db.ts), not 'failed', which stays
reserved for the run itself throwing (the outer catch, unchanged).
Tests (tests/deploy-log-reconcile-job.test.ts, new): one per loop, each with
a mutation counter-test confirmed to fail for the right reason (the element
after the failure was unreached; the reported count matched the plan instead
of the outcome), then reverted. $lib/server/db faked via
tests/helpers/db-fake.ts's shared registry (extended with the five exports
this job needs); $lib/server/deploy-log-store mocked directly since no other
test file claims it.
* fix(sse): reuse endFromResult on the streaming path so a failed run is recorded as failed
A failed docker compose run sends result:{success:false,error} and returns
normally -- it never throws. On the fire-and-forget streaming path (no Accept
header, the path every browser call takes), the .then() branch hardcoded
recorder.end(true) regardless of the result payload, so every such run was
persisted with status:"success" and errorMessage:null.
The JSON path already derives ok/error from the held-back result via
endFromResult(); reuse it here instead of duplicating success/failure logic,
using the result line the .then() branch already reads for completeJob().
* fix(deploy-run-record): redact known secret values out of the stored error text
DeployRunRecorder.end() wrote result.error verbatim into schedule_executions'
errorMessage/details on a failed run. That string is compose's raw, unredacted
stdout/stderr (deploy-recorder.ts's shouldRecord() deliberately excludes the
'result' event from the log file for this exact reason). GET
/api/schedules/executions returns errorMessage/details to every authenticated
user, with no stack/environment permission check -- so a secret that leaked
into compose's error output was readable across permission boundaries.
createRunRecorder() now takes the same merged secrets list its callers already
build for envHash, and end() runs the error text through the same redactLine()
the streamed lines already use before storing it. When redactLine withholds
the whole line (a too-short value matched), the row still closes "failed" with
a safe placeholder instead of no message at all -- a run stuck as "running"
forever is worse than one with a generic error.
* fix(stacks): auto-close after a successful deploy, stay open on failure
Operator decision (30.08.2026), after seeing the docked output live:
always keeping the modal open post-deploy was as wrong as always closing
it used to be. A plain save still closes as before. A deploy that
succeeded now closes too, just after a longer delay so the result is
still glimpsed. A deploy that failed keeps staying open.
shouldCloseAfterSave(deployed: boolean): boolean is replaced by
saveCloseTiming(deployed: boolean, ok: boolean): 'close' | 'close-delayed'
| 'stay-open', so the three operator-specified outcomes are each
individually representable and testable, instead of collapsing "closes"
into one boolean that couldn't distinguish "right away" from "after a
flash".
Verified there is no path at the two call sites where a failed deploy
resolves without throwing first: both POST /api/stacks and PUT
/api/stacks/{name}/compose always send an explicit `result` event with
success:false on failure, with an explicit `return` right after it, so
neither send() call falls through to the success path afterwards. The
'stay-open' branches at both call sites are therefore dead code today,
kept only so the switch stays exhaustive if that ever changes.
* feat(stack-modal): make the live output panel's height resizable
The output panel below the compose editor had a fixed height. Add a
horizontal drag divider (mirroring the existing vertical compose/env
split) so the panel can be resized between 15% and 70% of the combined
editor+output area, persisted to its own localStorage key.
Extract the clamp used by both splitters into clampSplitRatio() and fix
a latent bug along the way: Math.max(min, Math.min(max, value)) does
not clamp when value is NaN (Math.min/Math.max propagate NaN), which
happens for real when a container's measured size is briefly 0 during
a drag. The existing compose/env splitter had the same bug and now
uses the same fixed clamp.
* fix(stacks): record Save & redeploy (compose PUT) as a stack_deploy run
PUT /api/stacks/[name]/compose with restart:true redeployed the stack but
never built a run recorder, unlike the dedicated deploy endpoint and
stack-create-with-immediate-start. After a successful redeploy the modal
now auto-closes (recent change), so this was the only path where the
output of a successful deploy was fully lost -- no schedule_executions
row, no log file, nothing to look back at.
Wires createRunRecorder in identically to deploy/+server.ts: switches the
authoritative-paths lookup from getStackComposeFile to requireComposeFile
(same call, now also resolving secretVars/nonSecretVars), hashes the
just-submitted `content` (not a re-read off disk -- saveStackComposeFile
already wrote this exact string, and it's the same string handed to
deployStack), and hashes the merged effective env for envHash. Skipped
entirely when requireComposeFile unexpectedly fails right after a
successful save, so a run is never recorded with an empty secrets list.
restart:false (save only) is untouched -- no recorder is built, no row is
created.
Adds tests/helpers/authorize-fake.ts, mirroring db-fake.ts's dispatcher
pattern, and moves tests/deploy-endpoints.test.ts's inline authorize mock
onto it -- a second direct mock.module('$lib/server/authorize', ...) in
the new test file would have collided with that file's existing one.
bun test tests/: 2878 pass, 0 fail (baseline 2870 + this task's 8 new
tests). npm run check not yet run.
* fix(stack-modal): clamp the validate-panel width the same way, and rename accordingly
validatePanelWidth = Math.max(320, Math.min(560, w)) had the same NaN
gap as the two split-ratio clamps fixed earlier: if editorRowRef's
measured width is briefly 0 during a drag, w becomes NaN and the
"clamp" stays NaN.
clampSplitRatio never assumed a percentage, only value-vs-bounds
ordering, but its name did -- applying it to a pixel width buried that
under a misleading name. Renamed it (and its module/test file) to
clampNumber, with a doc comment that says plainly it works for both
percentage ratios and pixel distances, and a dedicated test proving
the pixel bounds (320-560) clamp exactly like the percentage bounds.
* feat(stack-modal): add a Deploys tab showing recorded run history
Adds a third history-style tab next to Backups: each recorded
stack-deploy run (GET /api/stacks/{name}/deploys, env required) as a
collapsed row -- status, relative time, duration, and a one-line
container/build summary -- expandable for the full picture: container
counts, whether the build came from cache or not, images built/pulled,
the deploy options used, a truncated-log notice, and on failure the
shortened last error line.
All of that text composition lives in deploy-run-view.ts as pure,
import-free functions (mirrors deploy-summary-core.ts's own reasoning:
no $lib/server import, so the test file runs under Bun without
dragging in better-sqlite3). The one case load-bearing enough to name
in its own doc comment: a run with no summary at all (compose produced
zero output lines) must read as "unknown", never silently as "zero
containers, nothing built" -- those are different facts.
No component test exists for DeploysPanel.svelte itself -- this repo
has no Svelte component test harness (no @testing-library/svelte, no
jsdom; @playwright/test is a dependency with zero test files). Manual
verification only for the component; the 37 new unit tests cover every
branch of the text-composition layer instead.
* perf(deploy-log-store): stop re-scanning deploy-logs/ on every appended line
DeployRunRecorder.line() called budgetExceeded() (a full readdir+stat over
every file in deploy-logs/) after every single line, so a deploy's cost grew
with both its own output AND the number of past runs' log files on disk.
Measured: 200 lines against 500 existing files took ~16.2s; against 50 files
~1.8s; with the scan removed entirely, ~49ms.
Replace that per-line full scan with SizeBudgetTracker: the directory total
excluding this run's own file is measured once, lazily, and cached; each
appended chunk's byte length is added to an in-memory running total (no
stat() involved); the cache is only re-measured once the run's own output
crosses 1 MiB since the last measurement. The budget itself is unchanged
(2 GiB, checked after each write per design doc §5) -- only how often the
directory gets re-scanned to check it.
This trades exactness for speed under concurrent writers: a run started
right after this run's baseline was measured is invisible to this run's
budget check until the next refresh. Documented on SizeBudgetTracker as an
intentional trade-off -- BUDGET_BYTES exists to catch a filled disk, not to
enforce a precise quota, and the old per-line scan had no locking against
concurrent appendRunLog() calls either.
Tests: SizeBudgetTracker's budget math (under/over/already-over-before-first-
byte), its refresh cadence (scans once for 200 small appends where the old
code would have scanned 200 times, and again once a single big append
crosses the refresh threshold), and the documented concurrent-writer gap.
Plus a DeployRunRecorder-level truncation test via an injected tiny-budget
tracker (SizeBudgetTracker now takes optional budgetBytes/measureOthers
params, DeployRunRecorder now takes an optional sizeBudget -- both test-only
seams, unused by production callers).
Refs branch-review.md H2.
* feat(stack-modal): load and show a deploy run's log on expand
Expanding a Deploys-tab row now fetches GET .../deploys/{id}/log into a
LogViewer, lazily -- never while the list itself loads. A run already
marked logMissing by deploy-log-reconcile skips the fetch entirely
instead of making a request that would just 404.
buildDeployLogPanelState (deploy-run-view.ts) turns the run into the
three independent facts the panel needs: whether the log is missing,
whether it was truncated, and whether the run has settled enough to
allow deleting it -- so a missing log never renders as an empty
viewer, and no delete action is offered while a run is still running.
* fix(deploys): scope stacks:view/edit permission checks to the run's own environment (H1)
GET/DELETE .../deploys/{runId} and GET .../deploys/{runId}/log checked
auth.can('stacks', 'view'|'edit') without an environmentId, which falls back
to the caller's globally-merged permission (OR across every role, per every
environment). A caller with stacks:view/edit scoped to environment A could
therefore read, or delete, a deploy run belonging to environment B, as long
as they also had SOME role granting stacks:view/edit somewhere -- and
loadOwnedDeployRun's own canAccessEnvironment check does not close this,
since it only confirms "has some role for this environment", not "has
stacks:view/edit specifically for this environment".
Add a second, run-scoped check via can('stacks', <action>, run.environmentId
?? undefined) AFTER loadOwnedDeployRun resolves the run -- in addition to,
not instead of, the existing un-scoped gate, so the pre-existing "no
permission anywhere" 403 tests keep their exact prior behavior. This is the
same derive-environment-then-check pattern api/git/stacks/[id]/+server.ts and
its env-files route already use for their own nullable environmentId. The
sibling list route (deploys/+server.ts) already documents and closes this
exact bug class for itself; this closes it for the three single-run routes
too.
Regression tests confirm the scoped check is what closes the gap: reverting
just the two source files turns the three "denied" tests red with the
pre-fix symptom (200 instead of 403), while the "allowed" tests stay green.
* fix(deploys): list route treats the local environment (env omitted) as reachable, not a 400 (H3)
On a single (local-only) environment install -- the normal case -- every
deploy run is recorded with environment_id = NULL: createRunRecorder()
writes it that way whenever the triggering request carried no envId, which
is always true for the local environment, because appendEnvParam()
($lib/stores/environment.ts) only appends an `env` query param when envId is
truthy. GET .../deploys required `env` as a parseable integer and 400'd
otherwise -- and there was no integer a caller could pass that matched a
NULL row (env=0 included; environment ids are never 0). The result: on a
fresh, single-environment Dockhand install, EVERY recorded deploy run was
completely unreachable through this endpoint.
The data layer already supported this -- getScheduleExecutions() maps
`environmentId: null` to `isNull(...)` -- only this route couldn't express
"give me the null-environment runs" at all.
Fix: two `env` shapes now both resolve to `environmentId: null` ("the local
environment") -- the parameter OMITTED entirely (the shape the UI actually
sends), and the literal string "null" (for direct/curl callers). Anything
else must still parse as an integer; a genuinely invalid value is still a
hard 400, so "list runs, unspecified environment" never silently means "all
environments merged". The permission check maps the null case to
`can('stacks', 'view', undefined)` -- the same "no environment context"
global-permission check api/git/stacks/[id]/+server.ts already uses for ITS
nullable environmentId -- so a caller still needs stacks:view somewhere to
see local-environment runs; canAccessEnvironment() is skipped for null (there
is no environment record to check access against).
Test coverage updated in the same file the pre-fix "missing env -> 400" and
"cross-environment leak" tests lived in, since both encoded the very
behavior this fixes (the leak test now confirms the null filter itself
still excludes a different, non-null environment's runs -- the security
property that motivated requiring `env` in the first place is unchanged).
Reverting just the route file turns the two H3-labeled tests red with the
pre-fix symptom (400 instead of 200).
* fix(openapi): move loadOwnedDeployRun-produced 400/404 into prose, closing the CI status-code-drift gate (B1)
`npm run generate:openapi:check` (.github/workflows/openapi.yml:49) failed
with 5 Gate-4 (status-code drift) issues: GET/DELETE .../deploys/{runId}
documented resp-400 and resp-404, and GET .../deploys/{runId}/log documented
resp-400, but all three of those specific codes are produced by the SHARED
loadOwnedDeployRun() helper (deploy-run-access.ts) -- not by a literal
`status: 400` / `error(400)` in the handler's OWN body text, which is all
scripts/generate-openapi.ts's static scanner (analyzeHandlerBody) can see.
statusDrift > 0 is a hard failure.
Fix: move that information into the `description:` field as prose instead of
resp-400/resp-404 lines -- the same pattern already used by
POST /api/backup/destinations/{id}/task for its own guard-produced codes.
Nothing is undocumented: the codes and their meaning are still explained,
just not as a formal schema-level response entry the generator expects to
find literally in the handler it's annotating. The log route's resp-404 line
stays as-is (its "Log not found" 404 IS literal in that handler's own body,
alongside loadOwnedDeployRun's 404 which is now covered by the same prose).
Also regenerates src/lib/openapi.generated.json and static/openapi.json
(--check writes both as a side effect of running) -- picks up this fix plus
unrelated, already-existing drift from earlier commits on this branch that
had never been regenerated (the deploy_log_reconcile schedule type additions
to two already-annotated schedule endpoints).
Verified: `npm run generate:openapi:check` was Exit 1 ("Hard failures: 1",
the 5 issues above) before this commit, at this branch's HEAD, in a fresh
clone with `npm install` -- Exit 0 ("Hard failures: 0") after.
tests/deploy-run-openapi-drift.test.ts reproduces Gate 4's own drift check
directly against these three routes' annotations (via the same
parseAnnotations/splitMethodBodies/analyzeHandlerBody the generator itself
uses) rather than shelling out to the full generator, which scans and
rewrites the entire route tree for what is a narrow, three-file regression.
Reverting just the two doc-comment edits reproduces the exact stale-code
sets from the CI report (["400","404"] and ["400"]).
* fix(stack-modal): stop duplicating a failed run's full error text
The expanded row's error box rendered the raw errorMessage (compose's
full stderr) verbatim, right above the new log panel from this same
task -- for a failed run, that meant showing almost the same text
twice. Task 17's own spec called for the shortened last error line
here (the same text the collapsed row already truncates to), not the
raw message; the box now shows view.errorSummary instead. The full
text has exactly one place now: the LogViewer beneath it.
* fix(stack-modal): fetch deploy runs for the local environment too
envId === null is not "no environment selected" -- per the list route's
H3 fix (.../deploys/+server.ts), it IS the local environment: a
single-environment install's own deploy calls never send an env param
either (appendEnvParam only appends one when envId is truthy), so
every one of its runs is recorded with environmentId === null, and the
route now resolves an omitted env to exactly that filter.
The guard that skipped the fetch and showed 'No environment selected.'
predates that fix, from when the endpoint hard-required env and would
400 without it. With the endpoint now accepting the omitted shape, the
guard silently kept the tab empty on the most common install shape
even though the API would have answered. Removed -- onMount fetches
unconditionally, appendEnvParam already produces the right URL for
both the local (null) and a specific (numeric) environment.
No pure-function decision to extract here: the branch this replaced
IS the fix, not a rule with two live outcomes -- appendEnvParam
already forms the correct request for both envId shapes, so there is
nothing left in the component to test beyond what deploy-endpoints.
test.ts already covers at the route.
* feat(git-stacks): record stack_deploy runs for every git-stack deploy trigger
deployGitStack() is the single function every git-stack deploy trigger funnels
through -- the manual Deploy button, a config-update-then-redeploy, both webhook
methods, create-and-deploy, and (via runGitStackSync) the cron scheduler and the
schedules page's "run now". Wire the stack_deploy run recorder in there, not at
a single HTTP route: wiring it only at the route level (the way compose PUT and
the non-git deploy endpoint already do it) would leave every other caller --
crucially the cron- and webhook-triggered ones, the truly unattended runs the
deploy-verlauf design doc's Meilenstein 5 is about -- unrecorded.
composeHash is taken from the content actually synced from git; envHash/secrets
mirror the same (deliberately approximate) DB-var resolution the non-git routes
already use. Every real call site now passes an explicit triggeredBy (manual/
webhook/cron) instead of the previous unconditional default, and the manual
routes forward auth.user?.id and an onLine callback so SSE progress streaming is
unchanged for HTTP callers.
Added tests/helpers/stacks-fake.ts (mirroring db-fake.ts's collision-safe
registration pattern) since syncGitStack -- deployGitStack's sync step -- is a
same-module collaborator that can't be faked via mock.module(), so the new test
drives a real git sync against a local repo served by a real `git daemon`
subprocess, faking only the $lib/server/stacks boundary (deployStack). Migrated
the existing compose-redeploy test to the same shared helper, since a second
direct mock.module('$lib/server/stacks', ...) call would have collided with it.
* docs(backups): note that a restore's redeploy has no stack_deploy run record
Backup restore's redeploy calls redeployStackFromDir directly, bypassing the
deployGitStack()/compose-route path createRunRecorder is wired into -- so it
gets no stack_deploy row and no log file. Left unrecorded on purpose (its own
piece of work, no structural blocker), but that was invisible in the code: no
comment, no marker, nothing to distinguish it from an oversight for a reader
who doesn't already know the history.
* fix(git-stacks): coerce forceRecreate to boolean for the run recorder's options
syncResult.updated is boolean | undefined; DeployRunOptions.forceRecreate wants a
strict boolean. Caught by a differential tsc run against the parent commit (the
only genuinely new type error introduced by the previous commit -- everything
else tsc reported was pre-existing, just line-shifted).
* fix(api): thread build/pull/forceRecreate through stack create & redeploy
POST /api/stacks and PUT /api/stacks/[name]/compose always deployed with
build:false and no pullPolicy, regardless of what the request sent -- a
stack whose compose declares a build: section silently never rebuilt on
"Create & Start" or "Save & redeploy" (the saved compose change just ran
against the stale local image). Both endpoints now accept pull, build,
and forceRecreate in the request body and forward them to deployStack()
and the run recorder, matching the contract the dedicated deploy endpoint
(POST /api/stacks/[name]/deploy) already uses.
PUT .../compose defaults forceRecreate to true when the field is omitted,
preserving its prior hardcoded behavior for a caller that has not been
updated to send it explicitly (env var changes need --force-recreate to
take effect). POST /api/stacks (Create & Start) has no such compatibility
concern -- there is nothing to recreate on a brand new stack -- so it
defaults all three to false when omitted.
pull:false intentionally leaves pullPolicy undefined rather than mapping
to a "never"/"missing" policy: deployStack() treats an undefined
pullPolicy as "do not pull", and its own post-deploy pending-update badge
reconciliation is gated on the same field, so unchecking Pull images also
means a stale "update available" badge on that stack will not clear until
the next pull. Documented at both call sites, not just left implicit.
Regenerated static/openapi.json and src/lib/openapi.generated.json for the
new pull/build/forceRecreate body fields on both @openapi blocks.
* feat(stacks): let Save & redeploy / Create & Start pick pull/build/force-recreate
"Save & redeploy" and "Create & Start" in the stack edit dialog silently
never rebuilt a service with a build: section (see the paired API commit).
Fixing the endpoints alone isn't enough without a way for the operator to
actually ask for a build -- both buttons are now wrapped by RedeployPopover
instead of deploying immediately, offering the same Pull/Build/Force
recreate checkboxes already used for redeploying from the stack grid.
RedeployPopover gains:
- triggerVariant="button" + triggerClass, so it can render as a full
labeled trigger styled like the shadcn Button component (buttonVariants)
instead of only the small icon-only trigger the stack grid rows use.
- defaultPull/defaultBuild/defaultForceRecreate + reason, re-seeded into
the checkboxes on every open (not just once at mount) so a caller whose
default tracks live content -- "does the compose have a build: section
right now" -- stays correct across repeated opens.
"Build images" is pre-checked whenever the compose file declares a build:
section for any service (compose-build-detect.ts, extracted as a plain
function with its own test rather than left inline in the component, the
same way save-close-policy.ts is). "Pull images" defaults to unchecked for
both buttons (unlike the grid's redeploy popover, which defaults it on) --
Save/Create already run against whatever images are already local, pulling
is an explicit extra step here. "Force recreate" defaults to checked for
Save & redeploy only, preserving its prior always-on behavior now that the
API also accepts a caller opting it off; Create & Start defaults it off
since there is nothing to recreate on a brand new stack.
pendingSaveOptions carries the chosen options through the path-change
confirmation dialog the same way pendingSaveRestart already does, so
re-entering handleSave() after the operator confirms a compose/env file
move still deploys with what they actually picked in the popover.
* chore: drop internal planning-doc references from comments
Comments and test descriptions referenced an internal milestone label and
design-doc section number that mean nothing to anyone outside this repo's
own planning process. Reworded to be self-contained -- no behavior change.
* refactor(stacks): split "Save & redeploy" / "Create & Start" into main + arrow
The previous shape had the whole button open the popover before deploying
anything -- correct behavior, wrong ergonomics. The common case (just save
and redeploy, or just create and start, with sane defaults) now takes one
click again: the label is a real, directly clickable button. A separate
small chevron button to its right opens the same Pull/Build/Force recreate
popover for the cases that need it. Two independently focusable <button>
elements sitting flush together, not one element whose behavior depends on
click position.
RedeployPopover's triggerVariant switches from 'button' (removed, was a
label-plus-icon trigger wrapping the whole action) to 'chevron' (a small
square trigger meant to sit attached to a separate main button the caller
renders itself -- the popover no longer owns the main button at all).
saveRedeployDefaults/createStartDefaults in StackModal are the single
source of truth for "what a direct click deploys with", shared between
each main button's own onclick and the popover's defaultPull/defaultBuild/
defaultForceRecreate props next to it, so the one-click path and the
pick-then-Deploy path can never quietly diverge on what "the default"
means.
The two existing icon-only redeploy triggers in the stack grid (RedeployPopover
used there via the unchanged 'icon' variant) are untouched -- they were
already the "click opens a menu" shape by design, since there is no separate
main button in a table row to split off from.
* feat(logs): follow the caller's own theme in LogViewer
LogViewer defaulted to a hardcoded dark theme regardless of the
editor's own light/dark toggle. Add a theme prop (defaulting to
'dark', so any caller that doesn't pass it renders unchanged) and
wire it through the two places that already have an editor theme in
scope: StackModal's docked output panel and, forwarded through a new
theme prop on DeploysPanel, the Deploys tab's expanded run log.
ComposeOutputModal has no editor in scope (opened straight from the
stack list) — its Dialog.Content already follows the app-wide
light/dark switch via bg-background, so it reads that switch once on
mount instead, mirroring the existing pattern in ComposeGraphViewer's
graphTheme.
Search-match highlighting gets a light-mode override, scoped to
.log-viewer-light so it doesn't touch the shared (unscoped)
.search-match rule other components such as LogsPanel still rely on:
the base rule's near-white text is illegible against the pale amber
highlight once the panel itself goes light.
* fix(stacks): pass DeploysPanel's theme prop under its own name
{editorTheme} is Svelte's shorthand for a prop literally named
editorTheme -- DeploysPanel's Props interface declares theme, not
editorTheme, so this silently passed an unknown prop and left theme
at its 'dark' default. svelte-check caught it: 'editorTheme' does not
exist in type 'Props'. bun test doesn't type-check props, which is
why it stayed green through the first commit.
* fix(sse): guard RunRecorder.end() against a second call if it throws
createJobResponse's fire-and-forget path calls recorder.end() in its
.then() branch. If that call itself throws, the rejection lands in
.catch(), which unconditionally called recorder.end() again -- breaking
the RunRecorder contract that end() runs exactly once.
Track whether the .then() branch already reached its end() call and
skip the .catch() call when it did, regardless of whether end() itself
resolved or rejected.
* test(fs-guard): remove redundant DATA_DIR assignment before the import
isProtectedPath() only reads process.env.DATA_DIR lazily inside its own
call, not at module-import time, so the beforeAll/afterAll pin already
in this file is sufficient on its own -- the extra assignment right
before the dynamic import did nothing.
Verified by measurement, not assumption: this file passes standalone,
paired with selfhst-icons.test.ts in both file orders, and as part of
the full suite, all without the removed line.
* refactor(components): move LogViewer out of the logs route
LogViewer.svelte lived under src/routes/logs/ but was never used by
that route (LogsPanel.svelte and +page.svelte render container/system
logs their own way). Its only three consumers -- ComposeOutputModal,
StackModal, and DeploysPanel, all under src/routes/stacks/ -- imported
it cross-route with a relative path, which is what shared components
under src/lib/components/ exist to avoid.
Moved it there and repointed all three imports at the $lib alias. No
internal change: every import inside LogViewer.svelte already used
$lib/... aliases, none were relative to its old location.
* test(types): fix svelte-check type errors in two test files
deploy-run-record-core.test.ts: d.summary is DeploySummary | undefined
on RunDetails, so d.summary.containersStarted possibly-undefined'd.
Switched to optional chaining -- the assertion stays just as strong
since expect(undefined).toBe(2) still fails, the RHS is a hard literal.
deploy-run-view.test.ts: both test.each((input, expected) => ...)
callbacks had their params implicitly typed 'any'. Both formatTrigger
and formatStatusLabel take/return plain strings, so annotated both
params 'string', matching the explicit-annotation style test.each
already uses elsewhere in this repo (stack-filename.test.ts).
* feat(settings): make deploy-log reconcile schedule configurable
setDeployLogReconcileCron/setDeployLogReconcileEnabled existed in db.ts
with no caller outside of it and the scheduler startup path -- there was
no way to disable or reschedule the daily reconcile job (which deletes
orphaned deploy-log files) short of writing to the settings table by
hand. Wire both into the general settings GET/POST route, the client
settings store, and a new toggle+cron block on the General settings
tab, following the existing scannerCleanupCron/scannerCleanupEnabled
pattern in the same files.
refreshSystemJobs() already read these getters before this change; it
just never had anything to react to.
* fix(format): read naive server timestamps as UTC in formatRelativeTime
formatRelativeTime() parsed its input with a bare `new Date(...)`, so a
naive PG timestamp (e.g. "2026-08-30 20:36:11.534" - no "Z"/offset, the
form Drizzle hands back for a `timestamp without time zone` column) got
read as browser-local time instead of UTC. A deploy from a couple of
minutes ago rendered as "2h ago" for anyone outside UTC.
date-format.ts already solved exactly this for formatDateTime() via
parseTimestamp() (#1183), but formatRelativeTime() lives in the
import-light utils/format module and was never switched over, even
though every caller passes the same raw server value to both functions
side by side.
Switch formatRelativeTime() to parseTimestamp() as well. Behavior is
unchanged for Date instances, epoch numbers, and already-zoned strings;
only the previously-mishandled naive-string case is fixed.
Added a TZ-forced (America/New_York) test block that reproduces the
bug and verifies the fix; reverting the change makes those three tests
fail with "in the future" (a 4h forward misinterpretation), confirming
they fail for the right reason.
* fix(tests): inject the reconcile job's log-store calls instead of mocking the module
mock.module() freezes a module's export shape for the whole test process, not
just the file that calls it. The reconcile job's test replaced
'$lib/server/deploy-log-store' with a two-export factory, so every module that
resolved the same specifier afterwards lost the other eight -- deploy-run-record.ts
imports appendRunLog, runLogFileName and SizeBudgetTracker from there and threw
"Export named ... not found".
Whether it triggered depended purely on the order bun walks tests/, which is why
the suite was green locally and red in CI.
The test's own comment reasoned that no other test file claimed that specifier.
That was the wrong question: the collision is with source modules, not test files.
runDeployLogReconcileJob() now takes its two filesystem calls as an optional deps
parameter, defaulting to the real ones. The test passes fakes and no longer
touches the module registry, so there is nothing left to collide.
* fix(deploy): redact provider-resolved secrets from stored deploy errors
Deploy run records (schedule_executions.errorMessage, readable by every
authenticated user via GET /api/schedules/executions) were redacted only
against the secret list a caller had at recorder-construction time --
DB-only nonSecretVars/secretVars, captured BEFORE deployStack() calls
resolveProviderEnvVars() and merges in the bound secret provider's
values (Bitwarden/1Password/Keepass/Proton/Vault/Infisical bulk pulls or
inline refs). A provider-resolved secret that surfaced in compose's raw
stderr therefore passed straight through DeployRunRecorder.end()'s
redaction and into errorMessage unredacted.
Add StackOperationResult.resolvedSecrets, populated once in deployStack()
(the single call site after resolveProviderEnvVars, covering both the
local and Hawser paths) from the post-resolution secretVars -- the same
set executeComposeCommand already redacts streamed lines against. Add
RunRecorder.addSecrets()/DeployRunRecorder.addSecrets() to extend the
redaction list after construction, and call it from every stack_deploy
run-record caller (the three createRunRecorder-based API routes and
deployGitStack, the funnel every git-stack deploy trigger -- including
the unattended cron/webhook ones -- goes through) right after
deployStack() returns and before the recorder is closed.
Regression tests assert on the stored value, not just field presence,
and were verified against a temporarily neutered addSecrets() (the
secret leaked verbatim, confirming they fail for the right reason)
before being restored to pass.
* fix(deploy-logs): scope deploy-log directory and size budget per environment
Before this fix, deploy-log-store.ts kept ALL environments' deploy-run log
files in a single shared deploy-logs/ directory and enforced a single
BUDGET_BYTES total across all of them (SizeBudgetTracker). A user with
deploy rights in one environment could fill that shared budget with a noisy
deploy, silently truncating deploy logs and summaries belonging to every
other environment -- a multi-tenant audit-trail integrity violation.
- deploy-log-store.ts: every path/size function now takes an `envId` and
resolves to deploy-logs/<envDirName(envId)>/ (envDirName(null) => "null").
SizeBudgetTracker measures "everything else" scoped to its own run's
environment only. Added envDirName/parseEnvDirName, listEnvDirNames() for
reconcile-job discovery, and a readRunLog() fallback to the pre-F5 flat
path (unreleased feature, no production data in that layout -- fallback
exists purely as a robustness measure).
- deploy-run-record.ts: DeployRunRecorder threads its envId through to
appendRunLog() and its default SizeBudgetTracker.
- db.ts: getScheduleExecutionIdsByType() now also returns environmentId, so
the reconcile job can group records per environment.
- deploy-log-reconcile.ts: reconciles PER ENVIRONMENT (union of directories
on disk and environments referenced by records), so an orphan file in one
environment can never be deleted -- or a record marked logMissing --
because of a same-numbered file/record in a different environment.
- Both single-run routes (GET/DELETE .../deploys/[runId], GET .../log) now
pass run.environmentId through to readRunLog()/deleteRunLog().
Tests: new cross-environment isolation coverage at the SizeBudgetTracker,
DeployRunRecorder, and reconcile-job levels, each proven against a reverted
(shared-directory) mutation to confirm it fails for the right reason before
being reverted back. Full suite: 3011 pass / 0 fail (was 2987 pass / 0 fail).
* fix(deploy): forbid deleting a non-terminal deploy run
DELETE /api/stacks/[name]/deploys/[runId] deleted the schedule_executions
row and the on-disk log file with no check on the run's status, so a
caller could delete their OWN still-running deploy's record. The
background deploy process keeps going after that: recorder.line()
recreates the just-deleted log file via appendRunLog(), recorder.end()'s
updateScheduleExecution() becomes a silent no-op (UPDATE WHERE id=x, 0
rows), and the default-enabled reconcile job then treats the recreated
file as an orphan and purges it -- erasing every trace of a deploy that
actually ran and had real host side effects.
Add isTerminalRunStatus() (deploy-run-access.ts) as an explicit allow-list
of the run-closing statuses (success/failed/skipped/warning/error/
cancelled/stale) and use it to gate the DELETE handler: a non-terminal
status (queued/running, or anything this code doesn't recognize -- the
status column is free text) now returns 409 and touches neither the log
file nor the database row. The check runs after the existing
ownership/permission checks, so an unauthorized or wrong-stack caller
still gets 403/404 rather than learning the run's status via a 409.
Deliberately its own allow-list rather than reusing
deploy-log-reconcile-core.ts's isEligibleForMissingMark(): that helper
fails OPEN for an unrecognized status because marking logMissing is
reversible metadata-only. Deleting a run is not reversible, so this
check fails CLOSED instead.
* test(deploy): guard that deployStack records the secret set it redacts against
A review found F4's single production line -- stacks.ts's
`result.resolvedSecrets = Object.values(secretVars)` -- had zero coverage:
deleting it left the whole targeted suite green. The two tests that reference
`resolvedSecrets` fake deployStack wholesale and hand-set the field, proving the
downstream recorder wiring but never that the real stacks.ts populates it.
A behaviour test of the real deployStack() is infeasible in this suite, established
empirically, not assumed:
- tests/helpers/stacks-fake.ts mock.module()s $lib/server/stacks at import time,
freezing that specifier for the whole Bun process; two suites load it, so
`import { deployStack }` resolves to the fake for the entire run, and Bun has no
per-file process isolation to escape it.
- pulling the real $lib/server/stacks into a test's import graph instantiates
better-sqlite3, which Bun does not support and crashes at load (reproduced).
So this follows the pattern the suite already uses for deployStack-internal
invariants (deploy-stack-online.test.ts, stack-create-start-build-options.test.ts):
a source guard, with comments stripped before matching and a vacuity guard. It pins
the wiring resolveProviderEnvVars -> secretVars -> executeComposeCommand (redaction)
AND -> result.resolvedSecrets (recording). Counter-tested: each assertion goes red
for its own mutation, including the exact deletion the review demonstrated.
* chore(openapi): regenerate spec after rebase onto upstream/main
* test(stacks): fake requireEnvAccess in the compose-endpoint auth stub
The old merge commit into this branch had manually patched this fake to
add requireEnvAccess (compose/+server.ts calls it, and authEnabled is
false here so the real implementation is a no-op) -- that fixup lived
only in the merge commit itself, not in any single commit, so a plain
rebase onto upstream/main dropped it. Restored; full suite is green
(3212 pass, 0 fail).
* docs(openapi): document deployLogReconcileCron/Enabled in the settings body
upstream's new Gate 7 (body-field drift check) caught that this PR's
deployLogReconcileCron/deployLogReconcileEnabled fields were read in
POST /api/settings/general but never listed in its @openapi body
annotation. Documented them and regenerated the spec; all openapi
gates and the full test suite (3221 pass, 0 fail) are green.
* fix(deploy-log-reconcile): protect an in-flight deploy's file from a TOCTOU orphan-delete
The reconcile job snapshots stack_deploy records once via
getScheduleExecutionIdsByType() before walking each environment's files.
A deploy whose record is created after that snapshot has no matching id
yet, but its log file can already exist by the time this environment's
listRunLogIds() runs -- indistinguishable from a genuine orphan by id
alone, and previously deleted on the spot while still being written to.
Add getRunLogMtimeMs() (deploy-log-store.ts) and use it as a second
signal alongside the records snapshot: a delete candidate whose file was
modified within the last 5 minutes is left alone this run instead of
deleted, since appendRunLog() keeps bumping a live deploy's mtime for as
long as it keeps writing. A later reconcile run will see the by-then-
created record and correctly never touch the file again -- nothing is
lost, it is just given one more cycle to prove itself real, the same
trade-off isEligibleForMissingMark() already makes on the other side of
this race.
* fix(git): record a stack_deploy run for the UI's manual deploy button
deployGitStack's doc comment claimed it was the single function every
git-stack deploy trigger funnels through, and wired the stack_deploy run
recorder in there on that basis. deployGitStackWithProgress -- the UI's
"Deploy" button (deploy-stream/+server.ts) -- runs its own clone/read/
deploy pipeline instead of calling deployGitStack (it needs its own
discrete onProgress stages, not deployGitStack's onLine streaming), so
it was the one caller that reached deployStack() without ever going
through createRunRecorder: webhook, cron, and config-update-triggered
deploys were recorded, the manual button was not.
Wire the same recorder setup into deployGitStackWithProgress, right
before its own deployStack() call, mirroring deployGitStack's shape
(composeHash/envHash/secrets from the DB-only vars, addSecrets() for the
provider-resolved values deployStack() only reveals after it returns,
end() closing the row on every path out). onLine mirrors output into the
run's log file without being forwarded to onProgress, so the existing
progress-stage streaming is untouched.
Corrected deployGitStack's doc comment, which is no longer accurate now
that a second call site exists, and extended the existing source-level
guard test (previously asserting exactly one createRunRecorder call in
the whole file) to check both functions individually.
This commit is contained in:
1 parent
be10bb098c
commit
b2eb0b5dcb
104 files changed
+17212
-605
No files matched your search
@@ -0,0 +1,444 @@
|
||||
<script lang="ts">
|
||||
import { RefreshCw, Copy, Download, WrapText, ArrowDownToLine, Search, ChevronUp, ChevronDown, X, Type, Eraser, Filter, Hash } from 'lucide-svelte';
|
||||
import { wrapHtmlLines } from '$lib/utils/log-lines';
|
||||
import { copyToClipboard } from '$lib/utils/clipboard';
|
||||
import { downloadFileName, stripAnsi } from '$lib/utils/log-download-name';
|
||||
import { isScrolledToBottom, shouldResetScrollPause } from '$lib/utils/scroll-position';
|
||||
import * as Select from '$lib/components/ui/select';
|
||||
import { appSettings, formatLogTimestamps } from '$lib/stores/settings';
|
||||
import { themeStore } from '$lib/stores/theme';
|
||||
import { getMonospaceFont } from '$lib/themes';
|
||||
import { AnsiUp } from 'ansi_up';
|
||||
const ansiUp = new AnsiUp();
|
||||
ansiUp.use_classes = true;
|
||||
|
||||
interface Props {
|
||||
logs: string;
|
||||
title?: string;
|
||||
loading?: boolean;
|
||||
autoRefresh?: boolean;
|
||||
autoScroll?: boolean;
|
||||
onRefresh?: () => void;
|
||||
onClear?: () => void;
|
||||
onAutoRefreshChange?: (value: boolean) => void;
|
||||
onAutoScrollChange?: (value: boolean) => void;
|
||||
class?: string;
|
||||
// Follows the caller's own light/dark switch, independent of the app-wide
|
||||
// theme -- e.g. the editor theme toggle in StackModal (`dockhand-editor-theme`).
|
||||
// Defaults to 'dark' so every existing caller that doesn't pass it renders
|
||||
// exactly as before.
|
||||
theme?: 'light' | 'dark';
|
||||
}
|
||||
|
||||
let {
|
||||
logs,
|
||||
title,
|
||||
loading = false,
|
||||
autoRefresh = true,
|
||||
autoScroll = true,
|
||||
onRefresh,
|
||||
onClear,
|
||||
onAutoRefreshChange,
|
||||
onAutoScrollChange,
|
||||
class: className = '',
|
||||
theme = 'dark'
|
||||
}: Props = $props();
|
||||
|
||||
let dark = $derived(theme === 'dark');
|
||||
|
||||
let logsRef: HTMLDivElement;
|
||||
let wordWrap = $state(true);
|
||||
let showLineNumbers = $state(typeof window !== 'undefined' && localStorage.getItem('dockhand-log-line-numbers') === 'true');
|
||||
let fontSize = $state(12);
|
||||
|
||||
// RAF-based auto-scroll
|
||||
let scrollRafPending = false;
|
||||
// True once the user has scrolled up to read earlier output. Cleared again once
|
||||
// they scroll back within range of the end, at which point auto-scroll resumes.
|
||||
let userScrolledUp = $state(false);
|
||||
// Set for the duration of our own programmatic scrollTop write, so the 'scroll'
|
||||
// event it triggers isn't mistaken for the user scrolling. Without this, a fast
|
||||
// enough stream of new lines can grow scrollHeight again between the write and
|
||||
// the event firing, leaving a gap wide enough to look like the user scrolled up.
|
||||
let isAutoScrolling = false;
|
||||
// Tracks the previous value of the autoScroll prop, purely so the effect below can
|
||||
// tell a false -> true edge (a fresh run starting) apart from autoScroll simply
|
||||
// staying true across renders. Left unset here rather than seeded from `autoScroll`
|
||||
// directly -- reading a prop in a plain top-level initializer only captures its
|
||||
// value at that one point in time, which svelte-check flags (state_referenced_locally)
|
||||
// because it usually signals a bug. The effect below seeds it on its first run instead,
|
||||
// where reading `autoScroll` is tracked.
|
||||
let previousAutoScroll: boolean | undefined;
|
||||
|
||||
// Search state
|
||||
let logSearchActive = $state(false);
|
||||
let logSearchQuery = $state('');
|
||||
let logSearchFilterMode = $state(typeof window !== 'undefined' && localStorage.getItem('dockhand-log-filter-mode') === 'true');
|
||||
let currentMatchIndex = $state(0);
|
||||
let matchCount = $state(0);
|
||||
let logSearchInputRef: HTMLInputElement;
|
||||
|
||||
const fontSizeOptions = [10, 12, 14, 16];
|
||||
|
||||
// Get terminal font family from theme preferences
|
||||
let terminalFontFamily = $derived(() => {
|
||||
const fontMeta = getMonospaceFont($themeStore.terminalFont);
|
||||
return fontMeta?.family || 'ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace';
|
||||
});
|
||||
|
||||
// A fresh run starting (autoScroll going false -> true) lifts a scroll-up pause left
|
||||
// over from an earlier run -- otherwise scrolling up once during a deploy pauses
|
||||
// auto-scroll for every later run of the same page session, since nothing else ever
|
||||
// clears userScrolledUp.
|
||||
$effect(() => {
|
||||
if (previousAutoScroll !== undefined && shouldResetScrollPause(previousAutoScroll, autoScroll)) {
|
||||
userScrolledUp = false;
|
||||
}
|
||||
previousAutoScroll = autoScroll;
|
||||
});
|
||||
|
||||
// Auto-scroll when logs change, unless the user scrolled up to read earlier output.
|
||||
$effect(() => {
|
||||
if (autoScroll && !userScrolledUp && logsRef && logs) {
|
||||
if (!scrollRafPending) {
|
||||
scrollRafPending = true;
|
||||
requestAnimationFrame(() => {
|
||||
if (logsRef) {
|
||||
isAutoScrolling = true;
|
||||
logsRef.scrollTop = logsRef.scrollHeight;
|
||||
// Let the 'scroll' event this write triggers reach handleLogScroll (and be
|
||||
// ignored there) before treating further events as user-initiated again.
|
||||
requestAnimationFrame(() => {
|
||||
isAutoScrolling = false;
|
||||
});
|
||||
}
|
||||
scrollRafPending = false;
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Pause auto-scroll once the user scrolls up, resume once they scroll back down.
|
||||
function handleLogScroll() {
|
||||
if (isAutoScrolling || !logsRef) return;
|
||||
const { scrollTop, scrollHeight, clientHeight } = logsRef;
|
||||
userScrolledUp = !isScrolledToBottom(scrollTop, scrollHeight, clientHeight);
|
||||
}
|
||||
|
||||
// Copy logs to clipboard
|
||||
async function copyLogs() {
|
||||
if (logs) {
|
||||
await copyToClipboard(stripAnsi(logs));
|
||||
}
|
||||
}
|
||||
|
||||
// Download logs as txt file
|
||||
function downloadLogs() {
|
||||
if (logs) {
|
||||
const blob = new Blob([stripAnsi(logs)], { type: 'text/plain' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = downloadFileName(title);
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
}
|
||||
|
||||
// Search functions
|
||||
function toggleLogSearch() {
|
||||
logSearchActive = !logSearchActive;
|
||||
if (logSearchActive) {
|
||||
setTimeout(() => logSearchInputRef?.focus(), 50);
|
||||
} else {
|
||||
logSearchQuery = '';
|
||||
currentMatchIndex = 0;
|
||||
matchCount = 0;
|
||||
}
|
||||
}
|
||||
|
||||
function closeLogSearch() {
|
||||
logSearchActive = false;
|
||||
logSearchQuery = '';
|
||||
logSearchFilterMode = false;
|
||||
currentMatchIndex = 0;
|
||||
matchCount = 0;
|
||||
}
|
||||
|
||||
function toggleSearchFilterMode() {
|
||||
logSearchFilterMode = !logSearchFilterMode;
|
||||
localStorage.setItem('dockhand-log-filter-mode', String(logSearchFilterMode));
|
||||
}
|
||||
|
||||
function navigateMatch(direction: 'prev' | 'next') {
|
||||
if (!logsRef || matchCount === 0) return;
|
||||
|
||||
const matches = logsRef.querySelectorAll('.search-match');
|
||||
if (matches.length === 0) return;
|
||||
|
||||
matches[currentMatchIndex]?.classList.remove('current-match');
|
||||
|
||||
if (direction === 'next') {
|
||||
currentMatchIndex = (currentMatchIndex + 1) % matches.length;
|
||||
} else {
|
||||
currentMatchIndex = (currentMatchIndex - 1 + matches.length) % matches.length;
|
||||
}
|
||||
|
||||
const currentEl = matches[currentMatchIndex];
|
||||
if (currentEl) {
|
||||
currentEl.classList.add('current-match');
|
||||
currentEl.scrollIntoView({ behavior: 'smooth', block: 'center' });
|
||||
}
|
||||
}
|
||||
|
||||
function handleLogSearchKeydown(e: KeyboardEvent) {
|
||||
if (e.key === 'Enter') {
|
||||
e.preventDefault();
|
||||
if (e.shiftKey) {
|
||||
navigateMatch('prev');
|
||||
} else {
|
||||
navigateMatch('next');
|
||||
}
|
||||
} else if (e.key === 'Escape') {
|
||||
closeLogSearch();
|
||||
}
|
||||
}
|
||||
|
||||
// Highlighted logs with search matches and ANSI color support
|
||||
let highlightedLogs = $derived(() => {
|
||||
let text = logs || '';
|
||||
if ($appSettings.formatLogTimestamps) {
|
||||
text = formatLogTimestamps(text);
|
||||
}
|
||||
|
||||
const query = logSearchQuery.trim();
|
||||
|
||||
// Filter lines before ANSI conversion (plain text matching)
|
||||
if (logSearchFilterMode && query) {
|
||||
const escapedForRegex = query.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
const filterRegex = new RegExp(escapedForRegex, 'i');
|
||||
const lines = text.split('\n');
|
||||
text = lines.filter(line => filterRegex.test(line)).join('\n');
|
||||
}
|
||||
|
||||
const withAnsi = ansiUp.ansi_to_html(text);
|
||||
if (!query) return withAnsi;
|
||||
|
||||
const escapedForRegex = query.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
const escapedQuery = escapedForRegex.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>');
|
||||
|
||||
// Split by HTML tags and only process text parts
|
||||
const parts = withAnsi.split(/(<[^>]*>)/);
|
||||
return parts.map(part => {
|
||||
if (part.startsWith('<')) return part;
|
||||
return part.replace(new RegExp(`(${escapedQuery})`, 'gi'), '<mark class="search-match">$1</mark>');
|
||||
}).join('');
|
||||
});
|
||||
|
||||
// Update match count after render
|
||||
$effect(() => {
|
||||
const html = highlightedLogs();
|
||||
|
||||
if (logSearchQuery && logsRef) {
|
||||
setTimeout(() => {
|
||||
const matches = logsRef.querySelectorAll('.search-match');
|
||||
matchCount = matches.length;
|
||||
currentMatchIndex = 0;
|
||||
if (matches.length > 0) {
|
||||
matches[0].classList.add('current-match');
|
||||
matches[0].scrollIntoView({ behavior: 'smooth', block: 'center' });
|
||||
}
|
||||
}, 100);
|
||||
} else {
|
||||
matchCount = 0;
|
||||
currentMatchIndex = 0;
|
||||
}
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col rounded-lg border overflow-hidden {dark ? 'log-viewer-dark bg-zinc-950 border-zinc-800' : 'log-viewer-light bg-gray-50 border-gray-300'} {className}">
|
||||
<!-- Header bar -->
|
||||
<div class="flex items-center justify-between px-3 py-1.5 border-b shrink-0 {dark ? 'border-zinc-800 bg-zinc-900/50' : 'border-gray-300 bg-gray-100'}">
|
||||
<div class="flex items-center gap-2">
|
||||
{#if loading}
|
||||
<RefreshCw class="w-3 h-3 animate-spin {dark ? 'text-zinc-500' : 'text-gray-400'}" />
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<!-- Auto-refresh button -->
|
||||
<button
|
||||
onclick={() => onAutoRefreshChange?.(!autoRefresh)}
|
||||
class="flex items-center gap-1 px-1.5 py-0.5 rounded text-xs transition-colors {autoRefresh ? (dark ? 'bg-amber-500/20 ring-1 ring-amber-500/50 text-amber-400' : 'bg-amber-500/30 ring-1 ring-amber-600/50 text-amber-700') : dark ? 'text-zinc-500 hover:text-zinc-300 hover:bg-zinc-800' : 'text-gray-500 hover:text-gray-700 hover:bg-gray-300'}"
|
||||
title="Toggle auto-refresh"
|
||||
>
|
||||
<RefreshCw class="w-3 h-3" />
|
||||
</button>
|
||||
<!-- Auto-scroll button -->
|
||||
<button
|
||||
onclick={() => onAutoScrollChange?.(!autoScroll)}
|
||||
class="flex items-center gap-1 px-1.5 py-0.5 rounded text-xs transition-colors {autoScroll ? (dark ? 'bg-amber-500/20 ring-1 ring-amber-500/50 text-amber-400' : 'bg-amber-500/30 ring-1 ring-amber-600/50 text-amber-700') : dark ? 'text-zinc-500 hover:text-zinc-300 hover:bg-zinc-800' : 'text-gray-500 hover:text-gray-700 hover:bg-gray-300'}"
|
||||
title="Toggle auto-scroll"
|
||||
>
|
||||
<ArrowDownToLine class="w-3 h-3" />
|
||||
</button>
|
||||
<!-- Font size -->
|
||||
<Select.Root type="single" value={String(fontSize)} onValueChange={(v) => fontSize = Number(v)}>
|
||||
<Select.Trigger class="h-6 w-16 text-xs px-1.5 {dark ? 'bg-zinc-800 border-zinc-700 text-zinc-300' : 'bg-white border-gray-300 text-gray-700'}">
|
||||
<Type class="w-3 h-3 mr-1 {dark ? 'text-zinc-400' : 'text-gray-400'}" />
|
||||
<span>{fontSize}px</span>
|
||||
</Select.Trigger>
|
||||
<Select.Content>
|
||||
{#each fontSizeOptions as size}
|
||||
<Select.Item value={String(size)} label="{size}px">
|
||||
<Type class="w-3 h-3 mr-1.5 text-muted-foreground" />
|
||||
{size}px
|
||||
</Select.Item>
|
||||
{/each}
|
||||
</Select.Content>
|
||||
</Select.Root>
|
||||
<!-- Word wrap -->
|
||||
<button
|
||||
onclick={() => wordWrap = !wordWrap}
|
||||
class="p-1 rounded transition-colors {wordWrap ? (dark ? 'bg-amber-500/20 ring-1 ring-amber-500/50' : 'bg-amber-500/30 ring-1 ring-amber-600/50') : ''} {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Toggle word wrap"
|
||||
>
|
||||
<WrapText class="w-3 h-3 transition-colors {wordWrap ? (dark ? 'text-amber-400' : 'text-amber-700') : dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
<!-- Line numbers -->
|
||||
<button
|
||||
onclick={() => { showLineNumbers = !showLineNumbers; localStorage.setItem('dockhand-log-line-numbers', String(showLineNumbers)); }}
|
||||
class="p-1 rounded transition-colors {showLineNumbers ? (dark ? 'bg-amber-500/20 ring-1 ring-amber-500/50' : 'bg-amber-500/30 ring-1 ring-amber-600/50') : ''} {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title={showLineNumbers ? 'Hide line numbers' : 'Show line numbers'}
|
||||
>
|
||||
<Hash class="w-3 h-3 transition-colors {showLineNumbers ? (dark ? 'text-amber-400' : 'text-amber-700') : dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
<!-- Search -->
|
||||
{#if logSearchActive}
|
||||
<div class="flex items-center gap-1 rounded px-1.5 py-0.5 {dark ? 'bg-zinc-800' : 'bg-gray-200'}">
|
||||
<Search class="w-3 h-3 {dark ? 'text-amber-400' : 'text-amber-700'}" />
|
||||
<input
|
||||
bind:this={logSearchInputRef}
|
||||
type="text"
|
||||
placeholder="Search..."
|
||||
bind:value={logSearchQuery}
|
||||
onkeydown={handleLogSearchKeydown}
|
||||
class="bg-transparent border-none outline-none text-xs w-20 {dark ? 'text-zinc-200 placeholder:text-zinc-500' : 'text-gray-800 placeholder:text-gray-400'}"
|
||||
/>
|
||||
<button
|
||||
onclick={toggleSearchFilterMode}
|
||||
class="p-0.5 rounded transition-colors {logSearchFilterMode ? (dark ? 'bg-amber-500/20 ring-1 ring-amber-500/50' : 'bg-amber-500/30 ring-1 ring-amber-600/50') : dark ? 'hover:bg-zinc-700' : 'hover:bg-gray-300'}"
|
||||
title={logSearchFilterMode ? 'Show all lines (filter mode active)' : 'Hide non-matching lines'}
|
||||
>
|
||||
<Filter class="w-3 h-3 transition-colors {logSearchFilterMode ? (dark ? 'text-amber-400' : 'text-amber-700') : dark ? 'text-zinc-400' : 'text-gray-500'}" />
|
||||
</button>
|
||||
{#if matchCount > 0}
|
||||
<span class="text-xs {dark ? 'text-zinc-400' : 'text-gray-500'}">{currentMatchIndex + 1}/{matchCount}</span>
|
||||
{:else if logSearchQuery}
|
||||
<span class="text-xs {dark ? 'text-zinc-500' : 'text-gray-400'}">0/0</span>
|
||||
{/if}
|
||||
<button onclick={() => navigateMatch('prev')} class="p-0.5 rounded {dark ? 'hover:bg-zinc-700' : 'hover:bg-gray-300'}" title="Previous">
|
||||
<ChevronUp class="w-3 h-3 {dark ? 'text-zinc-400' : 'text-gray-500'}" />
|
||||
</button>
|
||||
<button onclick={() => navigateMatch('next')} class="p-0.5 rounded {dark ? 'hover:bg-zinc-700' : 'hover:bg-gray-300'}" title="Next">
|
||||
<ChevronDown class="w-3 h-3 {dark ? 'text-zinc-400' : 'text-gray-500'}" />
|
||||
</button>
|
||||
<button onclick={closeLogSearch} class="p-0.5 rounded {dark ? 'hover:bg-zinc-700' : 'hover:bg-gray-300'}" title="Close">
|
||||
<X class="w-3 h-3 {dark ? 'text-zinc-400' : 'text-gray-500'}" />
|
||||
</button>
|
||||
</div>
|
||||
{:else}
|
||||
<button
|
||||
onclick={toggleLogSearch}
|
||||
class="p-1 rounded transition-colors {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Search logs"
|
||||
>
|
||||
<Search class="w-3 h-3 {dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
{/if}
|
||||
<!-- Copy -->
|
||||
<button
|
||||
onclick={copyLogs}
|
||||
class="p-1 rounded transition-colors {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Copy logs"
|
||||
>
|
||||
<Copy class="w-3 h-3 {dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
<!-- Download -->
|
||||
<button
|
||||
onclick={downloadLogs}
|
||||
class="p-1 rounded transition-colors {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Download logs"
|
||||
>
|
||||
<Download class="w-3 h-3 {dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
<!-- Clear -->
|
||||
{#if onClear}
|
||||
<button
|
||||
onclick={() => onClear?.()}
|
||||
class="p-1 rounded transition-colors {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Clear logs"
|
||||
>
|
||||
<Eraser class="w-3 h-3 {dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
{/if}
|
||||
<!-- Refresh -->
|
||||
{#if onRefresh}
|
||||
<button
|
||||
onclick={() => onRefresh?.()}
|
||||
class="p-1 rounded transition-colors {dark ? 'hover:bg-zinc-800' : 'hover:bg-gray-300'}"
|
||||
title="Refresh logs"
|
||||
>
|
||||
<RefreshCw class="w-3 h-3 {dark ? 'text-zinc-500 hover:text-zinc-300' : 'text-gray-500 hover:text-gray-700'}" />
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Logs content -->
|
||||
<div bind:this={logsRef} onscroll={handleLogScroll} class="flex-1 overflow-auto p-4">
|
||||
{#if logs}
|
||||
<pre class="{dark ? 'text-zinc-50' : 'text-gray-900'} {wordWrap ? 'whitespace-pre-wrap' : 'whitespace-pre'} {showLineNumbers ? 'show-line-numbers' : ''}" style="font-size: {fontSize}px; font-family: {terminalFontFamily()};">{@html wrapHtmlLines(highlightedLogs())}</pre>
|
||||
{:else if loading}
|
||||
<div class="flex items-center justify-center h-full {dark ? 'text-zinc-500' : 'text-gray-500'}">
|
||||
<RefreshCw class="w-5 h-5 animate-spin mr-2" />
|
||||
Loading logs...
|
||||
</div>
|
||||
{:else}
|
||||
<p class="text-sm {dark ? 'text-zinc-500' : 'text-gray-500'}">No logs available</p>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
:global(.search-match) {
|
||||
background-color: rgba(234, 179, 8, 0.4);
|
||||
color: #fef3c7;
|
||||
border-radius: 2px;
|
||||
padding: 1px 2px;
|
||||
box-shadow: 0 0 4px rgba(234, 179, 8, 0.5);
|
||||
}
|
||||
:global(.search-match.current-match) {
|
||||
background-color: rgba(234, 179, 8, 0.8);
|
||||
color: #1a1a1a;
|
||||
font-weight: 600;
|
||||
box-shadow: 0 0 8px rgba(234, 179, 8, 0.9), 0 0 16px rgba(234, 179, 8, 0.5);
|
||||
outline: 2px solid rgb(250, 204, 21);
|
||||
}
|
||||
|
||||
/* The base .search-match rule above pairs a pale amber background with
|
||||
near-white text (#fef3c7) -- fine against the dark log background this
|
||||
component defaults to, but low-contrast once the panel switches to its
|
||||
own light theme (see the `theme` prop). Scoped to .log-viewer-light so it
|
||||
never touches the dark styling above or other components sharing the
|
||||
(unscoped) .search-match class, such as LogsPanel. Higher specificity than
|
||||
the base rule -- (0,2,0) vs (0,1,0) -- so it wins regardless of source order.
|
||||
.current-match is left alone: its 0.8 background alpha already composites to
|
||||
a near-solid amber against either page background, so the dark (#1a1a1a) text
|
||||
it uses stays legible without a light-mode override. */
|
||||
:global(.log-viewer-light .search-match) {
|
||||
color: #78350f;
|
||||
box-shadow: 0 0 4px rgba(217, 119, 6, 0.35);
|
||||
}
|
||||
</style>
|
||||
@@ -22110,7 +22110,7 @@
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Schedule type (container_update, git_stack_sync, system_cleanup, env_update_check, image_prune, backup, repo_prune, repo_check, repo_verify)"
|
||||
"description": "Schedule type (container_update, git_stack_sync, system_cleanup, env_update_check, image_prune, backup, repo_prune, repo_check, repo_verify, deploy_log_reconcile)"
|
||||
},
|
||||
{
|
||||
"name": "id",
|
||||
@@ -22184,7 +22184,7 @@
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Schedule type (container_update, git_stack_sync, env_update_check, image_prune, backup, repo_prune, repo_check, repo_verify, system_cleanup)"
|
||||
"description": "Schedule type (container_update, git_stack_sync, env_update_check, image_prune, backup, repo_prune, repo_check, repo_verify, system_cleanup, deploy_log_reconcile)"
|
||||
},
|
||||
{
|
||||
"name": "id",
|
||||
@@ -22224,7 +22224,7 @@
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Invalid schedule id, unsupported type, or system_cleanup (cannot be paused)"
|
||||
"description": "Invalid schedule id, unsupported type, or system_cleanup/deploy_log_reconcile (cannot be paused)"
|
||||
},
|
||||
"404": {
|
||||
"description": "Schedule, backup config, or backup destination not found"
|
||||
@@ -23777,6 +23777,12 @@
|
||||
"defaultTrivyImage": {
|
||||
"type": "string"
|
||||
},
|
||||
"deployLogReconcileCron": {
|
||||
"type": "string"
|
||||
},
|
||||
"deployLogReconcileEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"downloadFormat": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -23895,6 +23901,8 @@
|
||||
"defaultScannerNetworkMode": "string",
|
||||
"defaultTrivyArgs": "string",
|
||||
"defaultTrivyImage": "string",
|
||||
"deployLogReconcileCron": "string",
|
||||
"deployLogReconcileEnabled": true,
|
||||
"downloadFormat": "string",
|
||||
"editorFont": "string",
|
||||
"eventCleanupCron": "string",
|
||||
@@ -24675,6 +24683,15 @@
|
||||
},
|
||||
"environmentId": {
|
||||
"type": "integer"
|
||||
},
|
||||
"pull": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"build": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"forceRecreate": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -24694,7 +24711,10 @@
|
||||
"secretProviderId": 0,
|
||||
"start": true,
|
||||
"envId": 0,
|
||||
"environmentId": 0
|
||||
"environmentId": 0,
|
||||
"pull": true,
|
||||
"build": true,
|
||||
"forceRecreate": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25036,6 +25056,15 @@
|
||||
},
|
||||
"secretProviderId": {
|
||||
"type": "integer"
|
||||
},
|
||||
"pull": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"build": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"forceRecreate": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -25050,7 +25079,10 @@
|
||||
"oldEnvPath": "string",
|
||||
"moveFromDir": "string",
|
||||
"restart": true,
|
||||
"secretProviderId": 0
|
||||
"secretProviderId": 0,
|
||||
"pull": true,
|
||||
"build": true,
|
||||
"forceRecreate": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25221,6 +25253,357 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/stacks/{name}/deploys": {
|
||||
"get": {
|
||||
"operationId": "get_api_stacks_name_deploys",
|
||||
"tags": [
|
||||
"stacks"
|
||||
],
|
||||
"summary": "List deploy runs recorded for a stack (no log text -- see GET .../{runId}/log)",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Stack name (from GET /api/stacks)"
|
||||
},
|
||||
{
|
||||
"name": "env",
|
||||
"in": "query",
|
||||
"required": false,
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
},
|
||||
"description": "Environment id the stack belongs to (from GET /api/environments). Omit, or pass the literal string \"null\", for the local/default environment (deploys triggered without an explicit env, the normal shape on a single-environment install)."
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Successful response",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"runs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer"
|
||||
},
|
||||
"environmentId": {
|
||||
"type": "integer"
|
||||
},
|
||||
"triggeredBy": {
|
||||
"type": "string"
|
||||
},
|
||||
"triggeredAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"duration": {
|
||||
"type": "integer"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"errorMessage": {
|
||||
"type": "string"
|
||||
},
|
||||
"details": {
|
||||
"type": "object",
|
||||
"properties": {}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"triggeredBy",
|
||||
"triggeredAt",
|
||||
"status"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"runs"
|
||||
]
|
||||
},
|
||||
"example": {
|
||||
"runs": [
|
||||
{
|
||||
"id": 0,
|
||||
"environmentId": 0,
|
||||
"triggeredBy": "string",
|
||||
"triggeredAt": "string",
|
||||
"startedAt": "string",
|
||||
"completedAt": "string",
|
||||
"duration": 0,
|
||||
"status": "string",
|
||||
"errorMessage": "string",
|
||||
"details": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "env query parameter must be an integer, or omitted/\"null\" for the local environment"
|
||||
},
|
||||
"401": {
|
||||
"description": "Authentication required"
|
||||
},
|
||||
"403": {
|
||||
"description": "Permission denied (needs stacks:view), or access denied to this environment"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"cookieAuth": []
|
||||
},
|
||||
{
|
||||
"bearerAuth": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"/api/stacks/{name}/deploys/{runId}": {
|
||||
"get": {
|
||||
"operationId": "get_api_stacks_name_deploys_runId",
|
||||
"tags": [
|
||||
"stacks"
|
||||
],
|
||||
"summary": "Get a single deploy run's metadata (no log text -- see GET .../{runId}/log)",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Stack name (from GET /api/stacks)"
|
||||
},
|
||||
{
|
||||
"name": "runId",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
},
|
||||
"description": "Run id (from GET /api/stacks/{name}/deploys)"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Successful response",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer"
|
||||
},
|
||||
"environmentId": {
|
||||
"type": "integer"
|
||||
},
|
||||
"triggeredBy": {
|
||||
"type": "string"
|
||||
},
|
||||
"triggeredAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"startedAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"completedAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"duration": {
|
||||
"type": "integer"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"errorMessage": {
|
||||
"type": "string"
|
||||
},
|
||||
"details": {
|
||||
"type": "object",
|
||||
"properties": {}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"triggeredBy",
|
||||
"triggeredAt",
|
||||
"status"
|
||||
]
|
||||
},
|
||||
"example": {
|
||||
"id": 0,
|
||||
"environmentId": 0,
|
||||
"triggeredBy": "string",
|
||||
"triggeredAt": "string",
|
||||
"startedAt": "string",
|
||||
"completedAt": "string",
|
||||
"duration": 0,
|
||||
"status": "string",
|
||||
"errorMessage": "string",
|
||||
"details": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Authentication required"
|
||||
},
|
||||
"403": {
|
||||
"description": "Permission denied (needs stacks:view for the run's own environment), or access denied to this environment"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"cookieAuth": []
|
||||
},
|
||||
{
|
||||
"bearerAuth": []
|
||||
}
|
||||
],
|
||||
"description": "400 (\"Invalid run id\") and 404 (\"Deploy run not found\", also returned for a run that belongs to a different stack) are produced by the shared loadOwnedDeployRun() helper (deploy-run-access.ts), not literally in this handler's body -- the openapi generator's static scanner cannot see status()/error() calls in a different file, hence the prose here instead of resp-400/resp-404 lines (same pattern as POST /api/backup/destinations/{id}/task's description)."
|
||||
},
|
||||
"delete": {
|
||||
"operationId": "delete_api_stacks_name_deploys_runId",
|
||||
"tags": [
|
||||
"stacks"
|
||||
],
|
||||
"summary": "Delete a deploy run's database record and its on-disk log file",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Stack name (from GET /api/stacks)"
|
||||
},
|
||||
{
|
||||
"name": "runId",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
},
|
||||
"description": "Run id (from GET /api/stacks/{name}/deploys)"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Successful response",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"success"
|
||||
]
|
||||
},
|
||||
"example": {
|
||||
"success": true
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Authentication required"
|
||||
},
|
||||
"403": {
|
||||
"description": "Permission denied (needs stacks:edit for the run's own environment), or access denied to this environment"
|
||||
},
|
||||
"409": {
|
||||
"description": "The run has not finished yet (status is not a terminal one) -- deleting a"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"cookieAuth": []
|
||||
},
|
||||
{
|
||||
"bearerAuth": []
|
||||
}
|
||||
],
|
||||
"description": "400 (\"Invalid run id\") and 404 (\"Deploy run not found\", also returned for a run that belongs to a different stack) are produced by the shared loadOwnedDeployRun() helper (deploy-run-access.ts), not literally in this handler's body -- see GET .../{runId}'s description for why that means prose here instead of resp-400/resp-404 lines."
|
||||
}
|
||||
},
|
||||
"/api/stacks/{name}/deploys/{runId}/log": {
|
||||
"get": {
|
||||
"operationId": "get_api_stacks_name_deploys_runId_log",
|
||||
"tags": [
|
||||
"stacks"
|
||||
],
|
||||
"summary": "Get a deploy run's log text (text/plain, from the on-disk file)",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Stack name (from GET /api/stacks)"
|
||||
},
|
||||
{
|
||||
"name": "runId",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
},
|
||||
"description": "Run id (from GET /api/stacks/{name}/deploys)"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Plain-text log content"
|
||||
},
|
||||
"401": {
|
||||
"description": "Authentication required"
|
||||
},
|
||||
"403": {
|
||||
"description": "Permission denied (needs stacks:view for the run's own environment), or access denied to this environment"
|
||||
},
|
||||
"404": {
|
||||
"description": "Deploy run not found (also returned for a run that belongs to a different stack), or log file missing"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"cookieAuth": []
|
||||
},
|
||||
{
|
||||
"bearerAuth": []
|
||||
}
|
||||
],
|
||||
"description": "400 (\"Invalid run id\") is produced by the shared loadOwnedDeployRun() helper (deploy-run-access.ts), not literally in this handler's body -- the openapi generator's static scanner cannot see status()/error() calls in a different file, hence the prose here instead of a resp-400 line (same pattern as POST /api/backup/destinations/{id}/task's description). The 404 below covers BOTH loadOwnedDeployRun's \"run not found / belongs to a different stack\" AND this handler's own \"log file missing\" -- the latter IS literal in this file's body, which is why it can stay a resp-404 line."
|
||||
}
|
||||
},
|
||||
"/api/stacks/{name}/down": {
|
||||
"post": {
|
||||
"operationId": "post_api_stacks_name_down",
|
||||
|
||||
@@ -969,6 +969,12 @@ function restorePorts(destination: any, access: { isEnterprise: boolean; canAcce
|
||||
log(`registered: composePath=${composePath} envPath=${envPath ?? '(none)'}`);
|
||||
},
|
||||
deploy: async () => {
|
||||
// Deliberately does NOT create a stack_deploy run record: this call goes
|
||||
// straight to redeployStackFromDir (stacks.ts), not through the
|
||||
// deployGitStack()/compose-route path that createRunRecorder is wired
|
||||
// into. A backup restore's redeploy could get its own recorder the same
|
||||
// way -- there is nothing structural stopping it -- it just hasn't been
|
||||
// done yet.
|
||||
const r = await redeployStackFromDir(name, stackDir, composeFileName, envId ?? undefined);
|
||||
if (!r.success) throw new Error(r.error || 'docker compose up failed');
|
||||
}
|
||||
|
||||
+93
-1
@@ -4276,7 +4276,7 @@ export async function saveDashboardPreferences(data: {
|
||||
// SCHEDULE EXECUTION OPERATIONS
|
||||
// =============================================================================
|
||||
|
||||
export type ScheduleType = 'container_update' | 'git_stack_sync' | 'system_cleanup' | 'env_update_check' | 'image_prune' | 'backup' | 'restore';
|
||||
export type ScheduleType = 'container_update' | 'git_stack_sync' | 'system_cleanup' | 'env_update_check' | 'image_prune' | 'backup' | 'restore' | 'stack_deploy' | 'deploy_log_reconcile';
|
||||
export type ScheduleTrigger = 'cron' | 'webhook' | 'manual' | 'startup';
|
||||
export type ScheduleStatus =
|
||||
| 'queued'
|
||||
@@ -4336,6 +4336,10 @@ export interface ScheduleExecutionFilters {
|
||||
scheduleType?: ScheduleType;
|
||||
scheduleId?: number;
|
||||
environmentId?: number | null;
|
||||
// Powers "runs for this stack" lookups (schedule_executions_entity_env_idx) --
|
||||
// paired with environmentId, an exact match on the (entity_name, environment_id)
|
||||
// index this filter was added for.
|
||||
entityName?: string;
|
||||
status?: ScheduleStatus;
|
||||
statuses?: ScheduleStatus[];
|
||||
triggeredBy?: ScheduleTrigger;
|
||||
@@ -4465,6 +4469,9 @@ export async function getScheduleExecutions(filters: ScheduleExecutionFilters =
|
||||
conditions.push(eq(scheduleExecutions.environmentId, filters.environmentId));
|
||||
}
|
||||
}
|
||||
if (filters.entityName !== undefined) {
|
||||
conditions.push(eq(scheduleExecutions.entityName, filters.entityName));
|
||||
}
|
||||
if (filters.status) {
|
||||
conditions.push(eq(scheduleExecutions.status, filters.status));
|
||||
}
|
||||
@@ -4527,6 +4534,44 @@ export async function getScheduleExecutions(filters: ScheduleExecutionFilters =
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* ALL execution ids (+ parsed details) for a given schedule type, unpaginated.
|
||||
*
|
||||
* Deliberately bypasses getScheduleExecutions()'s default 50-row page: a caller that
|
||||
* needs to reconcile every record against something else (deploy-log-reconcile.ts,
|
||||
* against files on disk) cannot afford to only see the most recent page -- an older
|
||||
* record just outside the window would look exactly like an orphan file with no
|
||||
* record, and the file behind it would be deleted even though a record exists.
|
||||
*
|
||||
* `environmentId` is included (F5 fix) so the caller can scope its reconciliation
|
||||
* PER ENVIRONMENT -- deploy-log-store.ts now keeps one log directory per environment
|
||||
* (see envDirName()), so a record's file can only ever be found in, or be deleted
|
||||
* from, ITS OWN environment's directory. Without environmentId here,
|
||||
* deploy-log-reconcile.ts would have to either reconcile everything as one flat pool
|
||||
* again (reintroducing the cross-environment mixing the size-budget fix closes) or
|
||||
* guess which environment a record belongs to.
|
||||
*/
|
||||
export async function getScheduleExecutionIdsByType(
|
||||
scheduleType: ScheduleType
|
||||
): Promise<Array<{ id: number; status: ScheduleStatus; details: any | null; environmentId: number | null }>> {
|
||||
const rows = await db
|
||||
.select({
|
||||
id: scheduleExecutions.id,
|
||||
status: scheduleExecutions.status,
|
||||
details: scheduleExecutions.details,
|
||||
environmentId: scheduleExecutions.environmentId
|
||||
})
|
||||
.from(scheduleExecutions)
|
||||
.where(eq(scheduleExecutions.scheduleType, scheduleType));
|
||||
|
||||
return rows.map((row: { id: number; status: string; details: string | null; environmentId: number | null }) => ({
|
||||
id: row.id,
|
||||
status: row.status as ScheduleStatus,
|
||||
details: row.details ? JSON.parse(row.details) : null,
|
||||
environmentId: row.environmentId
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Scheduled-task health for the metrics endpoint: execution counts by
|
||||
* (scheduleType, status), and per-type age (seconds) of the last run and last
|
||||
@@ -4699,9 +4744,12 @@ const SCHEDULE_CLEANUP_ENABLED_KEY = 'schedule_cleanup_enabled';
|
||||
const EVENT_CLEANUP_ENABLED_KEY = 'event_cleanup_enabled';
|
||||
const SCANNER_CLEANUP_CRON_KEY = 'scanner_cleanup_cron';
|
||||
const SCANNER_CLEANUP_ENABLED_KEY = 'scanner_cleanup_enabled';
|
||||
const DEPLOY_LOG_RECONCILE_CRON_KEY = 'deploy_log_reconcile_cron';
|
||||
const DEPLOY_LOG_RECONCILE_ENABLED_KEY = 'deploy_log_reconcile_enabled';
|
||||
const DEFAULT_SCHEDULE_CLEANUP_CRON = '0 3 * * *'; // Daily at 3 AM
|
||||
const DEFAULT_EVENT_CLEANUP_CRON = '30 3 * * *'; // Daily at 3:30 AM
|
||||
const DEFAULT_SCANNER_CLEANUP_CRON = '0 3 * * 0'; // Weekly Sunday at 3 AM
|
||||
const DEFAULT_DEPLOY_LOG_RECONCILE_CRON = '0 4 * * *'; // Daily at 4 AM
|
||||
|
||||
export async function getScheduleRetentionDays(): Promise<number> {
|
||||
const result = await db.select().from(settings).where(eq(settings.key, SCHEDULE_RETENTION_KEY));
|
||||
@@ -4879,6 +4927,50 @@ export async function setScannerCleanupEnabled(enabled: boolean): Promise<void>
|
||||
}
|
||||
}
|
||||
|
||||
export async function getDeployLogReconcileCron(): Promise<string> {
|
||||
const result = await db.select().from(settings).where(eq(settings.key, DEPLOY_LOG_RECONCILE_CRON_KEY));
|
||||
if (result[0]) {
|
||||
return result[0].value || DEFAULT_DEPLOY_LOG_RECONCILE_CRON;
|
||||
}
|
||||
return DEFAULT_DEPLOY_LOG_RECONCILE_CRON;
|
||||
}
|
||||
|
||||
export async function setDeployLogReconcileCron(cron: string): Promise<void> {
|
||||
const existing = await db.select().from(settings).where(eq(settings.key, DEPLOY_LOG_RECONCILE_CRON_KEY));
|
||||
if (existing.length > 0) {
|
||||
await db.update(settings)
|
||||
.set({ value: cron, updatedAt: new Date().toISOString() })
|
||||
.where(eq(settings.key, DEPLOY_LOG_RECONCILE_CRON_KEY));
|
||||
} else {
|
||||
await db.insert(settings).values({
|
||||
key: DEPLOY_LOG_RECONCILE_CRON_KEY,
|
||||
value: cron
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export async function getDeployLogReconcileEnabled(): Promise<boolean> {
|
||||
const result = await db.select().from(settings).where(eq(settings.key, DEPLOY_LOG_RECONCILE_ENABLED_KEY));
|
||||
if (result[0]) {
|
||||
return result[0].value === 'true';
|
||||
}
|
||||
return true; // Enabled by default
|
||||
}
|
||||
|
||||
export async function setDeployLogReconcileEnabled(enabled: boolean): Promise<void> {
|
||||
const existing = await db.select().from(settings).where(eq(settings.key, DEPLOY_LOG_RECONCILE_ENABLED_KEY));
|
||||
if (existing.length > 0) {
|
||||
await db.update(settings)
|
||||
.set({ value: enabled ? 'true' : 'false', updatedAt: new Date().toISOString() })
|
||||
.where(eq(settings.key, DEPLOY_LOG_RECONCILE_ENABLED_KEY));
|
||||
} else {
|
||||
await db.insert(settings).values({
|
||||
key: DEPLOY_LOG_RECONCILE_ENABLED_KEY,
|
||||
value: enabled ? 'true' : 'false'
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// EXTERNAL STACK PATHS
|
||||
// =============================================================================
|
||||
|
||||
@@ -488,7 +488,9 @@ export const scheduleExecutions = sqliteTable('schedule_executions', {
|
||||
logs: text('logs'), // Execution logs/output
|
||||
createdAt: text('created_at').default(sql`CURRENT_TIMESTAMP`)
|
||||
}, (table) => ({
|
||||
typeIdIdx: index('schedule_executions_type_id_idx').on(table.scheduleType, table.scheduleId)
|
||||
typeIdIdx: index('schedule_executions_type_id_idx').on(table.scheduleType, table.scheduleId),
|
||||
// Powers "runs for this stack/container" lookups without a full table scan.
|
||||
entityEnvIdx: index('schedule_executions_entity_env_idx').on(table.entityName, table.environmentId)
|
||||
}));
|
||||
|
||||
// =============================================================================
|
||||
|
||||
@@ -490,7 +490,9 @@ export const scheduleExecutions = pgTable('schedule_executions', {
|
||||
logs: text('logs'), // Execution logs/output
|
||||
createdAt: timestamp('created_at', { mode: 'string' }).defaultNow()
|
||||
}, (table) => ({
|
||||
typeIdIdx: index('schedule_executions_type_id_idx').on(table.scheduleType, table.scheduleId)
|
||||
typeIdIdx: index('schedule_executions_type_id_idx').on(table.scheduleType, table.scheduleId),
|
||||
// Powers "runs for this stack/container" lookups without a full table scan.
|
||||
entityEnvIdx: index('schedule_executions_entity_env_idx').on(table.entityName, table.environmentId)
|
||||
}));
|
||||
|
||||
// =============================================================================
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* Pure decision core for the deploy-log reconcile job.
|
||||
*
|
||||
* Compares the set of deploy-log files on disk against the set of run records in the
|
||||
* database and decides what to do about each side of the mismatch -- nothing else. It
|
||||
* touches neither the filesystem nor the database; the caller (deploy-log-reconcile.ts)
|
||||
* is responsible for gathering the two id sets and for carrying out the plan.
|
||||
*
|
||||
* The two failure modes are NOT symmetric, and that asymmetry is the whole point of
|
||||
* this module:
|
||||
* - A file without a record is deleted -- it is an orphan with nothing left to explain it.
|
||||
* - A record without a file is only marked, never deleted -- the metadata (when, who,
|
||||
* what was built, which digest) stays valuable even once the log text itself is gone.
|
||||
* Deleting the record here would be the "too eager" failure this job exists to avoid.
|
||||
*/
|
||||
|
||||
export interface ReconcileInput {
|
||||
/** Run ids that have a deploy-log file on disk. */
|
||||
fileIds: string[];
|
||||
/** Run ids that have a run record in the database. */
|
||||
recordIds: string[];
|
||||
}
|
||||
|
||||
export interface ReconcilePlan {
|
||||
/** Files to delete: present on disk, no matching record. */
|
||||
deleteFiles: string[];
|
||||
/** Records to mark (details.logMissing = true): present in the database, no matching file. */
|
||||
markMissing: string[];
|
||||
}
|
||||
|
||||
export function planReconcile(input: ReconcileInput): ReconcilePlan {
|
||||
const recordIdSet = new Set(input.recordIds);
|
||||
const fileIdSet = new Set(input.fileIds);
|
||||
|
||||
return {
|
||||
deleteFiles: input.fileIds.filter((id) => !recordIdSet.has(id)),
|
||||
markMissing: input.recordIds.filter((id) => !fileIdSet.has(id))
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a record in this status may be judged "missing its log file".
|
||||
*
|
||||
* A still in-progress run ('queued'/'running') may simply not have written its first
|
||||
* log line yet -- the schedule_executions row for a stack_deploy run is created before
|
||||
* the deploy has produced any output, and the log file itself only comes into existence
|
||||
* on the first line written (see deploy-log-reconcile.ts's module doc comment for the
|
||||
* verified timing this guards against). Marking it here would be wrong, and nothing
|
||||
* downstream ever re-checks or clears the flag once the file does appear -- so the
|
||||
* record would carry a false "log missing" permanently.
|
||||
*/
|
||||
export function isEligibleForMissingMark(status: string): boolean {
|
||||
return status !== 'queued' && status !== 'running';
|
||||
}
|
||||
@@ -0,0 +1,342 @@
|
||||
import { appendFile, mkdir, readFile, rm, readdir, stat } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
|
||||
const ID = /^[A-Za-z0-9_-]+$/;
|
||||
|
||||
/**
|
||||
* A run id comes from a route parameter, so it is validated rather than sanitised:
|
||||
* a value that is not a plain id is a bug or an attack, and neither should quietly
|
||||
* resolve to some other file.
|
||||
*/
|
||||
export function runLogFileName(runId: string): string {
|
||||
if (!ID.test(runId)) throw new Error(`invalid run id: ${JSON.stringify(runId)}`);
|
||||
return `${runId}.log`;
|
||||
}
|
||||
|
||||
/**
|
||||
* F5 fix: the directory name (under deploy-logs/) that holds ONE environment's OWN
|
||||
* run log files and counts against ONE environment's OWN size budget.
|
||||
*
|
||||
* Before this existed, every environment shared a SINGLE deploy-logs/ directory and a
|
||||
* single BUDGET_BYTES total (see SizeBudgetTracker below). A user with deploy rights
|
||||
* in ONE environment could fill that shared budget with a noisy deploy, which then
|
||||
* silently truncated the deploy logs and summaries of every OTHER environment -- a
|
||||
* multi-tenant audit-trail integrity violation, since a deploy log is exactly the
|
||||
* evidence an operator reaches for after something goes wrong. Scoping the directory
|
||||
* (and therefore the budget, since usedBytes()/usedBytesExcluding() only ever look
|
||||
* inside ONE environment's directory) per environment is the entire fix.
|
||||
*
|
||||
* `environmentId` always comes from a database column (`number | null`, see
|
||||
* ScheduleExecutionData in db.ts) -- never attacker-controlled free text reaching
|
||||
* this function directly. This still asserts the shape rather than trusting it
|
||||
* blindly, the same posture runLogFileName's ID regex takes for runId: a value that
|
||||
* is not a plain integer or null is a bug, and should throw rather than quietly
|
||||
* resolve to some unexpected directory name.
|
||||
*
|
||||
* The `null` case (an install with no environments / a run not attributed to one --
|
||||
* see createRunRecorder's `envId: number | null` input) gets the literal directory
|
||||
* name "null", not an omitted segment. `String(null) === 'null'` in JS already reads
|
||||
* exactly this way, so this is not inventing a new convention -- it is naming the one
|
||||
* `String(envId)` would already produce, so the special case is documented rather
|
||||
* than accidental.
|
||||
*/
|
||||
export function envDirName(envId: number | null): string {
|
||||
if (envId === null) return 'null';
|
||||
if (!Number.isInteger(envId)) {
|
||||
throw new Error(`invalid environment id: ${JSON.stringify(envId)}`);
|
||||
}
|
||||
return String(envId);
|
||||
}
|
||||
|
||||
/**
|
||||
* The inverse of envDirName() -- turns a directory name discovered on disk (via
|
||||
* listEnvDirNames()) back into the environmentId it represents. Used by the
|
||||
* reconcile job, which learns which environments have files by LISTING
|
||||
* deploy-logs/, rather than being handed a set of ids up front.
|
||||
*
|
||||
* Throws on anything that is not one of envDirName()'s own outputs. listEnvDirNames()
|
||||
* already filters to directories only (excluding the pre-F5 flat-layout .log files --
|
||||
* see legacyFlatRunLogPath), so in practice this only ever sees "null" or a
|
||||
* non-negative integer string; the throw is a defensive backstop against a
|
||||
* hand-created or otherwise unexpected directory being silently misread as some
|
||||
* arbitrary environment id.
|
||||
*/
|
||||
export function parseEnvDirName(name: string): number | null {
|
||||
if (name === 'null') return null;
|
||||
if (!/^\d+$/.test(name)) {
|
||||
throw new Error(`not an environment directory name: ${JSON.stringify(name)}`);
|
||||
}
|
||||
return Number(name);
|
||||
}
|
||||
|
||||
function deployLogsRoot(): string {
|
||||
return join(process.env.DATA_DIR || './data', 'deploy-logs');
|
||||
}
|
||||
|
||||
/** One environment's OWN log directory: deploy-logs/<envDirName(envId)>/. */
|
||||
function logDir(envId: number | null): string {
|
||||
return join(deployLogsRoot(), envDirName(envId));
|
||||
}
|
||||
|
||||
export function runLogPath(envId: number | null, runId: string): string {
|
||||
return join(logDir(envId), runLogFileName(runId));
|
||||
}
|
||||
|
||||
/**
|
||||
* The path a run's log file would have lived at BEFORE the F5 per-environment
|
||||
* scoping existed: deploy-logs/<runId>.log, no environment subdirectory. This branch
|
||||
* introduced deploy-log-store.ts in the first place -- there is no released version
|
||||
* of Dockhand that ever wrote logs in this flat layout, so there is no production
|
||||
* data to migrate, and appendRunLog()/deleteRunLog() never write here.
|
||||
*
|
||||
* readRunLog() still falls back to this path (see its doc comment) purely as a
|
||||
* robustness measure: a file placed here by hand -- a manual copy, a fixture, a
|
||||
* future migration script -- is still found instead of silently 404ing, rather than
|
||||
* this fix assuming its own directory layout is the only one that will ever exist on
|
||||
* disk. deleteRunLog() also clears it, so such a file cannot outlive the record that
|
||||
* pointed at it.
|
||||
*/
|
||||
function legacyFlatRunLogPath(runId: string): string {
|
||||
return join(deployLogsRoot(), runLogFileName(runId));
|
||||
}
|
||||
|
||||
/** Total bytes ONE environment's log directory may occupy before new writes are refused. */
|
||||
const BUDGET_BYTES = 2 * 1024 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* How many bytes of a run's OWN output SizeBudgetTracker lets accumulate against its
|
||||
* cached view of the rest of the directory before re-scanning that view. Bounds how
|
||||
* stale the cache can get for a single long-running deploy -- see SizeBudgetTracker's
|
||||
* doc comment for the concurrency trade-off this makes.
|
||||
*/
|
||||
const REFRESH_AFTER_OWN_BYTES = 1 * 1024 * 1024; // 1 MiB
|
||||
|
||||
export async function appendRunLog(envId: number | null, runId: string, chunk: string): Promise<void> {
|
||||
await mkdir(logDir(envId), { recursive: true });
|
||||
await appendFile(runLogPath(envId, runId), chunk);
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a run's log text. Tries the run's OWN environment directory first -- the
|
||||
* only place appendRunLog() ever writes -- and falls back to the pre-F5 flat path
|
||||
* (legacyFlatRunLogPath) if no file exists there. See that function's doc comment
|
||||
* for why the fallback exists despite there being no production data in that layout.
|
||||
* A file that exists at NEITHER path returns null, exactly as before F5.
|
||||
*/
|
||||
export async function readRunLog(envId: number | null, runId: string): Promise<string | null> {
|
||||
try {
|
||||
return await readFile(runLogPath(envId, runId), 'utf8');
|
||||
} catch {
|
||||
try {
|
||||
return await readFile(legacyFlatRunLogPath(runId), 'utf8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Removes a run's log file from its own environment directory AND from the pre-F5
|
||||
* flat legacy path (force: true -- a path that was never there is not an error), so
|
||||
* a run whose file happens to have been found via readRunLog()'s legacy fallback
|
||||
* does not survive its own deletion.
|
||||
*/
|
||||
export async function deleteRunLog(envId: number | null, runId: string): Promise<void> {
|
||||
await rm(runLogPath(envId, runId), { force: true });
|
||||
await rm(legacyFlatRunLogPath(runId), { force: true });
|
||||
}
|
||||
|
||||
export async function listRunLogIds(envId: number | null): Promise<string[]> {
|
||||
try {
|
||||
const names = await readdir(logDir(envId));
|
||||
return names.filter((n) => n.endsWith('.log')).map((n) => n.slice(0, -4));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Last-modified time (ms since epoch) of a run's log file, or null if it does not
|
||||
* exist. Used by the reconcile job (deploy-log-reconcile.ts) as a TOCTOU guard: the
|
||||
* set of stack_deploy records it compares files against is a single point-in-time
|
||||
* snapshot taken at the start of the job, while appendRunLog() keeps writing to (and
|
||||
* therefore keeps bumping the mtime of) a live deploy's file for as long as it runs.
|
||||
* A deploy that starts AFTER that snapshot has no record in it yet, but its file can
|
||||
* already exist by the time this environment's files are listed -- a file this fresh
|
||||
* is exactly what that race looks like, not an abandoned orphan.
|
||||
*/
|
||||
export async function getRunLogMtimeMs(envId: number | null, runId: string): Promise<number | null> {
|
||||
try {
|
||||
return (await stat(runLogPath(envId, runId))).mtimeMs;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Every environment directory name currently present under deploy-logs/. Used by the
|
||||
* reconcile job (deploy-log-reconcile.ts) to discover which environments have ANY
|
||||
* files on disk, without first needing to already know the full set of environment
|
||||
* ids from the database (an environment that was later deleted, but still has
|
||||
* orphaned log files, would otherwise never be reconciled).
|
||||
*
|
||||
* Filters to directories only via readdir's withFileTypes -- a stray FILE directly
|
||||
* under deploy-logs/ is the pre-F5 flat legacy layout (legacyFlatRunLogPath), not an
|
||||
* environment directory, and is intentionally excluded: F5 is a fresh, unreleased
|
||||
* feature with no production data in that layout to reconcile (see that function's
|
||||
* doc comment).
|
||||
*/
|
||||
export async function listEnvDirNames(): Promise<string[]> {
|
||||
try {
|
||||
const entries = await readdir(deployLogsRoot(), { withFileTypes: true });
|
||||
return entries.filter((e) => e.isDirectory()).map((e) => e.name);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
export async function usedBytes(envId: number | null): Promise<number> {
|
||||
let total = 0;
|
||||
for (const id of await listRunLogIds(envId)) {
|
||||
try {
|
||||
total += (await stat(runLogPath(envId, id))).size;
|
||||
} catch {
|
||||
/* raced with cleanup */
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Same as usedBytes(), but leaves excludeRunId's own file out of the sum. Used by
|
||||
* SizeBudgetTracker to get the size of "everything ELSE IN THIS ENVIRONMENT", so it
|
||||
* can add this run's own bytes on top from an in-memory counter instead of
|
||||
* re-stat'ing its own file.
|
||||
*/
|
||||
export async function usedBytesExcluding(envId: number | null, excludeRunId: string): Promise<number> {
|
||||
let total = 0;
|
||||
for (const id of await listRunLogIds(envId)) {
|
||||
if (id === excludeRunId) continue;
|
||||
try {
|
||||
total += (await stat(runLogPath(envId, id))).size;
|
||||
} catch {
|
||||
/* raced with cleanup */
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
export async function budgetExceeded(envId: number | null): Promise<boolean> {
|
||||
return (await usedBytes(envId)) > BUDGET_BYTES;
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks whether ONE run's output has pushed ITS OWN ENVIRONMENT's deploy-logs/<env>/
|
||||
* over BUDGET_BYTES, without doing a full directory scan (readdir + stat every file)
|
||||
* on every appended line.
|
||||
*
|
||||
* Before the scan-avoidance existed, deploy-run-record.ts's line() called
|
||||
* budgetExceeded() -- a full usedBytes() scan -- after EVERY line, so a deploy with N
|
||||
* output lines against a log directory holding M past runs' files cost O(N*M) stat
|
||||
* calls. Measured on this machine: 200 lines against 500 existing log files took
|
||||
* ~16.2s (~81ms/line); the same 200 lines with the directory scan removed entirely
|
||||
* took ~49ms total. That scan is needed to stay honest about the directory's actual
|
||||
* size, so it can't just be dropped -- it can only be done less often.
|
||||
*
|
||||
* How: the directory total EXCLUDING this run's own file (usedBytesExcluding) is
|
||||
* measured once, lazily, on the first recordAppend() call, and cached. Each
|
||||
* recordAppend() after that just adds the byte length the caller reports (bytes it
|
||||
* just wrote to ITS OWN file -- no stat() involved) to that cached total in memory.
|
||||
* The cache is re-measured every REFRESH_AFTER_OWN_BYTES bytes of this run's own
|
||||
* output, so a long deploy's view of "everything else" can't drift arbitrarily far
|
||||
* from reality -- at most REFRESH_AFTER_OWN_BYTES stale before the next refresh
|
||||
* catches up.
|
||||
*
|
||||
* F5 fix, ENVIRONMENT scoping: `envId` is threaded into the constructor and passed to
|
||||
* every measureOthers() call, so "everything else" means everything else IN THIS
|
||||
* RUN'S OWN ENVIRONMENT's directory -- never another environment's files. Before this,
|
||||
* `envId` did not exist here at all: every tracker (regardless of which environment
|
||||
* its run belonged to) measured the SAME shared deploy-logs/ directory against the
|
||||
* SAME BUDGET_BYTES total, so a noisy deploy in one environment could push every other
|
||||
* environment's tracker over budget too, truncating runs that never wrote anywhere
|
||||
* near BUDGET_BYTES themselves. See deploy-run-record.ts's module doc comment (F5
|
||||
* fix) and DeployRunRecorder's envId field for how this is wired end to end.
|
||||
*
|
||||
* Concurrency: OTHER runs in the SAME environment writing at the same time are
|
||||
* invisible between this tracker's refreshes. A second run in the same environment
|
||||
* that starts writing right after this one's baseline was measured can add up to its
|
||||
* own full output before the next refresh reflects it, so the total this tracker
|
||||
* checks against can undercount the real directory size by that much in the
|
||||
* meantime. This is an intentional trade-off: BUDGET_BYTES (2 GiB, deliberately
|
||||
* generous) exists to catch a filled disk, not to enforce an exact quota -- the
|
||||
* design doc calls for aborting "at the end of writing, not truncating mid-line",
|
||||
* never for a precise cutoff. The OLD per-line full rescan did not close this gap
|
||||
* either: budgetExceeded() reads the directory at the moment it's called, with no
|
||||
* locking against another run's concurrent appendRunLog() -- the old code just paid
|
||||
* for that same imprecision on every single line instead of once per MiB. This
|
||||
* concurrency trade-off is entirely orthogonal to the F5 environment-scoping fix
|
||||
* above: it was true before F5 (within the single shared directory) and remains true
|
||||
* after F5 (within one environment's own directory) -- F5 changes WHICH directory is
|
||||
* measured, not how fresh that measurement is.
|
||||
*/
|
||||
export class SizeBudgetTracker {
|
||||
/** "Everything else" (every OTHER run's file IN THIS RUN'S OWN ENVIRONMENT), last
|
||||
* measured by refresh(). NEVER includes this run's own file -- see
|
||||
* usedBytesExcluding() -- and NEVER includes another environment's files, since
|
||||
* usedBytesExcluding() only ever reads inside logDir(this.envId). */
|
||||
private cachedOtherBytes = 0;
|
||||
/** Every byte THIS run has appended so far, summed in memory as recordAppend()
|
||||
* is called. This is the run's true own total and is never reset -- it does not
|
||||
* need a re-scan to stay accurate, because the caller already knows exactly what
|
||||
* it just wrote. */
|
||||
private ownBytesTotal = 0;
|
||||
/** Own bytes appended since the LAST refresh() -- reset to 0 on every refresh.
|
||||
* Only exists to decide WHEN to refresh; the budget check itself always uses
|
||||
* ownBytesTotal, not this. */
|
||||
private ownBytesSinceLastScan = 0;
|
||||
private loaded = false;
|
||||
|
||||
constructor(
|
||||
/** F5 fix: which environment's directory/budget this tracker measures against.
|
||||
* Passed straight through to measureOthers() on every refresh() -- see the
|
||||
* class doc comment. */
|
||||
private readonly envId: number | null,
|
||||
private readonly runId: string,
|
||||
private readonly budgetBytes: number = BUDGET_BYTES,
|
||||
/** Test-only seam: lets tests count/replace the directory scan without
|
||||
* writing real files. Production code never passes this -- it always gets
|
||||
* the real usedBytesExcluding. */
|
||||
private readonly measureOthers: (
|
||||
envId: number | null,
|
||||
excludeRunId: string
|
||||
) => Promise<number> = usedBytesExcluding
|
||||
) {}
|
||||
|
||||
private async refresh(): Promise<void> {
|
||||
this.cachedOtherBytes = await this.measureOthers(this.envId, this.runId);
|
||||
this.ownBytesSinceLastScan = 0;
|
||||
this.loaded = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Call once per chunk appended to this run's own log file, with the number of
|
||||
* UTF-8 bytes just written (Buffer.byteLength(chunk, 'utf8') -- NOT chunk.length,
|
||||
* which counts UTF-16 code units and undercounts multi-byte characters). Returns
|
||||
* whether the directory total, including this run's own bytes so far, currently
|
||||
* exceeds the budget.
|
||||
*
|
||||
* ownBytesTotal is updated BEFORE the refresh decision, on purpose: a single
|
||||
* chunk large enough to cross REFRESH_AFTER_OWN_BYTES on its own must trigger its
|
||||
* refresh in this same call, not the next one -- otherwise a caller that stops
|
||||
* appending right after such a chunk would leave the tracker's view of
|
||||
* "everything else" stale for the rest of the run.
|
||||
*/
|
||||
async recordAppend(byteLength: number): Promise<boolean> {
|
||||
this.ownBytesTotal += byteLength;
|
||||
this.ownBytesSinceLastScan += byteLength;
|
||||
if (!this.loaded || this.ownBytesSinceLastScan >= REFRESH_AFTER_OWN_BYTES) {
|
||||
await this.refresh();
|
||||
}
|
||||
return this.cachedOtherBytes + this.ownBytesTotal > this.budgetBytes;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
// src/lib/server/deploy-recorder.ts
|
||||
/**
|
||||
* Decides what may be written to a persisted deploy log.
|
||||
*
|
||||
* Only redacted output lines qualify. The 'result' event is excluded on purpose: it
|
||||
* carries result.output (and, on the compose endpoint, result.error) -- compose's raw
|
||||
* stdout/stderr, which never passes through redactLine. Persisting it would turn a
|
||||
* ten-minute in-memory buffer into a permanent plaintext archive on disk.
|
||||
*
|
||||
* BOTH conditions below are load-bearing and are covered by separate tests. Do not
|
||||
* collapse them into the type check alone: that would admit any future event carrying
|
||||
* type: 'line', and no test would notice.
|
||||
*/
|
||||
export function shouldRecord(event: string | undefined, data: unknown): boolean {
|
||||
if (event !== 'progress') return false;
|
||||
if (typeof data !== 'object' || data === null) return false;
|
||||
return (data as { type?: string }).type === 'line';
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { getScheduleExecution, type ScheduleExecutionData } from './db';
|
||||
import type { AuthorizationContext } from './authorize';
|
||||
|
||||
/**
|
||||
* Loads a schedule_executions row and confirms it actually belongs to the named
|
||||
* stack before deciding whether the caller may reach it.
|
||||
*
|
||||
* Two checks, deliberately in this order:
|
||||
*
|
||||
* 1. Ownership -- the row must be a 'stack_deploy' execution whose entityName
|
||||
* matches the stack in the URL. A run for a DIFFERENT stack (or a
|
||||
* non-deploy schedule execution that happens to reuse the same numeric id
|
||||
* space) comes back as the exact same 404 as a run that doesn't exist at
|
||||
* all. A distinct response for "wrong stack" vs "no such run" would leak
|
||||
* that the id is valid for something else.
|
||||
*
|
||||
* 2. Environment access -- checked against the RUN's OWN environmentId, never
|
||||
* against a caller-supplied `env` query parameter. A caller could otherwise
|
||||
* pass an env they legitimately have access to while pointing runId at a
|
||||
* run that actually belongs to a DIFFERENT environment they have no access
|
||||
* to at all -- the same cross-env bypass class the backup routes' own
|
||||
* route-guards.ts closes for snapshot/config lookups (resolveSnapshotEnvId
|
||||
* + guardSnapshotEnvAccess). Fails closed: enterprise + inaccessible env
|
||||
* denies, even if that hides an otherwise-existing run.
|
||||
*
|
||||
* Shared by GET and DELETE .../deploys/{runId} AND GET .../deploys/{runId}/log
|
||||
* so the three routes can never drift into checking different things for the
|
||||
* same row. The stacks:view/stacks:edit permission gate itself is NOT part of
|
||||
* this helper -- each route checks that on its own, inline, before calling
|
||||
* this (see the three +server.ts files).
|
||||
*/
|
||||
/**
|
||||
* Statuses a stack_deploy run can be permanently closed under -- i.e. the deploy has
|
||||
* actually stopped running and nothing will update this row again on its own. Kept as
|
||||
* an explicit allow-list (not "everything except queued/running") on purpose: the
|
||||
* status column is free text (db.ts's ScheduleStatus comment), and a status this code
|
||||
* doesn't recognize must NOT be assumed finished just because it isn't literally
|
||||
* 'queued' or 'running' -- see isTerminalRunStatus()'s doc comment for what that
|
||||
* assumption would cost.
|
||||
*
|
||||
* This is deliberately its OWN small set, not deploy-log-reconcile-core.ts's
|
||||
* isEligibleForMissingMark() (which answers "queued"/"running" vs. everything else,
|
||||
* fail-open for an unrecognized status because marking logMissing is reversible and
|
||||
* merely metadata). Deleting a run is NOT reversible, so unlike that check, an
|
||||
* unrecognized status here must fail CLOSED (treated as still in progress), not open.
|
||||
*/
|
||||
const TERMINAL_RUN_STATUSES = new Set(['success', 'failed', 'skipped', 'warning', 'error', 'cancelled', 'stale']);
|
||||
|
||||
/**
|
||||
* Whether a stack_deploy run's status means the deploy is over and its record/log file
|
||||
* may safely be deleted.
|
||||
*
|
||||
* Fails CLOSED for anything not explicitly listed in TERMINAL_RUN_STATUSES -- including
|
||||
* 'queued', 'running', and any status this code has never been taught about (the status
|
||||
* column is free text). The alternative (terminal = "not queued/running") would let a
|
||||
* status this code doesn't recognize be deleted on the assumption that it must be
|
||||
* finished; there is no way to confirm that, and a wrong guess here is irreversible (see
|
||||
* loadOwnedDeployRun's DELETE caller in .../deploys/[runId]/+server.ts).
|
||||
*/
|
||||
export function isTerminalRunStatus(status: string): boolean {
|
||||
return TERMINAL_RUN_STATUSES.has(status);
|
||||
}
|
||||
|
||||
export async function loadOwnedDeployRun(
|
||||
stackName: string,
|
||||
runIdParam: string,
|
||||
auth: AuthorizationContext
|
||||
): Promise<{ run: ScheduleExecutionData } | { response: Response }> {
|
||||
const runId = parseInt(runIdParam, 10);
|
||||
if (isNaN(runId)) {
|
||||
return { response: json({ error: 'Invalid run id' }, { status: 400 }) };
|
||||
}
|
||||
|
||||
const run = await getScheduleExecution(runId);
|
||||
if (!run || run.scheduleType !== 'stack_deploy' || run.entityName !== stackName) {
|
||||
return { response: json({ error: 'Deploy run not found' }, { status: 404 }) };
|
||||
}
|
||||
|
||||
if (run.environmentId != null && auth.isEnterprise && !(await auth.canAccessEnvironment(run.environmentId))) {
|
||||
return { response: json({ error: 'Access denied to this environment' }, { status: 403 }) };
|
||||
}
|
||||
|
||||
return { run };
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import type { DeploySummary } from './deploy-summary-core';
|
||||
|
||||
/**
|
||||
* Assembles the `details` JSON stored on a stack_deploy schedule_executions row.
|
||||
*
|
||||
* Deliberately dependency-free (no db.ts, no filesystem) so it can be unit tested
|
||||
* under Bun -- see deploy-run-record.ts for the database-touching half that calls
|
||||
* this. Importing db.ts here would make this module unimportable under Bun
|
||||
* (better-sqlite3, ERR_DLOPEN_FAILED), breaking this file's own test and dragging
|
||||
* down every other test file that happens to run in the same process.
|
||||
*/
|
||||
|
||||
export interface DeployRunOptions {
|
||||
pull: boolean;
|
||||
build: boolean;
|
||||
forceRecreate: boolean;
|
||||
}
|
||||
|
||||
export interface BuildRunDetailsInput {
|
||||
options: DeployRunOptions;
|
||||
summary?: DeploySummary;
|
||||
exitCode: number;
|
||||
composeHash: string;
|
||||
envHash: string;
|
||||
logFile: string;
|
||||
/** Set when the run's output hit deploy-log-store's total-size budget and was cut off. */
|
||||
truncated?: boolean;
|
||||
}
|
||||
|
||||
export interface DeployRunDetails {
|
||||
options: DeployRunOptions;
|
||||
summary?: DeploySummary;
|
||||
exitCode: number;
|
||||
composeHash: string;
|
||||
envHash: string;
|
||||
logFile: string;
|
||||
truncated?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* The log file name is what ties this schedule_executions row to its log file on
|
||||
* disk (deploy-logs/<id>.log). Task 13's reconciler deletes any file whose id has
|
||||
* no matching record -- if the name were missing here, a live run's own file would
|
||||
* look orphaned and get eaten out from under it.
|
||||
*/
|
||||
export function buildRunDetails(input: BuildRunDetailsInput): DeployRunDetails {
|
||||
if (!input.logFile) {
|
||||
throw new Error('buildRunDetails: logFile is required to tie the record to its log file');
|
||||
}
|
||||
return {
|
||||
options: input.options,
|
||||
summary: input.summary,
|
||||
exitCode: input.exitCode,
|
||||
composeHash: input.composeHash,
|
||||
envHash: input.envHash,
|
||||
logFile: input.logFile,
|
||||
...(input.truncated ? { truncated: true } : {})
|
||||
};
|
||||
}
|
||||
|
||||
/** SHA-256 of the compose file content used for this deploy, hex-encoded. */
|
||||
export function hashComposeContent(content: string): string {
|
||||
return createHash('sha256').update(content).digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* SHA-256 fingerprint of the effective env for this deploy (every var -- secret and
|
||||
* non-secret alike -- that will actually reach the container), hex-encoded. Sorted by
|
||||
* key so the same env always hashes to the same value regardless of object key order.
|
||||
* This is a change-fingerprint, not a secret store -- the hash alone can't be turned
|
||||
* back into the values it was built from, so hashing secret values in is safe even
|
||||
* though logging them would not be. Callers merge whatever sources make up their
|
||||
* effective env (DB non-secret + secret vars, or a request body's raw/array vars)
|
||||
* into one Record before calling this.
|
||||
*/
|
||||
export function hashEnvFingerprint(vars: Record<string, string>): string {
|
||||
const serialized = Object.keys(vars).sort().map((k) => `${k}=${vars[k]}`).join('\n');
|
||||
return createHash('sha256').update(serialized).digest('hex');
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
import {
|
||||
createScheduleExecution,
|
||||
updateScheduleExecution,
|
||||
type ScheduleTrigger
|
||||
} from './db';
|
||||
import { appendRunLog, runLogFileName, SizeBudgetTracker } from './deploy-log-store';
|
||||
import { summarize } from './deploy-summary-core';
|
||||
import { buildRunDetails, type DeployRunOptions } from './deploy-run-record-core';
|
||||
import { redactLine } from './secret-redaction';
|
||||
import type { RunRecorder } from './sse';
|
||||
|
||||
/**
|
||||
* Stored instead of the raw error text when redactLine() had to withhold the whole
|
||||
* line (a too-short secret value matched, and redacting only that substring would
|
||||
* risk hitting an unrelated coincidence -- see secret-redaction.ts). A run without an
|
||||
* error message is unfortunate; a run with a leaked secret in a widely-readable field
|
||||
* is a security incident, so the row still closes "failed", just without the text.
|
||||
*/
|
||||
const ERROR_WITHHELD_MESSAGE = 'Deploy failed (error message withheld: contained a secret value)';
|
||||
|
||||
/**
|
||||
* The database-touching half of the stack_deploy run record. Split from
|
||||
* deploy-run-record-core.ts on purpose: this file imports db.ts (createScheduleExecution
|
||||
* pulls in better-sqlite3), which is not importable under Bun (ERR_DLOPEN_FAILED). Keeping
|
||||
* that import out of the pure -core module is what lets deploy-run-record-core.test.ts run
|
||||
* at all, and keeps it from dragging down unrelated test files sharing the same process.
|
||||
*
|
||||
* F5 fix: the caller-supplied `envId` (the environment the run belongs to, or null for a
|
||||
* run not attributed to one) is threaded through to appendRunLog()/SizeBudgetTracker
|
||||
* (deploy-log-store.ts) for every filesystem operation this class performs. Before this,
|
||||
* `envId` was accepted by createRunRecorder()'s input and stored on the schedule_executions
|
||||
* row (environmentId), but never reached the log file's PATH or its size budget -- every
|
||||
* environment's deploy output landed in the SAME deploy-logs/ directory and counted against
|
||||
* the SAME 2 GiB total (deploy-log-store.ts's BUDGET_BYTES), so a noisy deploy in one
|
||||
* environment could silently truncate another environment's deploy logs. See
|
||||
* deploy-log-store.ts's SizeBudgetTracker doc comment for the full explanation.
|
||||
*/
|
||||
|
||||
const TRUNCATION_NOTICE = '\n[deploy log truncated: size budget exceeded]\n';
|
||||
|
||||
export class DeployRunRecorder implements RunRecorder {
|
||||
private readonly executionId: number;
|
||||
private readonly runId: string;
|
||||
private readonly startedAtMs: number;
|
||||
private readonly options: DeployRunOptions;
|
||||
private readonly composeHash: string;
|
||||
private readonly envHash: string;
|
||||
private readonly userId?: number;
|
||||
/** F5 fix: which environment this run belongs to (or null) -- determines the ON-DISK
|
||||
* directory its log file lives in and which environment's size budget it counts
|
||||
* against (deploy-log-store.ts's logDir()/SizeBudgetTracker). Never used for anything
|
||||
* else here -- environmentId on the schedule_executions row itself is set separately,
|
||||
* by createRunRecorder()'s createScheduleExecution() call. */
|
||||
private readonly envId: number | null;
|
||||
/** Not readonly: addSecrets() extends this after construction -- see its doc comment. */
|
||||
private secrets: string[];
|
||||
private readonly lines: string[] = [];
|
||||
/** Chain of queued appends -- see line()'s doc comment for why this exists. */
|
||||
private tail: Promise<void> = Promise.resolve();
|
||||
private truncated = false;
|
||||
/** Tracks the deploy-logs/ size budget without a full directory re-scan on every
|
||||
* appended line -- see SizeBudgetTracker's doc comment (deploy-log-store.ts). */
|
||||
private readonly sizeBudget: SizeBudgetTracker;
|
||||
|
||||
constructor(
|
||||
executionId: number,
|
||||
startedAtMs: number,
|
||||
input: {
|
||||
options: DeployRunOptions;
|
||||
composeHash: string;
|
||||
envHash: string;
|
||||
userId?: number;
|
||||
secrets: string[];
|
||||
/** F5 fix: see the envId field's doc comment above. */
|
||||
envId: number | null;
|
||||
},
|
||||
/** Test-only seam: lets tests exercise the truncation path with a tiny budget
|
||||
* instead of writing gigabytes of real log data. createRunRecorder() never
|
||||
* passes this, so production always gets SizeBudgetTracker's real default
|
||||
* (BUDGET_BYTES, deploy-log-store.ts). */
|
||||
sizeBudget?: SizeBudgetTracker
|
||||
) {
|
||||
this.executionId = executionId;
|
||||
this.runId = String(executionId);
|
||||
this.startedAtMs = startedAtMs;
|
||||
this.options = input.options;
|
||||
this.composeHash = input.composeHash;
|
||||
this.envHash = input.envHash;
|
||||
this.userId = input.userId;
|
||||
this.secrets = input.secrets;
|
||||
this.envId = input.envId;
|
||||
this.sizeBudget = sizeBudget ?? new SizeBudgetTracker(this.envId, this.runId);
|
||||
}
|
||||
|
||||
/**
|
||||
* appendRunLog (deploy-log-store.ts) is async; this method is not. Without a
|
||||
* chain, two lines arriving close together could race and land in the file out
|
||||
* of order. Each call hangs its append off the tail of the previous one, so
|
||||
* writes always happen in the order line() was called -- and end() can await
|
||||
* `tail` to know every queued write has actually landed before it closes the row.
|
||||
*/
|
||||
line(line: string): void {
|
||||
if (this.truncated) return;
|
||||
this.tail = this.tail.then(async () => {
|
||||
if (this.truncated) return;
|
||||
// Pushed for the summary in the SAME guarded step as the file write, not
|
||||
// synchronously up in line() -- line() can be called many times before
|
||||
// the chain gets to run any of them, and truncation only actually takes
|
||||
// effect partway through that queue. Pushing here keeps `this.lines`
|
||||
// (what summarize() sees) exactly in sync with what really landed in the
|
||||
// file: a line that gets skipped because truncation already kicked in
|
||||
// isn't counted as if it had been written.
|
||||
this.lines.push(line);
|
||||
const chunk = line + '\n';
|
||||
await appendRunLog(this.envId, this.runId, chunk);
|
||||
// Checked AFTER writing, not before: a budget check ahead of the write
|
||||
// would either reject a legitimate line on stale size info, or need a
|
||||
// second read straight after anyway. Checking once, right after the
|
||||
// write that just happened, is both simpler and exactly what "abort at
|
||||
// the end of writing, not truncate mid-line" (design doc §5) asks for.
|
||||
// recordAppend() itself only re-scans the directory occasionally (see
|
||||
// SizeBudgetTracker) -- it does NOT re-stat the whole deploy-logs/
|
||||
// directory on every call, unlike the budgetExceeded() this replaced.
|
||||
if (await this.sizeBudget.recordAppend(Buffer.byteLength(chunk, 'utf8'))) {
|
||||
this.truncated = true;
|
||||
await appendRunLog(this.envId, this.runId, TRUNCATION_NOTICE);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* F4 fix: the caller (a route or deployGitStack) builds this recorder from the
|
||||
* DB-only vars it has BEFORE calling deployStack(), which then resolves the bound
|
||||
* secret provider's bulk/inline-ref values internally (resolveProviderEnvVars,
|
||||
* stacks.ts) -- entirely new values the caller never saw. Without this, a
|
||||
* provider-resolved secret that ends up in compose's raw stderr passes straight
|
||||
* through end()'s redaction (it redacts against `this.secrets` only) and into
|
||||
* errorMessage, which GET /api/schedules/executions returns to every authenticated
|
||||
* user. Callers pass deployStack()'s result.resolvedSecrets here, after the call
|
||||
* returns and before end() is invoked -- see stacks.ts's StackOperationResult.
|
||||
*/
|
||||
addSecrets(values: string[]): void {
|
||||
for (const value of values) {
|
||||
if (!this.secrets.includes(value)) this.secrets.push(value);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Called on every path out of the deploy (success, failure, or throw) -- including
|
||||
* when no line() ever arrived, e.g. a deploy that fails before compose produces any
|
||||
* output. That's why this never assumes `this.lines` is non-empty.
|
||||
*/
|
||||
async end(ok: boolean, exitCode?: number, error?: string): Promise<void> {
|
||||
// Wait for every queued append to land before summarizing and closing the
|
||||
// row -- otherwise a line still in flight could be missing from the summary,
|
||||
// or the row could close before its own last line is actually on disk.
|
||||
await this.tail;
|
||||
|
||||
const details = buildRunDetails({
|
||||
options: this.options,
|
||||
summary: this.lines.length > 0 ? summarize(this.lines) : undefined,
|
||||
// A real exit code (from the local/direct compose path) is preferred; the
|
||||
// Hawser path doesn't have one to give, so fall back to a code consistent
|
||||
// with ok/failed rather than leaving it out.
|
||||
exitCode: exitCode ?? (ok ? 0 : 1),
|
||||
composeHash: this.composeHash,
|
||||
envHash: this.envHash,
|
||||
logFile: runLogFileName(this.runId),
|
||||
truncated: this.truncated
|
||||
});
|
||||
|
||||
// "Who" is best-effort only (design doc §10.5: webhook/cron/startup have no
|
||||
// user, and auth can be disabled entirely) -- kept out of buildRunDetails'
|
||||
// pure, tested shape and merged in here instead.
|
||||
const detailsWithUser = this.userId !== undefined ? { ...details, userId: this.userId } : details;
|
||||
|
||||
// error is compose's raw, unredacted stdout/stderr (deploy-recorder.ts's
|
||||
// shouldRecord() deliberately excludes the 'result' event from the log file for
|
||||
// exactly this reason: it never passes through redactLine). This value goes into
|
||||
// errorMessage on schedule_executions, which GET /api/schedules/executions returns
|
||||
// to every authenticated user regardless of stack/environment permission -- so it
|
||||
// gets the same redaction the streamed lines already got before it's stored.
|
||||
const errorMessage = ok
|
||||
? null
|
||||
: (redactLine(error ?? 'Deploy failed', this.secrets) ?? ERROR_WITHHELD_MESSAGE);
|
||||
|
||||
await updateScheduleExecution(this.executionId, {
|
||||
status: ok ? 'success' : 'failed',
|
||||
completedAt: new Date().toISOString(),
|
||||
duration: Date.now() - this.startedAtMs,
|
||||
errorMessage,
|
||||
details: detailsWithUser
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the schedule_executions row for a stack deploy run and returns a RunRecorder
|
||||
* that mirrors qualifying progress lines to its log file and closes the row when the
|
||||
* deploy is over. See sse.ts for why this is built in the route/here, not in sse.ts itself.
|
||||
*/
|
||||
export async function createRunRecorder(input: {
|
||||
stackName: string;
|
||||
envId: number | null;
|
||||
userId?: number;
|
||||
triggeredBy: ScheduleTrigger;
|
||||
options: DeployRunOptions;
|
||||
composeHash: string;
|
||||
envHash: string;
|
||||
/** Every value (secret AND non-secret) that reaches the container -- same set the
|
||||
* caller already merged for envHash. Used to redact error/summary text on end(),
|
||||
* same secrets list stacks.ts uses for makeLineForwarder()/makeRedactedLineSink(). */
|
||||
secrets: string[];
|
||||
}): Promise<RunRecorder> {
|
||||
const execution = await createScheduleExecution({
|
||||
scheduleType: 'stack_deploy',
|
||||
scheduleId: 0, // there is no schedule -- see schema comment "or 0 for system jobs"
|
||||
environmentId: input.envId,
|
||||
entityName: input.stackName,
|
||||
triggeredBy: input.triggeredBy,
|
||||
status: 'running'
|
||||
// logs stays empty on purpose: the text lives in the file, not this column.
|
||||
});
|
||||
|
||||
// createScheduleExecution doesn't accept startedAt (it fills triggeredAt itself) --
|
||||
// same two-step shape as the system-cleanup.ts jobs this one is modeled on.
|
||||
const startedAtMs = Date.now();
|
||||
await updateScheduleExecution(execution.id, {
|
||||
startedAt: new Date(startedAtMs).toISOString()
|
||||
});
|
||||
|
||||
return new DeployRunRecorder(execution.id, startedAtMs, {
|
||||
options: input.options,
|
||||
composeHash: input.composeHash,
|
||||
envHash: input.envHash,
|
||||
userId: input.userId,
|
||||
secrets: input.secrets,
|
||||
// F5 fix: same envId already stored on the schedule_executions row above
|
||||
// (environmentId: input.envId) -- also threaded to the recorder so its log
|
||||
// file lands in, and counts against the budget of, THIS run's own
|
||||
// environment directory. See DeployRunRecorder's envId field doc comment.
|
||||
envId: input.envId
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* Condenses compose output into the few facts an operator actually asks after a deploy:
|
||||
* was anything built or did it all come from cache, was an image pulled or was it already
|
||||
* there, were containers recreated or merely restarted, and which digest is running now.
|
||||
*
|
||||
* Deliberately dependency-free so it can be unit tested under Bun, which cannot import
|
||||
* the modules that pull in the native database binding.
|
||||
*
|
||||
* Compose repeats state transitions when no TTY is attached -- the same "Created" line can
|
||||
* appear twice -- so every count is over a Set of names, never over line occurrences.
|
||||
*/
|
||||
export interface DeploySummary {
|
||||
containersCreated: number;
|
||||
containersRecreated: number;
|
||||
containersStarted: number;
|
||||
imagesBuilt: string[];
|
||||
imagesPulled: string[];
|
||||
buildSteps: number;
|
||||
buildStepsCached: number;
|
||||
digest?: string;
|
||||
}
|
||||
|
||||
const CONTAINER = /^\s*Container\s+(\S+)\s+(Created|Recreated|Started)\s*$/;
|
||||
const IMAGE = /^\s*Image\s+(\S+)\s+(Built|Pulled)\s*$/;
|
||||
const BUILD_DONE = /^#\d+\s+DONE\s/;
|
||||
const BUILD_CACHED = /^#\d+\s+CACHED\s*$/;
|
||||
const DIGEST = /exporting manifest list (sha256:[0-9a-f]{64})/;
|
||||
|
||||
export function summarize(lines: string[]): DeploySummary {
|
||||
const created = new Set<string>();
|
||||
const recreated = new Set<string>();
|
||||
const started = new Set<string>();
|
||||
const built = new Set<string>();
|
||||
const pulled = new Set<string>();
|
||||
let buildSteps = 0;
|
||||
let buildStepsCached = 0;
|
||||
let digest: string | undefined;
|
||||
|
||||
for (const line of lines) {
|
||||
const c = CONTAINER.exec(line);
|
||||
if (c) {
|
||||
const [, name, verb] = c;
|
||||
if (verb === 'Created') created.add(name);
|
||||
else if (verb === 'Recreated') recreated.add(name);
|
||||
else started.add(name);
|
||||
continue;
|
||||
}
|
||||
const i = IMAGE.exec(line);
|
||||
if (i) {
|
||||
const [, name, verb] = i;
|
||||
(verb === 'Built' ? built : pulled).add(name);
|
||||
continue;
|
||||
}
|
||||
if (BUILD_CACHED.test(line)) { buildSteps++; buildStepsCached++; continue; }
|
||||
if (BUILD_DONE.test(line)) { buildSteps++; continue; }
|
||||
const d = DIGEST.exec(line);
|
||||
if (d) digest = d[1];
|
||||
}
|
||||
|
||||
return {
|
||||
containersCreated: created.size,
|
||||
containersRecreated: recreated.size,
|
||||
containersStarted: started.size,
|
||||
imagesBuilt: [...built],
|
||||
imagesPulled: [...pulled],
|
||||
buildSteps,
|
||||
buildStepsCached,
|
||||
digest
|
||||
};
|
||||
}
|
||||
@@ -598,6 +598,12 @@ export async function getDockerConfig(envId?: number | null): Promise<DockerClie
|
||||
interface DockerFetchOptions extends RequestInit {
|
||||
/** Set to true for long-lived streaming connections */
|
||||
streaming?: boolean;
|
||||
/**
|
||||
* Called per output line while the request runs. Only the Hawser Edge (WebSocket) path
|
||||
* can deliver these: the agent sends 'stream' messages alongside a non-streaming request.
|
||||
* Every other transport is plain request/response and simply never calls it.
|
||||
*/
|
||||
onLine?: (line: string) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -828,7 +834,7 @@ export async function dockerFetch(
|
||||
|
||||
const startTime = Date.now();
|
||||
const config = await getDockerConfig(envId);
|
||||
const { streaming, ...fetchOptions } = options;
|
||||
const { streaming, onLine, ...fetchOptions } = options;
|
||||
const method = (options.method || 'GET').toUpperCase();
|
||||
|
||||
// Hawser Edge mode - route through WebSocket connection
|
||||
@@ -888,7 +894,8 @@ export async function dockerFetch(
|
||||
streaming || false,
|
||||
(streaming || path === '/_hawser/compose' || path.endsWith('/prune')) ? 300000 : 30000, // 5 min for streaming/compose/prune, 30s for normal
|
||||
isBinary,
|
||||
fetchOptions.signal ?? undefined
|
||||
fetchOptions.signal ?? undefined,
|
||||
onLine
|
||||
);
|
||||
const elapsed = Date.now() - startTime;
|
||||
// Only warn for slow requests, but skip /stats which is expected to be slow (5-10s)
|
||||
|
||||
@@ -16,6 +16,9 @@
|
||||
* config embeds remote credentials), at any depth. This blocks the credential
|
||||
* files inside a git-repos clone while still letting the working copies be
|
||||
* browsed (deploy verification, adoption).
|
||||
* - the deploy-log directory ($DATA_DIR/deploy-logs, see deploy-log-store.ts) —
|
||||
* deploy logs may carry secrets that survived redaction, and the file
|
||||
* browser is open by design, so this directory has to be named here too.
|
||||
*
|
||||
* isProtectedPath resolves symlinks (via the nearest existing ancestor) so a
|
||||
* symlink pointing into a protected location can't be used to bypass the check.
|
||||
@@ -38,12 +41,15 @@ function getDataDir(): string {
|
||||
return process.env.DATA_DIR || './data';
|
||||
}
|
||||
|
||||
/** Absolute paths under Dockhand's data dir that must never be exposed. */
|
||||
function protectedPaths(): { dbDir: string; keyFile: string } {
|
||||
/** Absolute paths Dockhand must never expose through the file browser. */
|
||||
function protectedPaths(): { dbDir: string; keyFile: string; deployLogsDir: string } {
|
||||
const dataDir = resolve(getDataDir());
|
||||
return {
|
||||
dbDir: join(dataDir, 'db'),
|
||||
keyFile: join(dataDir, KEY_FILE_NAME)
|
||||
keyFile: join(dataDir, KEY_FILE_NAME),
|
||||
// deploy logs may contain secrets that survived redaction; the file browser is open by
|
||||
// design, so this directory is named here (see deploy-log-store.ts).
|
||||
deployLogsDir: join(dataDir, 'deploy-logs')
|
||||
};
|
||||
}
|
||||
|
||||
@@ -78,18 +84,23 @@ function isInside(child: string, parent: string): boolean {
|
||||
|
||||
/**
|
||||
* Returns true if `requestedPath` resolves to a location that must never be read
|
||||
* through the file browser (Dockhand's DB/key, system secret trees, or any .ssh /
|
||||
* .git directory). Symlinks are resolved first so they can't bypass it.
|
||||
* through the file browser (Dockhand's DB/key, the deploy-log directory, system
|
||||
* secret trees, or any .ssh / .git directory). Symlinks are resolved first so
|
||||
* they can't bypass it.
|
||||
*/
|
||||
export function isProtectedPath(requestedPath: string): boolean {
|
||||
const { dbDir, keyFile } = protectedPaths();
|
||||
const { dbDir, keyFile, deployLogsDir } = protectedPaths();
|
||||
const resolved = safeResolve(requestedPath);
|
||||
|
||||
// Compare against symlink-resolved targets too: the request is resolved, so a
|
||||
// protected target that is itself a symlink (e.g. .encryption_key mounted from
|
||||
// a secret store, or a symlinked db/ volume) must be resolved as well or the
|
||||
// two never match. safeResolve tolerates a not-yet-existing target.
|
||||
if (resolved === safeResolve(keyFile) || isInside(resolved, safeResolve(dbDir))) {
|
||||
if (
|
||||
resolved === safeResolve(keyFile) ||
|
||||
isInside(resolved, safeResolve(dbDir)) ||
|
||||
isInside(resolved, safeResolve(deployLogsDir))
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
// Match each root both literally and symlink-resolved (e.g. macOS /etc -> /private/etc).
|
||||
|
||||
+193
-26
@@ -12,11 +12,16 @@ import {
|
||||
updateGitStack,
|
||||
upsertStackSource,
|
||||
getEnvironment,
|
||||
getSecretEnvVarsAsRecord,
|
||||
getNonSecretEnvVarsAsRecord,
|
||||
type GitRepository,
|
||||
type GitCredential,
|
||||
type GitStackWithRepo
|
||||
type GitStackWithRepo,
|
||||
type ScheduleTrigger
|
||||
} from './db';
|
||||
import { deployStack, getStackDir } from './stacks';
|
||||
import { createRunRecorder } from './deploy-run-record';
|
||||
import { hashComposeContent, hashEnvFingerprint } from './deploy-run-record-core';
|
||||
import { sendEventNotification } from './notifications';
|
||||
import { buildBasicAuthHeader } from './git-auth';
|
||||
import { assertSafeRepoUrl, assertSafeGitRef, repoFilePath, repoBaseEnvPath } from './git-url-safety';
|
||||
@@ -1315,8 +1320,45 @@ async function notifyGitSync(stackName: string, envId: number | null | undefined
|
||||
} catch { /* never changes the deploy outcome */ }
|
||||
}
|
||||
|
||||
export async function deployGitStack(stackId: number, options?: { force?: boolean }): Promise<{ success: boolean; output?: string; error?: string; skipped?: boolean }> {
|
||||
/**
|
||||
* The function every git-stack deploy trigger funnels through EXCEPT ONE: a
|
||||
* config-update-then-redeploy, both webhook methods, create-and-deploy, and (via
|
||||
* runGitStackSync) the cron scheduler and the schedules page's "run now" all reach
|
||||
* this function. That is why the stack_deploy run recorder is wired in HERE and not
|
||||
* at each of those call sites: wiring it at the route level only (the way
|
||||
* compose/+server.ts does it) would leave every OTHER caller -- crucially the cron-
|
||||
* and webhook-triggered ones, the truly unattended runs the design doc's Meilenstein 5
|
||||
* is about -- unrecorded. Duplicating the composeHash/envHash/secrets setup at
|
||||
* six-plus call sites would also drift out of sync with each other over time; one
|
||||
* implementation, reached from everywhere but one, cannot.
|
||||
*
|
||||
* The one exception is the UI's own "Deploy" button (deploy-stream/+server.ts),
|
||||
* which streams live per-step progress via deployGitStackWithProgress() below --
|
||||
* a separate pipeline (it re-clones and re-reads the compose file itself rather than
|
||||
* calling this function) that carries its OWN, separately-wired copy of the same
|
||||
* recorder setup, right at its own deployStack() call. See that function's doc
|
||||
* comment for why it is a second call site instead of routing through here.
|
||||
*/
|
||||
export async function deployGitStack(
|
||||
stackId: number,
|
||||
options?: {
|
||||
force?: boolean;
|
||||
/** Who/what triggered this deploy, for the stack_deploy run record. Defaults to
|
||||
* 'manual' for backward compatibility with existing callers/tests that predate
|
||||
* this parameter -- every REAL call site below passes its own explicit value. */
|
||||
triggeredBy?: ScheduleTrigger;
|
||||
/** Best-effort only, like every other stack_deploy record (design doc §10.5):
|
||||
* webhook/cron callers have no user and simply omit this. */
|
||||
userId?: number;
|
||||
/** Forwarded to deployStack() alongside the recorder's own line() call, so an
|
||||
* HTTP caller (the manual deploy route, the deployNow routes) can still stream
|
||||
* progress over SSE exactly as before -- recording never depends on whether
|
||||
* anyone is listening. */
|
||||
onLine?: (line: string) => void;
|
||||
}
|
||||
): Promise<{ success: boolean; output?: string; error?: string; skipped?: boolean }> {
|
||||
const force = options?.force ?? true; // Default to force for backward compatibility
|
||||
const triggeredBy: ScheduleTrigger = options?.triggeredBy ?? 'manual';
|
||||
|
||||
const gitStack = await getGitStack(stackId);
|
||||
if (!gitStack) {
|
||||
@@ -1386,21 +1428,82 @@ export async function deployGitStack(stackId: number, options?: { force?: boolea
|
||||
console.log(`${logPrefix} Compose filename:`, syncResult.composeFileName);
|
||||
console.log(`${logPrefix} Env filename:`, syncResult.envFileName ?? '(none)');
|
||||
|
||||
const result = await deployStack({
|
||||
name: gitStack.stackName,
|
||||
compose: syncResult.composeContent!,
|
||||
envId: gitStack.environmentId,
|
||||
sourceDir: syncResult.composeDir, // Copy entire directory from git repo
|
||||
composeFileName: syncResult.composeFileName, // Use original compose filename from repo
|
||||
envFileName: syncResult.envFileName, // Env file relative to compose dir (for --env-file flag, optional)
|
||||
forceRecreate,
|
||||
build: gitStack.buildOnDeploy,
|
||||
noBuildCache: gitStack.noBuildCache,
|
||||
pullPolicy: gitStack.repullImages ? 'always' : undefined,
|
||||
filesToDelete: syncResult.deletionPlan?.toDelete,
|
||||
isGitDeploy: true // suppress stack_* notification; we emit git_sync_* below
|
||||
// Same stack_deploy run record shape the non-git routes build (compose PUT,
|
||||
// dedicated deploy endpoint) -- see deploy-run-record.ts. composeHash is taken
|
||||
// from syncResult.composeContent, the EXACT string handed to deployStack right
|
||||
// below, not a redundant re-read off disk.
|
||||
//
|
||||
// envHash/secrets mirror the same (deliberately approximate) resolution the
|
||||
// non-git routes already use: getNonSecretEnvVarsAsRecord/getSecretEnvVarsAsRecord
|
||||
// keyed by stackName+envId -- the same DB-sourced values deployStack itself
|
||||
// resolves again internally at deploy time (stacks.ts). This does NOT include
|
||||
// secret-provider-injected vars or the git repo's own .env file content; that's
|
||||
// fine for the redaction list because deployStack's onLine output is
|
||||
// independently redacted against ITS OWN, more complete secrets set
|
||||
// (makeLineForwarder in stacks.ts) -- a leaked value in the log is still caught
|
||||
// even if it isn't one of the vars hashed here.
|
||||
const nonSecretVars = await getNonSecretEnvVarsAsRecord(gitStack.stackName, gitStack.environmentId);
|
||||
const secretVars = await getSecretEnvVarsAsRecord(gitStack.stackName, gitStack.environmentId);
|
||||
const effectiveEnvVars = { ...nonSecretVars, ...secretVars };
|
||||
|
||||
const recorder = await createRunRecorder({
|
||||
stackName: gitStack.stackName,
|
||||
envId: gitStack.environmentId ?? null,
|
||||
userId: options?.userId,
|
||||
triggeredBy,
|
||||
options: { pull: !!gitStack.repullImages, build: !!gitStack.buildOnDeploy, forceRecreate: !!forceRecreate },
|
||||
composeHash: hashComposeContent(syncResult.composeContent!),
|
||||
envHash: hashEnvFingerprint(effectiveEnvVars),
|
||||
secrets: Object.values(effectiveEnvVars)
|
||||
});
|
||||
|
||||
// Mirrors every redacted output line into the run record's log file, in addition
|
||||
// to (not instead of) forwarding it to the caller's own onLine (SSE progress) --
|
||||
// see the recorder doc comment above for why this can't be "record OR stream".
|
||||
const onLine = (line: string) => {
|
||||
recorder.line(line);
|
||||
options?.onLine?.(line);
|
||||
};
|
||||
|
||||
let result: Awaited<ReturnType<typeof deployStack>>;
|
||||
try {
|
||||
result = await deployStack({
|
||||
name: gitStack.stackName,
|
||||
compose: syncResult.composeContent!,
|
||||
envId: gitStack.environmentId,
|
||||
sourceDir: syncResult.composeDir, // Copy entire directory from git repo
|
||||
composeFileName: syncResult.composeFileName, // Use original compose filename from repo
|
||||
envFileName: syncResult.envFileName, // Env file relative to compose dir (for --env-file flag, optional)
|
||||
forceRecreate,
|
||||
build: gitStack.buildOnDeploy,
|
||||
noBuildCache: gitStack.noBuildCache,
|
||||
pullPolicy: gitStack.repullImages ? 'always' : undefined,
|
||||
filesToDelete: syncResult.deletionPlan?.toDelete,
|
||||
isGitDeploy: true, // suppress stack_* notification; we emit git_sync_* below
|
||||
onLine
|
||||
});
|
||||
} catch (error) {
|
||||
// deployStack has never been observed to throw here (every existing caller of
|
||||
// deployGitStack calls it without a try/catch around this point), but the
|
||||
// record must not be left "running" forever on the day that changes.
|
||||
await recorder.end(false, undefined, error instanceof Error ? error.message : String(error));
|
||||
throw error;
|
||||
}
|
||||
|
||||
// F4 fix: deployStack() resolves the bound secret provider's values internally,
|
||||
// AFTER `recorder` above was already built from the DB-only nonSecretVars/
|
||||
// secretVars this function read before calling it. Feed the provider-resolved
|
||||
// values in now, before recorder.end() below -- otherwise a provider secret
|
||||
// leaking into result.error (a real possibility: cron/webhook-triggered git
|
||||
// deploys are exactly the unattended runs most likely to use a bound provider)
|
||||
// would bypass end()'s redaction entirely. See deploy-run-record.ts.
|
||||
recorder.addSecrets(result.resolvedSecrets ?? []);
|
||||
|
||||
// Closed right after the result is known, before the post-success bookkeeping
|
||||
// below (deletion sync, upsertStackSource) -- those never fail the deploy itself,
|
||||
// so the recorded duration reflects the deploy, not the bookkeeping around it.
|
||||
await recorder.end(result.success, undefined, result.success ? undefined : result.error);
|
||||
|
||||
console.log(`${logPrefix} ----------------------------------------`);
|
||||
console.log(`${logPrefix} DEPLOY GIT STACK RESULT`);
|
||||
console.log(`${logPrefix} ----------------------------------------`);
|
||||
@@ -1523,6 +1626,17 @@ type ProgressCallback = (data: {
|
||||
error?: string;
|
||||
}) => void;
|
||||
|
||||
/**
|
||||
* The UI's own "Deploy" button (deploy-stream/+server.ts): runs its own clone/read/
|
||||
* deploy pipeline, reporting 5 discrete stages via onProgress, rather than calling
|
||||
* deployGitStack() -- that function's onLine streams raw compose output lines, a
|
||||
* different contract than the stage messages this one sends. Because of that split,
|
||||
* this is a SECOND call site that reaches deployStack() directly, and it carries its
|
||||
* own copy of the same stack_deploy run-record wiring deployGitStack() has (built
|
||||
* right before its own deployStack() call below) rather than inheriting it -- see
|
||||
* deployGitStack's doc comment for the full picture of which callers go through
|
||||
* which path.
|
||||
*/
|
||||
export async function deployGitStackWithProgress(
|
||||
stackId: number,
|
||||
onProgress: ProgressCallback
|
||||
@@ -1554,6 +1668,11 @@ export async function deployGitStackWithProgress(
|
||||
|
||||
const totalSteps = 5;
|
||||
|
||||
// Created just before the deployStack() call below, same as deployGitStack --
|
||||
// undefined here means the run never got that far (e.g. clone failed), exactly
|
||||
// mirroring deployGitStack's own "only record a run that is actually happening".
|
||||
let recorder: Awaited<ReturnType<typeof createRunRecorder>> | undefined;
|
||||
|
||||
try {
|
||||
// Step 1: Connecting
|
||||
onProgress({ status: 'connecting', message: 'Connecting to repository...', step: 1, totalSteps });
|
||||
@@ -1720,19 +1839,67 @@ export async function deployGitStackWithProgress(
|
||||
}
|
||||
}
|
||||
|
||||
const result = await deployStack({
|
||||
name: gitStack.stackName,
|
||||
compose: composeContent,
|
||||
envId: gitStack.environmentId,
|
||||
sourceDir: composeDir, // Copy entire directory from git repo
|
||||
composeFileName: progressComposeFileName, // Compose filename relative to source dir
|
||||
envFileName, // Env file relative to compose dir (for --env-file flag, optional)
|
||||
build: gitStack.buildOnDeploy,
|
||||
noBuildCache: gitStack.noBuildCache,
|
||||
pullPolicy: gitStack.repullImages ? 'always' : undefined,
|
||||
filesToDelete: deletionData.plan.toDelete
|
||||
// Same stack_deploy run record shape deployGitStack builds (see its doc
|
||||
// comment above deployGitStack) -- this manual "Deploy" button path runs its
|
||||
// own clone/read/deploy pipeline instead of calling deployGitStack (it needs
|
||||
// this function's own discrete onProgress stages, not deployGitStack's onLine
|
||||
// streaming), so it was the one caller of deployStack that never went through
|
||||
// createRunRecorder at all -- unlike every other trigger (webhook,
|
||||
// config-update-then-redeploy, and cron via runGitStackSync, all of which
|
||||
// funnel through deployGitStack). Wired in here, right before the SAME
|
||||
// deployStack() call deployGitStack wires it around, so it records without
|
||||
// touching onProgress's own streaming.
|
||||
const nonSecretVars = await getNonSecretEnvVarsAsRecord(gitStack.stackName, gitStack.environmentId);
|
||||
const secretVars = await getSecretEnvVarsAsRecord(gitStack.stackName, gitStack.environmentId);
|
||||
const effectiveEnvVars = { ...nonSecretVars, ...secretVars };
|
||||
|
||||
recorder = await createRunRecorder({
|
||||
stackName: gitStack.stackName,
|
||||
envId: gitStack.environmentId ?? null,
|
||||
triggeredBy: 'manual',
|
||||
// forceRecreate is never passed to deployStack below (unlike deployGitStack,
|
||||
// which computes it from forceRedeploy/gitUpdated) -- this path always omits
|
||||
// it, so the record should say so too rather than guessing.
|
||||
options: { pull: !!gitStack.repullImages, build: !!gitStack.buildOnDeploy, forceRecreate: false },
|
||||
composeHash: hashComposeContent(composeContent),
|
||||
envHash: hashEnvFingerprint(effectiveEnvVars),
|
||||
secrets: Object.values(effectiveEnvVars)
|
||||
});
|
||||
|
||||
let result: Awaited<ReturnType<typeof deployStack>>;
|
||||
try {
|
||||
result = await deployStack({
|
||||
name: gitStack.stackName,
|
||||
compose: composeContent,
|
||||
envId: gitStack.environmentId,
|
||||
sourceDir: composeDir, // Copy entire directory from git repo
|
||||
composeFileName: progressComposeFileName, // Compose filename relative to source dir
|
||||
envFileName, // Env file relative to compose dir (for --env-file flag, optional)
|
||||
build: gitStack.buildOnDeploy,
|
||||
noBuildCache: gitStack.noBuildCache,
|
||||
pullPolicy: gitStack.repullImages ? 'always' : undefined,
|
||||
filesToDelete: deletionData.plan.toDelete,
|
||||
// Mirrors deploy output into the run record's log file. Not forwarded to
|
||||
// onProgress -- that callback's contract is the 5 discrete stage messages
|
||||
// this function already sends above, not raw compose output lines.
|
||||
onLine: (line) => recorder?.line(line)
|
||||
});
|
||||
} catch (error) {
|
||||
// deployStack has never been observed to throw here, but the record must
|
||||
// not be left "running" forever on the day that changes (same posture
|
||||
// deployGitStack takes around its own deployStack call).
|
||||
await recorder.end(false, undefined, error instanceof Error ? error.message : String(error));
|
||||
throw error;
|
||||
}
|
||||
|
||||
// F4 fix (see deployGitStack's own addSecrets() call for the full rationale):
|
||||
// deployStack() resolves the bound secret provider's values internally, after
|
||||
// `recorder` above was already built from the DB-only vars read before calling
|
||||
// it. Feed the provider-resolved values in now, before recorder.end(), so a
|
||||
// provider secret leaking into result.error is still redacted.
|
||||
recorder.addSecrets(result.resolvedSecrets ?? []);
|
||||
await recorder.end(result.success, undefined, result.success ? undefined : result.error);
|
||||
|
||||
if (result.success) {
|
||||
// Deletion sync: persist manifest + log per-file change summary.
|
||||
// The change table was already shown before the deploy (#1260);
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* hawser-core.ts — the PURE parts of the Hawser Edge protocol message routing: no DB import,
|
||||
* so it is unit-testable directly (importing hawser.ts pulls in db/drizzle -> better-sqlite3,
|
||||
* which bun's test runner can't load). hawser.ts re-exports this so callers have one import site.
|
||||
*/
|
||||
|
||||
/** The minimal shape dispatchStreamMessage needs off an EdgeConnection. */
|
||||
export interface StreamDispatchTarget {
|
||||
pendingStreamRequests?: Map<string, { onData?: (data: string, stream?: 'stdout' | 'stderr') => void }>;
|
||||
// Line callbacks for non-streaming requests (e.g. compose) that still want to observe
|
||||
// 'stream' messages as they arrive. Separate from pendingStreamRequests, which belongs
|
||||
// to requests sent with streaming: true (see sendEdgeStreamRequest in hawser.ts).
|
||||
lineHandlers?: Map<string, (line: string) => void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Route a 'stream' message from the agent to whichever consumer is waiting for it.
|
||||
*
|
||||
* Two independent registries can claim a requestId: pendingStreamRequests (requests sent
|
||||
* with streaming: true) and lineHandlers (non-streaming requests, e.g. compose, that still
|
||||
* want to observe line-by-line output as it arrives). A requestId is only ever registered in
|
||||
* one of the two — pendingStreamRequests is checked first, matching the previous behaviour of
|
||||
* the inline 'stream' case this was extracted from.
|
||||
*
|
||||
* Compose requests are sent with streaming: false and land in pendingRequests only. Before
|
||||
* this function existed, the message dispatcher's 'stream' case looked up pendingStreamRequests
|
||||
* exclusively, so any line message for a non-streaming request vanished silently — no error, no
|
||||
* warning.
|
||||
*/
|
||||
export function dispatchStreamMessage(
|
||||
connection: StreamDispatchTarget,
|
||||
msg: { requestId: string; data: unknown }
|
||||
): void {
|
||||
const streamPending = connection.pendingStreamRequests?.get(msg.requestId);
|
||||
if (streamPending) {
|
||||
streamPending.onData?.(msg.data as string);
|
||||
return;
|
||||
}
|
||||
|
||||
const lineHandler = connection.lineHandlers?.get(msg.requestId);
|
||||
if (!lineHandler) return; // no handler registered: drop silently, this is the normal case
|
||||
|
||||
// Go marshals StreamMessage.Data ([]byte) as base64 -- sendEdgeStreamRequest's onData
|
||||
// callback does the same decode on the streaming path. Without this, the UI shows base64
|
||||
// gibberish instead of readable output.
|
||||
const text = typeof msg.data === 'string' ? Buffer.from(msg.data, 'base64').toString('utf8') : '';
|
||||
if (text) lineHandler(text);
|
||||
}
|
||||
@@ -14,6 +14,11 @@ import { isHealthTransition } from './subprocess-manager.js';
|
||||
import { pushMetric } from './metrics-store.js';
|
||||
import { secureGetRandomValues, secureRandomUUID } from './crypto-fallback.js';
|
||||
import { hashPassword, verifyPassword } from './auth.js';
|
||||
// The 'stream' message routing logic lives in a db-free core module so it stays unit-testable
|
||||
// (importing this file pulls in db/drizzle -> better-sqlite3, which bun's test runner can't
|
||||
// load). Re-exported so callers get it from one import site.
|
||||
import { dispatchStreamMessage } from './hawser-core.js';
|
||||
export { dispatchStreamMessage };
|
||||
|
||||
// Protocol constants
|
||||
export const HAWSER_PROTOCOL_VERSION = '1.0';
|
||||
@@ -48,6 +53,10 @@ export interface EdgeConnection {
|
||||
lastHeartbeat: number;
|
||||
pendingRequests: Map<string, PendingRequest>;
|
||||
pendingStreamRequests: Map<string, PendingStreamRequest>;
|
||||
// Line callbacks for non-streaming requests (e.g. compose) that still want to observe
|
||||
// 'stream' messages as they arrive. Separate from pendingStreamRequests, which belongs
|
||||
// to requests sent with streaming: true (see sendEdgeStreamRequest).
|
||||
lineHandlers?: Map<string, (line: string) => void>;
|
||||
pingInterval?: ReturnType<typeof setInterval>;
|
||||
lastMetrics?: {
|
||||
uptime?: number;
|
||||
@@ -514,7 +523,8 @@ export function handleEdgeConnection(
|
||||
connectedAt: new Date(),
|
||||
lastHeartbeat: Date.now(),
|
||||
pendingRequests: new Map(),
|
||||
pendingStreamRequests: new Map()
|
||||
pendingStreamRequests: new Map(),
|
||||
lineHandlers: new Map()
|
||||
};
|
||||
|
||||
edgeConnections.set(environmentId, connection);
|
||||
@@ -578,7 +588,8 @@ export async function sendEdgeRequest(
|
||||
streaming = false,
|
||||
timeout = 30000,
|
||||
isBinary = false,
|
||||
signal?: AbortSignal
|
||||
signal?: AbortSignal,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<EdgeResponse> {
|
||||
const connection = edgeConnections.get(environmentId);
|
||||
if (!connection) {
|
||||
@@ -587,9 +598,18 @@ export async function sendEdgeRequest(
|
||||
|
||||
const requestId = secureRandomUUID();
|
||||
|
||||
// A non-streaming request (compose) that wants to observe output as it happens. The agent
|
||||
// sends it as 'stream' messages, which dispatchStreamMessage routes here by requestId.
|
||||
// Registered before the message goes out, so no early line can miss the handler.
|
||||
if (onLine) {
|
||||
connection.lineHandlers ??= new Map();
|
||||
connection.lineHandlers.set(requestId, onLine);
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const timeoutHandle = setTimeout(() => {
|
||||
connection.pendingRequests.delete(requestId);
|
||||
connection.lineHandlers?.delete(requestId);
|
||||
if (streaming) {
|
||||
connection.pendingStreamRequests.delete(requestId);
|
||||
}
|
||||
@@ -607,6 +627,7 @@ export async function sendEdgeRequest(
|
||||
'abort',
|
||||
() => {
|
||||
connection.pendingRequests.delete(requestId);
|
||||
connection.lineHandlers?.delete(requestId);
|
||||
if (streaming) {
|
||||
connection.pendingStreamRequests.delete(requestId);
|
||||
}
|
||||
@@ -700,6 +721,7 @@ export async function sendEdgeRequest(
|
||||
const sent = globalThis.__hawserSendMessage(environmentId, messageStr);
|
||||
if (!sent) {
|
||||
connection.pendingRequests.delete(requestId);
|
||||
connection.lineHandlers?.delete(requestId);
|
||||
if (streaming) {
|
||||
connection.pendingStreamRequests.delete(requestId);
|
||||
}
|
||||
@@ -713,6 +735,7 @@ export async function sendEdgeRequest(
|
||||
const errorMsg = sendError instanceof Error ? sendError.message : String(sendError);
|
||||
console.error(`[Hawser Edge] Error sending message:`, errorMsg);
|
||||
connection.pendingRequests.delete(requestId);
|
||||
connection.lineHandlers?.delete(requestId);
|
||||
if (streaming) {
|
||||
connection.pendingStreamRequests.delete(requestId);
|
||||
}
|
||||
@@ -1261,6 +1284,10 @@ async function handleHawserWsMessage(ws: any, msg: any, connId: string, remoteIp
|
||||
if (pending) {
|
||||
clearTimeout(pending.timeout);
|
||||
connection.pendingRequests.delete(msg.requestId);
|
||||
// A non-streaming request that registered a line handler (compose, Task 9) is
|
||||
// done once its response arrives -- without this the map grows unbounded for
|
||||
// the lifetime of the process.
|
||||
connection.lineHandlers?.delete(msg.requestId);
|
||||
pending.resolve({
|
||||
statusCode: msg.statusCode,
|
||||
headers: msg.headers || {},
|
||||
@@ -1272,10 +1299,7 @@ async function handleHawserWsMessage(ws: any, msg: any, connId: string, remoteIp
|
||||
}
|
||||
|
||||
case 'stream': {
|
||||
const streamPending = connection.pendingStreamRequests.get(msg.requestId);
|
||||
if (streamPending) {
|
||||
streamPending.onData?.(msg.data);
|
||||
}
|
||||
dispatchStreamMessage(connection, msg);
|
||||
break;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* Pure child-process output collector. No imports, so it is unit-testable in
|
||||
* the bun runner without dragging in better-sqlite3 via stacks.ts -> ./db ->
|
||||
* ./db/drizzle (top-level `await seedDatabase()`, unsupported under Bun --
|
||||
* see tests/stacks-collect-process.test.ts for the reproduction).
|
||||
*/
|
||||
import type { ChildProcess } from 'node:child_process';
|
||||
|
||||
export type LineCallback = (line: string) => void;
|
||||
|
||||
/**
|
||||
* Collect stdout/stderr from a child process and wait for it to exit.
|
||||
* When `onLine` is provided, it is called for each complete line (split on '\n')
|
||||
* seen on either stdout or stderr, in arrival order, as the process runs — plus
|
||||
* once more for a trailing partial line (no terminating '\n') once the process closes.
|
||||
* Without `onLine`, behavior is unchanged.
|
||||
*/
|
||||
export function collectProcess(
|
||||
proc: ChildProcess,
|
||||
onLine?: LineCallback
|
||||
): Promise<{ exitCode: number; stdout: string; stderr: string }> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const stdoutChunks: Buffer[] = [];
|
||||
const stderrChunks: Buffer[] = [];
|
||||
let lineBuffer = '';
|
||||
|
||||
const emitLines = (chunk: string) => {
|
||||
if (!onLine) return;
|
||||
lineBuffer += chunk;
|
||||
const parts = lineBuffer.split('\n');
|
||||
lineBuffer = parts.pop() ?? ''; // last part may be incomplete
|
||||
for (const line of parts) onLine(line);
|
||||
};
|
||||
|
||||
proc.stdout?.on('data', (chunk: Buffer) => {
|
||||
stdoutChunks.push(chunk);
|
||||
emitLines(chunk.toString());
|
||||
});
|
||||
proc.stderr?.on('data', (chunk: Buffer) => {
|
||||
stderrChunks.push(chunk);
|
||||
emitLines(chunk.toString());
|
||||
});
|
||||
proc.on('error', reject);
|
||||
proc.on('close', (code) => {
|
||||
if (onLine && lineBuffer.length > 0) {
|
||||
onLine(lineBuffer);
|
||||
lineBuffer = '';
|
||||
}
|
||||
resolve({
|
||||
exitCode: code ?? 1,
|
||||
stdout: Buffer.concat(stdoutChunks).toString(),
|
||||
stderr: Buffer.concat(stderrChunks).toString()
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -18,11 +18,13 @@ import {
|
||||
getScheduleCleanupCron,
|
||||
getEventCleanupCron,
|
||||
getScannerCleanupCron,
|
||||
getDeployLogReconcileCron,
|
||||
getScheduleRetentionDays,
|
||||
getEventRetentionDays,
|
||||
getScheduleCleanupEnabled,
|
||||
getEventCleanupEnabled,
|
||||
getScannerCleanupEnabled,
|
||||
getDeployLogReconcileEnabled,
|
||||
getEnvironments,
|
||||
getEnvUpdateCheckSettings,
|
||||
getAllEnvUpdateCheckSettings,
|
||||
@@ -61,6 +63,7 @@ import {
|
||||
SYSTEM_VOLUME_HELPER_CLEANUP_ID,
|
||||
SYSTEM_SCANNER_CLEANUP_ID
|
||||
} from './tasks/system-cleanup';
|
||||
import { runDeployLogReconcileJob, DEPLOY_LOG_RECONCILE_ID } from './tasks/deploy-log-reconcile';
|
||||
|
||||
// Store all active cron jobs
|
||||
const activeJobs: Map<string, Cron> = new Map();
|
||||
@@ -70,6 +73,7 @@ let cleanupJob: Cron | null = null;
|
||||
let eventCleanupJob: Cron | null = null;
|
||||
let volumeHelperCleanupJob: Cron | null = null;
|
||||
let scannerCacheCleanupJob: Cron | null = null;
|
||||
let deployLogReconcileJob: Cron | null = null;
|
||||
|
||||
// Scheduler state
|
||||
let isRunning = false;
|
||||
@@ -170,6 +174,7 @@ export async function startScheduler(): Promise<void> {
|
||||
const scheduleCleanupCron = await getScheduleCleanupCron();
|
||||
const eventCleanupCron = await getEventCleanupCron();
|
||||
const scannerCleanupCron = await getScannerCleanupCron();
|
||||
const deployLogReconcileCron = await getDeployLogReconcileCron();
|
||||
const defaultTimezone = await getDefaultTimezone();
|
||||
|
||||
// Start system cleanup jobs (static schedules with default timezone)
|
||||
@@ -205,10 +210,19 @@ export async function startScheduler(): Promise<void> {
|
||||
});
|
||||
}
|
||||
|
||||
// Deploy log reconcile: own enabled flag + cron, same pattern as scanner cleanup
|
||||
const deployLogReconcileEnabled = await getDeployLogReconcileEnabled();
|
||||
if (deployLogReconcileEnabled) {
|
||||
deployLogReconcileJob = new Cron(deployLogReconcileCron, { timezone: defaultTimezone, legacyMode: false }, async () => {
|
||||
await runDeployLogReconcileJob('cron');
|
||||
});
|
||||
}
|
||||
|
||||
console.log(`[Scheduler] System schedule cleanup: ${scheduleCleanupCron} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] System event cleanup: ${eventCleanupCron} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Volume helper cleanup: every 30 minutes [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Scanner cache cleanup: ${scannerCleanupEnabled ? scannerCleanupCron : 'disabled'} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Deploy log reconcile: ${deployLogReconcileEnabled ? deployLogReconcileCron : 'disabled'} [${defaultTimezone}]`);
|
||||
|
||||
// Register all dynamic schedules from database
|
||||
await refreshAllSchedules();
|
||||
@@ -247,6 +261,10 @@ export function stopScheduler(): void {
|
||||
scannerCacheCleanupJob.stop();
|
||||
scannerCacheCleanupJob = null;
|
||||
}
|
||||
if (deployLogReconcileJob) {
|
||||
deployLogReconcileJob.stop();
|
||||
deployLogReconcileJob = null;
|
||||
}
|
||||
|
||||
// Stop all dynamic jobs
|
||||
for (const [key, job] of activeJobs.entries()) {
|
||||
@@ -668,6 +686,8 @@ export async function refreshSystemJobs(): Promise<void> {
|
||||
const eventCleanupCron = await getEventCleanupCron();
|
||||
const scannerCleanupCron = await getScannerCleanupCron();
|
||||
const scannerCleanupEnabled = await getScannerCleanupEnabled();
|
||||
const deployLogReconcileCron = await getDeployLogReconcileCron();
|
||||
const deployLogReconcileEnabled = await getDeployLogReconcileEnabled();
|
||||
const defaultTimezone = await getDefaultTimezone();
|
||||
|
||||
// Cleanup functions to pass to the job
|
||||
@@ -693,6 +713,9 @@ export async function refreshSystemJobs(): Promise<void> {
|
||||
if (scannerCacheCleanupJob) {
|
||||
scannerCacheCleanupJob.stop();
|
||||
}
|
||||
if (deployLogReconcileJob) {
|
||||
deployLogReconcileJob.stop();
|
||||
}
|
||||
|
||||
// Re-create with new timezone
|
||||
cleanupJob = new Cron(scheduleCleanupCron, { timezone: defaultTimezone, legacyMode: false }, async () => {
|
||||
@@ -713,10 +736,17 @@ export async function refreshSystemJobs(): Promise<void> {
|
||||
});
|
||||
}
|
||||
|
||||
if (deployLogReconcileEnabled) {
|
||||
deployLogReconcileJob = new Cron(deployLogReconcileCron, { timezone: defaultTimezone, legacyMode: false }, async () => {
|
||||
await runDeployLogReconcileJob('cron');
|
||||
});
|
||||
}
|
||||
|
||||
console.log(`[Scheduler] System schedule cleanup: ${scheduleCleanupCron} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] System event cleanup: ${eventCleanupCron} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Volume helper cleanup: every 30 minutes [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Scanner cache cleanup: ${scannerCleanupEnabled ? scannerCleanupCron : 'disabled'} [${defaultTimezone}]`);
|
||||
console.log(`[Scheduler] Deploy log reconcile: ${deployLogReconcileEnabled ? deployLogReconcileCron : 'disabled'} [${defaultTimezone}]`);
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
@@ -861,6 +891,23 @@ export async function triggerSystemJob(jobId: string): Promise<{ success: boolea
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Manually trigger the deploy log reconcile job.
|
||||
*
|
||||
* A dedicated function rather than a jobId branch in triggerSystemJob() above: this job
|
||||
* has its own scheduleType ('deploy_log_reconcile', not 'system_cleanup'), so its
|
||||
* DEPLOY_LOG_RECONCILE_ID (1) would otherwise collide with SYSTEM_SCHEDULE_CLEANUP_ID
|
||||
* (also 1) in that function's flat, type-less jobId dispatch.
|
||||
*/
|
||||
export async function triggerDeployLogReconcile(): Promise<{ success: boolean; executionId?: number; error?: string }> {
|
||||
try {
|
||||
runDeployLogReconcileJob('manual');
|
||||
return { success: true };
|
||||
} catch (error: any) {
|
||||
return { success: false, error: error.message };
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// UTILITY FUNCTIONS
|
||||
// =============================================================================
|
||||
@@ -881,6 +928,8 @@ export async function getSystemSchedules(): Promise<SystemScheduleInfo[]> {
|
||||
const scheduleCleanupEnabled = await getScheduleCleanupEnabled();
|
||||
const eventCleanupEnabled = await getEventCleanupEnabled();
|
||||
const scannerCleanupEnabled = await getScannerCleanupEnabled();
|
||||
const deployLogReconcileCron = await getDeployLogReconcileCron();
|
||||
const deployLogReconcileEnabled = await getDeployLogReconcileEnabled();
|
||||
|
||||
return [
|
||||
{
|
||||
@@ -922,13 +971,23 @@ export async function getSystemSchedules(): Promise<SystemScheduleInfo[]> {
|
||||
nextRun: scannerCleanupEnabled ? getNextRun(scannerCleanupCron)?.toISOString() ?? null : null,
|
||||
isSystem: true,
|
||||
enabled: scannerCleanupEnabled
|
||||
},
|
||||
{
|
||||
id: DEPLOY_LOG_RECONCILE_ID,
|
||||
type: 'deploy_log_reconcile' as const,
|
||||
name: 'Deploy log reconcile',
|
||||
description: 'Reconciles deploy-log files on disk against stack_deploy run records',
|
||||
cronExpression: deployLogReconcileCron,
|
||||
nextRun: deployLogReconcileEnabled ? getNextRun(deployLogReconcileCron)?.toISOString() ?? null : null,
|
||||
isSystem: true,
|
||||
enabled: deployLogReconcileEnabled
|
||||
}
|
||||
];
|
||||
}
|
||||
|
||||
export interface SystemScheduleInfo {
|
||||
id: number;
|
||||
type: 'system_cleanup';
|
||||
type: 'system_cleanup' | 'deploy_log_reconcile';
|
||||
name: string;
|
||||
description: string;
|
||||
cronExpression: string;
|
||||
|
||||
@@ -0,0 +1,321 @@
|
||||
/**
|
||||
* Deploy Log Reconcile Task
|
||||
*
|
||||
* Compares deploy-log files on disk (deploy-log-store.ts) against stack_deploy run
|
||||
* records in the database and reconciles the two: a file with no record is deleted (an
|
||||
* orphan with nothing left to explain it); a record with no file is only marked via
|
||||
* details.logMissing -- never deleted, since the metadata (when, who, what was built,
|
||||
* which digest) stays valuable even once the log text is gone. See
|
||||
* deploy-log-reconcile-core.ts for the decision itself; this file only gathers the two
|
||||
* id sets, guards the one case planReconcile can't know about (see below), and carries
|
||||
* out the resulting plan.
|
||||
*
|
||||
* F5 fix -- reconciled PER ENVIRONMENT, never across environments: deploy-log-store.ts
|
||||
* now keeps one log directory per environment (envDirName()/logDir()), because a SHARED
|
||||
* directory and a SHARED size budget let a noisy deploy in one environment silently
|
||||
* truncate another environment's deploy logs. This job has to honor that same boundary,
|
||||
* or it would reintroduce the exact cross-environment mixing the size-budget fix closes
|
||||
* -- e.g. deleting environment B's file because it happens to share a numeric run id
|
||||
* with an orphan in environment A, or marking environment A's record logMissing because
|
||||
* environment B happens to hold a same-numbered file. Every id set this file gathers
|
||||
* (files AND records) is therefore grouped by environment FIRST, and planReconcile()
|
||||
* (the pure decision core) is called ONCE PER ENVIRONMENT, on that environment's own two
|
||||
* sets only -- never once on everything flattened together.
|
||||
*
|
||||
* The set of environments to look at comes from BOTH sides on purpose:
|
||||
* - listEnvDirNames() (deploy-log-store.ts) -- every environment that currently has
|
||||
* ANY file on disk, including one that was later deleted from the database (its
|
||||
* orphaned files must still be reconciled, so its directory must still be visited).
|
||||
* - every DISTINCT environmentId among the stack_deploy records themselves -- an
|
||||
* environment whose only record has no file at all would otherwise never appear
|
||||
* via listEnvDirNames() (there is no directory for it to discover), and its record
|
||||
* would never be considered for markMissing.
|
||||
* The union of the two is walked once each; a directory name that cannot be parsed back
|
||||
* into an environment id (parseEnvDirName()) is skipped defensively rather than crashing
|
||||
* the whole job -- see the per-environment loop below.
|
||||
*
|
||||
* Record source, verified against Task 10's actual code (deploy-run-record.ts, read
|
||||
* directly from a sibling clone -- Task 10 had not landed on this branch at the time
|
||||
* this file was written, so it could not be imported here): createRunRecorder() creates
|
||||
* a schedule_executions row with scheduleType 'stack_deploy', scheduleId 0 (there is no
|
||||
* schedule, only individual runs), and DeployRunRecorder sets `this.runId =
|
||||
* String(executionId)` -- exactly the id this file reads via
|
||||
* getScheduleExecutionIdsByType('stack_deploy') and converts with String(r.id). Confirmed
|
||||
* correct, not assumed.
|
||||
*
|
||||
* That same reading surfaced a real race this job has to guard against: the row is
|
||||
* created with status 'running' BEFORE the deploy has produced a single line, and the
|
||||
* log file itself only comes into existence on the first DeployRunRecorder.line() call
|
||||
* (appendRunLog() -- deploy-log-store.ts). So there is a real, if narrow, window where a
|
||||
* 'running' record legitimately has no file yet. A naive reconcile firing in that window
|
||||
* would mark a deploy that is actively in progress as logMissing -- and nothing in this
|
||||
* job ever re-checks or clears that flag once the file does show up, so the record would
|
||||
* carry a false "log missing" forever. Records still in a non-terminal state ('queued'/
|
||||
* 'running') are therefore excluded from markMissing candidacy below, while still
|
||||
* counting toward recordIds (within their own environment's set) so an already-existing
|
||||
* file for them is protected from deletion either way.
|
||||
*
|
||||
* TOCTOU guard on the delete side: `records` (below) is read ONCE, at the very start
|
||||
* of this job, via a single getScheduleExecutionIdsByType() call -- a point-in-time
|
||||
* snapshot, not a live view. A deploy whose createRunRecorder() call happens AFTER
|
||||
* that snapshot has no row in `records` at all, yet its log file can already exist by
|
||||
* the time THIS environment's fileIds are listed a moment later (appendRunLog() writes
|
||||
* the file on the very first streamed line, which can arrive within the same tick).
|
||||
* Nothing in planReconcile() can tell that file apart from a genuine orphan -- both
|
||||
* simply have no matching id in `records`. Deleting it would tear a live deploy's log
|
||||
* out from under it while it is still being written to.
|
||||
*
|
||||
* Reported independently and confirmed by re-reading this exact snapshot-then-list
|
||||
* ordering: fixed by treating a delete candidate's file mtime as a second signal
|
||||
* alongside `records`, not by trying to make the DB read and the directory listing
|
||||
* atomic with each other (they are two different data sources; no single query spans
|
||||
* both). ORPHAN_DELETE_GRACE_MS below is deliberately generous compared to how long
|
||||
* this job actually takes to walk from its `records` snapshot to any one
|
||||
* environment's listRunLogIds() call (well under a second in the ordinary case) --
|
||||
* appendRunLog() keeps bumping a live deploy's file mtime for as long as it keeps
|
||||
* writing, so the grace period does not need to outlast the whole deploy, only the
|
||||
* gap between the snapshot and this environment being reached. A file newer than the
|
||||
* grace period is left alone THIS run; the NEXT scheduled run will see its by-then-
|
||||
* created record and correctly never consider it for deletion again -- nothing is
|
||||
* lost, the file is just given one more cycle to prove itself real, exactly the same
|
||||
* shape of trade-off isEligibleForMissingMark() already makes on the other side of
|
||||
* this same race (see above).
|
||||
*
|
||||
* Each element in both loops below is wrapped in its own try/catch. Without that, one
|
||||
* throwing deleteRunLog/updateScheduleExecution call would abort the whole run -- and
|
||||
* since the failing file/record is never removed/marked, every LATER run would fail at
|
||||
* the exact same element again, permanently. Reported and reproduced independently: a
|
||||
* harmless orphan file ordered before a failing one was still removed, everything after
|
||||
* it was not, and the failed run's own `details` came back `null` -- no record of what
|
||||
* had actually happened. Per-element isolation plus real counts (see below) fixes both --
|
||||
* and now applies WITHIN each environment's own loop, so a failure in one environment's
|
||||
* reconciliation cannot stop a later environment's elements from being tried either.
|
||||
*/
|
||||
|
||||
import type { ScheduleTrigger } from '../../db';
|
||||
import {
|
||||
getDeployLogReconcileEnabled,
|
||||
getScheduleExecutionIdsByType,
|
||||
createScheduleExecution,
|
||||
updateScheduleExecution,
|
||||
appendScheduleExecutionLog
|
||||
} from '../../db';
|
||||
import {
|
||||
listEnvDirNames,
|
||||
listRunLogIds,
|
||||
deleteRunLog,
|
||||
envDirName,
|
||||
parseEnvDirName,
|
||||
getRunLogMtimeMs
|
||||
} from '../../deploy-log-store';
|
||||
import { planReconcile, isEligibleForMissingMark } from '../../deploy-log-reconcile-core';
|
||||
|
||||
// System job ID (own scheduleType, so this never collides with system-cleanup.ts's ids)
|
||||
export const DEPLOY_LOG_RECONCILE_ID = 1;
|
||||
|
||||
/**
|
||||
* How much younger than "now" a delete candidate's file mtime may be before this job
|
||||
* still deletes it. See the TOCTOU guard in the module doc comment above: this only
|
||||
* needs to outlast the gap between the `records` snapshot and this environment's
|
||||
* listRunLogIds() call (ordinarily well under a second), never the whole deploy --
|
||||
* appendRunLog() keeps re-bumping a live deploy's mtime for as long as it keeps
|
||||
* writing. Generous on purpose; a false skip just costs one more reconcile cycle,
|
||||
* while a false delete destroys a live deploy's log outright.
|
||||
*/
|
||||
const ORPHAN_DELETE_GRACE_MS = 5 * 60 * 1000; // 5 minutes
|
||||
|
||||
/**
|
||||
* The filesystem calls this job makes, injectable so a test can force one element
|
||||
* to fail while its neighbours succeed, and so a test can populate a fake per-environment
|
||||
* file layout without touching the real filesystem.
|
||||
*
|
||||
* They are parameters rather than a mock.module() of '../../deploy-log-store' because that
|
||||
* call freezes the module's export shape for the WHOLE test process: deploy-run-record.ts
|
||||
* imports appendRunLog/runLogFileName/SizeBudgetTracker from the same specifier, and a
|
||||
* partial factory left those missing for every module that resolved it afterwards.
|
||||
*/
|
||||
export interface DeployLogReconcileDeps {
|
||||
listEnvDirNames: () => Promise<string[]>;
|
||||
listRunLogIds: (envId: number | null) => Promise<string[]>;
|
||||
deleteRunLog: (envId: number | null, runId: string) => Promise<void>;
|
||||
/** mtime (ms since epoch) of a delete candidate's file, or null if it is already gone. */
|
||||
getRunLogMtimeMs: (envId: number | null, runId: string) => Promise<number | null>;
|
||||
}
|
||||
|
||||
const defaultDeps: DeployLogReconcileDeps = {
|
||||
listEnvDirNames,
|
||||
listRunLogIds,
|
||||
deleteRunLog,
|
||||
getRunLogMtimeMs
|
||||
};
|
||||
|
||||
/**
|
||||
* Execute the deploy-log reconcile job.
|
||||
*/
|
||||
export async function runDeployLogReconcileJob(
|
||||
triggeredBy: ScheduleTrigger = 'cron',
|
||||
deps: DeployLogReconcileDeps = defaultDeps
|
||||
): Promise<void> {
|
||||
// Check if reconcile is enabled (skip check if manually triggered)
|
||||
if (triggeredBy === 'cron') {
|
||||
const enabled = await getDeployLogReconcileEnabled();
|
||||
if (!enabled) {
|
||||
return; // Skip execution if disabled
|
||||
}
|
||||
}
|
||||
|
||||
const startTime = Date.now();
|
||||
|
||||
// Create execution record
|
||||
const execution = await createScheduleExecution({
|
||||
scheduleType: 'deploy_log_reconcile',
|
||||
scheduleId: DEPLOY_LOG_RECONCILE_ID,
|
||||
environmentId: null,
|
||||
entityName: 'Deploy log reconcile',
|
||||
triggeredBy,
|
||||
status: 'running'
|
||||
});
|
||||
|
||||
await updateScheduleExecution(execution.id, {
|
||||
startedAt: new Date().toISOString()
|
||||
});
|
||||
|
||||
const log = async (message: string) => {
|
||||
console.log(`[Deploy Log Reconcile] ${message}`);
|
||||
await appendScheduleExecutionLog(execution.id, `[${new Date().toISOString()}] ${message}`);
|
||||
};
|
||||
|
||||
try {
|
||||
const records = await getScheduleExecutionIdsByType('stack_deploy');
|
||||
|
||||
// F5 fix: group records by their OWN environment (via envDirName -- the same
|
||||
// directory-name mapping deploy-log-store.ts uses on the filesystem side), so
|
||||
// each environment's records are only ever compared against that SAME
|
||||
// environment's files below -- never another environment's.
|
||||
const recordsByEnvKey = new Map<string, typeof records>();
|
||||
for (const record of records) {
|
||||
const key = envDirName(record.environmentId);
|
||||
const list = recordsByEnvKey.get(key);
|
||||
if (list) list.push(record);
|
||||
else recordsByEnvKey.set(key, [record]);
|
||||
}
|
||||
|
||||
// The set of environments to actually visit: every environment that has a
|
||||
// directory on disk (covers orphan files for an environment that no longer
|
||||
// has ANY record, e.g. a deleted environment) UNION every environment that has
|
||||
// at least one record (covers a record whose file has never been written, so
|
||||
// there is no directory for listEnvDirNames() to discover it by -- see the
|
||||
// module doc comment).
|
||||
const envKeysOnDisk = await deps.listEnvDirNames();
|
||||
const envKeys = new Set<string>([...envKeysOnDisk, ...recordsByEnvKey.keys()]);
|
||||
|
||||
let filesFoundTotal = 0;
|
||||
// Actually-performed counts, not the plan's sizes -- a failing element must not
|
||||
// be counted as done, and must not stop the elements after it (in this SAME
|
||||
// environment, or in a LATER environment) from being tried.
|
||||
let deletedCount = 0;
|
||||
let markedCount = 0;
|
||||
let skippedInProgress = 0;
|
||||
let skippedRecent = 0;
|
||||
let failedCount = 0;
|
||||
|
||||
for (const envKey of envKeys) {
|
||||
let envId: number | null;
|
||||
try {
|
||||
envId = parseEnvDirName(envKey);
|
||||
} catch {
|
||||
// A directory under deploy-logs/ that isn't one of envDirName()'s own
|
||||
// outputs (e.g. an unrelated file or a hand-created directory) is not an
|
||||
// environment this job knows how to reconcile -- skipped defensively
|
||||
// rather than crashing the whole run over it.
|
||||
await log(`Skipping unrecognized deploy-logs entry: ${envKey}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const envRecords = recordsByEnvKey.get(envKey) ?? [];
|
||||
const recordIds = envRecords.map((r) => String(r.id));
|
||||
const fileIds = await deps.listRunLogIds(envId);
|
||||
filesFoundTotal += fileIds.length;
|
||||
|
||||
const plan = planReconcile({ fileIds, recordIds });
|
||||
|
||||
for (const fileId of plan.deleteFiles) {
|
||||
try {
|
||||
// TOCTOU guard (see module doc comment): `records` was snapshotted
|
||||
// once, before this loop started. A file this fresh may belong to a
|
||||
// deploy whose record was created AFTER that snapshot -- not to an
|
||||
// abandoned orphan. Leave it for a later run, which will see its
|
||||
// by-then-existing record and correctly never touch it again.
|
||||
const mtimeMs = await deps.getRunLogMtimeMs(envId, fileId);
|
||||
if (mtimeMs !== null && Date.now() - mtimeMs < ORPHAN_DELETE_GRACE_MS) {
|
||||
skippedRecent++;
|
||||
continue;
|
||||
}
|
||||
await deps.deleteRunLog(envId, fileId);
|
||||
deletedCount++;
|
||||
} catch (error: any) {
|
||||
failedCount++;
|
||||
await log(`Failed to delete orphan file for run ${fileId} (environment ${envKey}): ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
const byId = new Map(envRecords.map((r) => [String(r.id), r]));
|
||||
for (const recordId of plan.markMissing) {
|
||||
const record = byId.get(recordId);
|
||||
// Still running/queued: it may simply not have written its first line yet
|
||||
// (see module doc comment). Leave it alone -- a later reconcile run will
|
||||
// catch it correctly once the run has actually finished.
|
||||
if (record && !isEligibleForMissingMark(record.status)) {
|
||||
skippedInProgress++;
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
await updateScheduleExecution(Number(recordId), {
|
||||
details: { ...(record?.details ?? {}), logMissing: true }
|
||||
});
|
||||
markedCount++;
|
||||
} catch (error: any) {
|
||||
failedCount++;
|
||||
await log(`Failed to mark record ${recordId} as logMissing (environment ${envKey}): ${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await log(
|
||||
`Found ${filesFoundTotal} deploy-log file(s) across ${envKeys.size} environment(s) and ${records.length} stack_deploy record(s)`
|
||||
);
|
||||
await log(
|
||||
`Reconcile complete: ${deletedCount} orphan file(s) deleted, ${markedCount} record(s) marked logMissing` +
|
||||
(skippedInProgress > 0 ? `, ${skippedInProgress} still-running record(s) skipped` : '') +
|
||||
(skippedRecent > 0 ? `, ${skippedRecent} recently-modified file(s) skipped (possible in-flight deploy)` : '') +
|
||||
(failedCount > 0 ? `, ${failedCount} element(s) failed` : '')
|
||||
);
|
||||
await updateScheduleExecution(execution.id, {
|
||||
// A run that hit per-element failures did complete, but not cleanly -- it
|
||||
// should not read as indistinguishable from one that had nothing go wrong.
|
||||
// 'warning' is exactly this repo's convention for "completed with a caveat"
|
||||
// (see the ScheduleStatus doc comment in db.ts), as opposed to 'failed',
|
||||
// which here is reserved for the run itself throwing (the outer catch below).
|
||||
status: failedCount > 0 ? 'warning' : 'success',
|
||||
completedAt: new Date().toISOString(),
|
||||
duration: Date.now() - startTime,
|
||||
details: {
|
||||
filesFound: filesFoundTotal,
|
||||
recordsFound: records.length,
|
||||
deletedFiles: deletedCount,
|
||||
markedRecords: markedCount,
|
||||
skippedInProgress,
|
||||
skippedRecent,
|
||||
failed: failedCount
|
||||
}
|
||||
});
|
||||
} catch (error: any) {
|
||||
await log(`Error: ${error.message}`);
|
||||
await updateScheduleExecution(execution.id, {
|
||||
status: 'failed',
|
||||
completedAt: new Date().toISOString(),
|
||||
duration: Date.now() - startTime,
|
||||
errorMessage: error.message
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -48,7 +48,13 @@ export async function runGitStackSync(
|
||||
// Deploy the git stack (only if there are changes). deployGitStack now emits the
|
||||
// git_sync_success/failed/skipped notification itself, so EVERY caller (webhook,
|
||||
// manual, this scheduler) notifies uniformly — we no longer dispatch here (#1295).
|
||||
const result = await deployGitStack(stackId, { force: false });
|
||||
//
|
||||
// triggeredBy is forwarded, not hardcoded: this function is itself called with
|
||||
// 'cron' (the nightly scheduler), 'manual' (the schedules page's "run now"), and
|
||||
// 'webhook' (scheduler/index.ts's triggerGitStackSyncFromWebhook) — passing the
|
||||
// value straight through gives the stack_deploy run record the same accurate
|
||||
// trigger this execution's own git_stack_sync record already gets above.
|
||||
const result = await deployGitStack(stackId, { force: false, triggeredBy });
|
||||
|
||||
if (result.success) {
|
||||
if (result.skipped) {
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* Minimum length for a secret to be replaced in place. Below it, a value like "test"
|
||||
* would match inside unrelated words ("latest", "testing"), so the whole line is
|
||||
* withheld instead — we cannot safely tell signal from coincidence.
|
||||
*/
|
||||
const MIN_REPLACEABLE_LENGTH = 8;
|
||||
|
||||
/** Placeholder written in place of a secret value. */
|
||||
const PLACEHOLDER = '***';
|
||||
|
||||
/**
|
||||
* Removes known secret values from a single output line.
|
||||
*
|
||||
* This does NOT pattern-match "things that look like secrets" — that approach fails for
|
||||
* every field nobody thought of. It replaces the exact values we injected ourselves and
|
||||
* therefore know.
|
||||
*
|
||||
* Returns the redacted line, or null when the line must be withheld entirely.
|
||||
*/
|
||||
export function redactLine(line: string, secrets: string[]): string | null {
|
||||
let result = line;
|
||||
|
||||
for (const secret of secrets) {
|
||||
const value = secret?.trim();
|
||||
if (!value) continue;
|
||||
|
||||
if (value.length < MIN_REPLACEABLE_LENGTH) {
|
||||
// Too short to replace without hitting unrelated substrings. If it appears at all,
|
||||
// withhold the line: a false positive costs one hidden line, a false negative leaks.
|
||||
if (line.includes(value)) return null;
|
||||
continue;
|
||||
}
|
||||
|
||||
result = result.split(value).join(PLACEHOLDER);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Bridges a raw line callback to a consumer, applying redaction on the way.
|
||||
* Withheld lines (redactLine returning null) are dropped silently — by design:
|
||||
* a suppressed line is cheaper than a leaked one.
|
||||
*/
|
||||
export function makeLineForwarder(
|
||||
consume: (line: string) => void,
|
||||
secrets: string[]
|
||||
): (line: string) => void {
|
||||
return (line: string) => {
|
||||
const safe = redactLine(line, secrets);
|
||||
if (safe !== null) consume(safe);
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Surfaces a final output block as a single redacted line, but ONLY when no line was
|
||||
* forwarded during the run (lineCount === 0). Some execution paths (e.g. Hawser's REST
|
||||
* call) report only a block, never individual lines -- without this, the operator sees
|
||||
* nothing while the run is in progress. A path that already forwarded its own lines
|
||||
* (or a future streaming path) must not get the block appended on top: that would show
|
||||
* the same text twice.
|
||||
*/
|
||||
export function surfaceBlockIfNoLines(
|
||||
onLine: ((line: string) => void) | undefined,
|
||||
lineCount: number,
|
||||
block: string | undefined,
|
||||
secrets: string[]
|
||||
): void {
|
||||
if (onLine && lineCount === 0 && block) {
|
||||
makeLineForwarder(onLine, secrets)(block);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A redacting line sink for callers that must also decide, once the run is over, whether to
|
||||
* surface the final output block.
|
||||
*
|
||||
* The two halves belong together because surfaceBlockIfNoLines needs the number of lines that
|
||||
* were actually forwarded, and only the sink knows it. Counting what the sender emitted would
|
||||
* be wrong: a line withheld by redaction never reached the operator, so for them the block is
|
||||
* still the only thing there is to see.
|
||||
*
|
||||
* `forward` is undefined when there is no consumer, so callers can pass it straight into an
|
||||
* optional callback parameter.
|
||||
*/
|
||||
export function makeRedactedLineSink(
|
||||
onLine: ((line: string) => void) | undefined,
|
||||
secrets: string[]
|
||||
): {
|
||||
forward: ((line: string) => void) | undefined;
|
||||
surfaceBlock: (block: string | undefined) => void;
|
||||
} {
|
||||
let forwarded = 0;
|
||||
const forward = onLine
|
||||
? makeLineForwarder((line) => {
|
||||
forwarded++;
|
||||
onLine(line);
|
||||
}, secrets)
|
||||
: undefined;
|
||||
|
||||
return {
|
||||
forward,
|
||||
surfaceBlock: (block) => surfaceBlockIfNoLines(onLine, forwarded, block, secrets)
|
||||
};
|
||||
}
|
||||
+95
-5
@@ -1,10 +1,53 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { createJob, appendLine, completeJob, failJob } from '$lib/server/jobs';
|
||||
import { prefersJSON } from '$lib/server/sse-parser';
|
||||
import { shouldRecord } from '$lib/server/deploy-recorder';
|
||||
|
||||
// Re-export pure parsing utilities (no server deps) for backward compat
|
||||
export { prefersJSON, sseToJSON } from '$lib/server/sse-parser';
|
||||
|
||||
/**
|
||||
* Optional hook that mirrors qualifying progress lines into a persisted run record and
|
||||
* is told how the run ended once it's done.
|
||||
*
|
||||
* Built in the ROUTE (see Task 10's deploy-run-record.ts), never here: constructing it
|
||||
* needs db.ts, and importing that here would make this module unimportable under Bun
|
||||
* (ERR_DLOPEN_FAILED) -- breaking every unit test that exercises createJobResponse
|
||||
* directly, this file's own included.
|
||||
*/
|
||||
export interface RunRecorder {
|
||||
/** Called for every send() whose (event, data) passes shouldRecord -- never raw. */
|
||||
line(line: string): void;
|
||||
/**
|
||||
* Extends the redaction secret list with values that were not yet known when the
|
||||
* recorder was constructed -- e.g. provider-resolved secrets (Bitwarden/1Password/
|
||||
* etc. bulk pulls or inline refs) that deployStack() only resolves internally,
|
||||
* strictly AFTER the caller built this recorder from the DB-only vars it had at the
|
||||
* time. Callers MUST call this (if they have anything to add) before end(), which is
|
||||
* the only method that reads the accumulated secret list -- see deploy-run-record.ts.
|
||||
*/
|
||||
addSecrets(values: string[]): void;
|
||||
/** Called exactly once, on every path out of the operation (success, failure, or throw). */
|
||||
end(ok: boolean, exitCode?: number, error?: string): Promise<void>;
|
||||
}
|
||||
|
||||
/** Extracts the redacted line string from a shouldRecord-qualifying { type: 'line', line } payload. */
|
||||
function extractLine(data: unknown): string | undefined {
|
||||
if (typeof data !== 'object' || data === null) return undefined;
|
||||
const line = (data as { line?: unknown }).line;
|
||||
return typeof line === 'string' ? line : undefined;
|
||||
}
|
||||
|
||||
/** Derives the RunRecorder.end() args from the JSON path's held-back result payload. */
|
||||
function endFromResult(resultData: unknown): { ok: boolean; error?: string } {
|
||||
if (typeof resultData !== 'object' || resultData === null) return { ok: false };
|
||||
const obj = resultData as { success?: unknown; error?: unknown };
|
||||
if (obj.success === true) return { ok: true };
|
||||
const { error } = obj;
|
||||
if (error === undefined) return { ok: false };
|
||||
return { ok: false, error: typeof error === 'string' ? error : String(error) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Job-based response for long-running operations.
|
||||
*
|
||||
@@ -16,10 +59,17 @@ export { prefersJSON, sseToJSON } from '$lib/server/sse-parser';
|
||||
*
|
||||
* The send() callback stores lines with { event, data } so the polling client
|
||||
* can reconstruct the same event stream semantics used by the old SSE flow.
|
||||
*
|
||||
* `recorder` is optional and, if omitted, changes nothing about the above -- see
|
||||
* RunRecorder's doc comment for why it's built by the caller, not here. When given, it
|
||||
* is fed from BOTH send() implementations below (JSON path and streaming path) so that
|
||||
* every caller of createJobResponse, not just the ones on the streaming path, gets its
|
||||
* progress lines mirrored.
|
||||
*/
|
||||
export function createJobResponse(
|
||||
operation: (send: (event: string, data: unknown) => void, isCancelled: () => boolean) => Promise<void>,
|
||||
request?: Request
|
||||
request?: Request,
|
||||
recorder?: RunRecorder
|
||||
): Response {
|
||||
// Backward compat: synchronous JSON path for explicit application/json callers
|
||||
if (prefersJSON(request)) {
|
||||
@@ -29,6 +79,10 @@ export function createJobResponse(
|
||||
let resultData: unknown = { success: false, error: 'No result' };
|
||||
let sentResult = false;
|
||||
const send = (event: string, data: unknown) => {
|
||||
if (recorder && shouldRecord(event, data)) {
|
||||
const line = extractLine(data);
|
||||
if (line !== undefined) recorder.line(line);
|
||||
}
|
||||
// Keep the last 'result' payload as the response body. A handler
|
||||
// may send progress events and then throw (e.g. a backup that
|
||||
// returns { status: 'error' } and rethrows), so preserve the
|
||||
@@ -40,6 +94,14 @@ export function createJobResponse(
|
||||
await operation(send, () => false);
|
||||
} catch (error) {
|
||||
if (!sentResult) resultData = { success: false, error: String(error) };
|
||||
} finally {
|
||||
// Awaited here, before enqueue/close: this async start() is the
|
||||
// only place on the JSON path where "the run is over" exists as a
|
||||
// moment in time, so the recorder's close is placed right on it.
|
||||
if (recorder) {
|
||||
const { ok, error } = endFromResult(resultData);
|
||||
await recorder.end(ok, undefined, error);
|
||||
}
|
||||
}
|
||||
controller.enqueue(encoder.encode(JSON.stringify(resultData)));
|
||||
controller.close();
|
||||
@@ -54,16 +116,44 @@ export function createJobResponse(
|
||||
const job = createJob();
|
||||
|
||||
const send = (event: string, data: unknown) => {
|
||||
if (recorder && shouldRecord(event, data)) {
|
||||
const line = extractLine(data);
|
||||
if (line !== undefined) recorder.line(line);
|
||||
}
|
||||
appendLine(job, { event, data });
|
||||
};
|
||||
|
||||
// Guards RunRecorder.end()'s "called exactly once" contract: if the .then()
|
||||
// branch below reaches its recorder.end() call and that call itself throws,
|
||||
// the rejection lands in .catch(), which must not call end() a second time.
|
||||
// Set right before invoking end() (not after it resolves), so a throwing
|
||||
// end() is still counted as "attempted" and never retried.
|
||||
let recorderEnded = false;
|
||||
|
||||
operation(send, () => job.cancelRequested === true)
|
||||
.then(() => {
|
||||
.then(async () => {
|
||||
const resultLine = job.lines.findLast((l) => l.event === 'result');
|
||||
completeJob(job, resultLine?.data ?? { success: true });
|
||||
const resultData = resultLine?.data ?? { success: true };
|
||||
completeJob(job, resultData);
|
||||
// Nothing on this path awaits createJobResponse's return value (it already
|
||||
// returned { jobId } below), so this runs after the response has gone out.
|
||||
// The record is still closed reliably -- just not before the client sees it.
|
||||
//
|
||||
// A failed compose run does not throw: it sends `result: {success:false,
|
||||
// error}` and returns normally, landing HERE, not in .catch() below. This is
|
||||
// the path every browser call takes (neither +page.svelte nor StackModal.svelte
|
||||
// send an Accept header) -- reuse the JSON path's endFromResult() instead of
|
||||
// hardcoding end(true), or every such run was recorded as a success.
|
||||
if (recorder) {
|
||||
recorderEnded = true;
|
||||
const { ok, error } = endFromResult(resultData);
|
||||
await recorder.end(ok, undefined, error);
|
||||
}
|
||||
})
|
||||
.catch((err: unknown) => {
|
||||
failJob(job, err instanceof Error ? err.message : String(err));
|
||||
.catch(async (err: unknown) => {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
failJob(job, message);
|
||||
if (recorder && !recorderEnded) await recorder.end(false, undefined, message);
|
||||
});
|
||||
|
||||
return json({ jobId: job.id });
|
||||
|
||||
+110
-45
@@ -10,6 +10,8 @@ import { join, resolve, dirname, basename, isAbsolute, normalize as pathNormaliz
|
||||
import { spawn as nodeSpawn } from 'node:child_process';
|
||||
import type { ChildProcess } from 'node:child_process';
|
||||
import { redactSecretVars } from './secret-redact';
|
||||
import { collectProcess } from './process-output-core';
|
||||
import { makeLineForwarder, makeRedactedLineSink } from './secret-redaction';
|
||||
import {
|
||||
applyFileDeletions,
|
||||
hashDirFiles,
|
||||
@@ -89,6 +91,24 @@ export interface StackOperationResult {
|
||||
command?: string;
|
||||
/** Result of applying git deletion sync (files removed / kept, with reasons) */
|
||||
deletion?: DeletionApplyResult;
|
||||
/**
|
||||
* The process's real exit code, when one exists to report -- the local/direct
|
||||
* compose path runs the command itself and knows it. Left unset on a timeout
|
||||
* (the process was killed, not exited) and on the Hawser path (the agent
|
||||
* protocol doesn't return one). Callers needing an exit code regardless
|
||||
* (deploy-run-record.ts) fall back to a value consistent with success/failure.
|
||||
*/
|
||||
exitCode?: number;
|
||||
/**
|
||||
* Set by deployStack() only: every secret value (DB AND provider-resolved --
|
||||
* Bitwarden/1Password/etc. bulk pulls or inline refs, resolveProviderEnvVars) that
|
||||
* actually reached the container for THIS run. Provider resolution happens inside
|
||||
* deployStack(), after any caller-built stack_deploy run recorder was already
|
||||
* constructed from DB-only vars -- callers MUST feed this into the recorder via
|
||||
* RunRecorder.addSecrets() before closing it, or a provider-resolved secret that
|
||||
* surfaces in compose's raw error text is stored unredacted (see deploy-run-record.ts).
|
||||
*/
|
||||
resolvedSecrets?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -147,6 +167,8 @@ export interface DeployStackOptions {
|
||||
* (Stack events and Git sync are separate user-facing groups). stack_events is
|
||||
* still recorded regardless. (#1295) */
|
||||
isGitDeploy?: boolean;
|
||||
/** Optional callback invoked per redacted output line as the compose command runs. */
|
||||
onLine?: (line: string) => void;
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
@@ -242,25 +264,11 @@ function isBinaryContent(bytes: Uint8Array): boolean {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Collect stdout/stderr from a child process and wait for it to exit.
|
||||
*/
|
||||
function collectProcess(proc: ChildProcess): Promise<{ exitCode: number; stdout: string; stderr: string }> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const stdoutChunks: Buffer[] = [];
|
||||
const stderrChunks: Buffer[] = [];
|
||||
proc.stdout?.on('data', (chunk: Buffer) => stdoutChunks.push(chunk));
|
||||
proc.stderr?.on('data', (chunk: Buffer) => stderrChunks.push(chunk));
|
||||
proc.on('error', reject);
|
||||
proc.on('close', (code) => {
|
||||
resolve({
|
||||
exitCode: code ?? 1,
|
||||
stdout: Buffer.concat(stdoutChunks).toString(),
|
||||
stderr: Buffer.concat(stderrChunks).toString()
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
// collectProcess lives in ./process-output-core (imported above) -- pure,
|
||||
// dependency-free, so it stays unit-testable without dragging in the DB
|
||||
// module chain. Re-exported here so existing call sites (loginToRegistries,
|
||||
// executeLocalCompose) are unaffected.
|
||||
export { collectProcess };
|
||||
|
||||
/**
|
||||
* Read all files from a directory as a map of relative path -> content.
|
||||
@@ -1162,6 +1170,8 @@ function findComposeOverrideFile(stackDir: string, composeFileName: string): str
|
||||
* @param secretVars - Secret environment variables (injected via shell env, NEVER written to disk)
|
||||
* @param workingDir - Optional working directory for compose execution (for imported stacks)
|
||||
* @param customComposePath - Optional path to existing compose file (for imported stacks, skips writing)
|
||||
* @param onLine - Optional callback invoked per output line, redacted against envVars/secretVars
|
||||
* (NOT spawnEnv — that also carries PATH/HOME/DOCKER_CONFIG and would over-redact)
|
||||
*/
|
||||
async function executeLocalCompose(
|
||||
operation: 'up' | 'down' | 'stop' | 'start' | 'restart' | 'pull' | 'build',
|
||||
@@ -1185,7 +1195,8 @@ async function executeLocalCompose(
|
||||
// direct-remote only: when the stack folder was staged to <remoteStackHostDir> on the target
|
||||
// host, rewrite the compose's same-dir relative binds (`./x`) to <remoteStackHostDir>/x so the
|
||||
// remote daemon binds the staged files. undefined = no staging, compose unchanged.
|
||||
remoteStackHostDir?: string
|
||||
remoteStackHostDir?: string,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const logPrefix = `[Stack:${stackName}]`;
|
||||
|
||||
@@ -1481,7 +1492,15 @@ async function executeLocalCompose(
|
||||
}, COMPOSE_TIMEOUT_MS);
|
||||
|
||||
try {
|
||||
const { exitCode: code, stdout, stderr } = await collectProcess(proc);
|
||||
// Do NOT use spawnEnv! It also carries PATH, HOME, DOCKER_CONFIG, DOCKER_API_VERSION.
|
||||
// HOME typically falls back to "/root" -- 5 characters, below MIN_REPLACEABLE_LENGTH.
|
||||
// Under our own rule, that would withhold EVERY line containing "/root".
|
||||
const secrets = [...Object.values(envVars ?? {}), ...Object.values(secretVars ?? {})]
|
||||
.filter((v): v is string => typeof v === 'string');
|
||||
const { exitCode: code, stdout, stderr } = await collectProcess(
|
||||
proc,
|
||||
onLine ? makeLineForwarder(onLine, secrets) : undefined
|
||||
);
|
||||
|
||||
console.log(`${logPrefix} ----------------------------------------`);
|
||||
console.log(`${logPrefix} COMPOSE PROCESS COMPLETE`);
|
||||
@@ -1510,7 +1529,8 @@ async function executeLocalCompose(
|
||||
return {
|
||||
success: true,
|
||||
output: stdout || stderr || `Stack "${stackName}" ${operation} completed successfully`,
|
||||
command: commandStr
|
||||
command: commandStr,
|
||||
exitCode: code
|
||||
};
|
||||
} else {
|
||||
// stderr can echo an interpolated secret value (e.g. a failing
|
||||
@@ -1520,7 +1540,8 @@ async function executeLocalCompose(
|
||||
success: false,
|
||||
output: redactSecretVars(stdout, secretVars),
|
||||
error: redactSecretVars(stderr, secretVars) || `docker compose ${operation} exited with code ${code}`,
|
||||
command: commandStr
|
||||
command: commandStr,
|
||||
exitCode: code
|
||||
};
|
||||
}
|
||||
} finally {
|
||||
@@ -1562,6 +1583,12 @@ async function executeLocalCompose(
|
||||
*
|
||||
* @param envVars - Non-secret environment variables (from .env file)
|
||||
* @param secretVars - Secret environment variables (injected via shell env on Hawser, NEVER in .env)
|
||||
* @param onLine - Called per redacted output line while the command runs. Hawser's
|
||||
* `/_hawser/compose` call is a single request/response, but an agent that understands
|
||||
* `streamOutput` sends its output alongside it as 'stream' messages, which the Edge
|
||||
* connection routes back here by requestId. An older agent sends none; for it the
|
||||
* response's `output` block is surfaced as one line instead. Never both -- see
|
||||
* makeRedactedLineSink.
|
||||
*/
|
||||
async function executeComposeViaHawser(
|
||||
operation: 'up' | 'down' | 'stop' | 'start' | 'restart' | 'pull' | 'build',
|
||||
@@ -1579,7 +1606,8 @@ async function executeComposeViaHawser(
|
||||
noBuildCache?: boolean,
|
||||
pullPolicy?: string,
|
||||
filesToDelete?: FileToDelete[],
|
||||
removeFiles?: boolean
|
||||
removeFiles?: boolean,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const logPrefix = `[Stack:${stackName}]`;
|
||||
// Import dockerFetch dynamically to avoid circular dependency
|
||||
@@ -1589,6 +1617,10 @@ async function executeComposeViaHawser(
|
||||
// Hawser will inject ALL these as shell environment variables (secrets are NOT written to .env)
|
||||
const allEnvVars = { ...(envVars || {}), ...(secretVars || {}) };
|
||||
const secretCount = secretVars ? Object.keys(secretVars).length : 0;
|
||||
// Unlike spawnEnv on the local path, allEnvVars is genuinely just the stack's own
|
||||
// variables -- no PATH/HOME/DOCKER_CONFIG that would withhold half the output.
|
||||
const secrets = Object.values(allEnvVars).filter((v): v is string => typeof v === 'string');
|
||||
const lines = makeRedactedLineSink(onLine, secrets);
|
||||
|
||||
console.log(`${logPrefix} ----------------------------------------`);
|
||||
console.log(`${logPrefix} EXECUTE COMPOSE VIA HAWSER`);
|
||||
@@ -1663,7 +1695,10 @@ async function executeComposeViaHawser(
|
||||
? filesToDelete.map(f => ({ path: f.path, sha256: f.hash }))
|
||||
: undefined,
|
||||
// Stack deletion (#1162): remove the agent-side stack dir on down
|
||||
removeFiles: removeFiles || false
|
||||
removeFiles: removeFiles || false,
|
||||
// Ask the agent to also send its output line by line while the command runs.
|
||||
// Old agents ignore the field and just return the block as before.
|
||||
streamOutput: !!onLine
|
||||
});
|
||||
|
||||
console.log(`${logPrefix} Sending request to Hawser agent...`);
|
||||
@@ -1672,7 +1707,8 @@ async function executeComposeViaHawser(
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body
|
||||
body,
|
||||
onLine: lines.forward
|
||||
},
|
||||
envId
|
||||
);
|
||||
@@ -1696,6 +1732,10 @@ async function executeComposeViaHawser(
|
||||
console.log(`${logPrefix} Error:`, result.error);
|
||||
}
|
||||
|
||||
// Only reaches the operator when the agent streamed nothing -- otherwise they would
|
||||
// see the whole run a second time, appended to the lines they already watched.
|
||||
lines.surfaceBlock(result.output);
|
||||
|
||||
// Git deletion sync: interpret the agent's report. An agent that supports
|
||||
// the feature always returns deletedFiles/skippedFiles (possibly empty
|
||||
// arrays) when filesToDelete was sent. An old agent ignores the field and
|
||||
@@ -1762,13 +1802,16 @@ async function executeComposeViaHawser(
|
||||
*
|
||||
* @param envVars - Non-secret environment variables (from .env file)
|
||||
* @param secretVars - Secret environment variables (from DB, injected via shell env)
|
||||
* @param onLine - Optional callback invoked per redacted output line as the command runs.
|
||||
* Forwarded to whichever execution path is chosen (local socket, direct, or Hawser).
|
||||
*/
|
||||
async function executeComposeCommand(
|
||||
operation: 'up' | 'down' | 'stop' | 'start' | 'restart' | 'pull' | 'build',
|
||||
options: ComposeCommandOptions,
|
||||
composeContent: string,
|
||||
envVars?: Record<string, string>,
|
||||
secretVars?: Record<string, string>
|
||||
secretVars?: Record<string, string>,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const { stackName, envId, forceRecreate, build, noBuildCache, pullPolicy, removeVolumes, stackFiles, workingDir, composePath, envPath, useOverrideFile, serviceName, composeFileName, filesToDelete, removeFiles } = options;
|
||||
|
||||
@@ -1795,7 +1838,9 @@ async function executeComposeCommand(
|
||||
serviceName,
|
||||
build,
|
||||
noBuildCache,
|
||||
pullPolicy
|
||||
pullPolicy,
|
||||
undefined, // remoteStackHostDir
|
||||
onLine
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1861,7 +1906,8 @@ async function executeComposeCommand(
|
||||
noBuildCache,
|
||||
pullPolicy,
|
||||
filesToDelete,
|
||||
removeFiles
|
||||
removeFiles,
|
||||
onLine
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1924,7 +1970,8 @@ async function executeComposeCommand(
|
||||
build,
|
||||
noBuildCache,
|
||||
pullPolicy,
|
||||
remoteStackHostDir
|
||||
remoteStackHostDir,
|
||||
onLine
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1956,7 +2003,9 @@ async function executeComposeCommand(
|
||||
serviceName,
|
||||
build,
|
||||
noBuildCache,
|
||||
pullPolicy
|
||||
pullPolicy,
|
||||
undefined, // remoteStackHostDir
|
||||
onLine
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -2414,7 +2463,8 @@ async function notifyStackLifecycle(stackName: string, envId: number | null | un
|
||||
|
||||
export async function startStack(
|
||||
stackName: string,
|
||||
envId?: number | null
|
||||
envId?: number | null,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const result = await requireComposeFile(stackName, envId);
|
||||
|
||||
@@ -2447,7 +2497,8 @@ export async function startStack(
|
||||
opts,
|
||||
result.content!,
|
||||
result.nonSecretVars,
|
||||
result.secretVars
|
||||
result.secretVars,
|
||||
onLine
|
||||
);
|
||||
await notifyStackLifecycle(stackName, envId, 'stack_started', startResult);
|
||||
return startResult;
|
||||
@@ -2459,7 +2510,8 @@ export async function startStack(
|
||||
*/
|
||||
export async function stopStack(
|
||||
stackName: string,
|
||||
envId?: number | null
|
||||
envId?: number | null,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const result = await requireComposeFile(stackName, envId);
|
||||
|
||||
@@ -2481,7 +2533,8 @@ export async function stopStack(
|
||||
{ stackName, envId, workingDir: result.stackDir, composePath: result.composePath, envPath: result.envPath, useOverrideFile: isGitStack, stackFiles: await lifecycleStackFiles(result.stackDir) },
|
||||
result.content!,
|
||||
result.nonSecretVars,
|
||||
result.secretVars
|
||||
result.secretVars,
|
||||
onLine
|
||||
);
|
||||
|
||||
// Stop any dynamically-spawned child containers not in the compose file
|
||||
@@ -2507,7 +2560,8 @@ export async function stopStack(
|
||||
export async function restartStack(
|
||||
stackName: string,
|
||||
envId?: number | null,
|
||||
mode: 'restart' | 'ordered' | 'recreate' = 'restart'
|
||||
mode: 'restart' | 'ordered' | 'recreate' = 'restart',
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const result = await requireComposeFile(stackName, envId);
|
||||
|
||||
@@ -2528,16 +2582,16 @@ export async function restartStack(
|
||||
|
||||
if (mode === 'recreate') {
|
||||
// Stop first, then bring up with --force-recreate to ensure new container IDs
|
||||
await executeComposeCommand('stop', opts, result.content!, result.nonSecretVars, result.secretVars);
|
||||
await executeComposeCommand('stop', opts, result.content!, result.nonSecretVars, result.secretVars, onLine);
|
||||
await applyProviderSecretsToComposeResult(result, stackName, envId, `[Stack:${stackName}]`);
|
||||
composeResult = await executeComposeCommand('up', { ...opts, forceRecreate: true }, result.content!, result.nonSecretVars, result.secretVars);
|
||||
composeResult = await executeComposeCommand('up', { ...opts, forceRecreate: true }, result.content!, result.nonSecretVars, result.secretVars, onLine);
|
||||
} else if (mode === 'ordered') {
|
||||
// Stop everything, then start in depends_on order (compose start honors the
|
||||
// dependency graph). Same container IDs, no recreate, no re-pull.
|
||||
await executeComposeCommand('stop', opts, result.content!, result.nonSecretVars, result.secretVars);
|
||||
composeResult = await executeComposeCommand('start', opts, result.content!, result.nonSecretVars, result.secretVars);
|
||||
await executeComposeCommand('stop', opts, result.content!, result.nonSecretVars, result.secretVars, onLine);
|
||||
composeResult = await executeComposeCommand('start', opts, result.content!, result.nonSecretVars, result.secretVars, onLine);
|
||||
} else {
|
||||
composeResult = await executeComposeCommand('restart', opts, result.content!, result.nonSecretVars, result.secretVars);
|
||||
composeResult = await executeComposeCommand('restart', opts, result.content!, result.nonSecretVars, result.secretVars, onLine);
|
||||
}
|
||||
|
||||
// Restart any dynamically-spawned child containers not in the compose file
|
||||
@@ -2553,7 +2607,8 @@ export async function restartStack(
|
||||
export async function downStack(
|
||||
stackName: string,
|
||||
envId?: number | null,
|
||||
removeVolumes = false
|
||||
removeVolumes = false,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<StackOperationResult> {
|
||||
const result = await requireComposeFile(stackName, envId);
|
||||
|
||||
@@ -2571,7 +2626,8 @@ export async function downStack(
|
||||
{ stackName, envId, removeVolumes, workingDir: result.stackDir, composePath: result.composePath, envPath: result.envPath, useOverrideFile: isGitStack, stackFiles: await lifecycleStackFiles(result.stackDir) },
|
||||
result.content!,
|
||||
result.nonSecretVars,
|
||||
result.secretVars
|
||||
result.secretVars,
|
||||
onLine
|
||||
);
|
||||
|
||||
// Remove any dynamically-spawned child containers not in the compose file
|
||||
@@ -2946,7 +3002,7 @@ async function reconcileStackPendingUpdates(stackName: string, envId: number): P
|
||||
* Uses stack locking to prevent concurrent deployments.
|
||||
*/
|
||||
export async function deployStack(options: DeployStackOptions): Promise<StackOperationResult> {
|
||||
const { name, compose, envId, sourceDir, forceRecreate, build, noBuildCache, pullPolicy, composePath, envPath, composeFileName, envFileName, filesToDelete, isGitDeploy } = options;
|
||||
const { name, compose, envId, sourceDir, forceRecreate, build, noBuildCache, pullPolicy, composePath, envPath, composeFileName, envFileName, filesToDelete, isGitDeploy, onLine } = options;
|
||||
const logPrefix = `[Stack:${name}]`;
|
||||
|
||||
console.log(`${logPrefix} ========================================`);
|
||||
@@ -3178,8 +3234,17 @@ export async function deployStack(options: DeployStackOptions): Promise<StackOpe
|
||||
cmdOptions,
|
||||
compose,
|
||||
composeEnvVars,
|
||||
secretVars
|
||||
secretVars,
|
||||
onLine
|
||||
);
|
||||
// F4 fix: `secretVars` here is POST-resolveProviderEnvVars (line ~3059 above) --
|
||||
// the same set executeComposeCommand just redacted streamed lines against. This
|
||||
// is the single call site inside deployStack(), so setting it here covers both
|
||||
// the local/direct compose path and the Hawser path uniformly. Callers (routes,
|
||||
// deployGitStack) feed this into their stack_deploy run recorder via
|
||||
// RunRecorder.addSecrets() before closing it -- see StackOperationResult's doc
|
||||
// comment and deploy-run-record.ts.
|
||||
result.resolvedSecrets = Object.values(secretVars);
|
||||
console.log(`${logPrefix} ========================================`);
|
||||
console.log(`${logPrefix} DEPLOY STACK RESULT`);
|
||||
console.log(`${logPrefix} ========================================`);
|
||||
|
||||
@@ -32,6 +32,8 @@ export interface AppSettings {
|
||||
eventCleanupEnabled: boolean;
|
||||
scannerCleanupCron: string;
|
||||
scannerCleanupEnabled: boolean;
|
||||
deployLogReconcileCron: string;
|
||||
deployLogReconcileEnabled: boolean;
|
||||
logBufferSizeKb: number; // legacy, retained for migration — UI uses logMaxLines
|
||||
logMaxLines: number; // line-count cap for the log buffer (replaces KB-based limit)
|
||||
defaultTimezone: string;
|
||||
@@ -77,6 +79,8 @@ const DEFAULT_SETTINGS: AppSettings = {
|
||||
eventCleanupEnabled: true,
|
||||
scannerCleanupCron: '0 3 * * 0',
|
||||
scannerCleanupEnabled: true,
|
||||
deployLogReconcileCron: '0 4 * * *',
|
||||
deployLogReconcileEnabled: true,
|
||||
logBufferSizeKb: 500,
|
||||
logMaxLines: 2000,
|
||||
defaultTimezone: 'UTC',
|
||||
@@ -167,6 +171,8 @@ function createSettingsStore() {
|
||||
eventCleanupEnabled: settings.eventCleanupEnabled ?? DEFAULT_SETTINGS.eventCleanupEnabled,
|
||||
scannerCleanupCron: settings.scannerCleanupCron ?? DEFAULT_SETTINGS.scannerCleanupCron,
|
||||
scannerCleanupEnabled: settings.scannerCleanupEnabled ?? DEFAULT_SETTINGS.scannerCleanupEnabled,
|
||||
deployLogReconcileCron: settings.deployLogReconcileCron ?? DEFAULT_SETTINGS.deployLogReconcileCron,
|
||||
deployLogReconcileEnabled: settings.deployLogReconcileEnabled ?? DEFAULT_SETTINGS.deployLogReconcileEnabled,
|
||||
logBufferSizeKb: settings.logBufferSizeKb ?? DEFAULT_SETTINGS.logBufferSizeKb,
|
||||
logMaxLines: deriveLogMaxLines(settings),
|
||||
defaultTimezone: settings.defaultTimezone ?? DEFAULT_SETTINGS.defaultTimezone,
|
||||
@@ -227,6 +233,8 @@ function createSettingsStore() {
|
||||
eventCleanupEnabled: updatedSettings.eventCleanupEnabled ?? DEFAULT_SETTINGS.eventCleanupEnabled,
|
||||
scannerCleanupCron: updatedSettings.scannerCleanupCron ?? DEFAULT_SETTINGS.scannerCleanupCron,
|
||||
scannerCleanupEnabled: updatedSettings.scannerCleanupEnabled ?? DEFAULT_SETTINGS.scannerCleanupEnabled,
|
||||
deployLogReconcileCron: updatedSettings.deployLogReconcileCron ?? DEFAULT_SETTINGS.deployLogReconcileCron,
|
||||
deployLogReconcileEnabled: updatedSettings.deployLogReconcileEnabled ?? DEFAULT_SETTINGS.deployLogReconcileEnabled,
|
||||
logBufferSizeKb: updatedSettings.logBufferSizeKb ?? DEFAULT_SETTINGS.logBufferSizeKb,
|
||||
logMaxLines: deriveLogMaxLines(updatedSettings),
|
||||
defaultTimezone: updatedSettings.defaultTimezone ?? DEFAULT_SETTINGS.defaultTimezone,
|
||||
@@ -416,6 +424,20 @@ function createSettingsStore() {
|
||||
return newSettings;
|
||||
});
|
||||
},
|
||||
setDeployLogReconcileCron: (value: string) => {
|
||||
update((current) => {
|
||||
const newSettings = { ...current, deployLogReconcileCron: value };
|
||||
saveSettings({ deployLogReconcileCron: value });
|
||||
return newSettings;
|
||||
});
|
||||
},
|
||||
setDeployLogReconcileEnabled: (value: boolean) => {
|
||||
update((current) => {
|
||||
const newSettings = { ...current, deployLogReconcileEnabled: value };
|
||||
saveSettings({ deployLogReconcileEnabled: value });
|
||||
return newSettings;
|
||||
});
|
||||
},
|
||||
setLogBufferSizeKb: (value: number) => {
|
||||
update((current) => {
|
||||
const newSettings = { ...current, logBufferSizeKb: value };
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* Clamps a proposed number into `[min, max]`.
|
||||
*
|
||||
* A naive `Math.max(min, Math.min(max, value))` looks like it clamps any input, but
|
||||
* it does not: if `value` is `NaN`, both `Math.min` and `Math.max` propagate the NaN
|
||||
* -- any NaN operand makes them return NaN, regardless of the other operand -- so the
|
||||
* "clamped" result is still NaN. That NaN then ends up as a CSS `height: NaN%` (or
|
||||
* `width: NaNpx`), which the browser treats as invalid and the panel collapses.
|
||||
*
|
||||
* This is not a theoretical edge case for a drag-resize handler: the proposed value is
|
||||
* usually computed from `(cursorPosition - containerEdge) / containerSize` (as a
|
||||
* percentage) or `containerEdge - cursorPosition` (as a pixel distance), and a
|
||||
* container's measured size is briefly `0` while a panel is transitioning (e.g. a
|
||||
* panel appearing/disappearing) -- dividing by that zero produces NaN either way.
|
||||
* `min`/`max` may be a percentage range (e.g. 15-70 for a split ratio) or a pixel
|
||||
* range (e.g. 320-560 for a panel width) -- the clamp itself does not care about the
|
||||
* unit, only about ordering `value` against two bounds, so both use it unchanged.
|
||||
*
|
||||
* `Infinity`/`-Infinity` do not need the same special-casing: they are well-ordered
|
||||
* numbers, so `Math.min`/`Math.max` clamp them to `max`/`min` correctly on their own.
|
||||
* Only `NaN` breaks the ordinary clamp.
|
||||
*/
|
||||
export function clampNumber(value: number, min: number, max: number): number {
|
||||
if (Number.isNaN(value)) {
|
||||
return min;
|
||||
}
|
||||
return Math.max(min, Math.min(max, value));
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import yaml from 'js-yaml';
|
||||
|
||||
/**
|
||||
* Whether a compose file declares a `build:` section for any service -- used to
|
||||
* pre-check "Build images" in the Save & redeploy / Create & Start popover
|
||||
* (RedeployPopover's defaultBuild).
|
||||
*
|
||||
* `build:` can be a bare string (`build: ./dir`) or an object
|
||||
* (`build: {context: ., dockerfile: ...}`) -- this checks truthiness, the same way
|
||||
* ComposeGraphViewer.svelte's `config.image || (config.build ? 'build' : 'custom')`
|
||||
* already does, not `typeof`, so both forms are detected equally.
|
||||
*
|
||||
* Parse errors (mid-edit, invalid YAML) fall back to false rather than throwing: an
|
||||
* unparseable draft shouldn't crash the checkbox, and "Build images" can still be
|
||||
* checked manually while the operator is still typing.
|
||||
*/
|
||||
export function hasBuildSection(composeContent: string): boolean {
|
||||
try {
|
||||
const parsed = yaml.load(composeContent) as { services?: Record<string, { build?: unknown }> } | undefined;
|
||||
const services = parsed?.services;
|
||||
if (!services || typeof services !== 'object') return false;
|
||||
return Object.values(services).some((svc) => !!svc?.build);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
/**
|
||||
* Turns one recorded stack-deploy run (GET /api/stacks/{name}/deploys) into the
|
||||
* display strings the Deploys tab shows for it. Every function here is pure text
|
||||
* composition -- no fetch, no store, no formatting that depends on user settings
|
||||
* (relative/exact timestamps stay in the component, which already has
|
||||
* formatRelativeTime/formatDateTime available for that).
|
||||
*
|
||||
* Deliberately dependency-free (mirrors deploy-summary-core.ts / run-status.ts):
|
||||
* no import from $lib/server (better-sqlite3 isn't importable under Bun, and would
|
||||
* drag down every other test file sharing this process) and no import from
|
||||
* $lib/stores or $app -- those need a mounted Svelte/SvelteKit runtime this test
|
||||
* file doesn't have. The shapes below intentionally mirror (not import)
|
||||
* deploy-run-record-core.ts / deploy-summary-core.ts on the server: client code
|
||||
* never imports $lib/server, even for types (see StackModal.svelte's own local
|
||||
* BackupConfig for the same pattern).
|
||||
*/
|
||||
|
||||
export interface DeployRunOptions {
|
||||
pull: boolean;
|
||||
build: boolean;
|
||||
forceRecreate: boolean;
|
||||
}
|
||||
|
||||
export interface DeployRunSummary {
|
||||
containersCreated: number;
|
||||
containersRecreated: number;
|
||||
containersStarted: number;
|
||||
imagesBuilt: string[];
|
||||
imagesPulled: string[];
|
||||
buildSteps: number;
|
||||
buildStepsCached: number;
|
||||
digest?: string;
|
||||
}
|
||||
|
||||
export interface DeployRunDetails {
|
||||
options?: DeployRunOptions;
|
||||
summary?: DeployRunSummary;
|
||||
/** Set when the run's output hit deploy-log-store's size budget and was cut off. */
|
||||
truncated?: boolean;
|
||||
/** Set by deploy-log-reconcile once it confirms the on-disk log file is gone
|
||||
* (see deploy-log-reconcile.ts) -- never cleared, and never set for a run
|
||||
* that is still in progress (the reconcile job skips those). */
|
||||
logMissing?: boolean;
|
||||
}
|
||||
|
||||
export interface DeployRun {
|
||||
id: number;
|
||||
triggeredBy: string;
|
||||
/** Null for a run that never got past the queued state. Display-only here --
|
||||
* formatting (relative + exact) stays in the component, which already has
|
||||
* the user's time-format settings available for it. */
|
||||
startedAt?: string | null;
|
||||
duration: number | null;
|
||||
status: string;
|
||||
errorMessage?: string | null;
|
||||
details?: DeployRunDetails | null;
|
||||
}
|
||||
|
||||
/** Everything the Deploys tab renders for one run, pre-composed so the component
|
||||
* only ever binds -- it never branches on raw run/details/summary shape itself. */
|
||||
export interface DeployRunView {
|
||||
statusLabel: string;
|
||||
trigger: string;
|
||||
duration: string;
|
||||
containerSummary: string;
|
||||
buildStatus: string;
|
||||
optionChips: string[];
|
||||
imagesBuilt: string[];
|
||||
imagesPulled: string[];
|
||||
/** Shortened last line of errorMessage -- only ever set for a failed run. */
|
||||
errorSummary: string | null;
|
||||
truncated: boolean;
|
||||
}
|
||||
|
||||
export function formatStatusLabel(status: string): string {
|
||||
switch (status) {
|
||||
case 'success':
|
||||
return 'Success';
|
||||
case 'failed':
|
||||
return 'Failed';
|
||||
case 'running':
|
||||
return 'Running';
|
||||
default:
|
||||
// An unrecognized status is a schema drift worth seeing as-is, not one
|
||||
// more row silently rendered "Success" or hidden entirely.
|
||||
return status;
|
||||
}
|
||||
}
|
||||
|
||||
export function formatTrigger(triggeredBy: string): string {
|
||||
switch (triggeredBy) {
|
||||
case 'cron':
|
||||
return 'Scheduled';
|
||||
case 'webhook':
|
||||
return 'Webhook';
|
||||
case 'startup':
|
||||
return 'Startup';
|
||||
case 'manual':
|
||||
return 'Manual';
|
||||
default:
|
||||
return triggeredBy;
|
||||
}
|
||||
}
|
||||
|
||||
/** Mirrors ExecutionHistoryList.svelte's local formatDuration so the two run-history
|
||||
* surfaces (backups, deploys) read the same way. Kept here (not imported from
|
||||
* there) because that one is a component-local, unexported function. */
|
||||
export function formatRunDuration(ms: number | null | undefined): string {
|
||||
if (ms === null || ms === undefined) return '—';
|
||||
if (ms < 1000) return `${ms}ms`;
|
||||
if (ms < 60000) return `${(ms / 1000).toFixed(1)}s`;
|
||||
const totalSeconds = Math.floor(ms / 1000);
|
||||
const minutes = Math.floor(totalSeconds / 60);
|
||||
const seconds = totalSeconds % 60;
|
||||
return `${minutes}m ${seconds}s`;
|
||||
}
|
||||
|
||||
/**
|
||||
* A run with NO summary at all (compose produced zero output lines -- see
|
||||
* deploy-run-record.ts: `summary: this.lines.length > 0 ? summarize(this.lines) :
|
||||
* undefined`) is not the same fact as a run whose summary says every count is
|
||||
* zero. The first means "we don't know what happened"; the second means "we know,
|
||||
* and nothing changed". Collapsing them into one "0 containers" string would
|
||||
* report the first case as a fact it never established.
|
||||
*/
|
||||
export function formatContainerSummary(summary: DeployRunSummary | undefined): string {
|
||||
if (!summary) return 'No summary recorded for this run';
|
||||
const parts: string[] = [];
|
||||
if (summary.containersCreated > 0) parts.push(`${summary.containersCreated} created`);
|
||||
if (summary.containersRecreated > 0) parts.push(`${summary.containersRecreated} recreated`);
|
||||
if (summary.containersStarted > 0) parts.push(`${summary.containersStarted} started`);
|
||||
return parts.length > 0 ? parts.join(', ') : 'No container changes';
|
||||
}
|
||||
|
||||
/**
|
||||
* "Was anything built, and how much of that came from the build cache" -- the one
|
||||
* question this whole feature was built to answer (see the task brief). Same
|
||||
* unknown-vs-zero distinction as formatContainerSummary: no summary reads as
|
||||
* "unknown", a summary with buildSteps === 0 reads as "nothing built" -- never the
|
||||
* same string.
|
||||
*/
|
||||
export function formatBuildStatus(summary: DeployRunSummary | undefined): string {
|
||||
if (!summary) return 'Build status unknown';
|
||||
const { buildSteps, buildStepsCached } = summary;
|
||||
if (buildSteps === 0) return 'Nothing built';
|
||||
if (buildStepsCached === buildSteps) {
|
||||
return buildSteps === 1 ? 'Built from cache' : `Built from cache (${buildSteps} steps)`;
|
||||
}
|
||||
if (buildStepsCached === 0) {
|
||||
return buildSteps === 1 ? 'Built (1 step)' : `Built (${buildSteps} steps)`;
|
||||
}
|
||||
return `Built (${buildSteps} steps, ${buildStepsCached} from cache)`;
|
||||
}
|
||||
|
||||
export function formatOptionChips(options: DeployRunOptions | undefined): string[] {
|
||||
if (!options) return [];
|
||||
const chips: string[] = [];
|
||||
if (options.pull) chips.push('Pull');
|
||||
if (options.build) chips.push('Build');
|
||||
if (options.forceRecreate) chips.push('Force recreate');
|
||||
return chips;
|
||||
}
|
||||
|
||||
/**
|
||||
* The last non-empty line of a (possibly multi-line) error, cut to maxLen with a
|
||||
* trailing "…" marker. errorMessage is compose's raw stdout/stderr tail (already
|
||||
* redacted server-side, see deploy-run-record.ts) -- often several lines of
|
||||
* progress before the actual failure, so the LAST line is the one worth showing
|
||||
* collapsed; the full text stays available via the run's log file.
|
||||
*/
|
||||
export function lastErrorLine(errorMessage: string | null | undefined, maxLen = 200): string | null {
|
||||
if (!errorMessage) return null;
|
||||
const lines = errorMessage
|
||||
.split('\n')
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => l.length > 0);
|
||||
const last = lines.length > 0 ? lines[lines.length - 1] : errorMessage.trim();
|
||||
if (last.length === 0) return null;
|
||||
if (last.length <= maxLen) return last;
|
||||
return `${last.slice(0, maxLen - 1)}…`;
|
||||
}
|
||||
|
||||
/**
|
||||
* What the Deploys tab's expanded row shows for a run's LOG PANEL specifically --
|
||||
* a separate, narrower question from buildDeployRunView's "how does the row
|
||||
* summarize" above it. Three independent facts, each driving its own piece of
|
||||
* UI (see the component): whether the log is worth fetching at all, whether a
|
||||
* truncation notice belongs above it, and whether a delete action makes sense
|
||||
* for a run that might still be writing to that very file.
|
||||
*
|
||||
* logMissing short-circuits the fetch entirely -- a run already marked missing
|
||||
* by deploy-log-reconcile will 404 on GET .../log every time, so there is no
|
||||
* reason to make that round trip just to learn what the record already says.
|
||||
*/
|
||||
export interface DeployLogPanelState {
|
||||
/** True when the record already knows the log file is gone (details.logMissing).
|
||||
* The panel shows "log no longer available" and never fires a GET for it. */
|
||||
logMissing: boolean;
|
||||
/** True when the stored log was cut off at deploy-log-store's size budget --
|
||||
* same underlying flag DeployRunView.truncated surfaces for the collapsed
|
||||
* row, repeated here so the log panel doesn't need the raw run for it too. */
|
||||
truncated: boolean;
|
||||
/** False for a run still in progress. Its log file may still be growing on
|
||||
* disk, and deleting the record out from under an active write would race
|
||||
* the writer -- so no delete action is offered until the run has settled. */
|
||||
deletable: boolean;
|
||||
}
|
||||
|
||||
export function buildDeployLogPanelState(run: DeployRun): DeployLogPanelState {
|
||||
return {
|
||||
logMissing: run.details?.logMissing === true,
|
||||
truncated: run.details?.truncated === true,
|
||||
deletable: run.status !== 'running'
|
||||
};
|
||||
}
|
||||
|
||||
export function buildDeployRunView(run: DeployRun): DeployRunView {
|
||||
const summary = run.details?.summary;
|
||||
return {
|
||||
statusLabel: formatStatusLabel(run.status),
|
||||
trigger: formatTrigger(run.triggeredBy),
|
||||
duration: formatRunDuration(run.duration),
|
||||
containerSummary: formatContainerSummary(summary),
|
||||
buildStatus: formatBuildStatus(summary),
|
||||
optionChips: formatOptionChips(run.details?.options),
|
||||
imagesBuilt: summary?.imagesBuilt ?? [],
|
||||
imagesPulled: summary?.imagesPulled ?? [],
|
||||
errorSummary: run.status === 'failed' ? lastErrorLine(run.errorMessage) : null,
|
||||
truncated: run.details?.truncated === true
|
||||
};
|
||||
}
|
||||
+10
-1
@@ -1,3 +1,5 @@
|
||||
import { parseTimestamp } from './date-format';
|
||||
|
||||
/**
|
||||
* Format a byte count into a human-readable string.
|
||||
*/
|
||||
@@ -36,9 +38,16 @@ export function formatErrorLines(msg: string | null | undefined): string {
|
||||
* Compact relative time like "just now", "5m ago", "2d ago", "3mo ago". Returns ''
|
||||
* for an unparseable/NaN date and "in the future" for a timestamp ahead of now.
|
||||
* Kept in this import-light module so it is unit-testable; stores/settings re-exports it.
|
||||
*
|
||||
* Uses parseTimestamp so a naive server timestamp (PG `timestamp without time
|
||||
* zone`, e.g. "2026-08-30 20:36:11.534") is read as UTC instead of the
|
||||
* browser's local zone - see date-format.ts and #1183. Every caller of this
|
||||
* function passes exactly that kind of raw server value alongside
|
||||
* formatDateTime(), which already goes through parseTimestamp(); this kept
|
||||
* the two out of sync until now.
|
||||
*/
|
||||
export function formatRelativeTime(date: Date | string | number): string {
|
||||
const d = date instanceof Date ? date : new Date(date);
|
||||
const d = parseTimestamp(date);
|
||||
const ms = d.getTime();
|
||||
if (isNaN(ms)) return '';
|
||||
const sec = Math.round((Date.now() - ms) / 1000);
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
/**
|
||||
* Builds the file name offered when downloading a log. The title is caller-supplied
|
||||
* (a container name today, a stack name tomorrow), so path separators are stripped:
|
||||
* a download name is not a path, and nothing good comes of letting one look like one.
|
||||
*/
|
||||
export function downloadFileName(title: string | undefined): string {
|
||||
const safe = (title ?? '').replace(/[/\\]/g, '').replace(/\.\./g, '').trim();
|
||||
return safe ? `${safe}-logs.txt` : 'logs.txt';
|
||||
}
|
||||
|
||||
// A CSI sequence is ESC [ , then parameter bytes (0x30-0x3F, which includes '?' for
|
||||
// private modes), then intermediate bytes (0x20-0x2F), then one final byte (0x40-0x7E).
|
||||
// Matching the full grammar rather than just digits and semicolons matters here: compose's
|
||||
// progress display hides and shows the cursor with ESC[?25l / ESC[?25h on every redraw, and
|
||||
// '?' is a parameter byte. A pattern that misses those leaves them in the very output this
|
||||
// viewer exists to show.
|
||||
// eslint-disable-next-line no-control-regex
|
||||
const ANSI_PATTERN = /\u001b\[[0-?]*[ -\/]*[@-~]/g;
|
||||
|
||||
/**
|
||||
* Strips ANSI escape sequences (color codes, cursor movement, etc.) from log text.
|
||||
* Copied or downloaded logs are read outside a terminal -- a `.log` file full of
|
||||
* `\u001b[32m` sequences is unreadable with `cat` and unusable in a diff or grep.
|
||||
*/
|
||||
export function stripAnsi(text: string): string {
|
||||
return text.replace(ANSI_PATTERN, '');
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
/**
|
||||
* Formats the status line shown under a compose action's live output window.
|
||||
*
|
||||
* Pure and import-free on purpose (see run-status.test.ts / bun test tests/ —
|
||||
* modules pulled in from src/lib/server or SvelteKit-generated code don't load
|
||||
* under Bun's test runner). Keep it that way: no imports here.
|
||||
*/
|
||||
export interface RunStatus {
|
||||
running: boolean;
|
||||
ok?: boolean;
|
||||
ms?: number;
|
||||
exitCode?: number;
|
||||
}
|
||||
|
||||
export function formatRunStatus(status: RunStatus): string {
|
||||
if (status.running) return 'Running…';
|
||||
|
||||
const seconds = typeof status.ms === 'number' ? `${(status.ms / 1000).toFixed(1)}s` : undefined;
|
||||
|
||||
if (status.ok) {
|
||||
return seconds ? `Succeeded · ${seconds}` : 'Succeeded';
|
||||
}
|
||||
|
||||
const exitPart = typeof status.exitCode === 'number' ? `exit ${status.exitCode}` : undefined;
|
||||
const parts = ['Failed', exitPart, seconds].filter((p): p is string => p !== undefined);
|
||||
return parts.join(' · ');
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* Whether and when the stack modal should close itself after a save/deploy settles.
|
||||
*
|
||||
* Superseded design note: an earlier version of this function only distinguished
|
||||
* "deployed or not" and kept the modal open for every deploy, on the reasoning that
|
||||
* closing would throw away the compose output now rendered below the editor. The
|
||||
* operator saw that in practice (30.08.2026) and did not want it: a *successful*
|
||||
* deploy leaving the modal open forever, needing a manual close every time, was as
|
||||
* wrong as the previous behaviour of always closing regardless of outcome. Both
|
||||
* pre-existed this file; this is the fix for both, decided by the operator, not
|
||||
* inferred from the code.
|
||||
*/
|
||||
export type SaveCloseTiming = 'close' | 'close-delayed' | 'stay-open';
|
||||
|
||||
/**
|
||||
* - A plain save (`deployed` false) has nothing new to show, so it closes -- same as
|
||||
* before this file existed. The caller keeps its own short flash-then-close delay
|
||||
* (500ms) for this case; this function only says "close", not how soon.
|
||||
* - A deploy that succeeded (`deployed` true, `ok` true) closes too, but only after a
|
||||
* longer delay, so the success and the compose output are still glimpsed before the
|
||||
* modal goes away. Trade-off accepted knowingly, not an oversight: that output is
|
||||
* then gone until the deploy-history view (Milestone 3) exists to show it again.
|
||||
* - A deploy that failed (`deployed` true, `ok` false) stays open, full stop -- the
|
||||
* output and the error dialog on top of it are exactly what the operator needs to
|
||||
* read and act on.
|
||||
*
|
||||
* `ok` is meaningless when `deployed` is false (a plain save has no deploy result to
|
||||
* be ok or not) -- callers may pass anything there, it is never read for that case.
|
||||
*/
|
||||
export function saveCloseTiming(deployed: boolean, ok: boolean): SaveCloseTiming {
|
||||
if (!deployed) return 'close';
|
||||
return ok ? 'close-delayed' : 'stay-open';
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Whether a scrollable element counts as "at the bottom" for auto-scroll purposes.
|
||||
*
|
||||
* Used both to decide whether to pause auto-scroll (the user scrolled up to read
|
||||
* earlier output) and to decide when to resume it (the user scrolled back down).
|
||||
* A small threshold rather than an exact match: sub-pixel scroll positions and
|
||||
* fractional zoom levels mean scrollHeight - scrollTop - clientHeight is rarely
|
||||
* ever exactly 0 even when the element visually is at the end.
|
||||
*/
|
||||
export function isScrolledToBottom(
|
||||
scrollTop: number,
|
||||
scrollHeight: number,
|
||||
clientHeight: number,
|
||||
threshold = 50
|
||||
): boolean {
|
||||
return scrollHeight - scrollTop - clientHeight <= threshold;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a scroll-up pause should be lifted because a fresh run is starting.
|
||||
*
|
||||
* autoScroll going from false to true is that signal (a stack redeploy sets
|
||||
* `running = true`, which is what LogViewer's autoScroll prop is bound to, at
|
||||
* the exact moment its output is cleared and a new run begins) -- so a viewer
|
||||
* who scrolled up to read one run's output gets auto-scroll back for the next
|
||||
* one, rather than it staying paused for the rest of the page session.
|
||||
*
|
||||
* Deliberately an edge (false -> true), not a level (autoScroll is true): a
|
||||
* naive "reset whenever autoScroll is true" reads as similar but re-lifts the
|
||||
* pause on every single update while a run is in progress, which defeats the
|
||||
* pause entirely.
|
||||
*/
|
||||
export function shouldResetScrollPause(previousAutoScroll: boolean, autoScroll: boolean): boolean {
|
||||
return autoScroll && !previousAutoScroll;
|
||||
}
|
||||
@@ -5,9 +5,12 @@ import type { JobLine } from '$lib/server/jobs';
|
||||
* Drop-in replacement for readSSEResponse when the endpoint has been migrated to jobs.
|
||||
*
|
||||
* Returns the job's final result (equivalent to the 'result' event data in SSE).
|
||||
* With onLine, each 'progress' line message ({type:'line', line}) is reported as it
|
||||
* arrives — callers that don't need live output can omit it.
|
||||
*/
|
||||
export async function readJobResponse(
|
||||
response: Response
|
||||
response: Response,
|
||||
onLine?: (line: string) => void
|
||||
): Promise<{ success?: boolean; error?: string; [key: string]: unknown }> {
|
||||
// Fall through for non-JSON or error responses
|
||||
if (!response.ok) {
|
||||
@@ -22,8 +25,10 @@ export async function readJobResponse(
|
||||
|
||||
// If the response is a { jobId } shape, poll the job endpoint
|
||||
if (data && typeof data === 'object' && 'jobId' in data) {
|
||||
const result = await watchJob(data.jobId as string, () => {
|
||||
// readJobResponse callers don't need line-by-line updates
|
||||
const result = await watchJob(data.jobId as string, (msg) => {
|
||||
if (!onLine) return;
|
||||
const payload = msg?.data as { type?: string; line?: string } | undefined;
|
||||
if (payload?.type === 'line' && typeof payload.line === 'string') onLine(payload.line);
|
||||
});
|
||||
return result as { success?: boolean; error?: string; [key: string]: unknown };
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user