Initial text cleanup

This commit is contained in:
2026-08-19 12:55:19 -04:00
parent 819a4cce7a
commit cc84d70d52
26 changed files with 454 additions and 457 deletions

View File

@@ -5,7 +5,7 @@
# cannot even approximate it: every in-flight file sits in `spool/` regardless of
# how far it has got, because stages route by enqueueing a reference rather than
# by moving bytes (src/worker.jl header). A file's stage is a property of the
# queue holding it, so only the pipeline itself can see it — there is no
# queue holding it, so only the pipeline itself can see it. There is no
# directory to poll.
#
# So each stage keeps four counters, all incremented in `worker_loop` (the one
@@ -23,15 +23,15 @@
# utilization = (Δbusy_ns - Δblocked_ns) / (Δt * workers) ~1.0 ⇒ bottleneck
#
# Utilization is the useful one. Throughput alone can't distinguish a stage that
# is saturated from one that is merely starved by the stage ahead of it — both
# show the same files/s — whereas utilization separates them: the saturated stage
# is saturated from one that is merely starved by the stage ahead of it: both
# show the same files/s. Utilization separates them: the saturated stage
# sits near 1.0 with its queue backing up, the starved stage sits near 0.
#
# `blocked_ns` is what keeps that true downstream. Stages 1 and 3 apply blocking
# backpressure: when the next queue is full the handler parks and retries rather
# than dropping the file (see ROUTE_ENQUEUE_RETRY_SECONDS). That parked time is
# inside the handler, so counting it as busy would show stage 1 pinned at 1.0
# whenever stage 2 is the real bottleneck — every stage upstream of the jam would
# whenever stage 2 is the real bottleneck, and every stage upstream of the jam would
# look like the jam. Subtracting it leaves utilization meaning "doing its own
# work", and the blocked share becomes its own signal: a stage blocked 90% of the
# time is naming its successor as the bottleneck.
@@ -41,7 +41,7 @@
# a scrape is stateless and two readers can't disturb each other.
#
# Cost is a handful of atomic adds per file, against handlers that spawn
# exiftool — unmeasurable in practice, and the counters are never read on the
# exiftool: unmeasurable in practice, and the counters are never read on the
# hot path.
# Stage identity lives here, in declaration order, so the report, the JSON and
@@ -55,7 +55,7 @@ const STAGE_TITLES = (classify = "classify", enrich = "enrich",
Counters for one pipeline stage. All fields are atomic and monotonic: workers
only ever add, readers only ever read, so no lock is needed between them.
`in_flight` is the exception to monotonic — it goes up and down — and is the one
`in_flight` is the exception to monotonic. It goes up and down, and is the one
counter that is a *level* rather than a total.
"""
struct StageStats
@@ -74,8 +74,8 @@ StageStats() = StageStats(Threads.Atomic{Int}(0), Threads.Atomic{Int}(0),
"""
Counters for HTTP intake, which has no worker loop to hang them off.
`files` counts what was actually spooled *and* queued — the same thing the 202
body reports as `accepted` — so it lines up with stage 1's arrivals. Files
`files` counts what was actually spooled *and* queued, the same thing the 202
body reports as `accepted`, so it lines up with stage 1's arrivals. Files
dropped because the queue was full are counted separately in `rejected`, since a
run where intake outruns the pipeline should be visible as backpressure rather
than as slow intake.
@@ -96,7 +96,7 @@ from.
`since` matters to a reader computing rates from a single scrape: without it,
"1000 files completed" has no denominator. Readers that scrape twice should use
their own Δt instead — it excludes the time before they started watching.
their own Δt instead, which excludes the time before they started watching.
"""
struct Metrics
intake::IntakeStats
@@ -117,7 +117,7 @@ const METRICS = Metrics()
reset_metrics!()
Zero every counter and restart the measurement window. For tests, and for a
benchmark that wants totals covering only its own run — though a harness that
benchmark that wants totals covering only its own run, though a harness that
scrapes before and after can subtract instead, which is safer against a server
that is also serving someone else.
"""
@@ -153,14 +153,14 @@ the parked time to `stats.blocked_ns`.
This is the routing half of stages 1 and 3: a classified file is never dropped,
so a full downstream queue means waiting, not failing. Wrapping the retry loop
here — rather than repeating `while !enqueue! sleep end` at each call site — is
here, rather than repeating `while !enqueue! sleep end` at each call site, is
what makes that wait measurable at all, and keeps the three routing paths from
drifting into three different backoff behaviours.
`stats` is `nothing` for callers that are not a pipeline stage — startup
`stats` is `nothing` for callers that are not a pipeline stage. Startup
recovery (`recover_dir!`) uses the same never-drop retry, but it runs outside any
handler, so it has no `busy_ns` to charge against. Billing its wait to a stage's
`blocked_ns` would make `/stats` utilization — `(Δbusy_ns - Δblocked_ns) / …` —
`blocked_ns` would make `/stats` utilization, `(Δbusy_ns - Δblocked_ns) / …`,
go *negative* after a long recovery. Skipping the charge keeps `blocked_ns`
meaning exactly what its docs say: time a handler spent parked on its successor.
"""
@@ -183,7 +183,7 @@ end
Read every counter into a plain value tree for `GET /stats`.
`queues` is a NamedTuple of `JobQueue`s keyed by `STAGE_KEYS`, supplying each
stage's live depth and capacity — the counters above are about work *done*, and
stage's live depth and capacity. The counters above are about work *done*, and
a stage's depth is what says whether work is piling up in front of it.
The read is not atomic across stages: counters keep moving while we walk them,