Route by queue, not by directory: stop moving files between stages

A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.

Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.

Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.

The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.

Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.

Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.

src/stats.jl

@@ -1,10 +1,12 @@
 # Per-stage pipeline metrics.
 #
 # End-to-end throughput says how fast the pipeline is; it does not say which
-# stage is the reason. The external harness can't answer that either: `known/`,
-# `unknown/` and `text/` are *transient* — a file can pass through one between
-# two directory polls — so sampling directories from outside undercounts exactly
-# the stages we most want to measure. Only the pipeline itself sees every job.
+# stage is the reason. The external harness can't answer that either, and now
+# cannot even approximate it: every in-flight file sits in `spool/` regardless of
+# how far it has got, because stages route by enqueueing a reference rather than
+# by moving bytes (src/worker.jl header). A file's stage is a property of the
+# queue holding it, so only the pipeline itself can see it — there is no
+# directory to poll.
 #
 # So each stage keeps four counters, all incremented in `worker_loop` (the one
 # place every stage's work passes through) and read by `GET /stats`:
@@ -154,15 +156,22 @@ so a full downstream queue means waiting, not failing. Wrapping the retry loop
 here — rather than repeating `while !enqueue! sleep end` at each call site — is
 what makes that wait measurable at all, and keeps the three routing paths from
 drifting into three different backoff behaviours.
+
+`stats` is `nothing` for callers that are not a pipeline stage — startup
+recovery (`recover_dir!`) uses the same never-drop retry, but it runs outside any
+handler, so it has no `busy_ns` to charge against. Billing its wait to a stage's
+`blocked_ns` would make `/stats` utilization — `(Δbusy_ns - Δblocked_ns) / …` —
+go *negative* after a long recovery. Skipping the charge keeps `blocked_ns`
+meaning exactly what its docs say: time a handler spent parked on its successor.
 """
-function enqueue_blocking!(queue::JobQueue, job::Job, stats::StageStats;
+function enqueue_blocking!(queue::JobQueue, job::Job, stats::Union{StageStats,Nothing};
                            retry_seconds::Real)
     enqueue!(queue, job) && return nothing
     t0 = time_ns()
     while !enqueue!(queue, job)
         sleep(retry_seconds)
     end
-    Threads.atomic_add!(stats.blocked_ns, Int(time_ns() - t0))
+    stats === nothing || Threads.atomic_add!(stats.blocked_ns, Int(time_ns() - t0))
     return nothing
 end