jeff/file-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: jeff:fac3adbaf665f61f186d6db911038a679615673c
Branches Tags
jeff:main
...
compare: jeff:main
Branches Tags
jeff:main

14 Commits
fac3adbaf6 ... main

Author SHA1 Message Date
Jeffrey Ward
4aed713939 Renamed to DarkStruct 2026-08-26 10:11:29 -04:00
Jeffrey Ward
630bf5f9a7 Added RabbitMQ 2026-08-26 09:41:53 -04:00
Jeffrey Ward
cc84d70d52 Initial text cleanup 2026-08-19 12:55:19 -04:00
Jeffrey Ward
819a4cce7a Containerize the server: multi-stage build, compose, data volume 
The runtime image carries only what serving needs — the Julia runtime, an
already-precompiled depot, exiftool, and optionally github-linguist. The
package registry, git clones, and the Ruby/C toolchain that builds rugged
all stay in earlier stages.

Two things shape the build. Dependencies are instantiated and precompiled in
a layer keyed only on Project.toml/Manifest.toml, so a src/ edit rebuilds in
seconds rather than minutes; a stub src/FileServer.jl satisfies Pkg's
root-package check there. And github-linguist, the one heavy optional
dependency, is behind WITH_LINGUIST: it degrades gracefully (stage 4 keeps
natural-language enrichment and warns), so building it out is a supported
~160MB saving.

All stage directories are pointed under /data via the FS_*_DIR variables so
one volume holds the whole in-flight working set, and data/ is excluded from
the build context.

Claude-Session: https://claude.ai/code/session_01KMNwyqLS3ns6uwWHGDGdAQ
 2026-08-07 15:38:33 -04:00
Jeffrey Ward
4a22123001 Route by queue, not by directory: stop moving files between stages 
A file is now written once into spool/ and stays there for its whole time in
flight. Stages 1 and 3 hand work on by enqueueing the same Job reference, so
job.path is constant from intake until commit. The three inter-stage renames
(spool->known, spool->unknown, unknown->text) are gone, along with the
known/, unknown/ and text/ directories and their FS_*_DIR settings.

Terminal moves stay: done/, text_done/, binary/ and failed/ still receive the
file, and binary/ in particular must, since it is the corpus the offline
stage-5 discovery sweep reads.

Measured by bin/bench_stage1.jl (2000 x 64 KiB, min of 5): the removed rename
cost 11.6 us per file, the equal of the classifier itself. One stage-1 worker
goes from ~28.6 us/file (~35k files/s) to 11.70 us (85.4k files/s); 16 workers
reach 333k files/s. What is left is classify 10.30 us, the queue handoff
0.12 us, and the disabled @debug lines 0.29 us.

The stage a file has reached now lives only in the queue holding its
reference, and the queues are in-process, so a crash loses it: everything in
spool/ replays from stage 1. That is safe rather than merely tolerable —
classification and the UTF-8 sniff are pure functions of the file's bytes and
the terminal commits rename with force=true, so a replayed file lands where it
would have landed and overwrites its own sidecar. The per-stage directories
were standing in for a durable queue, and charging every file a rename per
stage to do it; src/queue.jl already defines the seam where a broker-backed
JobQueue restores exact resume properly.

Recovery had to change to match. All leftovers now funnel onto the single
stage-1 queue, so the old non-blocking recover_dir! would have capped a
4000-file recovery at 1000 and abandoned the rest. It now blocks on a full
queue, and run() spawns the worker pools before recovering so the consumers
drain it as we fill; reset_metrics! moves above the spawn accordingly.
enqueue_blocking! takes stats::Union{StageStats,Nothing} so recovery reuses
the never-drop retry without charging its wait to a stage's blocked_ns, which
would drive /stats utilization negative.

Tests assert the new invariant positively (the file stays put, the routed
reference is unchanged, no intermediate directory appears) and cover recovery
of more files than the queue can hold. Verified end to end against a real
server: 40 leftovers, stage-1 capacity 3, all recovered and drained to
text_done/ with spool/ and failed/ empty.
 2026-08-07 13:21:49 -04:00
Jeffrey Ward
341b61f806 Add stage-2 decomposition benchmark; fix run_with_timeout latency and enforceability 
bin/bench.jl reports stage 2 as a single throughput number, which can't
distinguish slow extraction from a slow spawn — and those have opposite fixes.
bin/bench_stage2.jl times each component in isolation, then times the real
handle_known_job end to end. It draws its corpus from real files (default
data/done) because random bytes make exiftool bail out early and understate the
stage by ~10x, and it prices both fork-free alternatives (batched, -stay_open)
so the cost of one-fork-per-file is a measurement rather than a guess.

The benchmark found stage 2 to be ~98% exiftool, and found two problems in
run_with_timeout, which stages 2 and 4 share:

1. The watchdog polled with sleep(0.1) and then joined the polling task, so
   every call paid the remainder of an in-flight sleep after the child had
   already exited: ~25 ms per file, and a measured 101 ms on a process that
   exits instantly. Replaced with a one-shot Timer cancelled when the child
   exits. Stage 2 goes from 164.6 ms to 138.3 ms per file (6 -> 8 files/s on one
   worker); the wrapper is now within noise of a bare Base.run.

2. Writing the missing tests showed the timeout was never enforceable, in the
   old implementation as much as the new. wait(proc) returns only once the
   captured stdout pipe closes, and grandchildren inherit that pipe, so
   signalling the child alone left the worker blocked until the whole process
   tree finished on its own — a `sh -c "trap '' TERM; sleep 30"` child ran the
   full 30 s against a 1 s timeout. The child now runs in its own process group
   and the timeout signals the group. The trade is that a hard crash of the
   server orphans an in-flight child rather than taking it down with it.

Three new tests cover the fast path, the timeout, and the SIGTERM-ignoring
escalation; the second was previously unexercised, which is why the bug stood.

Not addressed here, but measured and documented in the README: Perl interpreter
startup is 76.7 ms of the remaining 135.9 ms call, so a persistent exiftool
(-stay_open, 40.1 ms/file) would cut the stage by roughly another 70%. And
fsync_dir measures 1.75 us, too fast to be a real flush — commit_enriched!'s
durability may not hold on this filesystem, which is a correctness question
left open.

Claude-Session: https://claude.ai/code/session_01Xy9At7HNLHWNmfh1yw71Uy
 2026-08-03 11:28:36 -04:00
Jeffrey Ward
c692d14a2c Demote stage-1 per-file logging to @debug; add stage-1 decomposition benchmark 
bin/bench_stage1.jl takes handle_classify_job apart — filesize, read_features,
Lux.apply, classify, move_to, enqueue_blocking!, and the log lines — times each
in isolation, then times the real handler end to end under four loggers so the
parts can be checked against the whole.

It found that logging was stage 1's dominant cost: as @info the two per-file
lines cost ~71 us of the handler's ~118 us, roughly 6x the classifier (10.6 us)
and 6x the rename (11.6 us). Nearly all of it is ConsoleLogger formatting
(~64 us), not the FlushLogger's per-message flush (~8 us).

Demoting them to @debug takes stage 1 from 8.5k files/s to 35.3k files/s on one
worker (4.2x). The messages are still available with JULIA_DEBUG=FileServer,
which the benchmark also prices (133 us/file). What remains splits evenly
between the rename (11.7 us) and classify (10.7 us, itself 74% feature read),
so stage 1 is now filesystem-bound; its thread sweep peaks at ~4 workers.
 2026-08-03 00:10:32 -04:00
Jeffrey Ward
c5d488d9b4 Add per-stage throughput instrumentation; fix memory-benchmark accuracy 
End-to-end throughput says how fast the pipeline is, not which stage is the
reason. The four stages run concurrently behind their own queues, so the
pipeline's rate *is* the slowest stage's rate and the others are invisible in
it. Nothing outside the server can recover them either: known/, unknown/ and
text/ are transient, and a file can cross one between two directory polls, so an
external sampler misses exactly the stages worth measuring.

So the pipeline counts its own work, and bin/bench.jl turns two scrapes into
rates.

- src/stats.jl: per-stage counters (completed/failed, bytes, busy_ns,
  blocked_ns, in_flight) plus intake counters, monotonic since startup in the
  Prometheus style — rates are the reader's job, so a scrape is stateless and
  two readers can't disturb each other. Recorded in worker_loop, the one place
  every stage's work passes through, so a new stage is instrumented the moment
  it is wired up and never on the read path.
- src/server.jl: GET /stats. An ordinary Oxygen route (no body to stream),
  unlike /upload. Intake counts files at the point they become stage 1's
  problem, so intake totals and stage-1 arrivals refer to the same files.
- src/queue.jl: capacity(q) joins length on the introspection seam — a depth of
  900 means nothing without knowing whether the limit is 1000 or 1_000_000.

utilization = (busy - blocked) / (window * workers) is the number that names the
bottleneck: throughput alone can't tell a saturated stage from one starved by
the stage ahead of it, since both report the same files/s. blocked_ns is what
keeps that true. Stages 1 and 3 apply blocking backpressure — a full downstream
queue means parking, not dropping — and that wait is inside the handler, so
counting it as busy would pin stage 1 at 1.0 whenever stage 2 is the real jam,
making every stage upstream of a jam look like the jam. enqueue_blocking! wraps
the retry loop so the wait is measurable at all, and keeps the three routing
paths from drifting into three different backoff behaviours.

Measured (400 mixed files, 16 KiB, concurrency 16): stage 4 is the constraint at
0.87 utilization and 853 ms/file — github-linguist is a process spawn per file —
while stages 1 and 3 idle under 0.10. Verified the blocked accounting against a
deliberately starved server (FS_TEXT_WORKERS=1, FS_TEXT_QUEUE_CAPACITY=2): stage
3 reported 100% blocked at 0.0 utilization rather than looking saturated too.

bin/bench_model.jl: the classifier alone, no server or queue in the way, because
stage 1's 38.7 ms/file cannot plausibly be a 32-64-16-2 MLP. It isn't:
Lux.apply is 2.3 us, read_features 4.2-6.0 us (flat across 1 KiB - 256 MiB, as
the seek-to-tail design intends), classify() 8.4 us — so ~99.98% of stage 1 is
rename, logging and contention, and the file read costs 3x the inference. Two
findings: batching would buy ~13x (179 ns/file at batch 512 vs 2.34 us at batch
1), and inference does not scale past ~4 threads. A pure-compute control kernel
runs the same sweep to place the blame — it reaches 14.3x at 16 tasks on this
box, so the machine parallelizes and Lux.apply does not. BLAS threads and GC are
both ruled out; the cause is inside Lux and is not diagnosed here.

Three bugs in the memory measurement, all of which produced wrong answers that
looked plausible:

- detect_pid matched any process with the launch command in its argv, including
  the shell that started the server — one run reported 3.64 MiB as the server's
  memory. Candidates are now filtered by /proc/<pid>/comm, what the process is
  rather than what its arguments say; no pattern over argv can do that.
- Baseline RSS was read *before* clear_refs reset the peak counter, so the two
  numbers had different origins. The 2 GiB run reported -1.01 MiB of growth;
  reading the baseline after the reset makes it 31.3 MiB.
- Negative growth is now reported as "none measurable" rather than a negative
  figure, which reads as a memory saving.

Re-measuring with those fixed keeps the claim that matters — growth is flat in
file size (14-31 MiB from 256 MiB to 2 GiB), so nothing is buffering — but the
concurrency coefficient does not survive: a freshly started server settles
anywhere in an ~860-985 MiB band, so baseline variance is comparable to the
growth being measured, and the old table quoted megabyte precision the
measurement never supported. README now states the shape, requires a ~30s settle
before a memory run, and says plainly that linear-in-concurrency is
undemonstrated rather than leaving an authoritative-looking number.

README also gains a single runnable sequence for all three harnesses: the server
prerequisite was never shown inline, so following the benchmarking section
top-to-bottom just produced "cannot reach /health".

Tests: 276 pass (41 new) — the blocked-vs-busy split, worker_loop draining
in_flight through a throwing handler, and the JSON round-trip of the field names
bench.jl reads.
 2026-08-02 23:49:16 -04:00
Jeffrey Ward
0d8eba05b8 Stream multipart intake; add throughput/memory benchmark harness 
Adds a benchmark harness, which showed that intake buffered each upload
whole, then makes intake streaming so the service's flat-memory property
holds end to end rather than only for the queue and workers.

The measurement problem first: /upload returns 202 once bytes are spooled
and a reference is enqueued, so HTTP latency measures intake, not the
pipeline. bin/bench.jl instead uploads a corpus and polls the terminal
sinks until the count stops moving, reporting intake rate and end-to-end
rate separately, sampling server RSS (kernel VmHWM, reset per run) and
the intermediate stage depths so the bottleneck stage names itself.

That exposed the buffering: HTTP.jl read the body into req.body,
parse_multipart_form materialized each part, and read(p.data) copied
again before spool_file wrote it — a 256 MiB upload grew RSS ~700 MiB,
and 4 concurrent ones pushed a 950 MiB baseline past 2 GiB.

- src/multipart.jl: incremental multipart/form-data reader. Pulls fixed
  chunks off the socket and hands each part's bytes straight to a sink,
  so memory is bounded by FS_UPLOAD_CHUNK_BYTES (64 KiB), not file size.
  Interface is two calls in a loop (next_part! then write_part_body! /
  skip_part_body!) so the handler keeps ordinary control flow. Retains
  the last length(delimiter)-1 bytes so a delimiter split across chunks
  still parses; part headers are bounded by policy, not by chunking.
- src/server.jl: /upload is served by a stream handler. Oxygen's root
  handler wraps HTTP.streamhandler, which does request.body =
  read(stream) before dispatching — so no Oxygen route, not even a
  @stream route, can stream a body. root_stream_handler intercepts
  POST /upload at the stream level and delegates the rest to Oxygen
  unchanged; /upload is therefore absent from Oxygen's metrics and docs.
  A client hangup is classified as routine (info, not error) and answered
  best-effort; every exit path drains the body so keep-alive still works.
- src/spool.jl: spool_file(bytes) -> spool_stream(write_body!, ...),
  which removes a partial file on a failed or abandoned write, so restart
  recovery can never pick up a truncated upload as if it were complete.
- config.jl: FS_UPLOAD_CHUNK_BYTES, the intake memory dial.

Streaming changes the 503 contract: a buffered handler knew up front how
many files a request held, this one discovers them as they arrive. When
the queue fills mid-request it no longer abandons the connection — it
stops spooling (discarding remaining parts rather than writing files it
cannot queue), drains, and answers 503 with the accepted list. Files
already queued stay queued.

Measured after (fresh server, 64 KiB chunk): 256 MiB +21.8 MiB, 1 GiB
+20.8 MiB, 2 GiB +17.0 MiB at concurrency 1 — flat across a 32x size
range; 4 concurrent 256 MiB uploads +86.9 MiB, linear in concurrency.
A 2 GiB upload sustains 334 MiB/s. Small files did not regress (intake
107 -> 133 files/s, end-to-end 27.6 -> 32.9 files/s, p95 1930 -> 776 ms).
A --size sweep that slopes upward is now the regression signal.

- Tests (235 pass, 55 new): byte-exact round-trip of 10 files in one
  request, sizes straddling the chunk boundary (0/1/63/65535/65536/65537/
  131072/196615/1e6) plus a payload stuffed with near-boundary sequences;
  the same body parsed at chunk sizes 1..10000 to put the delimiter split
  at every offset; bounded allocation on a 16 MiB part; malformed and
  truncated bodies; spool_stream cleanup on a failed write.
- Known cosmetic caveat, documented: when a body is cut short, HTTP.jl's
  own closeread logs an EOFError after the handler returns, because
  Content-Length promised more than arrived. Not reachable from a
  handler; the old code logged the same thing without replying.
 2026-08-02 22:22:35 -04:00
Jeffrey Ward
e18ac45d70 Updated README 2026-07-25 09:59:04 -04:00
Jeffrey Ward
f4e3f5be0b Update README for stage-5 phase B (durable catalog + sweep runner) 2026-07-03 17:04:37 -04:00
Jeffrey Ward
584bad02a7 Add stage-5 phase B: durable single-owner format catalog 
Wraps the assign_file scoring core (cluster.jl) in the live catalog the
design's phase B calls for (DESIGN §5B/§9):

- src/catalog.jl: Catalog durable state (frozen-id clusters + sufficient
  stats + processed set + examples); sparse, sidecar-first durable
  save/load; incremental catalog_sweep! (deterministic CRP-predictive
  assignment of new binary/ files); offline compact! that seeds on first
  run and recompacts later; write_nominations! emitting one JSON per
  promotable cluster with a hex magic template.
- bin/cluster_sweep.jl: cron/periodic single-owner runner (--compact
  forces a recluster; first run auto-compacts to seed).
- config.jl: cluster_catalog_path + nominated_dir knobs (FS_CLUSTER_CATALOG,
  FS_NOMINATED_DIR), wired into config_from_env and ensure_dirs.
- Tests: durable round-trip, incremental sweep growth + idempotency,
  §10.1 nothing-from-noise end-to-end (zero promotions), a recurring
  format self-nominating, seed-then-live-assign (165 pass).
- DESIGN_clustering.md: mark phase B built.
 2026-07-03 17:03:33 -04:00
Jeffrey Ward
d9f32d9aaf Add stage-5 unknown-format discovery: header clustering + calibration 
Implements phase A of the DESIGN_clustering.md design: a Dirichlet-process
mixture of per-position categoricals over the first 32 header bytes (257-symbol
alphabet) that clusters the binary/ pile by file format, plus signature
extraction and promotion nomination. All base-Julia (a Lanczos loggamma keeps
the Dirichlet-multinomial marginal dependency-free).

- src/cluster.jl: header_symbols feature extraction, collapsed Gibbs sampler
  (phase A), sequential CRP-predictive assignment (phase B core), signatures/
  promotion, and ARI/V-measure calibration metrics.
- bin/cluster_calibrate.jl: grid-tunes hyperparameters against magic-collapsed
  ground truth and cross-checks a model-free NCD (gzip) baseline.
- FS_CLUSTER_*/FS_PROMOTE_* config knobs; wire cluster.jl into the module.
- Tests for the three DESIGN §10 assertions plus the model primitives.

Calibrated defaults (n=32, alpha=1.0, beta=0.1) recover known formats at
ARI 0.77 (0.885 excl. tar); docx+zip and the ELF family merge correctly and the
NCD baseline agrees. DESIGN §11 records the results and three assumptions the
data corrected (tar/ELF header-zero merge, the cold-start seeding deadlock, and
the Bernoulli signature / Occam-penalized restart scoring).
 2026-07-03 16:43:52 -04:00
Jeffrey Ward
2c8de488a1 Add design doc for stage-5 unknown-format clustering 2026-07-03 15:51:24 -04:00
35 changed files with 7853 additions and 467 deletions
Expand all files Collapse all files

16
.dockerignore Normal file
View File

@@ -0,0 +1,16 @@
# The pipeline working set (17G here) belongs in a volume, not the image.
data/
.git/
test/
# Benchmark / training / calibration harnesses are dev tools, not runtime.
bin/bench.jl
bin/bench_model.jl
bin/bench_stage1.jl
bin/bench_stage2.jl
bin/cluster_calibrate.jl
bin/train.jl
bin/send_dir.sh
Dockerfile
docker-compose.yml
.dockerignore
*.md

146
Dockerfile Normal file
View File

@@ -0,0 +1,146 @@
# Multi-stage build for the DarkStruct pipeline.
#
# The runtime image carries only what serving needs: the Julia runtime, a depot
# that is already instantiated *and* precompiled (so startup is load-only, no
# compilation), exiftool, and optionally github-linguist. Nothing from the build
# survives into it: no compilers, no package registry, no gem toolchain.
#
# docker build -t darkstruct .
# docker build --build-arg WITH_LINGUIST=false -t darkstruct . # ~200MB smaller
#
# github-linguist (stage-4 programming-language detection) is the one heavy
# optional dependency: it needs a Ruby toolchain to build rugged. It degrades
# gracefully: without it, text files still get natural-language enrichment and
# the server logs a warning at startup, so it can be built out.
ARG JULIA_VERSION=1.12.6
ARG DEBIAN_RELEASE=bookworm
ARG WITH_LINGUIST=true
# ---------------------------------------------------------------------------
# Stage 1: instantiate and precompile the Julia depot.
# ---------------------------------------------------------------------------
FROM julia:${JULIA_VERSION}-${DEBIAN_RELEASE} AS julia-depot
ENV JULIA_DEPOT_PATH=/opt/julia-depot \
JULIA_PKG_PRECOMPILE_AUTO=0
WORKDIR /app
# Manifest-first so a source-only edit doesn't re-resolve or rebuild the whole
# dependency tree. DarkStruct is its own project's root package, so Pkg insists
# on a module file being there; a stub satisfies it while the (slow, rarely
# invalidated) dependency layer is built; the real source lands below.
COPY Project.toml Manifest.toml ./
RUN mkdir -p src \
&& printf 'module DarkStruct\nend\n' > src/DarkStruct.jl \
&& julia --project=/app -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'
# Precompiling DarkStruct itself means the runtime image loads cached code
# instead of compiling Lux/Zygote on first request.
COPY src/ ./src/
RUN julia --project=/app -e 'using Pkg; Pkg.precompile()' \
&& julia --project=/app -e 'using DarkStruct'
# Everything the depot keeps for *resolving* packages is dead weight once the
# Manifest is pinned: the registry, git clones, scratch and log dirs.
RUN rm -rf /opt/julia-depot/registries \
/opt/julia-depot/clones \
/opt/julia-depot/scratchspaces \
/opt/julia-depot/logs \
&& find /opt/julia-depot/packages -type d \
\( -name test -o -name docs -o -name .git \) -prune -exec rm -rf {} + \
&& find /opt/julia-depot/artifacts -type d -name doc -prune -exec rm -rf {} +
# ---------------------------------------------------------------------------
# Stage 2: build github-linguist into a relocatable GEM_HOME (optional).
#
# Built against Debian's system Ruby, the same package the runtime installs,
# so rugged's native extension is ABI-compatible with the interpreter there.
# ---------------------------------------------------------------------------
FROM debian:${DEBIAN_RELEASE}-slim AS linguist
ARG WITH_LINGUIST
ENV GEM_HOME=/opt/gems
RUN set -eux; \
mkdir -p "$GEM_HOME"; \
if [ "$WITH_LINGUIST" = "true" ]; then \
apt-get update; \
apt-get install -y --no-install-recommends \
ruby ruby-dev build-essential cmake pkg-config \
libicu-dev zlib1g-dev libssl-dev; \
gem install --no-document github-linguist; \
# Drop sources, docs and test fixtures from the installed gems; only the
# built .so and the .rb load path matter at runtime.
rm -rf "$GEM_HOME"/cache "$GEM_HOME"/doc; \
find "$GEM_HOME"/gems -type d \( -name test -o -name spec -o -name samples \) \
-prune -exec rm -rf {} +; \
find "$GEM_HOME"/extensions -type d -name tmp -prune -exec rm -rf {} +; \
rm -rf /var/lib/apt/lists/*; \
fi
# ---------------------------------------------------------------------------
# Stage 3: runtime.
# ---------------------------------------------------------------------------
FROM debian:${DEBIAN_RELEASE}-slim AS runtime
ARG WITH_LINGUIST
# exiftool is a hard startup prerequisite (stage-2 enrichment fails fast without
# it). Ruby + libicu are only needed to *run* the linguist gem built above.
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends \
libimage-exiftool-perl ca-certificates curl; \
if [ "$WITH_LINGUIST" = "true" ]; then \
apt-get install -y --no-install-recommends ruby libicu72; \
fi; \
rm -rf /var/lib/apt/lists/*
RUN useradd --system --uid 10001 --create-home --home-dir /app app
# Depot ownership matters: Julia rewrites a precompile cache it considers stale,
# and a read-only depot would turn that into a startup crash instead of a
# recompile. --chown does it in the copy, so it costs no extra layer.
COPY --from=julia-depot /usr/local/julia /usr/local/julia
COPY --from=julia-depot --chown=app:app /opt/julia-depot /opt/julia-depot
COPY --from=linguist /opt/gems /opt/gems
ENV PATH=/usr/local/julia/bin:/opt/gems/bin:$PATH \
GEM_HOME=/opt/gems \
GEM_PATH=/opt/gems \
JULIA_DEPOT_PATH=/opt/julia-depot \
JULIA_NUM_THREADS=auto \
JULIA_PKG_OFFLINE=true
WORKDIR /app
COPY Project.toml Manifest.toml ./
COPY src/ ./src/
COPY bin/server.jl bin/cluster_sweep.jl ./bin/
COPY model/classifier.jld2 ./model/
# Every stage directory hangs off /data so one volume covers the whole pipeline.
# 0.0.0.0 rather than the 127.0.0.1 default: the port is only reachable from
# outside if it is published.
ENV FS_HOST=0.0.0.0 \
FS_PORT=8080 \
FS_SPOOL_DIR=/data/spool \
FS_BINARY_DIR=/data/binary \
FS_DONE_DIR=/data/done \
FS_TEXT_DONE_DIR=/data/text_done \
FS_FAILED_DIR=/data/failed \
FS_CLUSTER_DIR=/data/binary \
FS_CLUSTER_CATALOG=/data/catalog.json \
FS_NOMINATED_DIR=/data/nominated \
FS_MODEL_PATH=/app/model/classifier.jld2
RUN mkdir -p /data && chown app:app /data
USER app
VOLUME ["/data"]
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
CMD curl -fsS "http://127.0.0.1:${FS_PORT}/health" || exit 1
ENTRYPOINT ["julia", "--project=/app", "-t", "auto"]
CMD ["bin/server.jl"]

12
Manifest.toml
View File

@@ -2,7 +2,7 @@
julia_version = "1.12.6" julia_version = "1.12.6"
manifest_format = "2.0" manifest_format = "2.0"
project_hash = "a623ff56053e3a56c1799a1cb2080ec48d933b73" project_hash = "d7c84379a20829ec867546b4a887bc3c10df8311"
[[deps.ADTypes]] [[deps.ADTypes]]
git-tree-sha1 = "d9aaef7c63466eee4de23b4d9dad03629df54bea" git-tree-sha1 = "d9aaef7c63466eee4de23b4d9dad03629df54bea"
@@ -15,6 +15,12 @@ weakdeps = ["ChainRulesCore", "ConstructionBase", "EnzymeCore"]
ADTypesConstructionBaseExt = "ConstructionBase" ADTypesConstructionBaseExt = "ConstructionBase"
ADTypesEnzymeCoreExt = "EnzymeCore" ADTypesEnzymeCoreExt = "EnzymeCore"
[[deps.AMQPClient]]
deps = ["Logging", "MbedTLS", "Sockets"]
git-tree-sha1 = "508457ed7a2afb432590247dc363fffc51f242fc"
uuid = "79c8b4cd-a41a-55fa-907c-fab5288e1383"
version = "0.5.1"
[[deps.AbstractFFTs]] [[deps.AbstractFFTs]]
deps = ["LinearAlgebra"] deps = ["LinearAlgebra"]
git-tree-sha1 = "d92ad398961a3ed262d8bf04a1a2b8340f915fef" git-tree-sha1 = "d92ad398961a3ed262d8bf04a1a2b8340f915fef"
@@ -308,8 +314,8 @@ weakdeps = ["HTTP"]
[deps.FileIO.extensions] [deps.FileIO.extensions]
HTTPExt = "HTTP" HTTPExt = "HTTP"
[[deps.FileServer]] [[deps.DarkStruct]]
deps = ["HTTP", "JLD2", "JSON3", "Logging", "Lux", "Optimisers", "Oxygen", "UUIDs", "Zygote"] deps = ["HTTP", "JLD2", "JSON3", "Languages", "Logging", "Lux", "Optimisers", "Oxygen", "Random", "UUIDs", "Zygote"]
path = "." path = "."
uuid = "b3f1c2d4-5e6a-4b7c-8d9e-0f1a2b3c4d5e" uuid = "b3f1c2d4-5e6a-4b7c-8d9e-0f1a2b3c4d5e"
version = "0.1.0" version = "0.1.0"

6
Project.toml
View File

@@ -1,9 +1,10 @@
name = "FileServer" name = "DarkStruct"
uuid = "b3f1c2d4-5e6a-4b7c-8d9e-0f1a2b3c4d5e" uuid = "b3f1c2d4-5e6a-4b7c-8d9e-0f1a2b3c4d5e"
version = "0.1.0" version = "0.1.0"
authors = ["wardjm@gmail.com"] authors = ["wardjm@gmail.com"]
[deps] [deps]
AMQPClient = "79c8b4cd-a41a-55fa-907c-fab5288e1383"
HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3" HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819" JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1" JSON3 = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
@@ -12,10 +13,12 @@ Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
Lux = "b2108857-7c20-44ae-9111-449ecde12c47" Lux = "b2108857-7c20-44ae-9111-449ecde12c47"
Optimisers = "3bd65402-5787-11e9-1adc-39752487f4e2" Optimisers = "3bd65402-5787-11e9-1adc-39752487f4e2"
Oxygen = "df9a0d86-3283-4920-82dc-4555fc0d1d8b" Oxygen = "df9a0d86-3283-4920-82dc-4555fc0d1d8b"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f" Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"
[compat] [compat]
AMQPClient = "0.5.1"
HTTP = "1.11.0" HTTP = "1.11.0"
JLD2 = "0.6.4" JLD2 = "0.6.4"
JSON3 = "1.14.3" JSON3 = "1.14.3"
@@ -24,6 +27,7 @@ Logging = "1.11.0"
Lux = "1.31.4" Lux = "1.31.4"
Optimisers = "0.4.7" Optimisers = "0.4.7"
Oxygen = "1.10.2" Oxygen = "1.10.2"
Random = "1.11.0"
UUIDs = "1.11.0" UUIDs = "1.11.0"
Zygote = "0.7.11" Zygote = "0.7.11"

803
README.md
View File

@@ -1,13 +1,13 @@
# FileServer # DarkStruct
A minimal Julia service that receives files over HTTP and hands them off to a A minimal Julia service that receives files over HTTP and hands them off to a
pool of worker threads for processing. The HTTP endpoint does no real work: it pool of worker threads for processing. The HTTP endpoint does no real work: it
spools each uploaded file to disk, pushes a lightweight reference onto a work spools each uploaded file to disk, pushes a lightweight reference onto a work
queue, and responds immediately — staying free to accept the next upload. queue, and responds immediately, staying free to accept the next upload.
The per-file "processing" runs each file through a small neural-network The per-file processing is deliberately thin. Each file runs through a small
classifier that labels it **known** (a file type resembling the training set) or neural-network classifier that labels it **known** (a file type resembling the
**unknown**, and logs the result. See "File classifier" below. training set) or **unknown**, and logs the result. See "File classifier" below.
## Architecture ## Architecture
@@ -20,14 +20,15 @@ enrichment mixes CPU with a subprocess):
POST /upload (multipart) POST /upload (multipart)
│ │
▼ ▼
┌─────────────────┐ spool bytes to disk ┌─────────────────┐ stream bytes to disk (never buffered)
│ HTTP handler │────────────────────────► data/spool/<uuid>-<name> │ HTTP handler │────────────────────────► data/spool/<uuid>-<name>
│ (Oxygen.jl) │ │ (streaming) │ the file's home for its
└────────┬─────────┘ enqueue reference (non-blocking) └────────┬─────────┘ enqueue reference whole time in flight
│ │ │ (non-blocking)
▼ ▼ ▼ │
202 + job IDs ┌────────────────────┐ 202 + job IDs ▼
(503 if full) │ stage-1 queue │ classification (503 if full) ┌────────────────────┐
│ stage-1 queue │ classification
└─────────┬──────────┘ └─────────┬──────────┘
│ dequeue │ dequeue
┌───────────────────┼───────────────────┐ ┌───────────────────┼───────────────────┐
@@ -35,9 +36,10 @@ enrichment mixes CPU with a subprocess):
classify wkr 1 classify wkr 2 … classify wkr N classify wkr 1 classify wkr 2 … classify wkr N
│ │
┌────────────┴────────────┐ ┌────────────┴────────────┐
:unknown :known :unknown :known the file itself never moves;
│ move to data/unknown/, │ move to data/known/, then │ │ routing is the enqueue alone
▼ then enqueue (blocking) ▼ enqueue (blocking backpressure) │ enqueue (blocking) │ enqueue (blocking backpressure)
▼ ▼
┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐
│ unknown queue │ │ known queue │ enrichment │ unknown queue │ │ known queue │ enrichment
└─────────┬──────────┘ └─────────┬──────────┘ └─────────┬──────────┘ └─────────┬──────────┘
@@ -47,10 +49,10 @@ enrichment mixes CPU with a subprocess):
unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M unk 1 unk 2 … unk K known wkr 1 known wkr 2 … known wkr M
│ binary-vs-text sniff │ exiftool → normalized sidecar │ binary-vs-text sniff │ exiftool → normalized sidecar
├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name> ├─► data/binary/<uuid>-<name> success ──┴──► data/done/<uuid>-<name>
│ (terminal) data/done/<uuid>-<name>.meta.json │ (terminal — moved) data/done/<uuid>-<name>.meta.json
│ (sidecar-first commit) │ (sidecar-first commit)
│ :text move to data/text/, failure ───────► data/failed/<uuid>-<name> │ :text enqueue (blocking failure ───────► data/failed/<uuid>-<name>
▼ then enqueue (blocking backpressure) ▼ backpressure)
┌────────────────────┐ ┌────────────────────┐
│ text queue │ language enrichment │ text queue │ language enrichment
└─────────┬──────────┘ └─────────┬──────────┘
@@ -63,6 +65,16 @@ enrichment mixes CPU with a subprocess):
(sidecar-first commit) (sidecar-first commit)
``` ```
A file is written once, into `data/spool/`, and stays there for its entire time
in the pipeline. Stages hand it on by enqueueing its small `Job` reference, never
by moving bytes: the queue holding the reference *is* the record of which stage
the file has reached. The only move is the last one, into a terminal sink
(`data/done/`, `data/text_done/`, `data/binary/`) or into `data/failed/` if a
worker throws. Stage 1 used to rename each file into `data/known/` or
`data/unknown/` first, and stage 3 into `data/text/`. Those three directories are
gone, and with them 11.6 µs per file, the equal of the classifier itself. Stage 1
now runs at ~85k files/s on one worker instead of ~35k.
Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1 Stages 2 (known-file enrichment) and 3 (content triage) run in parallel: stage 1
feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language feeds both the known and unknown queues. Stage 3 in turn feeds stage 4 (language
enrichment) for every file it sorts as text. enrichment) for every file it sorts as text.
@@ -70,18 +82,36 @@ enrichment) for every file it sorts as text.
Key properties: Key properties:
- **Fast intake:** the queue only ever carries small references; file bytes live - **Fast intake:** the queue only ever carries small references; file bytes live
on disk, so memory stays flat regardless of file size. on disk, so memory stays flat regardless of file size. This holds end to end:
intake streams each upload from the socket to the spool file a chunk at a
time (`FS_UPLOAD_CHUNK_BYTES`, default 64 KiB) rather than buffering the body,
and every worker reads only a bounded prefix. Measured: uploads of 256 MiB,
1 GiB and 2 GiB each grow resident memory by ~20 MiB, a flat line in file
size. See "Streaming intake" and "Benchmarking" below.
- **Backpressure:** each queue is bounded (default 1000). When the *intake* queue - **Backpressure:** each queue is bounded (default 1000). When the *intake* queue
is full, uploads get `503 Service Unavailable`. When the *known* queue is full, is full, uploads get `503 Service Unavailable`. When the *known* queue is full,
the stage-1 worker blocks and retries (a classified file is never dropped). the stage-1 worker blocks and retries (a classified file is never dropped).
- **Crash-resilient:** files survive on disk. On startup, recovery is - **Crash-resilient:** files survive on disk. On startup everything left in
stage-aware: leftovers in `data/spool/` re-enter classification, `data/known/` `data/spool/` is re-enqueued (`recovered` in the log) and **replays from stage
re-enter enrichment, `data/unknown/` re-enter content triage, and `data/text/` 1**. That is safe rather than merely tolerable: classification and the
re-enter language enrichment (`recovered` / `recovered_known` / binary/text sniff are pure functions of the file's bytes, and the terminal
`recovered_unknown` / `recovered_text` in the log), so a file resumes at its commits rename with `force=true`, so a replayed file lands where it would have
correct stage instead of restarting from scratch. landed and overwrites its own sidecar. Recovery *blocks* on a full queue rather
than dropping the excess, and runs with the worker pools already live, so a
backlog larger than one queue's capacity takes longer to re-drive but none of
it is abandoned.
The cost of replay is redoing stages a file had already cleared. That is a
property of the *queue*, not of the directory layout: the default queues are
in-process (`src/queue.jl`), so a crash destroys the only record of how far
each file got. Per-stage directories used to stand in for that record, at the
price of a rename per file per stage on the hot path: a permanent cost on
every file, to buy a cheaper restart. The seam in `src/queue.jl` is where that
is actually fixed: run with `FS_QUEUE_BACKEND=rabbitmq` and a job stays unacked
until its handler commits, so a restart resumes each file at the stage it had
reached instead of re-driving `spool/` from stage 1. See "Queue backends".
- **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`) - **Graceful shutdown:** SIGINT (Ctrl-C) and SIGTERM (systemd/Docker/k8s `stop`)
both stop accepting uploads, then drain the stages *in order* — close the both stop accepting uploads, then drain the stages *in order*: close the
stage-1 queue and wait out the classify workers (the only producer of the known stage-1 queue and wait out the classify workers (the only producer of the known
*and* unknown queues), then close those queues and wait out the enrich and *and* unknown queues), then close those queues and wait out the enrich and
content-triage workers (content triage being the only producer of the text content-triage workers (content triage being the only producer of the text
@@ -90,26 +120,66 @@ Key properties:
- **Safe filenames:** client-supplied names are sanitized and prefixed with a - **Safe filenames:** client-supplied names are sanitized and prefixed with a
server-minted UUID before touching the filesystem (no path traversal). server-minted UUID before touching the filesystem (no path traversal).
### Streaming intake
The upload endpoint never holds a file in memory. Bytes go socket → spool file in
`FS_UPLOAD_CHUNK_BYTES` chunks, so resident memory per in-flight upload is set by
the chunk size, not the file size. A 2 GiB upload costs about what a 2 KiB one
does. Two pieces make that work, and both are deliberate:
- **`src/multipart.jl`, an incremental multipart parser.** HTTP.jl's
`parse_multipart_form` takes the *complete* body as a byte vector, so using it
means every file in the request is in memory at once (and copied again per
part). The reader here pulls fixed-size chunks and hands each part's bytes
straight to its spool file. Its interface is two calls in a loop
(`next_part!` then `write_part_body!`, or `skip_part_body!`), so the handler
keeps ordinary control flow instead of inverting into callbacks. The subtle
part is that a boundary delimiter can straddle two chunks, so the buffer always
retains the last `length(delimiter)-1` bytes; the test suite parses the same
body at chunk sizes from 1 byte upward to put that split at every offset.
- **`/upload` bypasses Oxygen's router.** Oxygen's root handler wraps
`HTTP.streamhandler`, which does `request.body = read(stream)` *before*
dispatching, even for an Oxygen `@stream` route, so no route can stream an
upload. `run` therefore passes its own `handler` to `serve`
(`root_stream_handler`), which intercepts `POST /upload` at the stream level and
delegates everything else to Oxygen unchanged. The trade-off: `/upload` is
absent from Oxygen's built-in metrics and docs.
Streaming also changes what the endpoint can promise. A buffered handler knows up
front how many files a request holds; this one discovers them as they arrive. So
when the intake queue fills mid-request it does not abandon the connection: it
stops spooling (discarding the remaining parts rather than writing files it can't
queue), drains the body, and answers `503` with the `accepted` list of whatever
got in first. Files already queued stay queued, and the client can retry the rest.
A client that hangs up mid-upload is treated as routine: the partial spool file is
removed (so restart recovery can never pick up a truncated upload as if it were
complete) and the event is logged `upload aborted by client`. One cosmetic caveat,
like the SIGTERM one below: when a request body is cut short, HTTP.jl's own
`closeread` logs an `EOFError` after the handler returns, because the connection
promised more bytes via `Content-Length` than arrived. It's harmless noise from
inside HTTP.jl: the partial file is already cleaned up and the connection closed.
### Metadata enrichment (stage 2) ### Metadata enrichment (stage 2)
Files the classifier labels **known** are handed to a second pool that extracts Files the classifier labels **known** are handed to a second pool that extracts
metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`) — metadata with [`exiftool`](https://exiftool.org/) (`exiftool -json -G -n`),
chosen because no native Julia library comes close to its multi-format coverage. chosen because no native Julia library comes close to its multi-format coverage.
The output is normalized into a small, stable, documented schema and written as a The output is normalized into a small, stable, documented schema and written as a
JSON **sidecar** next to the file in `data/done/`, e.g. JSON **sidecar** next to the file in `data/done/`, e.g.
`data/done/<uuid>-<name>.meta.json`. The original bytes are never modified. `data/done/<uuid>-<name>.meta.json`. The original bytes are never modified.
> **Prerequisite:** `exiftool` must be on `PATH` (e.g. `apt install > **Prerequisite:** `exiftool` must be on `PATH` (e.g. `apt install
> libimage-exiftool-perl`). The server **fails fast at startup** if it's missing. > libimage-exiftool-perl`). The server fails fast at startup if it's missing.
Sidecar top-level fields (all nullable — present only when available), plus the Sidecar top-level fields (all nullable, present only when available), plus the
complete raw `exiftool` object under `raw`: complete raw `exiftool` object under `raw`:
| Field | Meaning | | Field | Meaning |
|---|---| |---|---|
| `id`, `original_name` | job id and client-supplied name | | `id`, `original_name` | job id and client-supplied name |
| `file_type`, `mime_type` | e.g. `PDF` / `application/pdf` | | `file_type`, `mime_type` | e.g. `PDF` / `application/pdf` |
| `file_size` | bytes (authoritative, from intake — not exiftool) | | `file_size` | bytes (authoritative, from intake, not exiftool) |
| `created_date`, `modified_date` | content timestamps | | `created_date`, `modified_date` | content timestamps |
| `author` | person (`Author`/`Artist`/`By-line`) | | `author` | person (`Author`/`Artist`/`By-line`) |
| `created_by` | authoring app/tool (`Producer`/`CreatorTool`/`Creator`/`Software`/…) | | `created_by` | authoring app/tool (`Producer`/`CreatorTool`/`Creator`/`Software`/…) |
@@ -122,12 +192,12 @@ complete raw `exiftool` object under `raw`:
Each normalized field is a coalesce over a priority list of exiftool tags Each normalized field is a coalesce over a priority list of exiftool tags
(`src/metadata.jl`); extend a field by appending tag names. If extraction fails (`src/metadata.jl`); extend a field by appending tag names. If extraction fails
or `exiftool` times out (`FS_EXIFTOOL_TIMEOUT`, default 30s), the file still or `exiftool` times out (`FS_EXIFTOOL_TIMEOUT`, default 30s), the file still
completes to `data/done/` with a **degraded sidecar** — `file_size`/`file_type` completes to `data/done/` with a **degraded sidecar** (`file_size`/`file_type`
plus an `error` note — rather than being quarantined, because it's still a wanted plus an `error` note) rather than being quarantined, because it's still a wanted
known file. Only genuine I/O errors (can't write the sidecar or move the file) known file. Only genuine I/O errors (can't write the sidecar or move the file)
send it to `data/failed/`. send it to `data/failed/`.
The sidecar is committed **before** the file is moved into `data/done/`, so a The sidecar is committed *before* the file is moved into `data/done/`, so a
file's presence there always implies its sidecar is already present; a crash in file's presence there always implies its sidecar is already present; a crash in
between leaves only a harmless orphan sidecar, and recovery re-enriches between leaves only a harmless orphan sidecar, and recovery re-enriches
idempotently. idempotently.
@@ -137,20 +207,23 @@ idempotently.
Files the classifier labels **unknown** are handed to a third pool that sorts Files the classifier labels **unknown** are handed to a third pool that sorts
them into two coarse buckets so downstream tooling can treat them differently: them into two coarse buckets so downstream tooling can treat them differently:
- **`data/binary/`** — the file looks like binary data. - **binary:** the file looks like binary data. This is the end of the live path,
- **`data/text/`** — the file looks like text. so the file is committed to `data/binary/`, which doubles as the corpus for the
offline stage-5 discovery sweep below.
- **text:** the file looks like text. Stage 4 is still to come, so nothing
moves; the file stays in `data/spool/` and its reference goes onto the
stage-4 queue, ending up in `data/text_done/` once enriched.
The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text The test is a **UTF-8 sniff**: read the first 8000 bytes and call the file text
when that window is valid UTF-8 and holds no control bytes outside the text-safe when that window is valid UTF-8 and holds no control bytes outside the text-safe
set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise set (tab, newline, CR, and friends, plus ESC for ANSI-colored logs); otherwise
binary. It's cheap (no full read) and Unicode-aware — unlike the older NUL-byte binary. It's cheap (no full read) and Unicode-aware. Where the older NUL-byte and
or printable-ASCII heuristics, it keeps non-ASCII text (accents, CJK, emoji) in printable-ASCII heuristics misfiled non-ASCII text, this keeps accents, CJK and
`text/` instead of misfiling it, while binary formats — which rarely form valid emoji in the text bucket, while binary formats, which rarely form valid UTF-8
UTF-8 near their start — still land in `binary/`. A NUL byte is valid UTF-8 but near their start, still read as binary. A NUL byte is valid UTF-8 but not a text
not a text control byte, so it still reads as binary. A multi-byte character control byte, so it too reads as binary. A multi-byte character split by the
split by the 8000-byte boundary is trimmed before the check so it isn't mistaken 8000-byte boundary is trimmed before the check so it isn't mistaken for malformed
for malformed bytes. An empty file is treated as text. `binary/` is terminal; bytes. An empty file is treated as text (`src/content.jl`).
`text/` is handed to stage 4 (`src/content.jl`).
### Language enrichment (stage 4) ### Language enrichment (stage 4)
@@ -158,12 +231,12 @@ Files that stage 3 sorts as **text** are handed to a fourth pool that identifies
their language and writes a `.meta.json` sidecar, mirroring the stage-2 their language and writes a `.meta.json` sidecar, mirroring the stage-2
known-file enrichment. Two detectors run per file: known-file enrichment. Two detectors run per file:
- **natural language** — [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s - **natural language:** [`Languages.jl`](https://github.com/JuliaText/Languages.jl)'s
`LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded `LanguageDetector` (a Julia port of the `whatlang` n-gram model) reads a bounded
prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English prefix (up to `LANG_SAMPLE_BYTES`, 64 KiB) and reports the language's English
name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess. name, ISO 639-3 code, and a confidence in `[0,1]`. Pure Julia, no subprocess.
The detector is built once at startup and shared read-only across the pool. The detector is built once at startup and shared read-only across the pool.
- **programming language** — the [`github-linguist`](https://github.com/github-linguist/linguist) - **programming language:** the [`github-linguist`](https://github.com/github-linguist/linguist)
CLI recognizes source and markup by extension + content heuristics (e.g. CLI recognizes source and markup by extension + content heuristics (e.g.
`Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as `Python`, `Markdown`). Plain prose reports as `Text` and unrecognized content as
`null`; both collapse to *no programming language*. `null`; both collapse to *no programming language*.
@@ -182,30 +255,176 @@ The sidecar schema:
| `error` | set if natural-language detection produced nothing | | `error` | set if natural-language detection produced nothing |
> **`github-linguist` and the git-repo quirk:** run against a path *inside* a git > **`github-linguist` and the git-repo quirk:** run against a path *inside* a git
> repository, linguist reads the file's committed git blob, not the on-disk bytes > repository, linguist reads the file's committed git blob, not the on-disk
> — and an untracked file (which everything under `data/` is) has no blob, so it > bytes, and an untracked file (which everything under `data/` is) has no blob,
> crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under > so it crashes. Stage 4 sidesteps this by copying each file to a fresh temp dir under
> `/tmp` (outside any repo, preserving the name so extension heuristics still > `/tmp` (outside any repo, preserving the name so extension heuristics still
> fire) and pointing linguist there. > fire) and pointing linguist there.
> >
> Programming-language detection is **best-effort**: if `github-linguist` is > Programming-language detection is best-effort: if `github-linguist` is
> missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or > missing (a startup warning, not a fatal error, unlike `exiftool`), fails, or
> times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply > times out (`FS_LINGUIST_TIMEOUT`, default 30s), `programming_language` is simply
> `null` and the file still completes. Natural-language detection failing produces > `null` and the file still completes. Natural-language detection failing produces
> a **degraded sidecar** (with an `error` note) rather than a quarantine, because > a **degraded sidecar** (with an `error` note) rather than a quarantine, because
> the file is still wanted text. > the file is still wanted text.
Like stage 2, the sidecar is committed **before** the file is moved into Like stage 2, the sidecar is committed *before* the file is moved into
`data/text_done/`, so the file's presence there always implies its sidecar is `data/text_done/`, so the file's presence there always implies its sidecar is
present; recovery re-enriches idempotently (`src/language.jl`). present; recovery re-enriches idempotently (`src/language.jl`).
## The queue seam (→ RabbitMQ later) ### Unknown-format discovery (stage 5, offline)
The HTTP handler and workers only ever call `enqueue!`, `dequeue!`, and The `binary/` sink from stage 3 is the pile of genuinely *unrecognized* files.
`close!` on a `JobQueue` (see `src/queue.jl`). Today that's an in-process Stage 5 mines it for recurring new file formats by clustering files on their
`ChannelQueue`. To move to RabbitMQ (or any broker), implement a new `JobQueue` header bytes into a growing catalog of discovered formats, each with a magic-byte
subtype with those three methods and swap the construction in `run` — no handler signature that can eventually be promoted into the classifier's fast path. Unlike
or worker code changes. stages 1–4 it is not on the request hot path: it is a single-owner *batch*
process (the catalog is mutable shared state, the opposite of the stateless
classifier), and because promotion is human-gated nothing here is
latency-sensitive. The full rationale, and the assumptions we deliberately
rejected, live in [`model/DESIGN_clustering.md`](model/DESIGN_clustering.md).
The model (`src/cluster.jl`, base-Julia, no extra deps) is a Dirichlet-process
mixture of **per-position categoricals** over the first 32 header bytes, on a
257-symbol alphabet (byte `0–255` plus a `past-EOF` symbol so short fixed-length
formats are modeled honestly). Bytes are treated as categorical rather than
numeric (`0x89` and `0x88` are not "close"), so this deliberately does *not*
reuse the classifier's `[0,1]` byte scaling. A fixed uniform *background*
component absorbs structureless (compressed/encrypted) blobs so they don't mint spurious
clusters. A cluster's spiked positions become a libmagic-style signature;
clusters with enough members and enough fixed positions self-**nominate** for
promotion (a human does the one irreversible step, redefining "known").
**Status:** both phases are implemented and calibrated. Phase A (offline Gibbs)
is the science; phase B (`src/catalog.jl`) is the live catalog: a durable
single-owner state that sweeps `binary/`, folds each new file into a cluster with
the deterministic CRP-predictive rule, and writes promotion nominations.
The catalog process is run periodically (cron), single-threaded. It is the only
writer of the catalog, so it needs no locking:
```bash
julia --project=. bin/cluster_sweep.jl # incremental live sweep of new binary/ files
julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs re-cluster (seed / recompact)
```
The catalog is a single durable file (`FS_CLUSTER_CATALOG`, default
`data/catalog.json`) committed with the same sidecar-first
temp→fsync→rename→fsync-dir discipline as the stage-2 sidecars, so a crash can
neither corrupt it nor lose a write. On the first run (empty catalog) the sweep
auto-promotes to a `--compact` pass to seed clusters; later runs assign
incrementally, touching only files they have not seen. A cluster that clears the
member/magic thresholds writes a nomination (a hex magic template, member count,
and example filenames) into `FS_NOMINATED_DIR` (default `data/nominated/`) for a
human to glance at and promote. Under the calibrated `bg_mass > α`, the live
sweep never mints single-file clusters; genuinely new formats surface from the
periodic `--compact` re-clustering of the background residue, not the live path.
Calibration is its own offline script, like training, and never in the request
path. It is scored against magic-collapsed ground truth (so `docx`≡`zip` and the whole
ELF family count as one format each, which is the *correct* answer, not an error):
```bash
julia --project=. bin/cluster_calibrate.jl [training_set_dir] # defaults to ../training_set
```
It grid-tunes the hyperparameters to maximize Adjusted Rand Index against known
formats and cross-checks against a model-free NCD (gzip) baseline. On the 700-file
training corpus the calibrated defaults (`n=32`, `α=1.0`, `β=0.1`) recover the
known formats at **ARI 0.77** (0.885 excluding tar), with `gzip`, `pkzip`
(`docx`+`zip` merged), and `jpeg` forming clean, promotable clusters; the NCD
baseline agrees. See `DESIGN_clustering.md` §11 for the full results, including the
one known limitation: ELF and these tarballs share a long run of header
zero-padding and merge. The v2 fix is inverse-entropy position weighting.
## Queue backends
The HTTP handler and the workers only ever call `enqueue!`, `dequeue!`, `ack!`,
`nack!` and `close!` on a `JobQueue` (`src/queue.jl`). Two implementations sit
behind those five methods, chosen at startup by `FS_QUEUE_BACKEND`:
| | `channel` (default) | `rabbitmq` |
|---|---|---|
| Where jobs live | in-process, bounded buffer | durable broker queues, persistent messages |
| Crash recovery | everything in `spool/` replays from stage 1 | each file resumes at the stage it had reached |
| External dependency | none | a RabbitMQ broker |
| Capacity | hard limit, enforced per enqueue | advisory, checked against a polled depth |
| Delivery | exactly once (nothing to redeliver) | at least once |
`ack!` is the whole difference. A job is settled only after its handler commits,
so a crash mid-enrichment leaves that job on the enrich queue and the restart
picks it up there — not at stage 1, and not lost. `ChannelQueue` implements
`ack!`/`nack!` as no-ops, which is honest rather than lazy: an in-process queue
has no delivery to settle, and its recovery story is `recover_dir!`.
### Running it
```bash
# broker + server, both in compose
docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up
# just the broker, with the server on the host (5672 and the management UI on
# 15672 are published for exactly this)
docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up -d rabbitmq
FS_QUEUE_BACKEND=rabbitmq \
FS_AMQP_URL=amqp://darkstruct:darkstruct@localhost:5672/ \
julia --project=. -t auto bin/server.jl
```
Queues are named `<FS_AMQP_PREFIX>.<stage>` — `darkstruct.classify`,
`.enrich`, `.triage`, `.language` — so the broker's queue list reads like the
pipeline, and `/stats` reports each one's depth from a once-a-second poll.
### What it guarantees, and what it deliberately doesn't
- **At least once, not exactly once.** Stages 1 and 3 publish downstream *then*
ack upstream, so a crash in that window redelivers a job that was already
routed. Safe for the same reason replay is: every stage is a pure function of
the file's bytes and every commit is idempotent. The one new case is that a
duplicate can run *concurrently* with the original and find the file already
committed; `worker_loop` recognises a vanished `job.path` and settles it
quietly, rather than quarantining a file that in fact succeeded.
- **Publisher confirms on intake only.** A `202` means the broker has the file,
because a client may delete its copy on the strength of it. That costs a round
trip per file and serializes intake publishes; `FS_AMQP_CONFIRMS=false` turns
it off. Inter-stage publishes are fire-and-forget by design — a lost one leaves
its source job unacked, and redelivery repairs it for free, on a handoff that
otherwise costs 0.12 µs.
- **Advisory capacity.** `enqueue!` compares `FS_QUEUE_CAPACITY` against a depth
polled once a second (and adjusted locally in between), so the existing `503`
and `blocked_ns` behaviour still works, but a burst can overshoot by up to a
poll interval. Making the limit real would mean `x-max-length` with
`overflow: reject-publish`, which needs a confirm per message to detect.
- **No reconnect.** A dropped connection invalidates every in-flight delivery
tag, so reconnecting would silently reprocess whatever the workers were
holding. Instead the server drains and exits non-zero, and the restart policy
brings it back to redelivered messages and an intact `spool/`.
- **No dead-letter queue.** A failed job is quarantined to `failed/` and then
acked: the file's story and the message's story end in the same place. A DLQ
would hold messages pointing at files that had already moved.
- **One consumer process.** `Job` is a claim check carrying a *local* path, so a
second server on another host would be handed jobs whose files it cannot see.
The broker buys durable resume across restarts, not horizontal scale; scaling
out would additionally need `spool/` on shared storage and an aggregated
`/stats`.
- **Restart recovery skips `spool/`.** The broker is the record of what is in
flight, so re-driving `spool/` would duplicate the whole backlog on every
restart. `FS_RECOVER_SPOOL=true` forces it, for the one case the broker cannot
cover: it was purged or recreated and the spooled files are all that is left.
### Tests
The pure parts (URL parsing, the wire format, backend selection, the duplicate
branch) run in the normal suite. The round trip against a real broker is gated:
```bash
docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up -d rabbitmq
FS_TEST_AMQP_URL=amqp://darkstruct:darkstruct@localhost:5672/ \
julia --project=. -e 'using Pkg; Pkg.test()'
```
Without `FS_TEST_AMQP_URL` those tests are skipped with a notice, so the suite
still passes on a machine with no Docker.
## Running ## Running
@@ -214,7 +433,7 @@ or worker code changes.
julia --project=. -e 'using Pkg; Pkg.instantiate()' julia --project=. -e 'using Pkg; Pkg.instantiate()'
# external tools: exiftool (stage 2, required) and github-linguist (stage 4, # external tools: exiftool (stage 2, required) and github-linguist (stage 4,
# optional — programming-language detection). e.g. on Debian/Ubuntu: # optional, for programming-language detection). e.g. on Debian/Ubuntu:
# apt install libimage-exiftool-perl # apt install libimage-exiftool-perl
# gem install github-linguist # gem install github-linguist
@@ -222,6 +441,10 @@ julia --project=. -e 'using Pkg; Pkg.instantiate()'
julia --project=. -t auto bin/server.jl julia --project=. -t auto bin/server.jl
``` ```
By default the pipeline's queues are in-process, and nothing external is needed.
For durable queues that survive a crash, run against RabbitMQ instead — see
"Queue backends" above.
## Shutdown ## Shutdown
Both SIGINT and SIGTERM trigger the same idempotent graceful drain Both SIGINT and SIGTERM trigger the same idempotent graceful drain
@@ -229,12 +452,25 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
- **SIGINT** is caught as an `InterruptException` (we call - **SIGINT** is caught as an `InterruptException` (we call
`Base.exit_on_sigint(false)`), so shutdown is clean and quiet. `Base.exit_on_sigint(false)`), so shutdown is clean and quiet.
- **SIGTERM** can't be intercepted directly — Julia blocks it on worker threads
One caveat specific to the RabbitMQ backend: Julia delivers SIGINT to whichever
task happens to be running on thread 1, which is usually the main loop but is
not guaranteed to be — AMQPClient runs receiver tasks of its own, and an
interrupt that lands in one of those kills the broker connection instead of
reaching the main loop. That is not a stuck server: the depth poller notices
the closed connection within a second and asks for the same drain, so the
process still stops (within ~4s, exiting non-zero, and logging it as a lost
connection rather than an interrupt). **Under a process manager, prefer SIGTERM
for the broker backend** — it goes through `atexit`, which has no such lottery.
Shutdown on either signal also prints a few `Consumer ... task exiting` warnings
from AMQPClient, which are cosmetic: they are its consumer tasks noticing that
we cancelled them.
- **SIGTERM** can't be intercepted directly: Julia blocks it on worker threads
and handles it in its own runtime, so a user `signal()` handler never fires. and handles it in its own runtime, so a user `signal()` handler never fires.
Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path Instead we hook the drain into an `atexit` handler, which Julia's SIGTERM path
does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace does run. Caveat: Julia prints its own `signal 15: Terminated` backtrace
*before* `atexit` runs. It's harmless noise — the drain still completes right *before* `atexit` runs. It's harmless noise, and the drain still completes
after it — but if you want a fully quiet stop under a process manager, right after it, but for a fully quiet stop under a process manager,
configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker: configure it to send SIGINT instead (systemd: `KillSignal=SIGINT`; Docker:
`STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain `STOPSIGNAL SIGINT`). Give the stop timeout enough headroom to drain
in-flight work (systemd: `TimeoutStopSec`). in-flight work (systemd: `TimeoutStopSec`).
@@ -243,7 +479,7 @@ Both SIGINT and SIGTERM trigger the same idempotent graceful drain
Each file is scored by a fixed-structure neural network (Lux.jl) that answers a Each file is scored by a fixed-structure neural network (Lux.jl) that answers a
single binary question: is this file **known** (like the types in the training single binary question: is this file **known** (like the types in the training
set) or **unknown**? It's novelty detection, not exact file-typing — it won't set) or **unknown**? It's novelty detection, not exact file-typing: it won't
tell you "PDF", just "this looks like something I was trained on, or not". tell you "PDF", just "this looks like something I was trained on, or not".
- **Features:** the first 16 bytes + last 16 bytes of the file, each scaled - **Features:** the first 16 bytes + last 16 bytes of the file, each scaled
@@ -254,23 +490,26 @@ tell you "PDF", just "this looks like something I was trained on, or not".
- **Artifact:** trained weights live in `model/classifier.jld2` (committed), so - **Artifact:** trained weights live in `model/classifier.jld2` (committed), so
the server just loads them at startup. Missing/unreadable ⇒ the server fails the server just loads them at startup. Missing/unreadable ⇒ the server fails
fast rather than run without classification. fast rather than run without classification.
- **Effect today:** *annotate-only*. The class is logged - **Effect today:** *active routing*. The class is logged
(`classification=known|unknown`) but every file still moves to `done/`; the (`classification=known|unknown`) and drives the pipeline split: `:known` files
classifier can't misroute real files while it's unproven. go onto the stage-2 queue for metadata enrichment, `:unknown` files onto the
stage-3 queue for content triage. The class chooses the downstream stage (the
file itself stays in `data/spool/` either way); what's still unproven is the
model's *accuracy*, not whether the routing path runs.
The architecture and byte→feature mapping are defined once in `src/model.jl` and The architecture and byte→feature mapping are defined once in `src/model.jl` and
shared by the trainer and the server, so they can't drift apart. shared by the trainer and the server, so they can't drift apart.
### Training ### Training
Training is a separate, offline script — it never runs in the request path: Training is a separate, offline script. It never runs in the request path:
```bash ```bash
julia --project=. bin/train.jl <positives_dir> [negatives_dir] julia --project=. bin/train.jl <positives_dir> [negatives_dir]
``` ```
- **positives_dir** — every file in it (≥32 bytes) is a "known" example. - **positives_dir:** every file in it (≥32 bytes) is a "known" example.
- **negatives_dir** *(optional)* — a grab-bag of *other* real file types used as - **negatives_dir** *(optional)*: a grab-bag of *other* real file types used as
"unknown" examples. Negatives are generated ~1:1 with positives, split 50/50 "unknown" examples. Negatives are generated ~1:1 with positives, split 50/50
between uniform-random byte vectors and grab-bag files. With no grab-bag dir, between uniform-random byte vectors and grab-bag files. With no grab-bag dir,
negatives are all random (weaker: the net may just learn "high entropy = negatives are all random (weaker: the net may just learn "high entropy =
@@ -296,17 +535,30 @@ init, so the artifact is exactly regenerable from the same inputs.
| `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) | | `FS_UNKNOWN_QUEUE_CAPACITY` | `1000` | Max pending triage jobs (then backpressure) |
| `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks | | `FS_TEXT_WORKERS` | `nthreads()` | Stage-4 (language enrichment) worker tasks |
| `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) | | `FS_TEXT_QUEUE_CAPACITY` | `1000` | Max pending language jobs (then backpressure) |
| `FS_SPOOL_DIR` | `data/spool` | Incoming files (pending classification) | | `FS_SPOOL_DIR` | `data/spool` | Every in-flight file, at every stage |
| `FS_KNOWN_DIR` | `data/known` | Classified-known, awaiting enrichment |
| `FS_UNKNOWN_DIR` | `data/unknown` | Classified-unknown, awaiting content triage |
| `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary | | `FS_BINARY_DIR` | `data/binary` | Stage-3 sink: unknown files that look binary |
| `FS_TEXT_DIR` | `data/text` | Classified-text, awaiting language enrichment |
| `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) | | `FS_DONE_DIR` | `data/done` | Enriched known files (+ `.meta.json`) |
| `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) | | `FS_TEXT_DONE_DIR` | `data/text_done` | Enriched text files (+ `.meta.json`) |
| `FS_FAILED_DIR` | `data/failed` | Files whose processing threw | | `FS_FAILED_DIR` | `data/failed` | Files whose processing threw |
| `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup | | `FS_MODEL_PATH` | `model/classifier.jld2` | Classifier artifact loaded at startup |
| `FS_UPLOAD_CHUNK_BYTES` | `65536` | Socket read size at intake; bounds intake memory per in-flight upload |
| `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed | | `FS_EXIFTOOL_TIMEOUT` | `30` | Seconds before a stuck exiftool is killed |
| `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed | | `FS_LINGUIST_TIMEOUT` | `30` | Seconds before a stuck github-linguist is killed |
| `FS_CLUSTER_DIR` | `data/binary` | Stage-5 input: the unknown/binary pile to sweep |
| `FS_CLUSTER_N` | `32` | Header bytes modeled per file |
| `FS_CLUSTER_ALPHA` | `1.0` | CRP concentration (propensity to spawn new formats) |
| `FS_CLUSTER_PSEUDOCOUNT` | `0.1` | Dirichlet pseudocount β (calibrated) |
| `FS_CLUSTER_BG_MASS` | `5.0` | Mass of the uniform background component |
| `FS_PROMOTE_MIN_MEMBERS` | `20` | Cluster size threshold for promotion nomination |
| `FS_PROMOTE_MIN_MAGIC` | `3` | Required fixed signature positions to nominate |
| `FS_CLUSTER_CATALOG` | `data/catalog.json` | Durable stage-5 catalog file (single-owner) |
| `FS_NOMINATED_DIR` | `data/nominated` | One JSON per self-nominated cluster (human promote gate) |
| `FS_QUEUE_BACKEND` | `channel` | `channel` (in-process) or `rabbitmq` (durable); see "Queue backends" |
| `FS_AMQP_URL` | `amqp://guest:guest@localhost:5672/` | Broker connection, credentials included |
| `FS_AMQP_PREFIX` | `darkstruct` | Queue names are `<prefix>.<stage>` |
| `FS_AMQP_PREFETCH` | worker count | Unacked messages the broker hands one stage at a time |
| `FS_AMQP_CONFIRMS` | `true` | Wait for a publisher confirm before a `202` (intake only) |
| `FS_RECOVER_SPOOL` | `false` | Re-drive `spool/` at startup even on the broker backend (use after a purged broker) |
> To get real parallelism, start Julia with enough threads (`-t N`) to cover all > To get real parallelism, start Julia with enough threads (`-t N`) to cover all
> pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS` > pools. If `FS_WORKERS + FS_KNOWN_WORKERS + FS_UNKNOWN_WORKERS + FS_TEXT_WORKERS`
@@ -323,6 +575,9 @@ curl http://127.0.0.1:8080/health
# upload one or more files (multipart/form-data) # upload one or more files (multipart/form-data)
curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload curl -F "a=@report.pdf" -F "b=@data.csv" http://127.0.0.1:8080/upload
# 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]} # 202 {"accepted":[{"id":"<uuid>","name":"report.pdf"}, ...]}
# per-stage counters
curl http://127.0.0.1:8080/stats
``` ```
Each file in a request becomes its own job. Responses: Each file in a request becomes its own job. Responses:
@@ -332,25 +587,417 @@ Each file in a request becomes its own job. Responses:
- `503 Service Unavailable` — queue full, retry later - `503 Service Unavailable` — queue full, retry later
- `500 Internal Server Error` — failed to write a file to disk - `500 Internal Server Error` — failed to write a file to disk
### `GET /stats` — per-stage counters
The pipeline counts its own work (`src/stats.jl`), because nothing outside it
can. Every in-flight file sits in `data/spool/` no matter which stage it has
reached. The stage is a property of the queue holding its reference, and only
the pipeline can see that; there is no directory to poll.
```jsonc
{
"now": 1785725300.5, "since": 1785725291.9, "uptime_seconds": 8.5,
"intake": { "requests": 400, "files": 400, "bytes": 6553600, "rejected": 0 },
"stages": [
{ "stage": 4, "name": "language", "workers": 16,
"queue_depth": 184, "queue_capacity": 1000,
"completed": 200, "failed": 0, "bytes": 3276800,
"busy_seconds": 170.5, // summed handler time across the pool
"blocked_seconds": 0.0, // of that, time parked on a full downstream queue
"in_flight": 3 }
]
}
```
Counters are monotonic since startup, Prometheus-style. Rates are the reader's
job, so a scrape is stateless and two readers can't disturb each other. Take two
scrapes Δt apart and subtract:
```
throughput = Δcompleted / Δt
utilization = (Δbusy_seconds − Δblocked_seconds) / (Δt × workers)
```
**Utilization is the number that names the bottleneck.** In a pipeline every
stage completes the same files, so at steady state they all report near-identical
files/s no matter which one is the constraint; what separates them is how hard
each pool worked to keep up. The bottleneck sits near 1.0 with its queue backing
up while its neighbours idle.
`blocked_seconds` is what keeps that true. Stages 1 and 3 apply *blocking*
backpressure (a full downstream queue means the handler parks and retries rather
than dropping the file), and that wait happens inside the handler. Counting it as
busy would pin stage 1 at 1.0 whenever stage 2 is the real jam, making every
stage upstream of a jam look like the jam. Subtracted out, utilization means
"doing its own work", and a high blocked share becomes its own signal: a stage
blocked 90% of the time is naming its successor.
The counters are a handful of atomic adds per file, recorded in `worker_loop`:
the one place every stage's work passes through, so a new stage is instrumented
the moment it is wired up, and never on the read path.
## Benchmarking (throughput + memory)
There are five harnesses. Only the first needs a running server:
| script | measures | server? |
|---|---|---|
| `bin/bench.jl` (below) | intake, end-to-end and per-stage throughput; server RSS | **yes** |
| [`bin/bench_stage1.jl`](#stage-1-component-benchmark-binbench_stage1jl) | stage 1 taken apart: classify vs. rename vs. enqueue vs. logging | no |
| [`bin/bench_stage2.jl`](#stage-2-component-benchmark-binbench_stage2jl) | stage 2 taken apart: exiftool spawn vs. extraction vs. commit | no |
| [`bin/bench_model.jl`](#model-microbenchmark-binbench_modeljl) | the classifier alone: inference, feature reads, thread scaling | no |
| [`bin/cluster_calibrate.jl`](#unknown-format-discovery-stage-5-offline) | stage-5 clustering quality vs. an NCD baseline | no |
Running all of them from a clean checkout:
```bash
julia --project=. -e 'using Pkg; Pkg.instantiate()' # once
# 1. the model, on its own; no server involved
julia --project=. -t auto bin/bench_model.jl
# 1b. stage 1 taken apart; also no server
julia --project=. -t auto bin/bench_stage1.jl
# 1c. stage 2 taken apart; needs a directory of real files, not generated ones
julia --project=. -t auto bin/bench_stage2.jl
# 2. the pipeline. Start the server in one terminal…
julia --project=. -t auto bin/server.jl
# …and drive it from another. Restart the server between memory runs: Julia's
# GC returns memory to the OS lazily, so a second run starts inflated.
julia --project=. -t auto bin/bench.jl --files 2000 --size 8k --concurrency 32
julia --project=. -t auto bin/bench.jl --files 2 --size 1g --concurrency 1 # memory
julia --project=. -t auto bin/bench.jl --corpus ../training_set --concurrency 16
# 3. stage-5 clustering quality (offline, needs a labelled corpus)
julia --project=. bin/cluster_calibrate.jl ../training_set
```
The test suite is `julia --project=. -t auto -e 'using Pkg; Pkg.test()'`.
`bin/bench.jl` must run on the same machine as the server (it reads the sink dirs
and `/proc`); otherwise it only speaks HTTP: `/upload`, `/health`, and `/stats`
for the per-stage numbers.
Three properties of this design dictate how it measures:
- **HTTP latency is not throughput.** `/upload` returns `202` once the bytes are
spooled and a reference is enqueued, and all four stages run *after* the
response, so `ab`/`hey`/`wrk` would only ever measure intake. The bench
instead uploads a corpus and polls the terminal sinks (`done/`, `text_done/`, `binary/`,
`failed/`) until the count stops moving, and reports both numbers separately:
intake rate *and* end-to-end completion rate. It also samples `spool/`, whose
peak depth is the high-water mark of files in flight. But *which* stage they
are waiting on comes from `/stats`, not from disk.
- **End-to-end throughput doesn't name the slow stage.** The four stages run
concurrently behind their own queues, so the pipeline's rate *is* the slowest
stage's rate and the others are invisible in it. The bench scrapes
[`/stats`](#get-stats--per-stage-counters) before and after the run and
subtracts, giving each stage its own throughput, mean service time and
utilization:
```
PER-STAGE (server counters, delta over the end-to-end window)
stage files/s MiB/s svc ms util blocked peak queue failed
1 classify 32.52 0.51 38.7 0.08 0.0% 209/1000 0
2 enrich 0.24 0.0 541.0 0.01 0.0% 0/1000 0
3 triage 32.27 0.5 33.9 0.07 0.0% 83/1000 0
4 language 16.26 0.25 852.6 0.87 0.0% 184/1000 0
bottleneck stage 4 (language) at 87.0% utilization of 16 worker(s)
```
(400 mixed files, 16 KiB each, concurrency 16. Stage 4 is the constraint,
`github-linguist` being a process spawn per file at ~850 ms, and the 200 text
files queue up behind it while stages 1 and 3 idle at under 10%.) Read *down*
the util column, not the files/s column: each stage sees a different subset of
the corpus, so a low rate can just mean little work was routed there.
- **Memory should be flat in file size, and the sweep is what proves it.** Both
halves of the pipeline are bounded: the workers read bounded prefixes (16+16
bytes to classify, 8 KB to sniff, 64 KB to language-detect), and intake streams
each upload to disk a chunk at a time. So peak RSS should track *concurrency*,
not size. Measured on this machine (16 threads, 64 KiB chunk):
| upload size | concurrency | RSS growth |
|---|---|---|
| 256 MiB × 4 | 1 | 14.0 / 20.4 MiB (two runs) |
| 1 GiB × 2 | 1 | 22.3 MiB |
| 2 GiB × 1 | 1 | 31.3 MiB |
| 256 MiB × 4 | 4 | none measurable (peak stayed under the baseline) |
**Read the shape, not the digits.** Growth is flat across an 8× range of file
sizes: 14–31 MiB whether the upload is 256 MiB or 2 GiB. That is the claim
that matters, because nothing scales with file size, so nothing is buffering.
The absolute figures are *not* precise to the megabyte. A freshly started server
settles anywhere in an ~860–985 MiB band, so run-to-run variance in the
baseline is comparable to the growth being measured; the residual is GC churn
from the chunk reads, not retained buffers.
Two consequences worth knowing before quoting these numbers:
- **Let the server settle ~30s after startup** before a memory run, or the
baseline is sampled mid-fall and the run reports less growth than it caused
(or none at all, as the concurrency-4 row above did).
- **Linear-in-concurrency is not currently demonstrated.** An earlier
measurement put 4 concurrent 256 MiB uploads at 86.9 MiB (21.7 per in-flight
upload), but that run predates a fix to how the baseline was sampled: it was
read *before* the peak counter was reset, against a different origin. The
re-measurement above cannot reproduce it, and the growth sits under the noise
floor. Expect concurrency to cost memory; don't trust a specific coefficient
without a quieter machine or many more runs.
Before intake was streamed, the same 256 MiB upload grew RSS by ~700 MiB and 4
concurrent ones pushed a 950 MiB baseline past 2 GiB. So a `--size` sweep that
slopes upward is the regression signal: something has started buffering bodies
again.
Flags: `--files`, `--size` (`8k`/`64m`/`1g`), `--concurrency`, `--kind`
(`binary`/`text`/`mixed`, which chooses the stages that get loaded), `--corpus
DIR` (real files, the only way to exercise stage 2's exiftool path), `--pid`, `--no-mem`,
`--no-stats`, `--sample-ms`, `--timeout`, `--json PATH`. Full list in the script
header. The `--json` output carries the per-stage numbers too, so a sweep can be
compared run to run.
Two caveats the script reports rather than hides: it counts sink *deltas*, so it
warns if the pipeline isn't idle at the start (in-flight leftovers would be
counted as its own throughput); and because Julia's GC returns memory to the OS
lazily, a second run in the same process starts from an inflated baseline. It
resets the kernel's peak-RSS counter (`/proc/<pid>/clear_refs`) per run and flags
a drifted baseline, but for a clean growth figure restart the server between
memory runs.
### Stage-1 component benchmark (`bin/bench_stage1.jl`)
`bin/bench.jl` reports stage 1 as one number and `bin/bench_model.jl` takes the
*classifier* apart, but stage 1 is more than the model. Per file it also
renames the file into its stage directory, pushes a reference onto the
downstream queue, and logs. `bin/bench_stage1.jl` times each of those in
isolation, then times the real `handle_classify_job` end to end so the parts can
be checked against the whole:
```bash
julia --project=. -t auto bin/bench_stage1.jl
```
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12; 2000 ×
64 KiB files, minimum of 5 trials):
| component | per file | share of the handler |
|---|---|---|
| `classify()` | 10.7 µs | 38% |
| ↳ `read_features` | 7.9 µs | 28% |
| ↳ `Lux.apply` | 2.3 µs | 8% |
| `move_to` (rename) | 11.7 µs | 41% |
| `enqueue_blocking!` | 0.12 µs | 0.4% |
| per-file logging (disabled `@debug`) | 0.29 µs | 1% |
| **`handle_classify_job`** | **28.3 µs** | 100% |
**This benchmark is why stage 1's per-file log lines are `@debug` rather than
`@info`.** As `@info` they cost ~71 µs of the handler's ~118 µs, about 6× the
classifier and 6× the rename, and nearly all of it was `ConsoleLogger`
*formatting* (~64 µs), not the `FlushLogger`'s per-message flush (~8 µs on top).
Demoting them took stage 1 from 8.5k files/s to 35.3k files/s on a single worker,
a 4.2× speedup for no algorithmic change. The script still prices a formatted
line, so the cost of turning them back on is visible: running the handler under
`JULIA_DEBUG=DarkStruct` measures 133 µs per file, a 4.7× slowdown. That is the
trade. Per-file tracing is available when you want it and off by default, with
`GET /stats` giving per-file observability that is counted, not formatted.
What's left is evenly split between the rename and the classifier, and neither
has an easy 2×. Two things worth knowing:
- **The rename, not the model, is the single largest component** (11.7 µs), and
it's a plain `mv` within one filesystem. Inside `classify`, the same pattern
holds: 7.9 µs of the 10.7 µs is `read_features` (the `open`, the two reads
and the `seek`) against 2.3 µs of actual inference. Stage 1 is now a
filesystem-bound stage with a neural network attached, not the reverse.
- **Stage 1 now peaks at ~4 workers.** With the logger removed from the hot path
the sweep reads 35.0k/s at 1 worker, 66.6k/s at 2, **73.3k/s at 4**, then
*falls back* to 60.1k/s at 8 and 53.3k/s at 16, because every worker renaming
into the same two directories contends on the same directory inode. That ceiling
coincides with the one `bin/bench_model.jl` finds for inference, so ~4 is the
number from both directions: raising `FS_WORKERS` past it costs throughput.
Reported times are the minimum over trials. Flags: `--files`, `--reps`,
`--trials`, `--size`, `--dir`, `--model`, `--threads`, `--no-threads`,
`--json PATH`.
### Stage-2 component benchmark (`bin/bench_stage2.jl`)
Stage 2 is the one stage whose cost is dominated by something outside Julia
entirely: it forks `exiftool`, a Perl program, once per file. `bin/bench.jl`
reports the stage as a single throughput number, which can't distinguish "the
extraction is slow" from "the *spawn* is slow", and those have opposite fixes.
`bin/bench_stage2.jl` times each piece in isolation, then times the real
`handle_known_job` end to end:
```bash
julia --project=. -t auto bin/bench_stage2.jl
```
Two things make this benchmark different from the stage-1 one:
- **The corpus must be real files.** exiftool's cost depends on what it finds; a
file of random bytes bails out early and understates the stage by ~10×. The
default corpus is `data/done`, files that already went through stage 2 on this
machine. `--corpus PATH` points it elsewhere.
- **It prices the alternatives to one-fork-per-file**, because if the fork
dominates then the only fixes are to stop paying it per file. `exiftool
(batched Nx)` runs the whole corpus through one process; `exiftool
(-stay_open)` keeps one process alive and feeds it one file at a time over a
pipe, which is the shape a streaming pipeline could actually adopt. Both are
measured, not assumed.
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12,
exiftool 12.40; 150 real files / 102 MiB, minimum of 2 trials):
| component | per file | share of the handler |
|---|---|---|
| `run_exiftool()` | 135.9 ms | 98% |
| ↳ bare fork + Perl boot (`exiftool -ver`) | 76.7 ms | 55% |
| ↳ `JSON3.read` + tag map | 10 µs | 0.0% |
| `normalize_metadata` | 1.7 µs | 0.0% |
| `commit_enriched!` (sidecar + fsyncs + rename) | 2.0 ms | 1.5% |
| per-file logging (`@info`, flush→file) | 47 µs | 0.0% |
| **`handle_known_job`** | **138.3 ms** | 100% |
| *alt:* `exiftool -stay_open` | 40.1 ms | 29% |
| *alt:* `exiftool` batched 150× | 37.3 ms | 27% |
**Stage 2 is exiftool and nothing else.** Everything the Julia code does
(parsing, normalizing, the durable sidecar-first commit, the log line) sums to
about 1.5% of the stage. There is no point optimizing any of it.
**More than half the stage is interpreter startup, not metadata extraction.**
The bare `exiftool -ver` (fork, Perl boot, module loads, read no file) costs
76.7 ms against a 135.9 ms full call. Both fork-free alternatives agree on what's
left: ~37–40 ms of actual work per file. So a persistent exiftool would cut the
stage by ~70%, and `-stay_open` gets there without giving up the one-file-in,
one-result-out shape the pipeline needs. That remains the single biggest
available win in this stage; it is measured here but not yet implemented.
**This benchmark is also why `run_with_timeout` no longer polls.** The original
watchdog polled with `sleep(0.1)` and then joined the polling task, so every call
paid the remainder of an in-flight sleep *after* the child had already exited:
~25 ms per file here, and a measured 101 ms on a process that exits instantly.
Replacing it with a one-shot `Timer` took the stage from 164.6 ms to 138.3 ms per
file (6→8 files/s on one worker) and cost nothing in behavior. Stage 4 shares the
wrapper and got the same fix for free.
Writing the missing tests for that wrapper turned up a second, worse problem:
**the timeout was never enforceable.** `wait(proc)` returns only once the
captured stdout pipe closes, and any grandchild inherits that pipe, so
signalling the child alone left the worker blocked until the whole process tree
finished on its own (a `sh -c "trap '' TERM; sleep 30"` child ran the full 30 s
against a 1 s timeout, under both the old and new watchdog). The child now runs
in its own process group and the timeout signals the group. The trade is that a
hard crash of the server orphans an in-flight child rather than taking it down;
these children are short-lived and timeout-bounded, which is the cheaper side of
it.
**Stage 2 scales to ~8 workers, then flattens**: 8 files/s at 1 worker, 14 at 2,
28 at 4, **49 at 8**, and 49 at 16. The machine runs out of cores to run Perl
on, which is exactly what you'd expect of a stage that is ~100% subprocess. Note
that the sweep pulls from a shared counter rather than splitting the corpus into
contiguous slices: per-file exiftool time spans two orders of magnitude on a real
corpus (one 2.1 s archive among 48 files), and a static split reports a scaling
ceiling that is really just load imbalance.
One caveat the numbers raise but don't answer: **`fsync_dir` measures 1.75 µs**,
which is far too fast to be a real disk flush. The durability that
`commit_enriched!` is written for may not survive power loss on this filesystem,
even though the code is correct. That's a correctness question, not a speed one,
and it is not yet resolved.
Reported times are the minimum over trials. Flags: `--files`, `--reps`,
`--trials`, `--corpus`, `--dir`, `--timeout`, `--threads`, `--no-threads`,
`--no-stay-open`, `--json PATH`.
### Model microbenchmark (`bin/bench_model.jl`)
`bin/bench.jl` reports stage 1 as a single number: the wall time of
`handle_classify_job`, which is a feature read, an inference, a rename, a
(disabled) debug line, and whatever contention the other three pools create.
That's the right number for capacity planning and the wrong one for "is the
model slow?".
`bin/bench_model.jl` answers that separately, with no server, queue, or HTTP
involved:
```bash
julia --project=. -t auto bin/bench_model.jl
```
Measured on this machine (Ryzen 7 2700X, 8 cores/16 threads, Julia 1.12):
| what | per file | notes |
|---|---|---|
| `Lux.apply`, batch 1 | **2.3 µs** | 768 B allocated per call |
| `read_features` | **4.2–6.0 µs** | flat across a 262,144× size range (1 KiB → 256 MiB) |
| `classify()` | **8.4 µs** | 73% feature read, 27% inference |
So the model is not the pipeline's problem, by three orders of magnitude: the
same run measured stage 1 at 38.7 ms per file, ~4,500× the 8.4 µs `classify()`
costs. Whatever stage 1 spends its time on, it isn't the network. (That 38.7 ms
predates the `@debug` demotion above and is a whole-pipeline figure. It includes
time the stage-1 worker spends *blocked* on a full downstream queue, which is why
it is three orders of magnitude above the 28.3 µs the handler costs in
isolation. For the uncontended split, see
[the stage-1 decomposition](#stage-1-component-benchmark-binbench_stage1jl).)
Two findings worth acting on if stage 1 ever *does* become the constraint:
- **Batching would buy ~13×.** A 32×1 matmul wastes most of a BLAS call:
batch 64 costs 280 ns/file and batch 512 costs 179 ns/file, against 2.34 µs
one at a time. The pipeline classifies strictly one file per job today, so it
pays the worst row in that table.
- **Inference does not scale past ~4 threads.** Concurrent `Lux.apply` on the
shared read-only `Classifier` peaks around 1.2M files/s at 4 tasks and then
*falls back* to single-thread throughput at 16. The script runs a pure-compute
control kernel through the same sweep to place the blame: the control reaches
14.3× at 16 tasks (90% efficiency) on the same box, so the machine
parallelizes and `Lux.apply` doesn't. GC is only ~1% of it, so allocation
pressure isn't the explanation either. The cause is inside Lux/BLAS and is
not diagnosed here. The practical consequence: raising `FS_WORKERS` past ~4
adds no classification throughput.
Reported times are the minimum over trials, because for a microbenchmark the
floor is the signal and everything above it is scheduler and GC noise. Every
timed loop stores its result in a sink so a pure call can't be hoisted out.
Flags: `--model`, `--reps`, `--trials`, `--batches`, `--sizes`, `--no-threads`,
`--json PATH`.
## Layout ## Layout
``` ```
src/ src/
FileServer.jl module + run() (startup, recovery, workers, serve, shutdown) DarkStruct.jl module + run() (startup, recovery, workers, serve, shutdown)
config.jl Config struct + env parsing config.jl Config struct + env parsing
job.jl Job (the queue reference) job.jl Job (the queue reference)
queue.jl JobQueue seam + in-process ChannelQueue queue.jl JobQueue seam + in-process ChannelQueue
spool.jl filename sanitizing, spool/move, startup recovery rabbit.jl RabbitMQ-backed JobQueue: durable queues, acks, confirms
stats.jl per-stage counters behind GET /stats (throughput, utilization)
multipart.jl streaming multipart/form-data reader (intake never buffers a file)
spool.jl filename sanitizing, streaming spool/move, startup recovery
model.jl NN architecture + byte→feature mapping (shared with trainer) model.jl NN architecture + byte→feature mapping (shared with trainer)
classify.jl load artifact + classify a file at inference time classify.jl load artifact + classify a file at inference time
metadata.jl exiftool extraction + normalized sidecar (stage 2) metadata.jl exiftool extraction + normalized sidecar (stage 2)
content.jl binary-vs-text sniff for unknown files (stage 3) content.jl binary-vs-text sniff for unknown files (stage 3)
language.jl natural + programming language enrichment for text (stage 4) language.jl natural + programming language enrichment for text (stage 4)
cluster.jl header-byte clustering model + Gibbs + scoring core (stage 5, science)
catalog.jl durable single-owner format catalog + sweep + nominations (stage 5, phase B)
worker.jl parametrized worker loop + classify/enrich/triage/language handlers worker.jl parametrized worker loop + classify/enrich/triage/language handlers
server.jl HTTP routes/handlers server.jl HTTP routes + the streaming /upload handler
docker-compose.yml the server, in-process queues
docker-compose.rabbitmq.yml overlay: adds the broker and switches the backend
bin/ bin/
server.jl entry point server.jl entry point
train.jl offline training script → model/classifier.jld2 bench.jl throughput + memory harness against a running server
bench_model.jl classifier microbenchmark (inference, feature reads, scaling)
bench_stage1.jl stage-1 decomposition (classify vs. rename vs. enqueue vs. logging)
train.jl offline training script → model/classifier.jld2
cluster_calibrate.jl offline stage-5 hyperparameter calibration + NCD baseline
cluster_sweep.jl stage-5 phase-B runner: sweep binary/, update catalog, write nominations
model/ model/
classifier.jld2 committed trained weights (loaded at startup) classifier.jld2 committed trained weights (loaded at startup)
DESIGN_clustering.md stage-5 design rationale + calibration results
``` ```

784
bin/bench.jl Executable file
View File

@@ -0,0 +1,784 @@
#!/usr/bin/env julia
#
# bench.jl: measure end-to-end throughput and server memory for a running DarkStruct.
#
# Two things make this pipeline awkward to benchmark with off-the-shelf tools
# (ab/hey/wrk), and both shape what this script does:
#
# 1. HTTP latency is not throughput. /upload returns 202 as soon as the bytes
# are spooled and a reference is enqueued, and all four stages run afterwards.
# So real throughput is the *arrival rate at the terminal sinks*
# (done/, text_done/, binary/, failed/), not the response rate. We upload a
# corpus, then poll the sinks until the file count stops moving.
#
# 2. End-to-end throughput doesn't name the slow stage. The four stages run
# concurrently behind their own queues, so the pipeline's rate is the
# slowest stage's rate and the others are invisible. Directory polling can't
# recover them either: every in-flight file sits in spool/ whatever stage it
# is at, since stages route by enqueueing rather than by moving bytes. So the
# server keeps per-stage counters
# (src/stats.jl) and we scrape GET /stats before and after: the deltas give
# each stage's throughput, mean service time, and worker utilization, and
# utilization is what actually names the bottleneck (see `stage_report`).
#
# 3. Memory should be flat in file size, and that claim needs checking on two
# axes. The workers read bounded prefixes (16+16 bytes to classify, 8 KB to
# sniff, 64 KB to language-detect), and intake streams each upload from the
# socket to the spool file a chunk at a time (FS_UPLOAD_CHUNK_BYTES, see
# src/multipart.jl). So peak RSS should track *concurrency*, not file size:
# sweeping --size at fixed --concurrency should be a flat line, and that is
# the regression this measures. We sample the server's RSS throughout and
# report the high-water mark.
#
# (Before intake was streamed it buffered each upload whole, several times
# over, and a 256 MiB upload grew RSS by ~700 MiB. If a --size sweep ever
# slopes upward again, something has started buffering.)
#
# The harness talks to the server only over HTTP (/upload, /health, /stats) and
# reads the on-disk sink layout; it must run on the same machine (sink dirs and
# /proc). If /stats is missing (an older build) everything else still works and
# the per-stage section is skipped.
#
# Usage:
# julia --project=. -t auto bin/bench.jl [options]
#
# --url URL server base URL (default: $FS_URL or http://127.0.0.1:8080)
# --files N number of files to upload (default: 200)
# --size S size of each generated file, e.g. 4k, 512k, 8m, 1g (default: 64k)
# --concurrency J uploads in flight at once (default: 8)
# --kind K binary | text | mixed: what to generate (default: binary)
# --corpus DIR upload an existing directory instead of generating
# (the only way to exercise stage 2: point it at real known files)
# --keep-corpus don't delete the generated corpus on exit
# --pid PID server pid for memory sampling (default: autodetect)
# --no-mem skip memory sampling entirely
# --no-stats skip the per-stage /stats scrape and its table
# --sample-ms MS sink/RSS sampling interval (default: 200)
# --timeout SEC give up after this long with no drain progress (default: 120)
# --json PATH also write the results as JSON
# --force skip the corpus-size safety check
#
# Examples:
# # throughput: many small files, high concurrency
# julia --project=. -t auto bin/bench.jl --files 2000 --size 8k --concurrency 32
#
# # memory: flat in file size? sweep --size with concurrency pinned
# julia --project=. -t auto bin/bench.jl --files 4 --size 256m --concurrency 1
# julia --project=. -t auto bin/bench.jl --files 2 --size 1g --concurrency 1
# julia --project=. -t auto bin/bench.jl --files 8 --size 1g --concurrency 8
#
# # stage 2 (exiftool) with real known files
# julia --project=. -t auto bin/bench.jl --corpus ../training_set --concurrency 16
using HTTP
using JSON3
using Random
# ---------------------------------------------------------------- option parsing
const DEFAULTS = Dict{String,Any}(
"url" => get(ENV, "FS_URL", "http://127.0.0.1:8080"),
"files" => 200,
"size" => 64 * 1024,
"concurrency" => 8,
"kind" => "binary",
"corpus" => nothing,
"keep-corpus" => false,
"pid" => nothing,
"no-mem" => false,
"no-stats" => false,
"sample-ms" => 200,
"timeout" => 120,
"json" => nothing,
"force" => false,
)
const FLAGS = ("keep-corpus", "no-mem", "no-stats", "force")
# Approximate RSS of a freshly started server (Lux + the loaded classifier + the
# language detector, measured on Julia 1.12 / -t auto). Only used to notice that
# a baseline is inflated by a previous run's un-returned GC memory, so a rough
# figure is enough.
const FRESH_RSS_HINT = 950 * 1024^2
"Parse `4k`/`8M`/`1g`/`4096` into a byte count."
function parse_size(s::AbstractString)::Int
m = match(r"^(\d+(?:\.\d+)?)\s*([kKmMgG]?)[bB]?$", strip(s))
m === nothing && error("bad --size: $s (expected e.g. 512, 64k, 8m, 1g)")
mult = Dict('k' => 1024, 'm' => 1024^2, 'g' => 1024^3)
scale = isempty(m[2]) ? 1 : mult[lowercase(m[2])[1]]
return round(Int, parse(Float64, m[1]) * scale)
end
function parse_args(argv)::Dict{String,Any}
opts = copy(DEFAULTS)
i = 1
while i <= length(argv)
a = argv[i]
startswith(a, "--") || error("unexpected argument: $a (see the header of $(PROGRAM_FILE))")
key = a[3:end]
haskey(opts, key) || error("unknown option: $a")
if key in FLAGS
opts[key] = true
i += 1
continue
end
i + 1 <= length(argv) || error("option --$key needs a value")
val = argv[i+1]
opts[key] = key == "size" ? parse_size(val) :
key in ("files", "concurrency", "sample-ms") ? parse(Int, val) :
key == "timeout" ? parse(Float64, val) :
key == "pid" ? parse(Int, val) :
val
i += 2
end
opts["kind"] in ("binary", "text", "mixed") ||
error("--kind must be binary, text or mixed (got $(opts["kind"]))")
opts["files"] >= 1 || error("--files must be >= 1")
opts["concurrency"] >= 1 || error("--concurrency must be >= 1")
return opts
end
# ------------------------------------------------------------------------- dirs
#
# Resolved from the same environment variables src/config.jl reads, so a server
# started with custom dirs is benchmarked correctly. Kept as a standalone table
# rather than `using DarkStruct` so the harness doesn't pay to load Lux.
sinkdirs() = (
done = get(ENV, "FS_DONE_DIR", "data/done"),
text_done = get(ENV, "FS_TEXT_DONE_DIR", "data/text_done"),
binary = get(ENV, "FS_BINARY_DIR", "data/binary"),
failed = get(ENV, "FS_FAILED_DIR", "data/failed"),
)
# One directory, not four. Files no longer move between stages: spool/ holds
# every in-flight file at every stage, and which stage it has reached lives in
# the queue holding its reference (src/worker.jl header). So this depth is
# "files in flight", full stop; per-stage depth comes from /stats, which is the
# only place that can see it at all.
stagedirs() = (spool = get(ENV, "FS_SPOOL_DIR", "data/spool"),)
"Count work items in `dir`, ignoring the .meta.json sidecars stages 2/4 write."
function count_files(dir::AbstractString)::Int
isdir(dir) || return 0
n = 0
for name in readdir(dir)
endswith(name, ".meta.json") && continue
isfile(joinpath(dir, name)) && (n += 1)
end
return n
end
counts(dirs) = NamedTuple{keys(dirs)}(map(count_files, values(dirs)))
total(c) = sum(values(c))
deltas(now_, base) = NamedTuple{keys(now_)}(map(-, values(now_), values(base)))
# ----------------------------------------------------------------------- memory
"Read (VmRSS, VmHWM) in bytes for `pid`, or `nothing` if unreadable."
function read_rss(pid::Int)
rss = hwm = nothing
try
for line in eachline("/proc/$pid/status")
if startswith(line, "VmRSS:")
rss = parse(Int, split(line)[2]) * 1024
elseif startswith(line, "VmHWM:")
hwm = parse(Int, split(line)[2]) * 1024
end
end
catch
return nothing
end
return (rss === nothing || hwm === nothing) ? nothing : (rss, hwm)
end
"""
Find the running server process, or `nothing`.
`pgrep -f` matches against the whole command line, which catches more than the
server: any shell launched with the command in its own argv (`sh -c 'julia …
bin/server.jl > log'`, a `setsid`/`nohup` wrapper, even the terminal running the
benchmark) matches the same pattern. Sampling one of those reports a few MiB of
shell as the server's memory: a wrong answer that looks plausible, which is the
worst kind.
So candidates are filtered by what each process *is* (`/proc/<pid>/comm`, the
executable name) rather than by what its arguments say. No pattern over argv can
make that distinction.
"""
function detect_pid()
out = try
readchomp(`pgrep -f "bin/server.jl"`)
catch
return nothing
end
candidates = parse.(Int, split(out))
pids = filter(candidates) do pid
comm = try
readchomp("/proc/$pid/comm")
catch
return false # exited between pgrep and here
end
startswith(comm, "julia")
end
isempty(pids) && return nothing
length(pids) > 1 && @warn "multiple server processes matched; sampling the first" pids
return first(pids)
end
"""
Reset the kernel's peak-RSS counter so VmHWM reflects only this run.
Without it, VmHWM carries the high-water mark from startup (model load) or from
an earlier benchmark, which would silently dominate a small run's result.
Requires the server to run as the same user; on failure we say so and fall back
to sampled VmRSS, which can miss a spike between samples.
"""
function reset_peak_rss(pid::Int)::Bool
try
write("/proc/$pid/clear_refs", "5")
return true
catch
return false
end
end
# ------------------------------------------------------------- per-stage counters
#
# The server exposes monotonic counters at GET /stats (src/stats.jl). Rates are
# ours to compute: scrape once before the run and once after, subtract, divide by
# the elapsed *server* clock so a slow scrape doesn't distort the window.
"Fetch and parse GET /stats, or `nothing` if the server doesn't serve it."
function scrape_stats(url::String)
try
resp = HTTP.get(string(rstrip(url, '/'), "/stats");
status_exception = false, retry = false, readtimeout = 5)
resp.status == 200 || return nothing
return JSON3.read(String(resp.body))
catch
return nothing
end
end
"One stage's activity between two scrapes."
struct StageDelta
stage::Int
name::String
workers::Int
completed::Int
failed::Int
bytes::Int
busy::Float64 # summed handler seconds across all workers in the pool
blocked::Float64 # of `busy`, seconds parked on a full downstream queue
peak_depth::Int # deepest its queue got, from the sampler
capacity::Int
end
files_per_sec(d::StageDelta, window) = d.completed / max(window, 1e-9)
mib_per_sec(d::StageDelta, window) = d.bytes / max(window, 1e-9) / 1024^2
"Mean wall time one file spends in one worker of this stage."
service_ms(d::StageDelta) = d.completed == 0 ? NaN :
(d.busy - d.blocked) / d.completed * 1000
"""
Fraction of the pool's capacity spent doing this stage's own work.
Blocked time is subtracted first: a stage parked on a full downstream queue is
waiting, not working, and leaving it in would light up every stage upstream of a
jam as though each were the jam.
"""
utilization(d::StageDelta, window) =
(d.busy - d.blocked) / max(window * d.workers, 1e-9)
blocked_share(d::StageDelta) = d.busy <= 0 ? 0.0 : d.blocked / d.busy
"Subtract two scrapes into per-stage deltas, folding in sampled peak depths."
function stage_deltas(before, after, peak_depth::Dict{Int,Int})
out = StageDelta[]
for (b, a) in zip(before.stages, after.stages)
push!(out, StageDelta(a.stage, String(a.name), a.workers,
a.completed - b.completed,
a.failed - b.failed,
a.bytes - b.bytes,
a.busy_seconds - b.busy_seconds,
a.blocked_seconds - b.blocked_seconds,
get(peak_depth, Int(a.stage), 0),
a.queue_capacity))
end
return out
end
"JSON has no NaN. A stage that completed nothing has no service time, and `null`
is the honest way to say that; writing NaN just makes JSON3 throw."
json_num(x::Real) = isfinite(x) ? x : nothing
pad(s, n) = rpad(string(s), n)
lpad_(s, n) = lpad(string(s), n)
"""
Print the per-stage table and say which stage is the bottleneck.
The verdict reads utilization, not throughput: in a pipeline every stage
completes the same files, so at steady state they all report nearly the same
files/s regardless of which one is the constraint. What separates them is how
hard each pool had to work to keep up: the bottleneck is pinned near 1.0 while
its neighbours idle.
"""
function stage_report(deltas::Vector{StageDelta}, window::Float64)
println("PER-STAGE (server counters, delta over the end-to-end window)")
println(" stage files/s MiB/s svc ms util blocked peak queue failed")
for d in deltas
svc = service_ms(d)
println(" $(d.stage) $(pad(d.name, 10)) " *
lpad_(fmt(files_per_sec(d, window)), 8) * " " *
lpad_(fmt(mib_per_sec(d, window)), 8) * " " *
lpad_(isnan(svc) ? "—" : fmt(svc, 1), 8) * " " *
lpad_(fmt(utilization(d, window)), 6) * " " *
lpad_(fmt(blocked_share(d) * 100, 0) * "%", 7) * " " *
lpad_("$(d.peak_depth)/$(d.capacity)", 11) * " " *
lpad_(d.failed, 7))
end
# Per-stage files/s are not comparable across rows and saying so costs one
# line: the stages process different subsets (stage 2 only known files,
# stage 4 only text), so a low rate can mean "little work arrived here"
# rather than "slow". Utilization is the column that compares.
println(" (files/s counts only files routed to that stage; svc is per-file wall time in")
println(" one worker; util = (busy − blocked) / (window × workers))")
worked = filter(d -> d.completed > 0, deltas)
if isempty(worked)
println(" (no stage completed a file in this window)")
return nothing
end
top = argmax(d -> utilization(d, window), worked)
println(" bottleneck stage $(top.stage) ($(top.name)) at " *
"$(fmt(utilization(top, window) * 100, 0))% utilization of " *
"$(top.workers) worker(s)")
if utilization(top, window) < 0.5
println(" ...but no stage is near saturated: the pipeline is " *
"waiting on intake,\n not on itself. Raise --concurrency " *
"or --files to load it properly.")
end
for d in worked
blocked_share(d) > 0.25 && println(" ! stage $(d.stage) ($(d.name)) spent " *
"$(fmt(blocked_share(d) * 100, 0))% of its time parked on a full downstream " *
"queue.\n It is being held up by the stage after it, not doing that work itself.")
end
return nothing
end
# ------------------------------------------------------------------------ corpus
const WORDS = split("the quick brown fox jumps over a lazy dog while parsing " *
"headers and spooling bytes onto disk for later enrichment " *
"because throughput matters more than latency here")
"Write one file of exactly `size` bytes, in bounded chunks so the generator
itself never holds a whole 1 GB file in memory."
function write_file(path::AbstractString, size::Int, kind::Symbol, rng)
chunk = 1024 * 1024
open(path, "w") do io
remaining = size
while remaining > 0
n = min(chunk, remaining)
if kind === :binary
write(io, rand(rng, UInt8, n))
else
buf = IOBuffer()
while buf.size < n
print(buf, rand(rng, WORDS), rand(rng) < 0.06 ? ".\n" : " ")
end
write(io, take!(buf)[1:n])
end
remaining -= n
end
end
return nothing
end
"Generate the corpus and return (dir, paths, total_bytes)."
function make_corpus(opts)
n, size, kind = opts["files"], opts["size"], opts["kind"]
totalbytes = n * size
if totalbytes > 16 * 1024^3 && !opts["force"]
error("corpus would be $(human(totalbytes)) on disk; pass --force if that's intended")
end
dir = mktempdir(; prefix = "fsbench-")
rng = MersenneTwister(1234)
paths = String[]
for i in 1:n
k = kind == "mixed" ? (isodd(i) ? :binary : :text) :
kind == "text" ? :text : :binary
ext = k === :text ? "txt" : "bin"
path = joinpath(dir, "bench-$(lpad(i, 6, '0')).$ext")
write_file(path, size, k, rng)
push!(paths, path)
end
return dir, paths, totalbytes
end
function existing_corpus(dir::AbstractString)
isdir(dir) || error("--corpus is not a directory: $dir")
paths = sort(filter(isfile, readdir(dir; join = true)))
isempty(paths) && error("--corpus directory is empty: $dir")
return paths, sum(filesize, paths)
end
# ----------------------------------------------------------------------- upload
struct Upload
status::Int # HTTP status, or 0 if the request threw
accepted::Int # jobs the server actually queued (from the 202/503 body)
seconds::Float64
end
"POST one file as multipart/form-data and report what the server accepted."
function upload_one(url::String, path::String)::Upload
t0 = time()
try
form = HTTP.Form(["file" => HTTP.Multipart(basename(path), open(path, "r"),
"application/octet-stream")])
resp = HTTP.post(url, [], form; status_exception = false, retry = false)
# 202 and 503 both carry an `accepted` array: a partially-accepted batch
# still queued those jobs, and they will show up in the sinks.
acc = try
length(JSON3.read(String(resp.body)).accepted)
catch
resp.status == 202 ? 1 : 0
end
return Upload(resp.status, acc, time() - t0)
catch e
e isa InterruptException && rethrow()
@warn "upload failed" file = basename(path) exception = e
return Upload(0, 0, time() - t0)
end
end
"""
Upload every path, at most `concurrency` in flight.
A bounded set of worker tasks pulling from a shared index keeps exactly
`concurrency` requests in flight for the whole run, unlike batching, where each
batch stalls on its slowest (largest) file and the real concurrency sags.
"""
function upload_all(url::String, paths::Vector{String}, concurrency::Int)
endpoint = string(rstrip(url, '/'), "/upload")
results = Vector{Upload}(undef, length(paths))
next = Threads.Atomic{Int}(1)
@sync for _ in 1:min(concurrency, length(paths))
Threads.@spawn while true
i = Threads.atomic_add!(next, 1)
i > length(paths) && break
results[i] = upload_one(endpoint, paths[i])
end
end
return results
end
# ------------------------------------------------------------------- formatting
function human(bytes::Real)
b = Float64(bytes)
for unit in ("B", "KiB", "MiB", "GiB", "TiB")
(abs(b) < 1024 || unit == "TiB") && return "$(round(b; digits = 2)) $unit"
b /= 1024
end
end
fmt(x::Real, digits::Int = 2) = string(round(Float64(x); digits = digits))
function percentile(sorted::Vector{Float64}, p::Float64)
isempty(sorted) && return NaN
idx = clamp(ceil(Int, p * length(sorted)), 1, length(sorted))
return sorted[idx]
end
# ------------------------------------------------------------------------- main
function main(argv)
opts = parse_args(argv)
url = string(opts["url"])
# Fail fast and clearly if there's no server, rather than reporting a run of zeros.
try
HTTP.get(string(rstrip(url, '/'), "/health"); retry = false, readtimeout = 5)
catch e
println(stderr, "cannot reach $url/health. Is the server running?")
println(stderr, " start it with: julia --project=. -t auto bin/server.jl")
return 1
end
sinks, stages = sinkdirs(), stagedirs()
# Leftovers mid-pipeline would land in the sinks during our window and be
# counted as our throughput, so say so up front rather than quietly skewing.
pending = total(counts(stages))
pending > 0 && @warn "pipeline is not idle: $pending file(s) still in spool/; " *
"throughput will include their completions"
# --- corpus
generated = opts["corpus"] === nothing
corpusdir, paths, corpusbytes = if generated
print("generating corpus: $(opts["files"]) × $(human(opts["size"])) ($(opts["kind"]))… ")
t = time()
d, p, b = make_corpus(opts)
println("done in $(fmt(time() - t))s → $d")
d, p, b
else
p, b = existing_corpus(String(opts["corpus"]))
println("using corpus: $(length(p)) file(s), $(human(b)) from $(opts["corpus"])")
String(opts["corpus"]), p, b
end
try
pid = opts["no-mem"] ? nothing : something(opts["pid"], detect_pid(), Some(nothing))
if pid === nothing && !opts["no-mem"]
@warn "could not find the server process; skipping memory (pass --pid PID)"
end
baseline_rss = nothing
peak_reset = false
if pid !== nothing
r = read_rss(pid)
r === nothing && (@warn "cannot read /proc/$pid/status; skipping memory"; pid = nothing)
if pid !== nothing
peak_reset = reset_peak_rss(pid)
peak_reset || @warn "could not reset the peak-RSS counter (/proc/$pid/clear_refs); " *
"reporting sampled RSS only"
# Baseline is read *after* the reset, not before. VmHWM restarts
# from whatever RSS is at the moment of the reset, so a baseline
# sampled earlier is measured against a different origin, and if
# the GC hands memory back in between, the run reports negative
# growth, which is nonsense on its face.
r2 = read_rss(pid)
baseline_rss = r2 === nothing ? r[1] : r2[1]
end
end
if Threads.nthreads() == 1 && opts["size"] > 64 * 1024^2
@warn "running with 1 thread and large files: the sampler shares a thread with " *
"blocking file reads, so the RSS curve will be coarse. Prefer -t auto."
end
base_sinks = counts(sinks)
interval = opts["sample-ms"] / 1000
# --- per-stage counters: the "before" half of the delta.
stats_before = opts["no-stats"] ? nothing : scrape_stats(url)
if stats_before === nothing && !opts["no-stats"]
@warn "no /stats endpoint on this server; skipping the per-stage table " *
"(the server predates src/stats.jl)"
end
# --- sampler: RSS curve, spool depth, and queue depths.
stop = Threads.Atomic{Bool}(false)
rss_samples = Float64[]
depth_max = Dict(k => 0 for k in keys(stages))
queue_peak = Dict{Int,Int}()
sampler = Threads.@spawn begin
while !stop[]
if pid !== nothing
r = read_rss(pid)
r !== nothing && push!(rss_samples, Float64(r[1]))
end
d = counts(stages)
for k in keys(d)
depth_max[k] = max(depth_max[k], getfield(d, k))
end
# Queue depth, unlike directory depth, can't be missed by a slow
# sample in the same way, since a file's *reference* sits in the
# queue for the whole time it waits, so this is the depth the stage
# table reports.
if stats_before !== nothing
s = scrape_stats(url)
s === nothing || for st in s.stages
k = Int(st.stage)
queue_peak[k] = max(get(queue_peak, k, 0), Int(st.queue_depth))
end
end
sleep(interval)
end
end
# --- intake
println("uploading $(length(paths)) file(s) at concurrency $(opts["concurrency"])…")
t_start = time()
ups = upload_all(url, paths, opts["concurrency"])
t_intake_end = time()
accepted = sum(u.accepted for u in ups)
n_202 = count(u -> u.status == 202, ups)
n_503 = count(u -> u.status == 503, ups)
n_err = count(u -> !(u.status in (202, 503)), ups)
intake_secs = t_intake_end - t_start
lat = sort([u.seconds for u in ups])
println(" intake: $accepted job(s) accepted in $(fmt(intake_secs))s " *
"($(fmt(accepted / max(intake_secs, 1e-9))) files/s, " *
"$(fmt(corpusbytes / max(intake_secs, 1e-9) / 1024^2)) MiB/s)")
n_503 > 0 && println(" backpressure: $n_503 request(s) got 503 (intake queue full)")
n_err > 0 && println(" errors: $n_err request(s) failed or returned an unexpected status")
# --- drain: poll the terminal sinks until they stop moving.
println("draining (polling sinks every $(opts["sample-ms"])ms)…")
completed = 0
t_last_progress = time()
t_last_completion = t_intake_end
timed_out = false
while completed < accepted
sleep(interval)
c = total(deltas(counts(sinks), base_sinks))
if c > completed
completed = c
t_last_completion = time()
t_last_progress = t_last_completion
elseif time() - t_last_progress > opts["timeout"]
timed_out = true
break
end
end
# Scrape before stopping the sampler, so the window closes as near the
# last completion as we can manage.
stats_after = stats_before === nothing ? nothing : scrape_stats(url)
stop[] = true
wait(sampler)
sink_delta = deltas(counts(sinks), base_sinks)
e2e_secs = t_last_completion - t_start
# The stage window is the server's own clock across the two scrapes, not
# e2e_secs: it starts a scrape earlier and ends a scrape later, and using
# our wall time against its counters would misattribute the difference.
stage_window, stage_delta = if stats_after === nothing
(0.0, StageDelta[])
else
(Float64(stats_after.now - stats_before.now),
stage_deltas(stats_before, stats_after, queue_peak))
end
final_rss = pid === nothing ? nothing : read_rss(pid)
peak_rss = if final_rss !== nothing && peak_reset
final_rss[2] # kernel VmHWM: catches spikes between samples
elseif !isempty(rss_samples)
maximum(rss_samples)
else
nothing
end
# --- report
println()
println("=" ^ 68)
println("corpus $(length(paths)) file(s), $(human(corpusbytes)) total, " *
"$(human(corpusbytes / length(paths))) avg")
println("concurrency $(opts["concurrency"])")
println()
println("INTAKE (HTTP 202: bytes spooled, not processed)")
println(" accepted $accepted of $(length(paths)) [202: $n_202, 503: $n_503, error: $n_err]")
println(" wall $(fmt(intake_secs))s")
println(" rate $(fmt(accepted / max(intake_secs, 1e-9))) files/s, " *
"$(fmt(corpusbytes / max(intake_secs, 1e-9) / 1024^2)) MiB/s")
println(" latency p50 $(fmt(percentile(lat, 0.5) * 1000, 1))ms " *
"p95 $(fmt(percentile(lat, 0.95) * 1000, 1))ms " *
"max $(fmt(percentile(lat, 1.0) * 1000, 1))ms")
println()
println("END-TO-END (files reaching a terminal sink)")
println(" completed $completed of $accepted accepted" * (timed_out ? " ** TIMED OUT **" : ""))
println(" wall $(fmt(e2e_secs))s (first upload → last completion)")
println(" throughput $(fmt(completed / max(e2e_secs, 1e-9))) files/s, " *
"$(fmt(corpusbytes / max(e2e_secs, 1e-9) / 1024^2)) MiB/s")
println(" sinks done $(sink_delta.done) text_done $(sink_delta.text_done) " *
"binary $(sink_delta.binary) failed $(sink_delta.failed)")
println(" peak dir depth spool $(depth_max[:spool]) known $(depth_max[:known]) " *
"unknown $(depth_max[:unknown]) text $(depth_max[:text])")
println()
if !isempty(stage_delta)
stage_report(stage_delta, stage_window)
println()
end
if peak_rss !== nothing
println("SERVER MEMORY (pid $pid)")
println(" baseline RSS $(human(baseline_rss))")
println(" peak RSS $(human(peak_rss))" *
(peak_reset ? " (kernel VmHWM, reset at start)" : " (sampled; may miss spikes)"))
growth = peak_rss - baseline_rss
if growth <= 0
# RSS never got back to where it started, so the run's own cost
# is below the noise floor of the server settling after startup.
# Printing a negative "growth" would invite reading a memory
# *saving* into what is really "too small to measure here".
println(" growth none measurable (peak never exceeded the baseline)")
println(" The baseline was still falling when we sampled it. Give the")
println(" server ~30s to settle after startup for a comparable figure.")
else
println(" growth $(human(growth))")
println(" per in-flight $(human(growth / opts["concurrency"])) " *
"at $(human(corpusbytes / length(paths))) avg file size")
println(" (should not grow with file size: intake streams to disk)")
end
# Julia's GC returns memory to the OS lazily, so a second run on the
# same process starts from an inflated baseline and under-reports
# growth. Absolute peak is the number to trust across runs.
baseline_rss > 1.3 * FRESH_RSS_HINT &&
println(" ! baseline is well above a fresh start ($(human(FRESH_RSS_HINT))): the GC " *
"has not\n returned memory from earlier work. Compare " *
"absolute peak, or restart\n the server for a clean growth figure.")
else
println("SERVER MEMORY not sampled")
end
println("=" ^ 68)
sink_delta.failed > 0 &&
println("\nnote: $(sink_delta.failed) file(s) landed in $(sinks.failed); check the server log.")
timed_out &&
println("\nnote: drain stalled with $(accepted - completed) file(s) outstanding. " *
"Check the server log and spool/; raise --timeout if the pipeline is just slow.")
if opts["json"] !== nothing
result = (
url, concurrency = opts["concurrency"], kind = opts["kind"],
files = length(paths), corpus_bytes = corpusbytes,
avg_file_bytes = corpusbytes / length(paths),
intake = (; accepted, n_202, n_503, n_err, seconds = intake_secs,
files_per_sec = accepted / max(intake_secs, 1e-9),
p50_ms = json_num(percentile(lat, 0.5) * 1000),
p95_ms = json_num(percentile(lat, 0.95) * 1000),
max_ms = json_num(percentile(lat, 1.0) * 1000)),
end_to_end = (; completed, seconds = e2e_secs, timed_out,
files_per_sec = completed / max(e2e_secs, 1e-9),
mib_per_sec = corpusbytes / max(e2e_secs, 1e-9) / 1024^2,
sinks = sink_delta, peak_stage_depth = depth_max),
stages = [(; stage = d.stage, name = d.name, workers = d.workers,
completed = d.completed, failed = d.failed, bytes = d.bytes,
busy_seconds = d.busy, blocked_seconds = d.blocked,
window_seconds = stage_window,
files_per_sec = files_per_sec(d, stage_window),
mib_per_sec = mib_per_sec(d, stage_window),
service_ms = json_num(service_ms(d)),
utilization = utilization(d, stage_window),
blocked_share = blocked_share(d),
peak_queue_depth = d.peak_depth,
queue_capacity = d.capacity) for d in stage_delta],
memory = (; pid, baseline_rss, peak_rss, peak_is_kernel_hwm = peak_reset,
growth = peak_rss === nothing ? nothing : peak_rss - baseline_rss,
samples = rss_samples),
)
open(String(opts["json"]), "w") do io
JSON3.write(io, result)
end
println("\nwrote $(opts["json"])")
end
return timed_out ? 1 : 0
finally
if generated && !opts["keep-corpus"]
rm(corpusdir; recursive = true, force = true)
elseif generated
println("\nkept corpus: $corpusdir")
end
end
end
if abspath(PROGRAM_FILE) == @__FILE__
exit(main(ARGS))
end

427
bin/bench_model.jl Executable file
View File

@@ -0,0 +1,427 @@
#!/usr/bin/env julia
#
# bench_model.jl: microbenchmark the classifier in isolation, with no server,
# no queue, and no disk in the way.
#
# bin/bench.jl measures the *pipeline*: it reports stage 1 as one number, the
# wall time of `handle_classify_job`, which is feature reads + inference + a
# rename + a log line, under whatever thread contention the other three pools are
# creating. That number is the right one for capacity planning and the wrong one
# for answering "is the model slow?". This script answers that question by taking
# the model apart:
#
# read_features open, read 16 bytes, seek, read 16 bytes, scale
# Lux.apply the network itself, on a feature vector already in memory
# classify both together: what stage 1 actually calls per file
#
# Three properties are worth checking beyond the raw per-file cost:
#
# * Feature reads should be flat in file size. read_features seeks to the tail
# rather than slurping, so a 1 GiB file should cost the same as a 1 KiB one.
# (This is the same claim bin/bench.jl makes about memory, on the CPU axis.)
# * Batching should be much cheaper per file. A 32x1 matmul wastes most of a
# BLAS call; if batch-64 inference is many times cheaper per file, that is
# the headroom a batching stage-1 would buy, worth knowing before building
# one, since today the pipeline classifies strictly one file at a time.
# * Inference should scale across threads. `Classifier` is shared read-only by
# the whole stage-1 pool on the claim that Lux inference is pure. If per-file
# cost degrades as tasks are added, that claim holds but BLAS threading is
# fighting the worker pool, and stage-1 workers are contending, not scaling.
#
# Usage:
# julia --project=. -t auto bin/bench_model.jl [options]
#
# --model PATH classifier artifact (default: $FS_MODEL_PATH or model/classifier.jld2)
# --reps N inference calls per timed trial (default: 20000)
# --trials N timed trials; the minimum is reported (default: 5)
# --batches LIST batch sizes to sweep, comma-separated (default: 1,8,64,512)
# --sizes LIST file sizes for the read_features sweep (default: 1k,64k,4m,256m)
# --no-threads skip the thread-scaling sweep
# --json PATH also write the results as JSON
#
# Reported times are the *minimum* over trials: for a microbenchmark the floor is
# the signal and everything above it is scheduler and GC noise.
using JSON3
using Random
using Statistics
using Printf
# The script is run directly, not as part of the package, so pull in exactly the
# pieces the classifier needs. `Lux`/`JLD2` first, because model.jl and classify.jl both
# assume the including scope already has them (see the note at the top of model.jl).
using Lux
using JLD2
using LinearAlgebra
const SRC = joinpath(dirname(@__DIR__), "src")
include(joinpath(SRC, "model.jl"))
include(joinpath(SRC, "classify.jl"))
# ---------------------------------------------------------------- option parsing
const DEFAULTS = Dict{String,Any}(
"model" => get(ENV, "FS_MODEL_PATH", "model/classifier.jld2"),
"reps" => 20_000,
"trials" => 5,
"batches" => "1,8,64,512",
"sizes" => "1k,64k,4m,256m",
"no-threads" => false,
"json" => nothing,
)
const FLAGS = ("no-threads",)
function parse_size(s::AbstractString)::Int
m = match(r"^(\d+(?:\.\d+)?)\s*([kKmMgG]?)[bB]?$", strip(s))
m === nothing && error("bad size: $s (expected e.g. 512, 64k, 8m, 1g)")
mult = Dict('k' => 1024, 'm' => 1024^2, 'g' => 1024^3)
scale = isempty(m[2]) ? 1 : mult[lowercase(m[2])[1]]
return round(Int, parse(Float64, m[1]) * scale)
end
function parse_args(argv)
opts = copy(DEFAULTS)
i = 1
while i <= length(argv)
a = argv[i]
startswith(a, "--") || error("unexpected argument: $a")
key = a[3:end]
haskey(opts, key) || error("unknown option: $a")
if key in FLAGS
opts[key] = true; i += 1; continue
end
i + 1 <= length(argv) || error("option --$key needs a value")
opts[key] = key in ("reps", "trials") ? parse(Int, argv[i+1]) : argv[i+1]
i += 2
end
return opts
end
# ------------------------------------------------------------------- measurement
# Every timed loop stores its result here. Without a visible side effect the
# compiler is free to hoist a pure call out of the loop and we would be timing an
# empty `for`.
const SINK = Ref{Any}(nothing)
"""
measure(f, reps; trials) -> (ns_per_op, bytes_per_op)
Time `f` over `reps` calls, `trials` times, and report the fastest trial.
The first call is thrown away: it pays Julia's JIT compilation, which on a
function this small is orders of magnitude more than the thing being measured.
"""
function measure(f, reps::Int; trials::Int = 5)
SINK[] = f() # warm up (compile), and keep the result
best = Inf
for _ in 1:trials
GC.gc()
t0 = time_ns()
for _ in 1:reps
SINK[] = f()
end
best = min(best, (time_ns() - t0) / reps)
end
bytes = @allocated(SINK[] = f()) # one call, after warmup
return (Float64(best), Float64(bytes))
end
# ------------------------------------------------------------------- formatting
function human_time(ns::Real)
ns < 1_000 && return @sprintf("%.0f ns", ns)
ns < 1_000_000 && return @sprintf("%.2f µs", ns / 1e3)
ns < 1e9 && return @sprintf("%.2f ms", ns / 1e6)
return @sprintf("%.2f s", ns / 1e9)
end
human_bytes(b::Real) = b < 1024 ? @sprintf("%.0f B", b) :
b < 1024^2 ? @sprintf("%.1f KiB", b / 1024) :
@sprintf("%.1f MiB", b / 1024^2)
rate(ns::Real) = 1e9 / max(ns, 1e-9) # calls per second
function human_rate(r::Real)
r >= 1e6 && return @sprintf("%.2fM/s", r / 1e6)
r >= 1e3 && return @sprintf("%.1fk/s", r / 1e3)
return @sprintf("%.0f/s", r)
end
fmt2(x::Real) = @sprintf("%.2f", x)
# ------------------------------------------------------------- scaling control
"""
control_kernel(x) -> Float64
Pure arithmetic, no allocation, no library call, and deliberately dependent
(each step needs the last) so the compiler can't vectorize it away, and sized to
land in the same microsecond neighbourhood as one `Lux.apply`.
"""
function control_kernel(x::Float64)
a = x
@inbounds for i in 1:600
a = sqrt(a + i)
end
return a
end
"""
control_scaling(trials) -> Vector
Run `control_kernel` over the same task counts as the model sweep. This is the
machine's own ceiling for perfectly parallel work: if the control scales and the
model doesn't, the shortfall is the model's, and no amount of `FS_WORKERS` will
recover it.
"""
function control_scaling(trials::Int)
rows = []
base = 0.0
per_task = 200_000
for k in unique([1; 2; 4; 8; Threads.nthreads()])
k > Threads.nthreads() && continue
best = Inf
for _ in 1:trials
GC.gc()
t0 = time_ns()
@sync for _ in 1:k
Threads.@spawn begin
local acc = 0.0
for i in 1:per_task
acc += control_kernel(i % 97 + 1.0)
end
SINK[] = acc
end
end
best = min(best, Float64(time_ns() - t0))
end
r = k * per_task / (best / 1e9)
k == 1 && (base = r)
push!(rows, (; tasks = k, ops_per_sec = r, speedup = r / base))
end
return rows
end
# ------------------------------------------------------------------------- corpus
"""
Write a file of exactly `size` random bytes, in bounded chunks.
Content is random rather than zeros so the classifier sees a realistic input,
and so the filesystem can't cheat with a sparse file, which would make the tail
`seek` unrepresentatively fast.
"""
function write_file(path::AbstractString, size::Int, rng)
chunk = 1024 * 1024
open(path, "w") do io
remaining = size
while remaining > 0
n = min(chunk, remaining)
write(io, rand(rng, UInt8, n))
remaining -= n
end
end
return path
end
# ------------------------------------------------------------------------- main
function main(argv)
opts = parse_args(argv)
modelpath = String(opts["model"])
isfile(modelpath) || (println(stderr, "model artifact not found: $modelpath"); return 1)
reps, trials = opts["reps"], opts["trials"]
batches = [parse(Int, s) for s in split(String(opts["batches"]), ",")]
sizes = [parse_size(s) for s in split(String(opts["sizes"]), ",")]
clf = load_classifier(modelpath)
println("model $modelpath")
println("architecture $(FEATURE_DIM) → 64 → 16 → 2 (Dense/relu, raw logits)")
println("julia threads $(Threads.nthreads()) BLAS threads $(BLAS.get_num_threads())")
println("timing min of $trials trials × $reps reps")
println("=" ^ 72)
results = Dict{String,Any}()
# --- 1. inference alone, one file at a time: the number the pipeline pays.
x1 = rand(Float32, FEATURE_DIM, 1)
infer_ns, infer_bytes = measure(reps; trials) do
Lux.apply(clf.model, x1, clf.ps, clf.st)
end
println()
println("INFERENCE (Lux.apply, batch 1; features already in memory)")
println(" per call $(human_time(infer_ns)) $(human_rate(rate(infer_ns)))")
println(" allocations $(human_bytes(infer_bytes)) per call")
results["inference_batch1"] = (; ns = infer_ns, bytes = infer_bytes, per_sec = rate(infer_ns))
# --- 2. batching: how much of that is per-call overhead rather than math?
println()
println("INFERENCE BATCHED (same net, N files per apply)")
println(" batch per batch per file files/s speedup")
batch_rows = []
for b in batches
xb = rand(Float32, FEATURE_DIM, b)
ns, _ = measure(max(1, reps ÷ b); trials) do
Lux.apply(clf.model, xb, clf.ps, clf.st)
end
per_file = ns / b
@printf(" %8d %13s %12s %13s %7.1fx\n",
b, human_time(ns), human_time(per_file),
human_rate(rate(per_file)), infer_ns / per_file)
push!(batch_rows, (; batch = b, ns_per_batch = ns, ns_per_file = per_file,
files_per_sec = rate(per_file), speedup = infer_ns / per_file))
end
results["batched"] = batch_rows
println(" (a large speedup is headroom a batching stage 1 could claim; the pipeline")
println(" classifies one file per job today, so it pays the batch-1 row above)")
# --- 3. feature reads: should be flat in file size (seek, not slurp).
println()
println("FEATURE READS (read_features: 16 head + 16 tail bytes, scaled)")
println(" file size per call calls/s allocations")
read_rows = []
dir = mktempdir(; prefix = "fsmodel-")
try
rng = MersenneTwister(1234)
for sz in sizes
path = write_file(joinpath(dir, "f-$sz.bin"), sz, rng)
# Fewer reps for the big files: this touches the page cache, and the
# point is the shape of the curve, not another digit of precision.
r = max(200, reps ÷ 20)
ns, bytes = measure(() -> read_features(path), r; trials)
@printf(" %13s %14s %14s %14s\n",
human_bytes(sz), human_time(ns), human_rate(rate(ns)), human_bytes(bytes))
push!(read_rows, (; size_bytes = sz, ns, bytes, per_sec = rate(ns)))
end
finally
rm(dir; recursive = true, force = true)
end
results["read_features"] = read_rows
flat = length(read_rows) > 1 ?
maximum(r.ns for r in read_rows) / minimum(r.ns for r in read_rows) : 1.0
@printf(" spread across a %.0fx size range: %.1fx, %s\n",
maximum(sizes) / minimum(sizes), flat,
flat < 3 ? "flat, as designed (it seeks to the tail)" :
"NOT flat: something is reading more than 32 bytes")
# --- 4. classify(): what stage 1 calls, I/O and inference together.
println()
println("CLASSIFY (read_features + Lux.apply; one whole stage-1 file)")
dir2 = mktempdir(; prefix = "fsmodel-")
classify_ns = 0.0
try
path = write_file(joinpath(dir2, "sample.bin"), 64 * 1024, MersenneTwister(7))
classify_ns, classify_bytes = measure(() -> classify(clf, path), max(200, reps ÷ 20); trials)
println(" per file $(human_time(classify_ns)) $(human_rate(rate(classify_ns)))")
println(" allocations $(human_bytes(classify_bytes)) per file")
@printf(" split %.0f%% feature read, %.0f%% inference\n",
100 * (classify_ns - infer_ns) / classify_ns, 100 * infer_ns / classify_ns)
results["classify"] = (; ns = classify_ns, bytes = classify_bytes, per_sec = rate(classify_ns))
finally
rm(dir2; recursive = true, force = true)
end
# --- 5. thread scaling: does the shared read-only Classifier actually scale?
if !opts["no-threads"] && Threads.nthreads() > 1
println()
println("THREAD SCALING (concurrent Lux.apply on the one shared Classifier)")
println(" tasks files/s per file speedup efficiency GC")
thread_rows = []
base = 0.0
for k in unique([1; 2; 4; 8; Threads.nthreads()])
k > Threads.nthreads() && continue
# Work per task is held *constant* as tasks are added, so total work
# scales with `k`. Splitting a fixed total instead would shrink each
# task as the pool grows until `@spawn`/`@sync` overhead dominated,
# and the resulting curve would show a collapse that is the
# measurement's fault rather than the model's.
per_task = max(reps, 20_000)
# Each task gets its own input so we measure the model, not cache
# line ping-pong on a shared buffer.
xs = [rand(Float32, FEATURE_DIM, 1) for _ in 1:k]
best, best_gc = Inf, 0.0
for _ in 1:trials
GC.gc()
# Julia's GC stops the world, so it is the one cost that cannot
# be parallelized away: measuring its share here is what turns a
# bad efficiency number into a diagnosis (see the note below).
gc0 = Base.gc_num().total_time
t0 = time_ns()
@sync for t in 1:k
Threads.@spawn begin
local acc = 0.0f0
for _ in 1:per_task
y, _ = Lux.apply(clf.model, xs[t], clf.ps, clf.st)
acc += y[1] # consume the result
end
SINK[] = acc
end
end
elapsed = Float64(time_ns() - t0)
if elapsed < best
best = elapsed
best_gc = Float64(Base.gc_num().total_time - gc0)
end
end
files = k * per_task
fps = files / (best / 1e9)
k == 1 && (base = fps)
@printf(" %8d %13s %11s %7.2fx %10.0f%% %5.0f%%\n",
k, human_rate(fps), human_time(best / files), fps / base,
100 * fps / base / k, 100 * best_gc / best)
push!(thread_rows, (; tasks = k, files_per_sec = fps,
ns_per_file = best / files, speedup = fps / base,
gc_fraction = best_gc / best))
end
results["thread_scaling"] = thread_rows
# A poor scaling curve has two possible authors, the model or the box,
# and the table alone can't tell them apart. So run the same sweep on a
# kernel that is pure arithmetic with no allocation and no library
# underneath: whatever *it* achieves is this machine's ceiling for
# embarrassingly parallel work, and the gap between the two curves is
# the part that belongs to Lux.apply.
ctrl = control_scaling(trials)
results["control_scaling"] = ctrl
top = last(ctrl)
println(" control pure-compute kernel, same sweep: " *
"$(fmt2(top.speedup))x at $(top.tasks) tasks " *
"($(round(Int, 100 * top.speedup / top.tasks))% efficiency)")
model_top = last(thread_rows)
if model_top.speedup < 0.6 * top.speedup
println(" → the machine parallelizes; Lux.apply does not. Stage-1")
println(" workers past ~4 buy little, whatever FS_WORKERS says.")
else
println(" → inference tracks the machine's own scaling ceiling.")
end
gc_top = maximum(r.gc_fraction for r in thread_rows)
gc_top > 0.15 && println(" ! GC is $(round(Int, 100 * gc_top))% of the " *
"worst case: apply allocates per call, and\n" *
" collection stops every thread.")
end
println()
println("=" ^ 72)
println("Stage 1's cost per file in bin/bench.jl is this classify() figure plus a")
println("rename, a log line, and whatever contention the other three pools create.")
println("A large gap between the two is pipeline overhead, not the model.")
if opts["json"] !== nothing
results["meta"] = (; model = modelpath, feature_dim = FEATURE_DIM,
julia_threads = Threads.nthreads(),
blas_threads = BLAS.get_num_threads(),
reps, trials)
open(String(opts["json"]), "w") do io
JSON3.write(io, results)
end
println("\nwrote $(opts["json"])")
end
return 0
end
if abspath(PROGRAM_FILE) == @__FILE__
exit(main(ARGS))
end

535
bin/bench_stage1.jl Normal file
View File

@@ -0,0 +1,535 @@
#!/usr/bin/env julia
#
# bench_stage1.jl: take stage 1 apart and find the slowest component.
#
# bin/bench.jl reports stage 1 as a single number (the wall time of
# `handle_classify_job` under whole-pipeline contention) and bin/bench_model.jl
# takes the *classifier* apart. Neither answers "which part of stage 1 costs the
# most?", because stage 1 is more than the model. Per file it does:
#
# classify filesize + two 16-byte reads + a 32x1 forward pass
# read_features open, read head, seek, read tail, scale to Float32
# Lux.apply the network on a feature vector already in memory
# enqueue push a Job reference onto the downstream bounded queue
# logging two @info lines ("classified file", "routed to ...")
#
# This script times each of those in isolation, then times the real
# `handle_classify_job` end to end so the parts can be checked against the whole.
# The two knobs that most change the answer get their own sweeps:
#
# * Logger. The server runs `FlushLogger(ConsoleLogger(stderr))`, which formats
# and flushes every message. Under redirect (a log file, journald) that is a
# syscall per line, two lines per file, on the hot path. We time the handler
# under a null logger, a formatting-but-discarding logger, and the real
# flushing-to-file logger, so the cost of logging is a subtraction, not a
# guess. This sweep is what demoted stage 1's per-file lines to `@debug`
# (see the note in src/worker.jl); the standalone `logging (...)` rows below
# still price a *formatted* line, i.e. what those lines cost when switched
# back on with `JULIA_DEBUG=DarkStruct`, while the handler rows show what the
# stage pays with them off.
# * Concurrency. Components that own a lock (the queue's condition, the
# logger's stream) don't scale, and the ranking at one worker need not be the
# ranking at sixteen. The `--threads` sweep runs the full handler across
# worker counts.
#
# Usage:
# julia --project=. -t auto bin/bench_stage1.jl [options]
#
# --files N files per timed pass for whole-corpus benchmarks (default: 2000)
# --reps N calls per timed pass for per-call benchmarks (default: 20000)
# --trials N timed passes; the minimum is reported (default: 5)
# --size SPEC corpus file size (default: 64k)
# --dir PATH working directory for the corpus (default: a temp dir under data/)
# --model PATH classifier artifact (default: $FS_MODEL_PATH or model/classifier.jld2)
# --threads LIST worker counts for the concurrency sweep (default: 1,2,4,8,nthreads)
# --no-threads skip the concurrency sweep
# --json PATH also write the results as JSON
#
# Reported times are the *minimum* over trials: the floor is the signal and
# everything above it is scheduler, page-cache and GC noise.
using DarkStruct
using Lux
using JSON3
using Logging
using Printf
using Random
const FS = DarkStruct
# ---------------------------------------------------------------- option parsing
const DEFAULTS = Dict{String,Any}(
"files" => 2000,
"reps" => 20_000,
"trials" => 5,
"size" => "64k",
"dir" => nothing,
"model" => get(ENV, "FS_MODEL_PATH", "model/classifier.jld2"),
"threads" => nothing,
"no-threads" => false,
"json" => nothing,
)
const FLAGS = ("no-threads",)
const INTS = ("files", "reps", "trials")
function parse_size(s::AbstractString)::Int
m = match(r"^(\d+(?:\.\d+)?)\s*([kKmMgG]?)[bB]?$", strip(s))
m === nothing && error("bad size: $s (expected e.g. 512, 64k, 8m, 1g)")
mult = Dict('k' => 1024, 'm' => 1024^2, 'g' => 1024^3)
scale = isempty(m[2]) ? 1 : mult[lowercase(m[2])[1]]
return round(Int, parse(Float64, m[1]) * scale)
end
function parse_args(argv)
opts = copy(DEFAULTS)
i = 1
while i <= length(argv)
a = argv[i]
startswith(a, "--") || error("unexpected argument: $a")
key = a[3:end]
haskey(opts, key) || error("unknown option: $a")
if key in FLAGS
opts[key] = true; i += 1; continue
end
i + 1 <= length(argv) || error("option --$key needs a value")
opts[key] = key in INTS ? parse(Int, argv[i+1]) : argv[i+1]
i += 2
end
return opts
end
# ------------------------------------------------------------------- measurement
# Every timed loop stores its result here. Without a visible side effect the
# compiler is free to hoist a pure call out of the loop and we would be timing an
# empty `for`.
const SINK = Ref{Any}(nothing)
"""
best_of(pass, prepare; trials) -> ns_per_op
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed: that is where a benchmark resets whatever its
last pass consumed (today: the queues). `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
this small is orders of magnitude more than the thing being measured.
"""
function best_of(pass, prepare; trials::Int)
best = Inf
for t in 0:trials
prepare()
GC.gc()
t0 = time_ns()
n = pass()
dt = Float64(time_ns() - t0)
t == 0 && continue # warm-up: compiled, not measured
best = min(best, dt / n)
end
return best
end
noop() = nothing
# ------------------------------------------------------------------- formatting
function human_time(ns::Real)
ns < 1_000 && return @sprintf("%.0f ns", ns)
ns < 1_000_000 && return @sprintf("%.2f µs", ns / 1e3)
ns < 1e9 && return @sprintf("%.2f ms", ns / 1e6)
return @sprintf("%.2f s", ns / 1e9)
end
function human_rate(r::Real)
r >= 1e6 && return @sprintf("%.2fM/s", r / 1e6)
r >= 1e3 && return @sprintf("%.1fk/s", r / 1e3)
return @sprintf("%.0f/s", r)
end
rate(ns::Real) = 1e9 / max(ns, 1e-9)
rule(n = 78) = println("-" ^ n)
function header(title)
println()
println(title)
rule()
end
# ------------------------------------------------------------------------- corpus
"""
Write a file of exactly `size` random bytes, in bounded chunks.
Content is random rather than zeros so the classifier sees a realistic input,
and so the filesystem can't cheat with a sparse file, which would make the tail
`seek` unrepresentatively fast.
"""
function write_file(path::AbstractString, size::Int, rng)
chunk = 1024 * 1024
open(path, "w") do io
remaining = size
while remaining > 0
n = min(chunk, remaining)
write(io, rand(rng, UInt8, n))
remaining -= n
end
end
return path
end
"""
make_corpus(cfg, n, size, rng) -> Vector{Job}
Create `n` spooled files and the `Job` references a stage-1 worker would dequeue
for them: the exact input `handle_classify_job` sees.
"""
function make_corpus(cfg::FS.Config, n::Int, size::Int, rng)
jobs = FS.Job[]
for i in 1:n
id, path = FS.spool_path(cfg, @sprintf("bench-%06d.bin", i))
write_file(path, size, rng)
push!(jobs, FS.Job(id, basename(path), path, size, time()))
end
return jobs
end
# There is no `respool!` step any more. Stage 1 used to consume its input by
# renaming each file into known/ or unknown/, so every repeated pass had to put
# the corpus back first. It now routes by enqueueing alone and leaves the file in
# spool/, so the corpus is reusable as-is and the only per-pass reset is draining
# the queues.
"Drain a queue without blocking, so the next pass starts from empty."
function drain!(q::FS.ChannelQueue)
while length(q) > 0
FS.dequeue!(q)
end
return nothing
end
# --------------------------------------------------------------------- loggers
"""
with_logger_named(name, path, f)
Run `f` under one of the three loggers the cost of logging is bracketed by:
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`DarkStruct.run` installs, under the redirect it was written for.
* `:debug` the same, at `Debug` level: the stage's per-file lines are
`@debug`, so this is the equivalent of running the server with
`JULIA_DEBUG=DarkStruct` and the only setting under which they
are emitted at all.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
return with_logger(f, NullLogger())
elseif which === :format
return with_logger(f, ConsoleLogger(devnull))
elseif which === :flush || which === :debug
level = which === :debug ? Logging.Debug : Logging.Info
return open(path, "w") do io
with_logger(f, FS.FlushLogger(ConsoleLogger(io, level)))
end
end
error("unknown logger: $which")
end
# ------------------------------------------------------------------ components
"""
component_rows(cfg, clf, jobs, opts) -> Vector
Time each piece of stage 1 on its own. Per-call pieces (`filesize`,
`read_features`, `Lux.apply`, `classify`, the log lines) run `reps` times over
the corpus; whole-corpus pieces (`enqueue_blocking!`, the full handler) run once
per corpus file, with an untimed queue drain between passes.
"""
function component_rows(cfg::FS.Config, clf::FS.Classifier, jobs::Vector{FS.Job}, opts)
reps, trials = opts["reps"], opts["trials"]
nfiles = length(jobs)
paths = [j.path for j in jobs]
rows = []
# A feature vector already in memory, so the inference row measures the net
# and not the disk read in front of it.
feats = FS.read_features(paths[1])
x = reshape(feats, FS.FEATURE_DIM, 1)
logger = ConsoleLogger(devnull) # components other than the log rows: quiet
# --- filesize: the stat() read_features does before touching the bytes
push!(rows, (; name = "filesize (stat)", part = "classify",
ns = best_of(noop; trials) do
@inbounds for i in 1:reps
SINK[] = filesize(paths[(i - 1) % nfiles + 1])
end
reps
end))
# --- read_features: open + head read + seek + tail read + scale
push!(rows, (; name = "read_features", part = "classify",
ns = best_of(noop; trials) do
@inbounds for i in 1:reps
SINK[] = FS.read_features(paths[(i - 1) % nfiles + 1])
end
reps
end))
# --- Lux.apply: the network alone, features already in memory
push!(rows, (; name = "Lux.apply (1x32)", part = "classify",
ns = best_of(noop; trials) do
for _ in 1:reps
SINK[] = Lux.apply(clf.model, x, clf.ps, clf.st)
end
reps
end))
# --- classify: read_features + apply + argmax, what the handler calls
push!(rows, (; name = "classify (total)", part = "classify",
ns = best_of(noop; trials) do
@inbounds for i in 1:reps
SINK[] = FS.classify(clf, paths[(i - 1) % nfiles + 1])
end
reps
end))
# Stage 1 used to rename each file into known/ or unknown/ before enqueueing
# it, and that rename was measured here as its own line item. It is gone:
# routing is the enqueue alone, and a file does not move until it is
# committed to a terminal sink (src/worker.jl header). So `route` below is
# now just the queue handoff.
# --- enqueue: lock, push, notify on an uncontended, non-full queue
q = FS.ChannelQueue(nfiles + 1)
stats = FS.StageStats()
push!(rows, (; name = "enqueue_blocking!", part = "route",
ns = best_of(() -> drain!(q); trials) do
@inbounds for job in jobs
SINK[] = FS.enqueue_blocking!(q, job, stats;
retry_seconds = FS.ROUTE_ENQUEUE_RETRY_SECONDS)
end
nfiles
end))
drain!(q)
# --- the two @info lines, under each of the three loggers
job1 = jobs[1]
logfile = joinpath(cfg.spool_dir, "..", "bench_stage1.log")
for (which, label) in ((:null, "logging (NullLogger)"),
(:format, "logging (format only)"),
(:flush, "logging (flush→file)"))
ns = with_logger_named(which, logfile) do
best_of(noop; trials) do
for _ in 1:reps
@info "classified file" worker=1 id=job1.id name=job1.original_name size=job1.size classification=:unknown
@info "routed to content triage" worker=1 id=job1.id dest=job1.path
end
reps
end
end
push!(rows, (; name = label, part = "log", ns))
end
rm(logfile; force = true)
return rows, logger
end
"""
handler_rows(cfg, jobs, opts) -> Vector
Time the real `handle_classify_job` end to end under each logger. The difference
between the rows is the cost logging adds to a file; the `:flush` row is what the
running server actually pays.
"""
function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
trials = opts["trials"]
nfiles = length(jobs)
logfile = joinpath(dirname(cfg.spool_dir), "bench_stage1.log")
known = FS.ChannelQueue(nfiles + 1)
unknown = FS.ChannelQueue(nfiles + 1)
stats = FS.StageStats()
rows = []
for (which, label) in ((:null, "handle_classify_job (NullLogger)"),
(:format, "handle_classify_job (format only)"),
(:flush, "handle_classify_job (flush→file)"),
(:debug, "handle_classify_job (JULIA_DEBUG)"))
ns = with_logger_named(which, logfile) do
best_of(() -> (drain!(known); drain!(unknown)); trials) do
@inbounds for job in jobs
FS.handle_classify_job(job, cfg, 1, known, unknown, stats)
end
nfiles
end
end
push!(rows, (; name = label, part = "total", ns))
end
drain!(known); drain!(unknown)
rm(logfile; force = true)
return rows
end
"""
thread_rows(cfg, jobs, opts) -> Vector
Run the full handler across worker counts, under the server's real logger. A
component that owns a lock (the queue's condition variable, the logger's
stream) stops scaling here even though it looked cheap single-threaded, so this
is where the single-thread ranking gets checked against the deployed one.
"""
function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
trials = opts["trials"]
nfiles = length(jobs)
counts = opts["threads"] === nothing ?
unique([1; 2; 4; 8; Threads.nthreads()]) :
[parse(Int, s) for s in split(String(opts["threads"]), ",")]
counts = sort(unique(filter(k -> 1 <= k <= Threads.nthreads(), counts)))
logfile = joinpath(dirname(cfg.spool_dir), "bench_stage1.log")
known = FS.ChannelQueue(nfiles + 1)
unknown = FS.ChannelQueue(nfiles + 1)
stats = FS.StageStats()
rows = []
base = 0.0
for k in counts
ns = with_logger_named(:flush, logfile) do
best_of(() -> (drain!(known); drain!(unknown)); trials) do
# Static split: each task takes a contiguous slice, so the only
# sharing between workers is the state the server also shares:
# the classifier, the queues, the logger, the filesystem.
chunk = cld(nfiles, k)
@sync for t in 1:k
lo = (t - 1) * chunk + 1
hi = min(t * chunk, nfiles)
lo > hi && continue
Threads.@spawn begin
@inbounds for i in lo:hi
FS.handle_classify_job(jobs[i], cfg, t, known, unknown, stats)
end
end
end
nfiles
end
end
r = rate(ns) # files/sec aggregate (ns is already per file, wall-clock)
k == counts[1] && (base = r)
push!(rows, (; workers = k, ns, files_per_sec = r, speedup = r / base))
end
drain!(known); drain!(unknown)
rm(logfile; force = true)
return rows
end
# ------------------------------------------------------------------- reporting
function print_components(rows, total_ns)
@printf("%-34s %-9s %12s %10s %9s\n", "component", "part", "per file", "rate", "% total")
rule()
for r in rows
@printf("%-34s %-9s %12s %10s %8.1f%%\n", r.name, r.part, human_time(r.ns),
human_rate(rate(r.ns)), 100 * r.ns / total_ns)
end
end
function print_threads(rows)
@printf("%-9s %12s %12s %9s\n", "workers", "per file", "throughput", "speedup")
rule()
for r in rows
@printf("%-9d %12s %12s %8.2fx\n", r.workers, human_time(r.ns),
human_rate(r.files_per_sec), r.speedup)
end
end
# ------------------------------------------------------------------------- main
function main(argv)
opts = parse_args(argv)
modelpath = String(opts["model"])
isfile(modelpath) || (println(stderr, "model artifact not found: $modelpath"); return 1)
size = parse_size(String(opts["size"]))
nfiles, trials = opts["files"], opts["trials"]
root = opts["dir"] === nothing ?
mktempdir(pwd(); prefix = "bench_stage1_") : String(opts["dir"])
owned = opts["dir"] === nothing
cfg = FS.Config(
spool_dir = joinpath(root, "spool"),
failed_dir = joinpath(root, "failed"),
model_path = modelpath,
)
for d in (cfg.spool_dir, cfg.failed_dir)
mkpath(d)
end
clf = FS.load_classifier(modelpath)
FS.CLASSIFIER[] = clf # handle_classify_job reads the global, as in the server
println("stage-1 component benchmark")
rule()
@printf("%-22s %s\n", "julia threads", Threads.nthreads())
@printf("%-22s %s\n", "model", modelpath)
@printf("%-22s %s\n", "corpus", "$(nfiles) files x $(size) B in $(root)")
@printf("%-22s %s\n", "reps / trials", "$(opts["reps"]) / $(trials)")
rng = MersenneTwister(0x5741524d)
jobs = make_corpus(cfg, nfiles, size, rng)
try
comps, _ = component_rows(cfg, clf, jobs, opts)
handlers = handler_rows(cfg, jobs, opts)
# The denominator is the handler as the server actually runs it: the real
# flushing logger at Info level, one worker. Percentages are shares of
# that, so they are directly comparable and the parts can be checked
# against the whole. The JULIA_DEBUG row is deliberately *not* the
# baseline. It is the opt-in configuration, and letting it set the scale
# would make every other component look free.
total = only(r.ns for r in handlers if r.name == "handle_classify_job (flush→file)")
header("Components (single worker)")
print_components(comps, total)
header("Whole handler, by logger")
print_components(handlers, total)
# Stage 1's own log lines are `@debug`, so what the deployed handler pays
# for them is the disabled-macro cost, not a formatted line.
accounted = sum(r.ns for r in comps if r.name in
("classify (total)", "enqueue_blocking!", "logging (NullLogger)"))
println()
@printf("accounted: %s of %s (%.0f%%); unaccounted overhead %s\n",
human_time(accounted), human_time(total), 100 * accounted / total,
human_time(max(total - accounted, 0)))
threads = nothing
if !opts["no-threads"] && Threads.nthreads() > 1
threads = thread_rows(cfg, jobs, opts)
header("Full handler across workers (server logger)")
print_threads(threads)
end
if opts["json"] !== nothing
open(String(opts["json"]), "w") do io
JSON3.write(io, (;
julia_threads = Threads.nthreads(),
file_size = size, files = nfiles, reps = opts["reps"], trials,
components = comps, handlers, threads,
))
end
println("\nwrote ", opts["json"])
end
finally
owned && rm(root; recursive = true, force = true)
end
return 0
end
exit(main(ARGS))

700
bin/bench_stage2.jl Normal file
View File

@@ -0,0 +1,700 @@
#!/usr/bin/env julia
#
# bench_stage2.jl: take stage 2 apart and find the slowest component.
#
# bin/bench.jl reports stage 2 as a single number (its throughput and worker
# utilization under whole-pipeline contention). It doesn't say *which part* of
# the stage costs the most, and stage 2 is the one stage whose cost is dominated
# by something outside Julia entirely: it forks `exiftool`, a Perl program, once
# per file. Per file the stage does:
#
# build_metadata
# run_exiftool fork/exec exiftool -json -G -n, capture stdout
# run_with_timeout the watchdog wrapper around the subprocess
# JSON3.read parse the dump
# normalize_metadata coalesce ~20 tag names into the sidecar schema
# finalize_known! (commit_enriched!)
# JSON3.write serialize the sidecar payload
# write + fsync durably persist the sidecar bytes to a temp name
# mv + fsync_dir commit the sidecar, then persist the rename itself
# move_to rename spool/<f> -> done/<f>, the commit point
# logging one @info line ("enriched")
#
# This script times each of those in isolation, then times the real
# `handle_known_job` end to end so the parts can be checked against the whole.
#
# Three things here that the stage-1 benchmark has no equivalent of:
#
# * The corpus must be real files. exiftool's cost depends on what it finds;
# random bytes exit early and would understate the stage by a lot. The
# default corpus is `data/done`, files that already went through stage 2 on
# this machine, copied back into a scratch spool/ dir.
# * Two rows price the *alternatives* to one-fork-per-file, because if the
# fork dominates then the only fixes are to stop paying it per file:
# `exiftool (batched Nx)` runs the whole corpus through one process, and
# `exiftool (-stay_open)` keeps a single process alive and feeds it one file
# at a time over a pipe: the shape a streaming pipeline could actually use.
# Both are measured, not assumed.
# * `run_with_timeout` gets its own row *next to* a bare `Base.run` of the same
# command. The difference is what the watchdog costs, and its polling loop
# (`sleep(0.1)`) is a suspicious enough design to want measured rather than
# reasoned about.
#
# The `--threads` sweep runs the full handler across worker counts: subprocess
# spawning contends on things (the kernel's fork path, page cache, the logger's
# stream) that a single-threaded ranking can't reveal.
#
# Usage:
# julia --project=. -t auto bin/bench_stage2.jl [options]
#
# --files N corpus files per timed pass (default: 48). The concurrency
# sweep wants more than the component rows do. With a single
# 2 s file in the corpus, 48 files can't show more than ~4x no
# matter how many workers run, so pass --files 150 when the
# question is scaling.
# --reps N calls per timed pass for cheap, non-consuming benchmarks (default: 2000)
# --trials N timed passes; the minimum is reported (default: 3)
# --corpus PATH directory of real files to draw the corpus from (default: data/done)
# --dir PATH working directory for the corpus (default: a temp dir under data/)
# --timeout SEC exiftool timeout, as Config.exiftool_timeout (default: 30)
# --threads LIST worker counts for the concurrency sweep (default: 1,2,4,8,nthreads)
# --no-threads skip the concurrency sweep
# --no-stay-open skip the persistent-exiftool probe
# --json PATH also write the results as JSON
#
# Reported times are the *minimum* over trials: the floor is the signal and
# everything above it is scheduler, page-cache and GC noise.
using DarkStruct
using JSON3
using Logging
using Printf
using Random
const FS = DarkStruct
# ---------------------------------------------------------------- option parsing
const DEFAULTS = Dict{String,Any}(
"files" => 48,
"reps" => 2000,
"trials" => 3,
"corpus" => "data/done",
"dir" => nothing,
"timeout" => 30,
"threads" => nothing,
"no-threads" => false,
"no-stay-open" => false,
"json" => nothing,
)
const FLAGS = ("no-threads", "no-stay-open")
const INTS = ("files", "reps", "trials", "timeout")
function parse_args(argv)
opts = copy(DEFAULTS)
i = 1
while i <= length(argv)
a = argv[i]
startswith(a, "--") || error("unexpected argument: $a")
key = a[3:end]
haskey(opts, key) || error("unknown option: $a")
if key in FLAGS
opts[key] = true; i += 1; continue
end
i + 1 <= length(argv) || error("option --$key needs a value")
opts[key] = key in INTS ? parse(Int, argv[i+1]) : argv[i+1]
i += 2
end
return opts
end
# ------------------------------------------------------------------- measurement
# Every timed loop stores its result here. Without a visible side effect the
# compiler is free to hoist a pure call out of the loop and we would be timing an
# empty `for`.
const SINK = Ref{Any}(nothing)
"""
best_of(pass, prepare; trials) -> ns_per_op
Run `pass()` `trials` times and report the fastest, in nanoseconds per operation
(`pass` returns the number of operations it performed). `prepare()` runs before
each pass and is *not* timed: that is where a consuming benchmark puts the file
back where it started. `pass` comes first so callers can pass it as a `do` block.
The first pass is thrown away: it pays Julia's JIT compilation, which on calls
this small is orders of magnitude more than the thing being measured.
"""
function best_of(pass, prepare; trials::Int)
best = Inf
for t in 0:trials
prepare()
GC.gc()
t0 = time_ns()
n = pass()
dt = Float64(time_ns() - t0)
t == 0 && continue # warm-up: compiled, not measured
best = min(best, dt / n)
end
return best
end
noop() = nothing
# ------------------------------------------------------------------- formatting
function human_time(ns::Real)
ns < 1_000 && return @sprintf("%.0f ns", ns)
ns < 1_000_000 && return @sprintf("%.2f µs", ns / 1e3)
ns < 1e9 && return @sprintf("%.2f ms", ns / 1e6)
return @sprintf("%.2f s", ns / 1e9)
end
function human_rate(r::Real)
r >= 1e6 && return @sprintf("%.2fM/s", r / 1e6)
r >= 1e3 && return @sprintf("%.1fk/s", r / 1e3)
return @sprintf("%.0f/s", r)
end
rate(ns::Real) = 1e9 / max(ns, 1e-9)
rule(n = 84) = println("-" ^ n)
function header(title)
println()
println(title)
rule()
end
# ------------------------------------------------------------------------- corpus
"""
make_corpus(cfg, corpus_dir, n) -> Vector{Job}
Copy up to `n` real files from `corpus_dir` into `spool/` and build the `Job`
references a stage-2 worker would dequeue for them: the exact input
`handle_known_job` sees.
Real files, not generated ones: exiftool's cost is a function of what it can
parse, and a file of random bytes bails out early enough to understate the stage
by an order of magnitude. `.meta.json` sidecars are skipped, since they are stage-2
*output*, and enriching them would measure the wrong population.
"""
function make_corpus(cfg::FS.Config, corpus_dir::AbstractString, n::Int)
isdir(corpus_dir) || error("corpus dir not found: $corpus_dir")
names = filter(readdir(corpus_dir)) do f
!endswith(f, ".meta.json") && isfile(joinpath(corpus_dir, f))
end
isempty(names) && error("no usable files in corpus dir: $corpus_dir")
sort!(names) # deterministic selection across runs
length(names) > n && (names = names[1:n])
jobs = FS.Job[]
for (i, name) in enumerate(names)
src = joinpath(corpus_dir, name)
# Give it a fresh id/spool-style filename so nothing collides with the
# corpus the file came from.
id, spooled = FS.spool_path(cfg, @sprintf("s2-%04d-%s", i, basename(name)))
cp(src, spooled; force = true)
# Staged in spool/, not a known/ dir: stage 1 routes by enqueueing and
# leaves the bytes where intake put them, so this is the exact on-disk
# state a stage-2 worker dequeues into (src/worker.jl header).
push!(jobs, FS.Job(id, basename(name), spooled, filesize(spooled), time()))
end
return jobs
end
"""
respool!(cfg, jobs)
Put every corpus file back in `spool/`, wherever the last pass left it (done/ or
already home), and delete any sidecar it produced. This is the untimed `prepare`
step for benchmarks that consume their input by committing it. Stage 2 still
moves, because its move is the terminal commit, not an inter-stage hop.
"""
function respool!(cfg::FS.Config, jobs::Vector{FS.Job})
for job in jobs
base = basename(job.path)
for dir in (cfg.done_dir, cfg.failed_dir)
sidecar = joinpath(dir, string(base, ".meta.json"))
rm(sidecar; force = true)
rm(string(sidecar, ".tmp"); force = true)
end
isfile(job.path) && continue
for dir in (cfg.done_dir, cfg.failed_dir)
candidate = joinpath(dir, base)
if isfile(candidate)
mv(candidate, job.path; force = true)
break
end
end
end
return nothing
end
# --------------------------------------------------------------------- loggers
"""
with_logger_named(f, which, path)
Run `f` under one of the loggers the cost of logging is bracketed by:
* `:null` `NullLogger`: the `@info` macro's own overhead, nothing else.
* `:format` `ConsoleLogger` to `devnull`: message formatting and key/value
interpolation, but no I/O.
* `:flush` `FlushLogger(ConsoleLogger(io))` to a real file: what
`DarkStruct.run` installs, under the redirect it was written
for. Stage 2's per-file line is `@info`, not `@debug`, so this
row is what the deployed server actually pays.
"""
function with_logger_named(f, which::Symbol, path::AbstractString)
if which === :null
return with_logger(f, NullLogger())
elseif which === :format
return with_logger(f, ConsoleLogger(devnull))
elseif which === :flush
return open(path, "w") do io
with_logger(f, FS.FlushLogger(ConsoleLogger(io, Logging.Info)))
end
end
error("unknown logger: $which")
end
# --------------------------------------------------- exiftool spawn alternatives
"""
capture(cmd) -> Vector{UInt8}
Run `cmd` and return its stdout, tolerating a non-zero exit the way
`run_with_timeout` does. `read(cmd, String)` would throw instead, and a real
corpus makes that a question of when, not whether: exiftool exits 1 on a file
whose type it can't recognize, which in this pipeline is a routine outcome (it
yields a degraded sidecar, not a failure). This is `run_with_timeout` minus the
watchdog, so the gap between the two rows prices the watchdog exactly.
"""
function capture(cmd::Cmd)
out = IOBuffer()
proc = Base.run(pipeline(cmd; stdout = out, stderr = devnull); wait = false)
wait(proc)
return take!(out)
end
"""
batched_ns(paths, trials) -> ns_per_file
Run the whole corpus through *one* `exiftool` process and divide by the file
count. This is the floor for "what does exiftool cost if you stop paying the
interpreter startup per file": the fork, the Perl boot and the module loads are
paid once for the batch instead of once per file.
"""
function batched_ns(paths::Vector{String}, trials::Int)
return best_of(noop; trials) do
SINK[] = capture(`exiftool -json -G -n $paths`)
length(paths)
end
end
"""
stay_open_ns(paths, trials) -> ns_per_file (or nothing if unsupported)
Feed files one at a time to a single long-lived `exiftool -stay_open True -@ -`
process over a pipe, reading its `{ready}` sentinel after each. Unlike the
batched row this preserves the pipeline's actual shape (one file in, one result
out, arriving whenever it arrives) so it prices the realistic fix rather than
an unrealistic one.
"""
function stay_open_ns(paths::Vector{String}, trials::Int)
inp, outp = Pipe(), Pipe()
proc = Base.run(pipeline(`exiftool -stay_open True -@ -`;
stdin = inp, stdout = outp, stderr = devnull); wait = false)
close(inp.out); close(outp.in)
ask(path) = begin
write(inp, "-json\n-G\n-n\n", path, "\n-execute\n")
flush(inp)
readuntil(outp, "{ready}")
end
try
ask(paths[1]) # pay the one-time process startup untimed
return best_of(noop; trials) do
for p in paths
SINK[] = ask(p)
end
length(paths)
end
finally
try
write(inp, "-stay_open\nFalse\n"); flush(inp); close(inp)
wait(proc)
catch
kill(proc, Base.SIGKILL)
end
end
end
# ------------------------------------------------------------------ components
"""
component_rows(cfg, jobs, opts) -> Vector
Time each piece of stage 2 on its own. The subprocess rows run once per corpus
file (they cost milliseconds and don't need repetition); the in-memory and
filesystem rows run `reps` times; the committing rows run once per corpus file
with an untimed reset between passes.
"""
function component_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
reps, trials = opts["reps"], opts["trials"]
timeout = opts["timeout"]
nfiles = length(jobs)
paths = String[j.path for j in jobs]
rows = []
add!(name, part, ns) = push!(rows, (; name, part, ns))
# --- the bare interpreter: fork/exec + Perl boot, reading no file at all.
# Everything the real call does beyond this is actual work.
add!("exiftool -ver (spawn)", "extract", best_of(noop; trials) do
for _ in 1:nfiles
SINK[] = capture(`exiftool -ver`)
end
nfiles
end)
# --- the real command, run directly: spawn + parse the file, no watchdog.
add!("exiftool -json (raw run)", "extract", best_of(noop; trials) do
@inbounds for p in paths
SINK[] = capture(`exiftool -json -G -n $p`)
end
nfiles
end)
# --- the same command through the watchdog wrapper the stage actually uses.
# The gap to the row above is what the timeout costs.
add!("run_with_timeout", "extract", best_of(noop; trials) do
@inbounds for p in paths
SINK[] = FS.run_with_timeout(`exiftool -json -G -n $p`, timeout)
end
nfiles
end)
# --- run_exiftool: the wrapper plus JSON3.read plus the group-stripped Dict.
add!("run_exiftool (total)", "extract", best_of(noop; trials) do
@inbounds for p in paths
SINK[] = FS.run_exiftool(p, timeout)
end
nfiles
end)
# --- what a fork-free exiftool would cost, two ways (see the docstrings).
add!("exiftool (batched $(nfiles)x)", "alt", batched_ns(paths, trials))
if !opts["no-stay-open"]
try
add!("exiftool (-stay_open)", "alt", stay_open_ns(paths, trials))
catch e
@warn "persistent-exiftool probe failed; skipping" exception = e
end
end
# --- parsing alone, from bytes already captured: isolates JSON3 from the fork.
raw = [FS.run_with_timeout(`exiftool -json -G -n $p`, timeout) for p in paths]
valid = [b for b in raw if b !== nothing]
if !isempty(valid)
add!("JSON3.read (parse dump)", "extract", best_of(noop; trials) do
@inbounds for i in 1:reps
SINK[] = JSON3.read(String(copy(valid[(i - 1) % length(valid) + 1])))
end
reps
end)
end
# --- normalize_metadata: the ~20 tag coalesces, on a tag map already in memory.
bytags = [FS.run_exiftool(p, timeout) for p in paths]
good = [(j, b) for (j, b) in zip(jobs, bytags) if b !== nothing]
isempty(good) && error("exiftool produced no parseable output for any corpus file")
add!("normalize_metadata", "extract", best_of(noop; trials) do
@inbounds for i in 1:reps
j, b = good[(i - 1) % length(good) + 1]
SINK[] = FS.normalize_metadata(j, b)
end
reps
end)
# The sidecar payloads, built once, untimed: the commit rows below measure
# committing, not extracting.
metas = [FS.normalize_metadata(j, b) for (j, b) in good]
# --- serializing the sidecar (the raw dump makes this bigger than it looks).
add!("JSON3.write (sidecar)", "commit", best_of(noop; trials) do
@inbounds for i in 1:reps
SINK[] = JSON3.write(metas[(i - 1) % length(metas) + 1])
end
reps
end)
# --- sidecar bytes: open + write + flush + fsync, to a temp name.
# Non-consuming: same path rewritten each rep, as commit_enriched! does.
tmp = joinpath(cfg.done_dir, "bench_stage2_sidecar.tmp")
blobs = [JSON3.write(m) for m in metas]
# One pass over the real sidecar population, not `reps` of them. Two reasons,
# and the first is a correctness trap: thousands of back-to-back fsyncs
# saturate the device's write cache and each one starts waiting on the
# queue, which reported this row at 16 ms/file: eight times the whole
# `commit_enriched!` that contains it. The real stage fsyncs once per file
# with ~160 ms of exiftool between, and never queues that way. Second, real
# sidecars vary hugely in size (a zip's raw dump dwarfs a jpeg's), so the
# honest per-file number is one pass over all of them, not a cycle.
nio = length(blobs)
add!("write + fsync (sidecar)", "commit", best_of(noop; trials) do
@inbounds for i in 1:nio
open(tmp, "w") do io
write(io, blobs[(i - 1) % length(blobs) + 1])
flush(io)
FS.fsync_fd(fd(io))
end
end
nio
end)
rm(tmp; force = true)
# --- fsync_dir: persisting the rename itself, once per file in the real path.
add!("fsync_dir (done/)", "commit", best_of(noop; trials) do
for _ in 1:nio
SINK[] = FS.fsync_dir(cfg.done_dir)
end
nio
end)
# --- move_to: the rename spool/<f> -> done/<f>. Consuming: reset each pass.
add!("move_to (rename)", "commit", best_of(() -> respool!(cfg, jobs); trials) do
@inbounds for job in jobs
SINK[] = FS.move_to(cfg.done_dir, job)
end
nfiles
end)
# --- commit_enriched!: the whole sidecar-first commit, extraction excluded.
committable = [j for (j, _) in good]
add!("commit_enriched! (total)", "commit",
best_of(() -> respool!(cfg, jobs); trials) do
@inbounds for (k, job) in enumerate(committable)
SINK[] = FS.commit_enriched!(cfg.done_dir, job, metas[k])
end
length(committable)
end)
respool!(cfg, jobs)
# --- the one @info line, under each logger.
job1, meta1 = good[1][1], metas[1]
logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
for (which, label) in ((:null, "logging (NullLogger)"),
(:format, "logging (format only)"),
(:flush, "logging (flush→file)"))
ns = with_logger_named(which, logfile) do
best_of(noop; trials) do
for _ in 1:reps
@info "enriched" worker=1 id=job1.id dest=job1.path sidecar="x.meta.json" file_type=meta1.file_type created_by=meta1.created_by degraded=(meta1.error !== nothing)
end
reps
end
end
add!(label, "log", ns)
end
rm(logfile; force = true)
return rows
end
"""
handler_rows(cfg, jobs, opts) -> Vector
Time the real `handle_known_job` end to end under each logger. The difference
between the rows is the cost logging adds to a file; the `:flush` row is what the
running server actually pays.
"""
function handler_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
trials = opts["trials"]
nfiles = length(jobs)
logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
rows = []
for (which, label) in ((:null, "handle_known_job (NullLogger)"),
(:format, "handle_known_job (format only)"),
(:flush, "handle_known_job (flush→file)"))
ns = with_logger_named(which, logfile) do
best_of(() -> respool!(cfg, jobs); trials) do
@inbounds for job in jobs
FS.handle_known_job(job, cfg, 1)
end
nfiles
end
end
push!(rows, (; name = label, part = "total", ns))
end
respool!(cfg, jobs)
rm(logfile; force = true)
return rows
end
"""
thread_rows(cfg, jobs, opts) -> Vector
Run the full handler across worker counts, under the server's real logger. Stage
2 spends most of its time in a child process, so this is the sweep that matters
most: whether the stage scales is a question about the kernel's fork path and the
machine's cores, not about Julia.
Workers pull from a shared atomic counter rather than taking a contiguous slice.
That matches the server (its pool pulls from one queue), and it matters here in a
way it doesn't for stage 1: per-file exiftool time spans two orders of magnitude
on a real corpus (a single 2 s archive among 48 files), so a static split leaves
whichever worker drew it running alone while the rest idle, and the sweep would
report a scaling ceiling that is really just load imbalance.
"""
function thread_rows(cfg::FS.Config, jobs::Vector{FS.Job}, opts)
trials = opts["trials"]
nfiles = length(jobs)
counts = opts["threads"] === nothing ?
unique([1; 2; 4; 8; Threads.nthreads()]) :
[parse(Int, s) for s in split(String(opts["threads"]), ",")]
counts = sort(unique(filter(k -> 1 <= k <= Threads.nthreads(), counts)))
logfile = joinpath(dirname(cfg.spool_dir), "bench_stage2.log")
rows = []
base = 0.0
for k in counts
ns = with_logger_named(:flush, logfile) do
next = Threads.Atomic{Int}(1)
best_of(() -> (respool!(cfg, jobs); next[] = 1); trials) do
# Shared counter, not a contiguous slice: every worker takes the
# next unclaimed file the moment it frees up, exactly as the
# server's pool takes the next job off the known queue.
@sync for t in 1:k
Threads.@spawn begin
while true
i = Threads.atomic_add!(next, 1)
i > nfiles && break
@inbounds FS.handle_known_job(jobs[i], cfg, t)
end
end
end
nfiles
end
end
r = rate(ns) # files/sec aggregate (ns is already per file, wall-clock)
k == counts[1] && (base = r)
push!(rows, (; workers = k, ns, files_per_sec = r, speedup = r / base))
end
respool!(cfg, jobs)
rm(logfile; force = true)
return rows
end
# ------------------------------------------------------------------- reporting
function print_components(rows, total_ns)
@printf("%-34s %-9s %12s %10s %9s\n", "component", "part", "per file", "rate", "% total")
rule()
for r in rows
@printf("%-34s %-9s %12s %10s %8.1f%%\n", r.name, r.part, human_time(r.ns),
human_rate(rate(r.ns)), 100 * r.ns / total_ns)
end
end
function print_threads(rows)
@printf("%-9s %12s %12s %9s\n", "workers", "per file", "throughput", "speedup")
rule()
for r in rows
@printf("%-9d %12s %12s %8.2fx\n", r.workers, human_time(r.ns),
human_rate(r.files_per_sec), r.speedup)
end
end
# ------------------------------------------------------------------------- main
function main(argv)
opts = parse_args(argv)
try
FS.assert_exiftool()
catch e
println(stderr, sprint(showerror, e)); return 1
end
root = opts["dir"] === nothing ?
mktempdir(pwd(); prefix = "bench_stage2_") : String(opts["dir"])
owned = opts["dir"] === nothing
cfg = FS.Config(
spool_dir = joinpath(root, "spool"),
done_dir = joinpath(root, "done"),
failed_dir = joinpath(root, "failed"),
exiftool_timeout = opts["timeout"],
)
for d in (cfg.spool_dir, cfg.done_dir, cfg.failed_dir)
mkpath(d)
end
jobs = try
make_corpus(cfg, String(opts["corpus"]), opts["files"])
catch e
owned && rm(root; recursive = true, force = true)
println(stderr, sprint(showerror, e)); return 1
end
bytes = sum(j.size for j in jobs)
println("stage-2 component benchmark")
rule()
@printf("%-22s %s\n", "julia threads", Threads.nthreads())
@printf("%-22s %s\n", "exiftool", strip(read(`exiftool -ver`, String)))
@printf("%-22s %s\n", "corpus", "$(length(jobs)) files ($(round(bytes / 1024^2; digits=1)) MiB) from $(opts["corpus"])")
@printf("%-22s %s\n", "scratch", root)
@printf("%-22s %s\n", "reps / trials", "$(opts["reps"]) / $(opts["trials"])")
@printf("%-22s %s\n", "exiftool timeout", "$(opts["timeout"]) s")
try
comps = component_rows(cfg, jobs, opts)
handlers = handler_rows(cfg, jobs, opts)
# The denominator is the handler as the server actually runs it: the real
# flushing logger, one worker. Percentages are shares of that, so they are
# directly comparable and the parts can be checked against the whole.
total = only(r.ns for r in handlers if r.name == "handle_known_job (flush→file)")
header("Components (single worker)")
print_components(comps, total)
header("Whole handler, by logger")
print_components(handlers, total)
pick(name) = only(r.ns for r in comps if r.name == name)
accounted = pick("run_exiftool (total)") + pick("commit_enriched! (total)") +
pick("logging (flush→file)")
println()
@printf("accounted: %s of %s (%.0f%%); unaccounted overhead %s\n",
human_time(accounted), human_time(total), 100 * accounted / total,
human_time(max(total - accounted, 0)))
threads = nothing
if !opts["no-threads"] && Threads.nthreads() > 1
threads = thread_rows(cfg, jobs, opts)
header("Full handler across workers (server logger)")
print_threads(threads)
end
if opts["json"] !== nothing
open(String(opts["json"]), "w") do io
JSON3.write(io, (;
julia_threads = Threads.nthreads(),
files = length(jobs), corpus_bytes = bytes,
reps = opts["reps"], trials = opts["trials"],
exiftool_timeout = opts["timeout"],
components = comps, handlers, threads,
))
end
println("\nwrote ", opts["json"])
end
finally
owned && rm(root; recursive = true, force = true)
end
return 0
end
exit(main(ARGS))

217
bin/cluster_calibrate.jl Normal file
View File

@@ -0,0 +1,217 @@
#!/usr/bin/env julia
#
# Phase-A calibration for stage-5 header clustering (model/DESIGN_clustering.md
# §7). Runs labeled known files through the exact clustering pipeline, scores the
# recovered partition against magic-collapsed ground truth with ARI / V-measure,
# grid-tunes (α, β, bg_mass, n), and cross-checks the winning config against a
# model-free NCD (gzip) baseline (§8). The settings printed here are the ones the
# machine rediscovers known formats at; copy the winner into config.jl.
#
# julia --project=. bin/cluster_calibrate.jl [training_set_dir]
#
# Defaults to ../training_set. Prints a report; writes nothing.
using Random
using Printf
include(joinpath(@__DIR__, "..", "src", "cluster.jl"))
# --- ground truth: magic-collapsed classes, NOT extensions (DESIGN §7.2) -----
"""
truth_label(path) -> String
The magic-collapsed format class of a file, read from its actual bytes (so
docx≡zip and the whole ELF family merge, exactly the answer we want the
clustering to reproduce). `tar` is detected by the `ustar` magic at offset 257,
outside the model's front window, so tars are the accepted blind spot that
scatters to background.
"""
function truth_label(path::AbstractString)
b = zeros(UInt8, 262)
open(path) do io
chunk = read(io, 262)
copyto!(b, 1, chunk, 1, length(chunk))
end
b[1] == 0x1f && b[2] == 0x8b && return "gzip"
b[1] == 0x50 && b[2] == 0x4b && return "pkzip"
b[1] == 0x25 && b[2] == 0x50 && b[3] == 0x44 && b[4] == 0x46 && return "pdf"
b[1] == 0xff && b[2] == 0xd8 && b[3] == 0xff && return "jpeg"
b[1] == 0x7f && b[2] == 0x45 && b[3] == 0x4c && b[4] == 0x46 && return "elf"
(b[258] == 0x75 && b[259] == 0x73 && b[260] == 0x74 && b[261] == 0x61 && b[262] == 0x72) && return "tar"
return "other"
end
# --- NCD (Normalized Compression Distance) baseline, model-free (DESIGN §8) ---
"gzip-compressed size of a byte buffer, via the gzip CLI (no CodecZlib dep)."
function gz_size(bytes::Vector{UInt8})
out = IOBuffer()
open(pipeline(`gzip -c`; stdout=out); write=true) do io
write(io, bytes)
end
return length(take!(out))
end
"NCD(x,y) = (C(xy) - min(C(x),C(y))) / max(C(x),C(y)); 0 = identical, ~1 = unrelated."
function ncd(xb, yb, cx, cy)
cxy = gz_size(vcat(xb, yb))
return (cxy - min(cx, cy)) / max(cx, cy)
end
"""
ncd_1nn_purity(paths, truth; head_bytes) -> Float64
Fraction of files whose NCD-nearest neighbour shares its true label: a cheap,
O(N²) sanity read on how well raw gzip-similarity alone separates formats on the
same input. The Bayesian clusters should broadly agree; a big gap is a red flag
(DESIGN §10.3). Uses each file's first `head_bytes` so the giant files don't
dominate compression time.
"""
function ncd_1nn_purity(paths::Vector{String}, truth::Vector{String}; head_bytes::Int=4096)
bufs = map(paths) do p
open(io -> read(io, head_bytes), p)
end
csz = gz_size.(bufs)
N = length(paths)
correct = 0
for i in 1:N
best_j = 0; best_d = Inf
for j in 1:N
i == j && continue
d = ncd(bufs[i], bufs[j], csz[i], csz[j])
if d < best_d
best_d = d; best_j = j
end
end
best_j != 0 && truth[best_j] == truth[i] && (correct += 1)
end
return correct / N
end
"1-NN label purity of a *cluster* assignment vs truth (same yardstick as NCD's)."
function cluster_1nn_purity(pred::Vector{Int}, truth::Vector{String})
# For each file, its 'nearest neighbour' is any other file in the same
# cluster; purity = P(a random same-cluster neighbour shares the true label).
groups = Dict{Int,Vector{Int}}()
for (i, k) in enumerate(pred)
push!(get!(groups, k, Int[]), i)
end
correct = 0; total = 0
for (_, idxs) in groups
length(idxs) < 2 && continue
for i in idxs
same = count(j -> j != i && truth[j] == truth[i], idxs)
total += 1
same > 0 && (correct += 1)
end
end
return total == 0 ? 0.0 : correct / total
end
# --- data ---------------------------------------------------------------------
function load_corpus(dir::AbstractString)
paths = String[]
for name in readdir(dir; join=true)
isfile(name) && push!(paths, name)
end
truth = truth_label.(paths)
return paths, truth
end
# --- grid search --------------------------------------------------------------
function evaluate(X, truth; α, β, bg_mass, sweeps, restarts, seed)
r = gibbs_cluster(X; α=α, β=β, bg_mass=bg_mass, sweeps=sweeps,
restarts=restarts, rng=MersenneTwister(seed))
pred = r.assignments
ari = adjusted_rand_index(truth, pred)
keep = truth .!= "tar"
ari_notar = adjusted_rand_index(truth[keep], pred[keep])
v, h, comp = v_measure(truth, pred)
return (; ari, ari_notar, v, h, comp, k=length(r.clusters),
bg=count(==(0), pred), result=r)
end
function main()
dir = length(ARGS) >= 1 ? ARGS[1] : joinpath(@__DIR__, "..", "..", "training_set")
isdir(dir) || error("training set dir not found: $dir")
paths, truth = load_corpus(dir)
classes = sort(unique(truth))
counts = [(c, count(==(c), truth)) for c in classes]
@printf("corpus: %d files from %s\n", length(paths), dir)
println("magic-collapsed truth classes: ", join(["$c=$n" for (c, n) in counts], " "))
println()
sweeps = 150
restarts = 6
seed = 20260703
# Grid. n is expensive to re-featurize, so loop it outermost. Ranges are
# centred where the coarse sweep found the optimum: small β (peaked
# per-position priors) is what separates formats whose headers differ in only
# a few magic bytes; large β over-merges. bg_mass barely moves the result
# here (almost nothing lands in background on this corpus), so it is fixed.
αs = [1.0, 2.0]
βs = [0.05, 0.08, 0.1, 0.15, 0.2]
bgs = [5.0]
ns = [32, 64]
println("grid search (sweeps=$sweeps, restarts=$restarts):")
@printf(" %-4s %-5s %-5s %-6s | %-6s %-8s %-6s %-6s %-6s %-4s %-4s\n",
"n", "alpha", "beta", "bgmss", "ARI", "ARI-tar", "V", "homog", "compl", "k", "bg")
results = Vector{Any}()
for n in ns
X = header_matrix(paths; n=n)
for α in αs, β in βs, bg in bgs
e = evaluate(X, truth; α=α, β=β, bg_mass=bg, sweeps=sweeps, restarts=restarts, seed=seed)
push!(results, (; n, α, β, bg, e))
@printf(" %-4d %-5.1f %-5.2f %-6.1f | %-6.3f %-8.3f %-6.3f %-6.3f %-6.3f %-4d %-4d\n",
n, α, β, bg, e.ari, e.ari_notar, e.v, e.h, e.comp, e.k, e.bg)
end
end
# Rank by ARI-excluding-tar (tar is the accepted blind spot; scoring it would
# penalise the correct answer of scattering tars to background; DESIGN §7.2).
sort!(results; by=r -> r.e.ari_notar, rev=true)
best = results[1]
println()
@printf("BEST (by ARI excl. tar): n=%d α=%.1f β=%.2f bg_mass=%.1f\n",
best.n, best.α, best.β, best.bg)
@printf(" ARI=%.3f ARI(excl tar)=%.3f V=%.3f homogeneity=%.3f completeness=%.3f clusters=%d background=%d\n",
best.e.ari, best.e.ari_notar, best.e.v, best.e.h, best.e.comp, best.e.k, best.e.bg)
# Per-cluster composition of the winning partition, and promotion nominations.
pred = best.e.result.assignments
println("\nwinning partition, cluster composition (truth breakdown):")
for (id, c) in sort(collect(best.e.result.clusters); by=x -> -x[2].members)
members = [truth[i] for i in eachindex(pred) if pred[i] == id]
comp = sort([(l, count(==(l), members)) for l in unique(members)]; by=x -> -x[2])
sig = signature(c)
promo = is_promotable(c, sig; min_members=20, min_magic=3) ? " ✓NOMINATE" : ""
@printf(" cluster %-4d n=%-3d magic=%-2d %s%s\n",
id, c.members, magic_positions(sig),
join(["$l:$k" for (l, k) in comp], " "), promo)
end
nbg = count(==(0), pred)
bg_truth = [truth[i] for i in eachindex(pred) if pred[i] == 0]
bgc = sort([(l, count(==(l), bg_truth)) for l in unique(bg_truth)]; by=x -> -x[2])
@printf(" background n=%-3d %s\n", nbg, join(["$l:$k" for (l, k) in bgc], " "))
# NCD baseline cross-check on a subsample (O(N²), so keep it small).
println("\nNCD (gzip) baseline cross-check:")
subn = min(150, length(paths))
sub = shuffle(MersenneTwister(seed), collect(1:length(paths)))[1:subn]
subpaths = paths[sub]; subtruth = truth[sub]
ncd_pur = ncd_1nn_purity(subpaths, subtruth)
Xsub = header_matrix(subpaths; n=best.n)
rsub = gibbs_cluster(Xsub; α=best.α, β=best.β, bg_mass=best.bg,
sweeps=sweeps, restarts=restarts, rng=MersenneTwister(seed))
bay_pur = cluster_1nn_purity(rsub.assignments, subtruth)
@printf(" subsample=%d NCD 1-NN label purity=%.3f Bayesian same-cluster purity=%.3f\n",
subn, ncd_pur, bay_pur)
println(" (both high ⇒ header-byte signal agrees with model-free gzip similarity; DESIGN §10.3)")
end
main()

19
bin/cluster_sweep.jl Normal file
View File

@@ -0,0 +1,19 @@
#!/usr/bin/env julia
#
# Stage-5 phase-B runner (model/DESIGN_clustering.md §9): the single-owner,
# periodic/cron process that sweeps `binary/`, folds new files into the durable
# format catalog by sequential CRP-predictive assignment, and (re)writes
# promotion nominations. Run it single-threaded on a schedule: it is the ONLY
# writer of the catalog, so no locking is needed.
#
# julia --project=. bin/cluster_sweep.jl # incremental live sweep
# julia --project=. bin/cluster_sweep.jl --compact # offline Gibbs (seed / recompact)
#
# On a fresh catalog (nothing processed yet) the incremental sweep would send
# every file to background (there are no clusters to match), so the first run
# auto-promotes to a compaction pass to seed the catalog. Configure via the
# FS_CLUSTER_* / FS_NOMINATED_DIR env vars (see src/config.jl).
using DarkStruct
DarkStruct.cluster_sweep_cli(ARGS)

2
bin/send_dir.sh
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# #
# send_dir.sh — upload every file in a directory to a running FileServer. # send_dir.sh — upload every file in a directory to a running DarkStruct.
# #
# Each file is POSTed to /upload as multipart/form-data. By default one file per # Each file is POSTed to /upload as multipart/form-data. By default one file per
# request (so you get per-file feedback); use -b to batch all files into a # request (so you get per-file feedback); use -b to batch all files into a

4
bin/server.jl
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env julia #!/usr/bin/env julia
# Entry point. Run with: julia --project -t auto bin/server.jl # Entry point. Run with: julia --project -t auto bin/server.jl
using FileServer using DarkStruct
FileServer.run() DarkStruct.run()

55
docker-compose.rabbitmq.yml Normal file
View File

@@ -0,0 +1,55 @@
# Overlay that swaps the in-process queues for durable RabbitMQ ones.
#
# docker compose up # channels
# docker compose -f docker-compose.yml -f docker-compose.rabbitmq.yml up
#
# The base file is left alone deliberately: `docker compose up` still brings up
# exactly what it always did, so neither backend is the awkward one to run.
#
# What the broker changes: a job stays unacked until its handler commits, so a
# restart resumes each file at the stage it had reached instead of re-driving
# everything in spool/ from stage 1. See "Queue backends" in the README for the
# guarantees, and for the deliberate holes.
services:
rabbitmq:
image: rabbitmq:4-management
environment:
# The built-in guest/guest only authenticates over loopback, so it cannot
# work across the compose network. Override these for anything real; they
# are here so the stack comes up with one command, not as a credential.
RABBITMQ_DEFAULT_USER: "darkstruct"
RABBITMQ_DEFAULT_PASS: "darkstruct"
ports:
# 5672 is published so you can run the server on the host
# (`julia --project=. -t auto bin/server.jl`) against this broker; 15672 is
# the management UI, which is most of the reason to run a broker you can
# look at. Neither needs publishing for the composed server to work.
- "5672:5672"
- "15672:15672"
volumes:
# Durable queues and persistent messages are only as durable as what they
# are written to: without this, `docker compose down` discards the backlog
# the acks exist to protect.
- rabbitmq-data:/var/lib/rabbitmq
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
interval: 10s
timeout: 5s
retries: 12
start_period: 30s
restart: unless-stopped
darkstruct:
environment:
FS_QUEUE_BACKEND: "rabbitmq"
FS_AMQP_URL: "amqp://darkstruct:darkstruct@rabbitmq:5672/"
depends_on:
rabbitmq:
# The server retries a broker that isn't up yet (AMQP_CONNECT_RETRY_SECONDS),
# so this gate is belt-and-braces rather than load-bearing — but it keeps
# a cold start quiet instead of a screenful of retry lines.
condition: service_healthy
volumes:
rabbitmq-data:

27
docker-compose.yml Normal file
View File

@@ -0,0 +1,27 @@
services:
darkstruct:
build:
context: .
args:
# false drops the Ruby toolchain and github-linguist (~160MB smaller).
# Stage-4 text files then get natural-language enrichment but no
# programming language; the server warns at startup and carries on.
WITH_LINGUIST: "true"
image: darkstruct:latest
ports:
- "8080:8080"
volumes:
# Every pipeline stage (spool, done, text_done, binary, failed, nominated)
# lives under /data, so one volume holds the whole in-flight working set.
- fs-data:/data
environment:
# The four pools each default to nthreads(), which oversubscribes the
# threads Julia actually has. Size them for the host instead.
FS_WORKERS: "4"
FS_KNOWN_WORKERS: "4"
FS_UNKNOWN_WORKERS: "2"
FS_TEXT_WORKERS: "2"
restart: unless-stopped
volumes:
fs-data:

314
model/DESIGN_clustering.md Normal file
View File

@@ -0,0 +1,314 @@
# Stage-5: Unknown-format discovery by Bayesian header clustering
Status: phases A and B implemented and calibrated (`src/cluster.jl` +
`src/catalog.jl`, `bin/cluster_calibrate.jl` + `bin/cluster_sweep.jl`, tests in
`test/runtests.jl`). Phase A (offline Gibbs) is calibrated; phase B's durable
single-owner catalog, incremental sweep, and nomination writer are now built on
top of the `assign_file` scoring core. This is the product of a design
interview. It captures the decisions and, just as importantly, the assumptions
we *rejected*, so they don't get silently reintroduced. §11 records
what building it actually taught us, including three assumptions in this document
that the data corrected.
## 1. Goal
Discover recurring new file formats hiding in the `binary/` bucket (the
`:unknown` sink from `classify.jl` → stage-3 triage). A genuinely novel format is
a plausible proxy for a genuinely novel producing application, but we do not
try to identify producers directly (see §3). The output is a **growing catalog of
discovered formats**, each with a magic-byte signature that can be promoted into
the classifier's fast path.
Task shape (settled): **unsupervised clustering with an unknown number of
clusters.** Not pairwise "same producer" scoring, not classification against a
fixed label set.
## 2. Two phases: build (A) then run (B)
**(A) Batch, offline, the science.** Cluster the accumulated pile from scratch.
Its job is *not* to be the catalog; it is to (i) prove the header-byte signal
actually separates formats, cross-checked against an NCD baseline (§8), and
(ii) calibrate hyperparameters against known formats (§7). Ship this first,
because it de-risks (B). If (A)'s clusters are garbage, (B)'s machinery is wasted.
**(B) Online, live, the catalog.** The target deliverable. A persistent catalog
where each discovered format has a durable, frozen ID and stored sufficient
statistics. New unknown files are scored against existing clusters; only
genuinely novel ones spawn a new entry. Clusters that accumulate enough evidence
are nominated for promotion into the classifier (§6).
## 3. What we are and are NOT clustering
We cluster by file format, not by producer. The first-*n* header bytes are
format-mandated and producer-invariant: every valid PNG shares the same magic
regardless of which program wrote it; a PDF's producer string lives deep inside
the file, not in the header. Producer identity, where recoverable at all, is
`exiftool`'s job (stage 2), not this stage's.
Corollary already visible in `../training_set`: extension labels are not
header-format labels. `docx` *is* a PK zip; `so`/`o`/`elf`/`out` are all ELF.
Merging those is correct, not error (see §7).
## 4. Model: DP mixture of per-position categoricals
A cluster is a product of independent per-position categorical distributions
over the first *n* header bytes. Position *i* carries a distribution `θᵢ` over a
**257-symbol alphabet**: byte values `0–255`, plus symbol `256 = "past EOF"`.
- Invariant positions (magic bytes) learn a spiked `θᵢ`; variable positions
(lengths, timestamps) learn a flat one. A cluster's signature = the vector of
modal symbols + per-position peakedness. That signature **is a magic-number
template**, which is the entire reason for the categorical choice.
- `257` alphabet handles short files honestly: a format that is always 20 bytes
produces a spiked "past-EOF" at positions 20–31, which is real, discriminative
signal. No zero-padding (would collide `0x00` padding with real `0x00` bytes).
**Priors:** Dirichlet on each `θᵢ` (conjugate to Categorical); **Dirichlet
process (CRP)** over cluster assignments → unknown *k* falls out natively.
**Why categorical, not Euclidean.** Bytes are categorical, not ordinal: `0x89`
and `0x88` are not "close," `0x00` and `0xFF` are not "far." k-means / Gaussian
mixtures over scaled bytes assert a metric that does not exist in header space.
**Do not reuse `model.jl`'s `[0,1]` byte scaling here.** That scaling is correct
for the Lux net and wrong for this model. We need the raw `0–255` byte as a
categorical index.
### 4a. Background component (high-entropy handling)
Add a fixed, non-adaptive uniform component (each position uniform over 257)
as the "junk drawer." Compressed/encrypted/structureless blobs are ~uniform after
any magic and would otherwise either (i) mint a singleton per file or (ii)
collapse into one flat cluster that then matches everything. The background
absorbs them cleanly.
Two populations, to be precise:
- **Structured prefix + random tail** (gzip `1f 8b`, PK zip, zstd, most encrypted
*containers*): peaked at positions 0–3, flat after. These form **real clusters
for free**: genuine discoveries, no special handling.
- **Uniform from byte 0** (raw encrypted streams, key material): nothing in the
header to cluster on → absorbed by background.
The background is never promotable. But it is not a silent sink: its
size / growth / entropy histogram is surfaced as a first-class signal ("12% of
this week's unknowns are structureless"). If sub-clustering the structureless
residue ever matters, that needs a *different* feature (byte histogram / entropy),
a separate v3 model. Header bytes genuinely cannot do it.
### 4b. Feature window
**Front-only, `n = 32`** (config knob; try 64 if under-resolved). Magic lives at
offset 0. Tail window deferred to v2: a minority of formats have trailers
(ZIP EOCD, ID3v1, PDF `%%EOF`); add as an independent *second block* of positions
only if real trailer-formats show up in the residue.
**Known blind spot: tar.** `ustar` magic is at offset 257, outside the window,
so all 100 training tars scatter to background. Accepted for v1, since tar is
already a *known* format and discovery doesn't need it. General lesson: a
minority of formats put magic at a fixed deeper offset; the fix (if ever needed)
is a sparse probe window at that offset (e.g. bytes 257–262 as a third block),
rather than densely modeling 257 front bytes, which would 8× every cluster's
`n×257` sufficient-stat table to catch one format.
## 5. Inference: different mode per phase (resolves the Bayesian-vs-catalog tension)
A sampler yields a *posterior over partitions*; a catalog needs *one partition
with durable IDs*. Two MCMC gotchas: label switching (cluster #3 is not a
stable identity across iterations/runs) and distribution-not-answer (1000
partitions, not one). We sidestep both by using two inference modes:
- **Phase (A), offline:** full collapsed Gibbs sampler over the
Dirichlet-Categorical (conjugacy → ~100 lines, no continuous approximation,
unknown *k* native). Used to validate signal, tune `α` + Dirichlet strength,
and seed the initial catalog (summarize to a point partition once, via a
VI/Binder loss over the posterior similarity matrix, tolerated because it is
offline, never in the hot path).
- **Phase (B), live:** deterministic sequential CRP-predictive assignment.
Each catalog cluster stores per-position 257-count vectors (sufficient stats).
A new file's CRP predictive probability of joining each existing cluster vs.
the background vs. spawning a new cluster is computed; assign to the argmax.
A new cluster is minted only if the new-cluster evidence beats the background
by a margin. IDs are frozen at birth → no label switching. This is exactly
the Gibbs predictive rule with existing assignments held fixed: the same math,
not an ad-hoc hack.
- **Periodic compaction, offline:** re-run Gibbs seeded from the current catalog
to merge drifted clusters / split bloated ones.
## 6. Promotion (closing the loop to the classifier)
**Layered known-check at ingest** becomes:
1. Match against promoted signatures (exact, fast), which runs *before* the net.
2. Else the Lux `:known` / `:unknown` classifier.
3. Else route to `binary/` for this stage.
**Promotion = append a magic-byte signature to a registry.** A cluster's spiked
positions (posterior max-prob `> ~0.9`) become required bytes; flat positions
become wildcards, giving a libmagic-style signature. This is a **data change, not a
retrain**; interpretable, auditable, reversible. Retraining the Lux net is a
separate, *optional periodic* activity using accumulated signature-labeled files,
never the promotion mechanism itself.
**Nominate automatically, activate by hand.** A cluster crossing the thresholds
(`≥ N` members, starting at `N ≈ 20–50`, a loose dial since a human is the
backstop; `≥ ~3` magic positions; and not the background) is written to a
`nominated/` registry with its signature, member count, and example files. A human glance
promotes it into the active set. Human gate guards the one hard-to-reverse action
(redefining "known"); everything upstream stays automatic.
## 7. Calibration: recover known formats, then trust on unknowns
Do not pick priors blind. We have ground truth: `../training_set` (100 each of
tgz/tar/pdf/docx, 98 zip, 93 jpg, ELF family) and the `data/done` corpus.
1. Run labeled known files through the exact clustering pipeline.
2. Ground truth = magic-collapsed classes, *not* extensions:
`{gzip (tgz), PKzip (docx≡zip), ELF (so/o/elf/out/x86_64), JPEG, PDF, tar}`.
Merging docx+zip and the ELF family is the correct answer; scoring
against raw extensions would penalize correctness and mistune `α`.
3. Measure recovered-vs-truth agreement with Adjusted Rand Index / V-measure.
4. **Grid-tune `α` and the Dirichlet pseudocount to maximize agreement**, which
finds the settings at which the machine rediscovers formats we already know.
5. Freeze, deploy on the `:unknown` pile.
Splitting docx from zip is a later tier: the discriminating info
(central-directory filenames like `word/document.xml`) sits at a *variable
offset*, not a fixed position. That is a different feature problem, deferred.
## 8. Julia package surface
- **Hand-rolled collapsed Gibbs** over Dirichlet-Categorical. Recommended: the
conjugacy makes it short/fast; we own the online + promotion logic; no library
impedance. `Distributions.jl` for `Dirichlet`/`Categorical` primitives.
- **`CodecZlib`** for the NCD (Normalized Compression Distance) baseline:
model-free gzip-similarity clustering. Excellent at format grouping and a great
step-(A) sanity check, but O(N²), non-generative (no signature, no online
scoring, no promotion) → baseline only, cannot be the catalog.
- **`Clustering.jl`** for `randindex` / `vmeasure`, the §7 calibration metric,
plus a throwaway k-modes-ish baseline. Not the real model (its k-means is
the Euclidean trap of §4).
- **`Turing.jl`**, considered and rejected: discrete assignment latents + DP are
awkward, and we'd still hand-roll the online path. Overkill.
## 9. Architecture: single-owner batch stage, NOT inline inference
The classifier is stateless, immutable, shared read-only across worker threads
(see `classify.jl`). The catalog is the opposite: mutable, learned, shared, since
every assigned file updates a cluster's counts. It therefore must not copy the
classifier's concurrency model (concurrent workers → lock contention, torn reads
of sufficient stats, CRP assignment against stale mass).
Because promotion is human-gated, nothing here is latency-sensitive. So:
- Workers stay stateless. They deposit `:unknown` files into `binary/` exactly as
today. No catalog access on the hot path.
- A separate stage-5 process (periodic / cron, single-threaded) owns the
catalog exclusively: sweeps newly-arrived `binary/` files, runs sequential
CRP-predictive assignment, updates sufficient stats, writes nominations.
**One writer, zero locks, no cross-thread shared mutable state.**
- The catalog is a durable file mutated by one process, so reuse the stage-2
**sidecar-first durable-commit** discipline (`commit_enriched!`: temp write →
fsync bytes → rename → fsync dir) so a crash can't corrupt it or lose a rename.
This slots in as a batch stage, matching how stages 2/3/4 already work. New
`Config` knobs follow the existing `FS_*` env-override convention (e.g.
`FS_CLUSTER_DIR`, `FS_CLUSTER_N`, `FS_CLUSTER_ALPHA`, `FS_CLUSTER_PSEUDOCOUNT`,
`FS_PROMOTE_MIN_MEMBERS`).
## 10. Concrete test assertions (write these first)
1. **Discovers nothing from noise.** Current `data/binary` = 20 small random
blobs + 1 giant PDF. Correct output: PDF is a singleton that never promotes
(N=1), 20 blobs absorbed by background, zero promoted clusters. Any
promoted cluster from this pile = broken.
2. **Recovers known formats.** On a `../training_set` sample, calibrated settings
cluster into the ~6 magic-collapsed classes with high ARI (docx+zip merged,
ELF family merged, tar in background as the accepted blind spot).
3. **NCD agreement.** Step-(A) Bayesian clusters broadly agree with the NCD
baseline on the same input; large disagreement is a red flag to investigate
before trusting the generative model.
## 11. Implementation status & calibration results (v1)
**Shipped.** `src/cluster.jl` holds feature extraction (`header_symbols`, 257-symbol
alphabet), collapsed Gibbs (`gibbs_cluster`, phase A), the sequential
CRP-predictive rule (`assign_file`, phase B core), signatures/promotion
(`signature`, `is_promotable`), and calibration metrics (`adjusted_rand_index`,
`v_measure`). All base-Julia: a base-only Lanczos `loggamma` keeps the
Dirichlet-multinomial marginal dependency-free (no Manifest churn). Config knobs
`FS_CLUSTER_*` (§9) added. `bin/cluster_calibrate.jl` runs the §7 grid and the §8
NCD baseline. Concrete §10 assertions are in the test suite (hermetic synthetic
corpora, so they need neither `../training_set` nor gzip).
**Calibrated defaults** (grid over the 700-file `training_set`, ranked by ARI
excluding tar): n=32, α=1.0, β=0.1, bg_mass=5.0 → ARI 0.77 (0.885 excl.
tar), V-measure 0.83, homogeneity 0.87. Clusters are clean and promotable:
`pkzip:197` (docx+zip correctly merged, §7.2 ✓), `gzip:100`, `jpeg`, and several
`pdf` clusters all self-nominate. The NCD baseline agrees (§10.3): on a
150-file subsample, NCD 1-NN label purity 0.90 vs. the model's same-cluster
purity 0.987. The generative header model separates formats at least as well as
model-free gzip similarity.
### Three assumptions the data corrected
1. **Tar is not in the background here; it merges into ELF.** §4b/§7 assumed
tar's `ustar`-at-257 magic is out of window so tars scatter to background. But
98/100 tars in the corpus are Hex/Elixir package tarballs whose *first
archived file is named `VERSION`* → a constant, strongly-peaked `VERSION\0`
prefix at offset 0. They do form a peaked cluster, but it merges with ELF,
because ELF's ident padding and tar's name-field zero-padding give the two a
long shared run of `0x00` in bytes 5–31; they differ in only ~3 magic bytes,
and 32 equally-weighted positions let ~20 shared zeros outvote 3 real ones. No
β both separates ELF/tar and keeps the other formats whole. The honest v1
position: this is the *same* "tar is hard" reality §4b flagged, just wearing a
different mask. Fix (v2): weight positions by inverse entropy so a
low-information shared-zero run stops dominating a few high-information magic
bytes. This generalizes beyond tar and is the highest-value next lever.
2. **You cannot cold-start every point in the background.** A natural reading of
§4a/§5 is "everything starts in the junk drawer, real clusters condense out."
That deadlocks: at a format's first file a fresh cluster and the background
are *both* uniform, so with the `bg_mass ≥ α` that §4a needs for absorption,
the background always wins and no cluster is ever seeded. Fix: **initialize
every file in its own singleton**; same-format singletons merge and snowball,
while a lone random-blob singleton dissolves on resample and is reclaimed by
the (stickier) background. Absorption still works, just not as the *initial*
state.
3. **Two pieces of math that look optional but aren't.** (a) Signature peakedness
is a Bernoulli question ("is this position fixed to byte v?"), so it uses a
2-way posterior `(count+β)/(members+2β)`, not the 257-way mixture
predictive, because the alphabet-wide denominator drags even a unanimous position
below 0.9 once β<1, which would make promotion *impossible*. (b) Ranking Gibbs
restarts needs the collapsed Dirichlet-multinomial marginal (with its
`loggamma` normalizer / Occam penalty); a plain product-of-predictives score
omits the penalty and actively rewards merging everything into one blob
(observed, then fixed).
### Known v1 limitations (accepted)
- **β=0.1 over-splits** PDF and JPEG into several *pure* sub-clusters (e.g. PDF by
version byte). This costs completeness/ARI but not the mission: each sub-cluster
still carries valid magic and promotes independently, and a human dedupes
overlapping `%PDF-1.x` nominations at the gate.
- The point partition is the best of N Gibbs restarts by marginal likelihood,
a MAP-style stand-in for the VI/Binder posterior summary §5 defers. Adequate,
because the formats are strongly separated; revisit if compaction (§5) needs it.
- Phase B's live single-owner batch process (§9) and the durable catalog file
are implemented in `src/catalog.jl` (the `Catalog` durable state, the
incremental `catalog_sweep!`, the offline `compact!` seed/recompaction, and
`write_nominations!`), driven by `bin/cluster_sweep.jl` (cron/periodic; the
first run auto-compacts to seed, subsequent runs sweep incrementally). The
catalog is persisted with the stage-2 sidecar-first temp→fsync→rename→fsync-dir
discipline. As §5B predicts, under the calibrated `bg_mass > α` the live sweep
never mints single-file clusters: new formats are discovered by the offline
`compact!` re-clustering the background residue, not by the live path.
## Open items (deferred, intentionally)
- **v2, now top priority: inverse-entropy position weighting** (unblocks ELF/tar
and any format pair that shares a long constant run; see §11).
- v2: tail-window block; sparse deep-offset probe (tar-class).
- v3: sub-clustering structureless high-entropy residue (needs entropy/histogram
feature, not header bytes).
- Later tier: docx-vs-zip split via variable-offset central-directory names.
- Periodic Lux retrain from accumulated signature-labeled files.

254
src/DarkStruct.jl Normal file
View File

@@ -0,0 +1,254 @@
module DarkStruct
using Logging
using Random
using UUIDs
using HTTP
using JSON3
using Oxygen
using Lux
using JLD2
using Languages
include("multipart.jl") # streaming multipart reader (defines UPLOAD_CHUNK_BYTES, used by config.jl)
include("config.jl")
include("job.jl")
include("queue.jl")
include("stats.jl") # per-stage counters behind GET /stats (needs Config/Job/JobQueue)
include("rabbit.jl") # RabbitMQ-backed JobQueue (needs Config/Job/JobQueue/STAGE_KEYS)
include("spool.jl")
include("model.jl") # build_model() + read_features(); shared with bin/train.jl
include("classify.jl") # Classifier + load_classifier/classify (needs model.jl)
include("metadata.jl") # exiftool extraction + sidecar enrichment (stage 2)
include("content.jl") # binary-vs-text triage for unknown files (stage 3)
include("language.jl") # natural + programming language enrichment for text (stage 4)
include("cluster.jl") # unknown-format discovery by header clustering (stage 5, science)
include("catalog.jl") # durable single-owner format catalog (stage 5, phase B; needs cluster.jl + metadata.jl fsync)
include("worker.jl")
# Globals the HTTP handlers read at request time. Set once in `run`, before the
# server starts accepting connections. Declared after the includes above so the
# `Config`/`JobQueue` types exist.
#
# Typed as the abstract `JobQueue` because which implementation these hold is a
# runtime choice (`FS_QUEUE_BACKEND`). That makes `enqueue!` a dynamic dispatch
# on the intake path — nanoseconds against a handler that writes a file to disk,
# and the price of the backend being swappable at all.
const CONFIG = Ref{Config}()
const QUEUE = Ref{JobQueue}() # stage-1 (classification) queue; HTTP intake enqueues here
const KNOWN_QUEUE = Ref{JobQueue}() # stage-2 (enrichment) queue; stage-1 workers enqueue here
const UNKNOWN_QUEUE = Ref{JobQueue}() # stage-3 (content triage) queue; stage-1 workers enqueue here
const TEXT_QUEUE = Ref{JobQueue}() # stage-4 (language enrichment) queue; stage-3 workers enqueue here
const CLASSIFIER = Ref{Classifier}() # loaded once at startup, shared read-only across workers
const DETECTOR = Ref{LanguageDetector}() # natural-language detector; built once at startup, shared read-only
include("server.jl") # registers routes (references CONFIG/QUEUE at call time)
export run
# When stderr is a live terminal Julia flushes each write, but when it's
# redirected to a file or pipe (a log file, `tee`, journald, a container log
# driver) Julia block-buffers it, so a long-running server's logs sit unseen in
# the buffer until it fills or the process exits, making it look like nothing is
# happening. This wrapper delegates to a normal logger and flushes after every
# message so output appears immediately wherever stderr is pointed.
struct FlushLogger{L<:AbstractLogger} <: AbstractLogger
inner::L
end
Logging.min_enabled_level(f::FlushLogger) = Logging.min_enabled_level(f.inner)
Logging.shouldlog(f::FlushLogger, args...) = Logging.shouldlog(f.inner, args...)
Logging.catch_exceptions(f::FlushLogger) = Logging.catch_exceptions(f.inner)
function Logging.handle_message(f::FlushLogger, args...; kwargs...)
Logging.handle_message(f.inner, args...; kwargs...)
flush(f.inner.stream)
return nothing
end
"""
run(; overrides...)
Start DarkStruct: build config, ensure directories, recover any leftover
spooled files, spawn the worker pool, then serve HTTP until interrupted
(Ctrl-C / SIGINT or SIGTERM). On shutdown it stops accepting uploads, drains the
queue, waits for workers to finish in-flight files, and exits cleanly.
Keyword `overrides` (e.g. `port=9000`) take precedence over environment vars.
"""
function run(; overrides...)
# By default a Julia script exits immediately on SIGINT (Ctrl-C). Disable
# that so the interrupt surfaces as a catchable InterruptException, letting
# us drain the queue gracefully below.
Base.exit_on_sigint(false)
# Flush every log message so output is visible in real time even when stderr
# is redirected to a file/pipe (see FlushLogger). Set before anything logs.
global_logger(FlushLogger(ConsoleLogger(stderr)))
cfg = config_from_env(; overrides...)
ensure_dirs(cfg)
# All pools draw from the same OS threads. Warn on the *combined* size (still
# allowed): oversubscription just means tasks share threads, not a failure.
total_workers = cfg.worker_count + cfg.known_worker_count + cfg.unknown_worker_count + cfg.text_worker_count
if total_workers > Threads.nthreads()
@warn "combined worker count exceeds available threads; workers will share threads (start Julia with -t N for real parallelism)" classify_workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count total=total_workers nthreads=Threads.nthreads()
end
# exiftool is a hard prerequisite for stage-2 enrichment. Fail fast at
# startup rather than discover it missing on the first known file.
assert_exiftool()
# github-linguist powers stage-4 *programming*-language detection, but it's
# best-effort (natural-language enrichment stands on its own), so a missing
# binary is a warning, not a fatal error; per-file lookups degrade to none.
linguist_available() || @warn "github-linguist not found on PATH; stage-4 text files will have no programming language (install it to enable)"
# The one place the backend choice is made. Everything downstream — handlers,
# workers, /stats — sees only `JobQueue` (src/queue.jl).
# Set if the broker connection dies under us. This backend does not
# reconnect — see `poll_loop!` — so the honest response is to drain and let
# the restart policy bring us back to redelivered messages.
connection_lost = Threads.Atomic{Bool}(false)
backend = cfg.queue_backend === :rabbitmq ? connect_backend(cfg) : nothing
queues = backend === nothing ?
(classify = ChannelQueue(cfg.queue_capacity),
enrich = ChannelQueue(cfg.known_queue_capacity),
triage = ChannelQueue(cfg.unknown_queue_capacity),
language = ChannelQueue(cfg.text_queue_capacity)) :
open_queues(backend, cfg; on_lost = () -> (connection_lost[] = true))
queue, known_queue = queues.classify, queues.enrich
unknown_queue, text_queue = queues.triage, queues.language
CONFIG[] = cfg
QUEUE[] = queue
KNOWN_QUEUE[] = known_queue
UNKNOWN_QUEUE[] = unknown_queue
TEXT_QUEUE[] = text_queue
# Load the classifier before serving. Fail fast: a server that silently
# doesn't classify is a worse surprise than a clear startup error.
CLASSIFIER[] = load_classifier(cfg.model_path)
@info "loaded classifier" path=cfg.model_path
# Build the natural-language detector once (it loads the whatlang n-gram
# model) and share it read-only across the stage-4 pool, like the classifier.
DETECTOR[] = LanguageDetector()
@info "loaded language detector"
@info "starting darkstruct" backend=cfg.queue_backend host=cfg.host port=cfg.port workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count capacity=cfg.queue_capacity known_capacity=cfg.known_queue_capacity unknown_capacity=cfg.unknown_queue_capacity text_capacity=cfg.text_queue_capacity
# Zero the counters here, not at module load: `since` should mean "serving
# since", so a scrape's totals cover the run, not the minutes spent loading
# the classifier. It must also come *before* the pools start, since recovery
# below runs against live workers. The replayed files are real work and
# belong in the totals, but only from this point on.
reset_metrics!()
# Each pool gets its stage's counters (src/stats.jl); `worker_loop` records
# into them, `GET /stats` reads them out. Stage 1 and 3 also hand theirs to
# their handler, which charges time parked on a full downstream queue to
# `blocked_ns` so it isn't mistaken for work.
st = METRICS.stages
workers = [Threads.@spawn worker_loop(i, cfg, queue,
(job, c, wid) -> handle_classify_job(job, c, wid, known_queue, unknown_queue, st.classify),
st.classify)
for i in 1:cfg.worker_count]
known_workers = [Threads.@spawn worker_loop(i, cfg, known_queue, handle_known_job, st.enrich)
for i in 1:cfg.known_worker_count]
unknown_workers = [Threads.@spawn worker_loop(i, cfg, unknown_queue,
(job, c, wid) -> handle_unknown_job(job, c, wid, text_queue, st.triage),
st.triage)
for i in 1:cfg.unknown_worker_count]
text_workers = [Threads.@spawn worker_loop(i, cfg, text_queue,
(job, c, wid) -> handle_text_job(job, c, wid, DETECTOR[]),
st.language)
for i in 1:cfg.text_worker_count]
# Recovery runs *after* the pools are live, and blocks rather than dropping
# when the queue is full. `spool/` holds every in-flight file now, so a crash
# under load can leave far more leftovers than `queue_capacity`; with the
# consumers already running the queue drains as we fill it, so a big backlog
# takes longer to re-drive but none of it is abandoned. Serving hasn't
# started yet, so intake cannot race us for the directory.
#
# Everything re-enters at stage 1 and replays: in-process queues die with the
# process, so a crash loses the only record of how far each file had got.
# Safe, because every stage is a pure function of the file's bytes and every
# commit is idempotent; see the header of src/worker.jl.
#
# Broker-backed, this is exactly the wrong thing to do: the broker still
# holds every unacked job at the stage it reached, so re-driving `spool/`
# would duplicate the entire in-flight backlog on every restart and drag
# finished stages back to the start. `FS_RECOVER_SPOOL` forces it anyway, for
# the one case the broker cannot cover: it was purged or recreated, and the
# spooled files are the only surviving record of the work.
if cfg.queue_backend === :channel || cfg.recover_spool
recovered = recover_dir!(cfg.spool_dir, queue)
recovered > 0 && @info "recovered leftover files; replaying from stage 1" recovered=recovered
else
in_flight = count(p -> isfile(p) && !endswith(p, ".meta.json"),
readdir(cfg.spool_dir; join = true))
@info "broker holds the in-flight record; not re-driving spool/ (set FS_RECOVER_SPOOL=true if the broker was purged)" spooled=in_flight
end
register_routes()
# `handler` replaces Oxygen's root stream handler so POST /upload can read its
# body incrementally instead of having it buffered into memory first; every
# other route still goes through Oxygen (see `root_stream_handler`).
serve(; host = cfg.host, port = cfg.port, async = true, show_banner = false,
handler = root_stream_handler)
# Idempotent graceful drain: stop accepting uploads, let workers finish the
# buffered jobs, then exit. Called from two places:
# * the `finally` below, for SIGINT (Ctrl-C) and normal return, and
# * an `atexit` hook, for SIGTERM (systemd/Docker/k8s `stop`).
# We can't intercept SIGTERM directly: Julia blocks it on worker threads and
# handles it in its own runtime, so a user signal() handler never fires. But
# Julia's SIGTERM path runs `atexit` hooks, which gives us a reliable seam.
drained = Threads.Atomic{Bool}(false)
function drain()
Threads.atomic_xchg!(drained, true) && return # run at most once
@info "draining queues and stopping workers"
terminate() # 1. stop accepting new HTTP requests
close!(queue) # 2. no new classify jobs; stage-1 drains buffered
foreach(wait, workers) # 3. wait out stage-1, the ONLY producer of BOTH the
# known and unknown queues, so nothing else enqueues
close!(known_queue) # 4. now safe to close the queues stage-1 fed
close!(unknown_queue)
foreach(wait, known_workers) # 5. wait out stage-2 (terminal) and stage-3; stage-3 is
foreach(wait, unknown_workers) # the ONLY producer of the text queue
close!(text_queue) # 6. now safe to close the queue stage-3 fed
foreach(wait, text_workers) # 7. wait out stage-4
backend === nothing || close_backend!(backend) # 8. release the broker connection
@info "shutdown complete"
end
atexit(drain)
# Two ways out of here: an interrupt, or the broker connection dying.
#
# SIGINT is delivered to whichever task happens to be running on thread 1,
# which is the main task in the common case but is not guaranteed to be:
# AMQPClient runs its own receiver tasks, and an interrupt that lands in one
# of those kills the connection instead of reaching this loop. That is not a
# hole so much as the same door: the poller sees the dead connection within
# a second and sets `connection_lost`, so the drain happens either way. Under
# a process manager, prefer SIGTERM for the broker backend — it goes through
# Julia's `atexit`, which has no such lottery (see "Shutdown" in the README).
try
while !connection_lost[]
sleep(0.5) # interruptible; SIGINT throws in here
end
@error "stopping: broker connection closed"
catch e
e isa InterruptException || rethrow(e)
@info "shutdown requested (SIGINT)"
finally
drain()
end
# A lost broker is a failure, and saying so in the exit status is what lets a
# supervisor tell "asked to stop" from "fell over".
connection_lost[] && exit(1)
return nothing
end
end # module

178
src/FileServer.jl
View File

@@ -1,178 +0,0 @@
module FileServer
using Logging
using UUIDs
using HTTP
using JSON3
using Oxygen
using Lux
using JLD2
using Languages
include("config.jl")
include("job.jl")
include("queue.jl")
include("spool.jl")
include("model.jl") # build_model() + read_features(); shared with bin/train.jl
include("classify.jl") # Classifier + load_classifier/classify (needs model.jl)
include("metadata.jl") # exiftool extraction + sidecar enrichment (stage 2)
include("content.jl") # binary-vs-text triage for unknown files (stage 3)
include("language.jl") # natural + programming language enrichment for text (stage 4)
include("worker.jl")
# Globals the HTTP handlers read at request time. Set once in `run`, before the
# server starts accepting connections. Declared after the includes above so the
# `Config`/`ChannelQueue` types exist.
const CONFIG = Ref{Config}()
const QUEUE = Ref{ChannelQueue}() # stage-1 (classification) queue; HTTP intake enqueues here
const KNOWN_QUEUE = Ref{ChannelQueue}() # stage-2 (enrichment) queue; stage-1 workers enqueue here
const UNKNOWN_QUEUE = Ref{ChannelQueue}() # stage-3 (content triage) queue; stage-1 workers enqueue here
const TEXT_QUEUE = Ref{ChannelQueue}() # stage-4 (language enrichment) queue; stage-3 workers enqueue here
const CLASSIFIER = Ref{Classifier}() # loaded once at startup, shared read-only across workers
const DETECTOR = Ref{LanguageDetector}() # natural-language detector; built once at startup, shared read-only
include("server.jl") # registers routes (references CONFIG/QUEUE at call time)
export run
# When stderr is a live terminal Julia flushes each write, but when it's
# redirected to a file or pipe (a log file, `tee`, journald, a container log
# driver) Julia block-buffers it — so a long-running server's logs sit unseen in
# the buffer until it fills or the process exits, making it look like nothing is
# happening. This wrapper delegates to a normal logger and flushes after every
# message so output appears immediately wherever stderr is pointed.
struct FlushLogger{L<:AbstractLogger} <: AbstractLogger
inner::L
end
Logging.min_enabled_level(f::FlushLogger) = Logging.min_enabled_level(f.inner)
Logging.shouldlog(f::FlushLogger, args...) = Logging.shouldlog(f.inner, args...)
Logging.catch_exceptions(f::FlushLogger) = Logging.catch_exceptions(f.inner)
function Logging.handle_message(f::FlushLogger, args...; kwargs...)
Logging.handle_message(f.inner, args...; kwargs...)
flush(f.inner.stream)
return nothing
end
"""
run(; overrides...)
Start the file server: build config, ensure directories, recover any leftover
spooled files, spawn the worker pool, then serve HTTP until interrupted
(Ctrl-C / SIGINT or SIGTERM). On shutdown it stops accepting uploads, drains the
queue, waits for workers to finish in-flight files, and exits cleanly.
Keyword `overrides` (e.g. `port=9000`) take precedence over environment vars.
"""
function run(; overrides...)
# By default a Julia script exits immediately on SIGINT (Ctrl-C). Disable
# that so the interrupt surfaces as a catchable InterruptException, letting
# us drain the queue gracefully below.
Base.exit_on_sigint(false)
# Flush every log message so output is visible in real time even when stderr
# is redirected to a file/pipe (see FlushLogger). Set before anything logs.
global_logger(FlushLogger(ConsoleLogger(stderr)))
cfg = config_from_env(; overrides...)
ensure_dirs(cfg)
# All pools draw from the same OS threads. Warn on the *combined* size (still
# allowed): oversubscription just means tasks share threads, not a failure.
total_workers = cfg.worker_count + cfg.known_worker_count + cfg.unknown_worker_count + cfg.text_worker_count
if total_workers > Threads.nthreads()
@warn "combined worker count exceeds available threads; workers will share threads (start Julia with -t N for real parallelism)" classify_workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count total=total_workers nthreads=Threads.nthreads()
end
# exiftool is a hard prerequisite for stage-2 enrichment. Fail fast at
# startup rather than discover it missing on the first known file.
assert_exiftool()
# github-linguist powers stage-4 *programming*-language detection, but it's
# best-effort (natural-language enrichment stands on its own), so a missing
# binary is a warning, not a fatal error — per-file lookups degrade to none.
linguist_available() || @warn "github-linguist not found on PATH; stage-4 text files will have no programming language (install it to enable)"
queue = ChannelQueue(cfg.queue_capacity)
known_queue = ChannelQueue(cfg.known_queue_capacity)
unknown_queue = ChannelQueue(cfg.unknown_queue_capacity)
text_queue = ChannelQueue(cfg.text_queue_capacity)
CONFIG[] = cfg
QUEUE[] = queue
KNOWN_QUEUE[] = known_queue
UNKNOWN_QUEUE[] = unknown_queue
TEXT_QUEUE[] = text_queue
# Load the classifier before serving. Fail fast: a server that silently
# doesn't classify is a worse surprise than a clear startup error.
CLASSIFIER[] = load_classifier(cfg.model_path)
@info "loaded classifier" path=cfg.model_path
# Build the natural-language detector once (it loads the whatlang n-gram
# model) and share it read-only across the stage-4 pool, like the classifier.
DETECTOR[] = LanguageDetector()
@info "loaded language detector"
# Stage-aware recovery: re-drive each stage's leftovers onto its own queue so
# files resume where they were, not from scratch. spool/ → stage-1,
# known/ → stage-2, unknown/ → stage-3.
recovered = recover_dir!(cfg.spool_dir, queue)
recovered_known = recover_dir!(cfg.known_dir, known_queue)
recovered_unknown = recover_dir!(cfg.unknown_dir, unknown_queue)
recovered_text = recover_dir!(cfg.text_dir, text_queue)
@info "starting file-server" host=cfg.host port=cfg.port workers=cfg.worker_count known_workers=cfg.known_worker_count unknown_workers=cfg.unknown_worker_count text_workers=cfg.text_worker_count capacity=cfg.queue_capacity known_capacity=cfg.known_queue_capacity unknown_capacity=cfg.unknown_queue_capacity text_capacity=cfg.text_queue_capacity recovered=recovered recovered_known=recovered_known recovered_unknown=recovered_unknown recovered_text=recovered_text
workers = [Threads.@spawn worker_loop(i, cfg, queue,
(job, c, wid) -> handle_classify_job(job, c, wid, known_queue, unknown_queue))
for i in 1:cfg.worker_count]
known_workers = [Threads.@spawn worker_loop(i, cfg, known_queue, handle_known_job)
for i in 1:cfg.known_worker_count]
unknown_workers = [Threads.@spawn worker_loop(i, cfg, unknown_queue,
(job, c, wid) -> handle_unknown_job(job, c, wid, text_queue))
for i in 1:cfg.unknown_worker_count]
text_workers = [Threads.@spawn worker_loop(i, cfg, text_queue,
(job, c, wid) -> handle_text_job(job, c, wid, DETECTOR[]))
for i in 1:cfg.text_worker_count]
register_routes()
serve(; host = cfg.host, port = cfg.port, async = true, show_banner = false)
# Idempotent graceful drain: stop accepting uploads, let workers finish the
# buffered jobs, then exit. Called from two places:
# * the `finally` below, for SIGINT (Ctrl-C) and normal return, and
# * an `atexit` hook, for SIGTERM (systemd/Docker/k8s `stop`).
# We can't intercept SIGTERM directly — Julia blocks it on worker threads and
# handles it in its own runtime, so a user signal() handler never fires. But
# Julia's SIGTERM path runs `atexit` hooks, which gives us a reliable seam.
drained = Threads.Atomic{Bool}(false)
function drain()
Threads.atomic_xchg!(drained, true) && return # run at most once
@info "draining queues and stopping workers"
terminate() # 1. stop accepting new HTTP requests
close!(queue) # 2. no new classify jobs; stage-1 drains buffered
foreach(wait, workers) # 3. wait out stage-1 — the ONLY producer of BOTH the
# known and unknown queues — so nothing else enqueues
close!(known_queue) # 4. now safe to close the queues stage-1 fed
close!(unknown_queue)
foreach(wait, known_workers) # 5. wait out stage-2 (terminal) and stage-3 — stage-3 is
foreach(wait, unknown_workers) # the ONLY producer of the text queue
close!(text_queue) # 6. now safe to close the queue stage-3 fed
foreach(wait, text_workers) # 7. wait out stage-4
@info "shutdown complete"
end
atexit(drain)
try
while true
sleep(0.5) # interruptible; SIGINT throws in here
end
catch e
e isa InterruptException || rethrow(e)
@info "shutdown requested (SIGINT)"
finally
drain()
end
return nothing
end
end # module

404
src/catalog.jl Normal file
View File

@@ -0,0 +1,404 @@
# Stage-5 phase B: the live, single-owner format catalog (DESIGN §5B/§9).
#
# `cluster.jl` is the *science*: feature extraction, the collapsed Gibbs sampler
# (phase A, offline), and `assign_file` (the phase-B scoring core). This file is
# the *plumbing* that turns that core into a durable, growing catalog:
#
# * a `Catalog` = the surviving clusters' sufficient statistics + a record of
# which files have already been folded in + a few example filenames each;
# * `catalog_sweep!` = the phase-B loop. For every *new* file in `binary/`,
# run the deterministic CRP-predictive `assign_file` and fold it into the
# chosen cluster's stats (DESIGN §5B);
# * `compact!` = the offline Gibbs pass that *seeds* the catalog on first run
# and periodically re-clusters the pile (DESIGN §5, "seed the initial
# catalog" / "periodic compaction");
# * `write_nominations!` = surface promotable clusters to a human (DESIGN §6).
#
# Concurrency model is the deliberate opposite of the stateless classify workers
# (DESIGN §9): exactly ONE process owns the catalog, so there are no locks and no
# torn reads of sufficient stats. The catalog is a single durable file mutated by
# that one process; it is committed with the same sidecar-first temp→fsync→rename
# →fsync-dir discipline as the stage-2 sidecars (`commit_enriched!`), so a crash
# mid-write can neither corrupt it nor lose the rename. This file lives in the
# module (not dependency-flat like cluster.jl) because it needs JSON3 + the fsync
# helpers from metadata.jl.
"How many example filenames to retain per cluster (for the human promotion gate,
DESIGN §6). A handful is plenty to eyeball; the count/signature carry the weight."
const CATALOG_EXAMPLE_CAP = 8
"""
Catalog
The mutable phase-B state owned by the single sweep process:
* `n` : header window these clusters were built at (must match the
feature window used to score new files; frozen once seeded).
* `clusters` : id → `ClusterStats` (per-position 257-counts + member count).
Ids are frozen at birth and never renumbered, so there is no
label switching across sweeps (DESIGN §5).
* `examples` : id → up to `CATALOG_EXAMPLE_CAP` member filenames, for the
human nomination glance.
* `next_id` : the next fresh cluster id to hand out (monotone; retired ids are
never reused, keeping ids globally unique over the catalog's life).
* `processed` : basenames of every `binary/` file already folded in, so a sweep
is incremental: it touches only files it has not seen. This set
grows with the `binary/` pile it mirrors, at the same rate and
no faster, which is acceptable for v1 (DESIGN §9).
"""
mutable struct Catalog
n::Int
clusters::Dict{Int,ClusterStats}
examples::Dict{Int,Vector{String}}
next_id::Int
processed::Set{String}
end
"An empty catalog at header window `n` (no clusters seen yet)."
Catalog(n::Integer) = Catalog(Int(n), Dict{Int,ClusterStats}(),
Dict{Int,Vector{String}}(), 1, Set{String}())
"Record `name` as an example of cluster `id`, capped at `CATALOG_EXAMPLE_CAP`."
function record_example!(cat::Catalog, id::Integer, name::AbstractString)
ex = get!(cat.examples, id, String[])
length(ex) < CATALOG_EXAMPLE_CAP && push!(ex, String(name))
return nothing
end
# ---------------------------------------------------------------------------
# Durable persistence (reuse the stage-2 sidecar-first commit discipline)
# ---------------------------------------------------------------------------
# Count tables are stored sparsely, as only the non-zero
# `(position, symbol, count)` triples, because a cluster's n×257 table is
# overwhelmingly zero (a peaked
# magic byte touches one of 257 symbols per position). Sparse keeps the catalog
# file small and its load O(non-zeros), not O(n·257·K).
function _sparse_counts(counts::Matrix{Int})
n, A = size(counts)
triples = Vector{Vector{Int}}()
@inbounds for i in 1:n, v in 1:A
c = counts[i, v]
c != 0 && push!(triples, [i, v, c])
end
return triples
end
function _dense_counts(triples, n::Integer)
counts = zeros(Int, n, ALPHABET)
for t in triples
counts[Int(t[1]), Int(t[2])] = Int(t[3])
end
return counts
end
"Serialize a `Catalog` to a plain `NamedTuple` ready for `JSON3.write`."
function catalog_payload(cat::Catalog)
clusters = [(
id = id,
members = c.members,
counts = _sparse_counts(c.counts),
examples = get(cat.examples, id, String[]),
) for (id, c) in sort(collect(cat.clusters); by=first)]
return (
n = cat.n,
next_id = cat.next_id,
clusters = clusters,
processed = sort(collect(cat.processed)),
)
end
"""
save_catalog!(path, cat)
Durably write `cat` to `path` with the sidecar-first ordering (DESIGN §9): write
to a temp name, fsync the bytes, atomically rename into place, then fsync the
containing directory so the rename itself survives power loss. A crash can leave
at most a stale `.tmp`, never a torn catalog.
"""
function save_catalog!(path::AbstractString, cat::Catalog)
dir = dirname(path)
isempty(dir) || mkpath(dir)
tmp = string(path, ".tmp")
open(tmp, "w") do io
write(io, JSON3.write(catalog_payload(cat)))
flush(io)
fsync_fd(fd(io)) # persist bytes before the rename
end
mv(tmp, path; force=true) # atomic replace
fsync_dir(isempty(dir) ? "." : dir) # persist the rename itself
return path
end
"""
load_catalog(path; n) -> Catalog
Load the durable catalog from `path`, or return a fresh empty `Catalog(n)` if it
does not exist yet (first run). `n` is the configured header window used only for
the empty case; a loaded catalog keeps its own frozen `n`.
"""
function load_catalog(path::AbstractString; n::Integer=HEADER_N)
isfile(path) || return Catalog(n)
doc = JSON3.read(read(path, String))
cn = Int(doc.n)
cat = Catalog(cn)
cat.next_id = Int(doc.next_id)
for entry in doc.clusters
id = Int(entry.id)
c = ClusterStats(_dense_counts(entry.counts, cn), Int(entry.members))
cat.clusters[id] = c
cat.examples[id] = String[String(e) for e in entry.examples]
end
for name in doc.processed
push!(cat.processed, String(name))
end
return cat
end
# ---------------------------------------------------------------------------
# Listing the input pile
# ---------------------------------------------------------------------------
"""
binary_files(dir) -> Vector{String}
Sorted absolute paths of the regular files in `dir` to be swept, skipping
`.meta.json` sidecars and any `.tmp` scratch. Sorted so a sweep's sequential
CRP-predictive assignment (which is order-dependent) is deterministic run to run.
"""
function binary_files(dir::AbstractString)
isdir(dir) || return String[]
paths = String[]
for name in readdir(dir; join=true)
isfile(name) || continue
(endswith(name, ".meta.json") || endswith(name, ".tmp")) && continue
push!(paths, name)
end
sort!(paths)
return paths
end
# ---------------------------------------------------------------------------
# Phase B: the incremental sweep (the deliverable)
# ---------------------------------------------------------------------------
"""
catalog_sweep!(cat, cfg) -> NamedTuple
Fold every *new* file in `cfg.cluster_dir` into `cat` using the deterministic
CRP-predictive rule (`assign_file`, DESIGN §5B), updating the chosen cluster's
sufficient statistics in place. Files already in `cat.processed` are skipped, so
repeated sweeps are incremental and idempotent over the pile.
Per file, `assign_file` returns the argmax component:
* an existing cluster id → the file joins it (`add!`);
* `0` (background) → counted, not clustered. A novel-but-unmatched file parks
here by design; genuinely new formats are discovered by the offline
`compact!` re-clustering the residue, not by single-file minting;
* `-1` (mint) → a fresh cluster is seeded with a frozen `next_id`. Under the
calibrated `bg_mass > α` this never fires on the live path (fresh and
background share the one-file likelihood, so background always wins); the
branch exists for correctness, not as a routine outcome (DESIGN §5B).
Mutates `cat` but does NOT persist it; the caller commits once, after the sweep.
Returns a summary of what happened this sweep.
"""
function catalog_sweep!(cat::Catalog, cfg::Config)
n_seen = 0; n_bg = 0; n_joined = 0; n_minted = 0
for path in binary_files(cfg.cluster_dir)
base = basename(path)
base in cat.processed && continue
x = header_symbols(path; n=cat.n)
ids = sort!(collect(keys(cat.clusters)))
k = assign_file(x, cat.clusters, ids;
α=cfg.cluster_alpha, β=cfg.cluster_pseudocount,
bg_mass=cfg.cluster_bg_mass)
if k == -1
id = cat.next_id
cat.next_id += 1
c = ClusterStats(cat.n)
add!(c, x)
cat.clusters[id] = c
record_example!(cat, id, base)
n_minted += 1
elseif k == 0
n_bg += 1
else
add!(cat.clusters[k], x)
record_example!(cat, k, base)
n_joined += 1
end
push!(cat.processed, base)
n_seen += 1
end
return (; n_seen, n_joined, n_bg, n_minted,
n_clusters=length(cat.clusters), n_processed=length(cat.processed))
end
# ---------------------------------------------------------------------------
# Offline seed / compaction (wraps the phase-A Gibbs sampler)
# ---------------------------------------------------------------------------
"""
compact!(cat, cfg; sweeps, restarts, rng) -> NamedTuple
Re-cluster the *entire* `binary/` pile with the offline collapsed Gibbs sampler
and adopt the winning partition as the catalog's clusters (DESIGN §5). This is
both the **seed** on first run (an empty catalog has no clusters, so the live
sweep alone would send everything to background) and the **periodic compaction**
that merges drifted clusters / splits bloated ones later.
Ids are taken from the Gibbs partition and frozen; because compaction re-derives
the whole partition, this is a wholesale replace of `clusters`/`examples`, and
every file in the pile is marked processed. Callers run this on an explicit
schedule (e.g. `--compact`), never on the latency path.
"""
function compact!(cat::Catalog, cfg::Config;
sweeps::Integer=150, restarts::Integer=6,
rng::AbstractRNG=Random.default_rng())
paths = binary_files(cfg.cluster_dir)
if isempty(paths)
return (; n_files=0, n_clusters=length(cat.clusters), n_bg=0)
end
X = header_matrix(paths; n=cat.n)
result = gibbs_cluster(X; α=cfg.cluster_alpha, β=cfg.cluster_pseudocount,
bg_mass=cfg.cluster_bg_mass, sweeps=sweeps,
restarts=restarts, rng=rng)
empty!(cat.clusters)
empty!(cat.examples)
empty!(cat.processed)
for (id, c) in result.clusters
cat.clusters[id] = c
end
cat.next_id = (isempty(result.clusters) ? 0 : maximum(keys(result.clusters))) + 1
for (j, path) in enumerate(paths)
base = basename(path)
push!(cat.processed, base)
z = result.assignments[j]
z > 0 && record_example!(cat, z, base)
end
n_bg = count(==(0), result.assignments)
return (; n_files=length(paths), n_clusters=length(cat.clusters), n_bg)
end
# ---------------------------------------------------------------------------
# Nominations (closing the loop to the classifier; DESIGN §6)
# ---------------------------------------------------------------------------
"Render a signature (from `signature`) into a human-readable hex template:
two hex digits for a required byte, `EOF` for a required past-EOF, `??` for a
wildcard. This is what a human eyeballs at the promotion gate."
function signature_hex(sig::AbstractVector)
parts = map(sig) do s
s === nothing ? "??" :
s == PAST_EOF ? "EOF" :
string(s; base=16, pad=2)
end
return join(parts, " ")
end
"The required (non-wildcard) positions of a signature as `(position, byte)`
records; `byte` is the raw 0–255 value, or the string `\"past_eof\"`."
function signature_magic(sig::AbstractVector)
magic = Vector{NamedTuple{(:position, :byte),Tuple{Int,Any}}}()
for (i, s) in enumerate(sig)
s === nothing && continue
push!(magic, (position=i, byte=(s == PAST_EOF ? "past_eof" : s)))
end
return magic
end
"""
write_nominations!(cat, cfg) -> Vector{String}
Write one JSON nomination per promotable cluster (`is_promotable`, DESIGN §6)
into `cfg.nominated_dir`, each carrying the cluster's signature (hex template +
required magic positions), member count, and example filenames: everything a
human needs to glance and promote. The background (id 0) is never a cluster here,
so it can never be nominated, by construction.
Nominations are rewritten every sweep (membership only grows), so each file is
durably replaced via the same temp→fsync→rename→fsync-dir commit as the catalog.
Returns the paths written. Non-promotable clusters are left alone; a cluster
that *was* nominated and later fell below threshold cannot happen (members only
grow), so there is nothing to retract.
"""
function write_nominations!(cat::Catalog, cfg::Config)
mkpath(cfg.nominated_dir)
written = String[]
for (id, c) in sort(collect(cat.clusters); by=first)
sig = signature(c) # default β; signature β is
# decoupled from clustering β (DESIGN §11.3a)
is_promotable(c, sig; min_members=cfg.promote_min_members,
min_magic=cfg.promote_min_magic) || continue
payload = (
cluster_id = id,
members = c.members,
magic_length = magic_positions(sig),
signature_hex = signature_hex(sig),
magic = signature_magic(sig),
examples = get(cat.examples, id, String[]),
)
dest = joinpath(cfg.nominated_dir, "cluster-$(id).json")
tmp = string(dest, ".tmp")
open(tmp, "w") do io
write(io, JSON3.write(payload))
flush(io)
fsync_fd(fd(io))
end
mv(tmp, dest; force=true)
push!(written, dest)
end
fsync_dir(cfg.nominated_dir)
return written
end
# ---------------------------------------------------------------------------
# Orchestration + CLI (the periodic single-owner process; DESIGN §9)
# ---------------------------------------------------------------------------
"""
run_cluster_sweep(cfg; compact=false, rng) -> NamedTuple
One end-to-end pass of the single-owner stage-5 process: load the durable
catalog, either `compact!` (offline Gibbs, used to seed on first run or to
recompact) or `catalog_sweep!` (the incremental live path), then durably persist
the catalog and (re)write nominations. This is the whole job the cron/periodic
runner performs; `bin/cluster_sweep.jl` is a thin shell around it.
`compact` is forced when the catalog is empty (no clusters AND nothing processed
yet): a first live sweep against no clusters would send every file to background,
so the catalog must be seeded by an offline Gibbs pass before it can assign.
"""
function run_cluster_sweep(cfg::Config; compact::Bool=false,
rng::AbstractRNG=Random.default_rng())
cat = load_catalog(cfg.cluster_catalog_path; n=cfg.cluster_n)
if cat.n != cfg.cluster_n
@warn "configured cluster_n differs from the catalog's frozen window; using the catalog's" catalog_n=cat.n configured_n=cfg.cluster_n
end
is_empty = isempty(cat.clusters) && isempty(cat.processed)
mode = (compact || is_empty) ? :compact : :sweep
summary = mode === :compact ? compact!(cat, cfg; rng=rng) : catalog_sweep!(cat, cfg)
save_catalog!(cfg.cluster_catalog_path, cat)
nominated = write_nominations!(cat, cfg)
return (; mode, summary, n_nominated=length(nominated), nominated,
n_clusters=length(cat.clusters), n_processed=length(cat.processed))
end
"""
cluster_sweep_cli(args)
Entry point for `bin/cluster_sweep.jl`. Builds a `Config` from the environment,
runs one `run_cluster_sweep`, and logs a one-line summary. `--compact` forces the
offline Gibbs re-cluster (seed / periodic compaction) instead of the incremental
live sweep.
"""
function cluster_sweep_cli(args::AbstractVector{<:AbstractString}=String[])
compact = "--compact" in args
cfg = config_from_env()
ensure_dirs(cfg)
@info "stage-5 sweep starting" catalog=cfg.cluster_catalog_path input=cfg.cluster_dir compact=compact
r = run_cluster_sweep(cfg; compact=compact)
@info "stage-5 sweep complete" mode=r.mode clusters=r.n_clusters processed=r.n_processed nominated=r.n_nominated summary=r.summary
return r
end

4
src/classify.jl
View File

@@ -8,7 +8,7 @@
An in-memory, ready-to-run classifier: the (stateless) Lux `model` plus the An in-memory, ready-to-run classifier: the (stateless) Lux `model` plus the
learned parameters `ps` and states `st` loaded from the artifact. Immutable and learned parameters `ps` and states `st` loaded from the artifact. Immutable and
shared read-only across all worker threads — Lux inference is a pure function shared read-only across all worker threads, since Lux inference is a pure function
over `ps`/`st`, so no locking is needed. over `ps`/`st`, so no locking is needed.
""" """
struct Classifier{M,P,S} struct Classifier{M,P,S}
@@ -21,7 +21,7 @@ end
load_classifier(path) -> Classifier load_classifier(path) -> Classifier
Rebuild the fixed architecture and load `ps`/`st` from the JLD2 artifact at Rebuild the fixed architecture and load `ps`/`st` from the JLD2 artifact at
`path`. Throws if the file is missing or unreadable — the server fails fast at `path`. Throws if the file is missing or unreadable, so the server fails fast at
startup rather than run silently without classification. startup rather than run silently without classification.
""" """
function load_classifier(path::AbstractString) function load_classifier(path::AbstractString)

518
src/cluster.jl Normal file
View File

@@ -0,0 +1,518 @@
# Stage-5: unknown-format discovery by Bayesian header clustering.
#
# See model/DESIGN_clustering.md for the full rationale. In brief: files that
# stage-3 sorted into `binary/` are the `:unknown` sink: genuinely
# unrecognized bytes. This stage clusters them by *file format* (not producer)
# using the first `HEADER_N` header bytes, modeled as a Dirichlet-process
# mixture of per-position categoricals over a 257-symbol alphabet
# (byte 0–255, plus symbol 257 = "past EOF"). Each cluster's signature is a
# magic-number template that can be promoted into the classifier's fast path.
#
# This file is deliberately dependency-light: everything below is base Julia
# (only `log`, no `SpecialFunctions`), so it drops into the existing module and
# the offline calibration script alike without touching the Manifest. The model
# is categorical on purpose. Do NOT reuse model.jl's [0,1] byte scaling here
# (that metric is meaningful for the Lux net and meaningless for header bytes,
# where 0x89 and 0x88 are not "close"; see DESIGN §4).
"Number of leading header bytes modeled per file (the feature window). DESIGN §4b."
const HEADER_N = 32
"Alphabet size: byte values 0–255 plus one extra symbol for 'past end of file'."
const ALPHABET = 257
"The 'past EOF' symbol (1-based index `ALPHABET`). A file shorter than a given
position emits this here: real, discriminative signal for fixed-length formats,
and it avoids colliding zero-padding with genuine 0x00 header bytes (DESIGN §4)."
const PAST_EOF = ALPHABET
# ---------------------------------------------------------------------------
# Feature extraction
# ---------------------------------------------------------------------------
"""
header_symbols(path; n=HEADER_N) -> Vector{Int}
Read the first `n` bytes of the file at `path` and map them to a length-`n`
vector of 1-based categorical symbols: byte value `b` → `b + 1` (so `1..256`),
and every position at or beyond end-of-file → `PAST_EOF` (`257`). Never reads
more than `n` bytes, so memory stays flat regardless of file size.
"""
function header_symbols(path::AbstractString; n::Integer=HEADER_N)
syms = fill(PAST_EOF, n)
open(path, "r") do io
bytes = read(io, n)
@inbounds for i in eachindex(bytes)
syms[i] = Int(bytes[i]) + 1
end
end
return syms
end
"""
header_matrix(paths; n=HEADER_N) -> Matrix{Int}
Stack `header_symbols` for every path into an `n × length(paths)` matrix (one
column per file), the input layout the Gibbs sampler and predictive scorer both
consume.
"""
function header_matrix(paths::AbstractVector{<:AbstractString}; n::Integer=HEADER_N)
X = Matrix{Int}(undef, n, length(paths))
for (j, p) in enumerate(paths)
X[:, j] = header_symbols(p; n=n)
end
return X
end
# ---------------------------------------------------------------------------
# Model: DP mixture of per-position categoricals (Dirichlet-Categorical)
# ---------------------------------------------------------------------------
"""
ClusterStats
Sufficient statistics for one cluster: a per-position count table `counts`
(`n × ALPHABET`; `counts[i, v]` = how many member files show symbol `v` at
position `i`) and the member count `members`. These are exactly what phase-B
persists per catalog entry, and everything the collapsed predictive needs.
A slot with `members == 0` is inactive (reusable): the Gibbs sweep prunes
emptied clusters without renumbering, so surviving cluster ids stay stable.
"""
mutable struct ClusterStats
counts::Matrix{Int} # n × ALPHABET
members::Int
end
ClusterStats(n::Integer) = ClusterStats(zeros(Int, n, ALPHABET), 0)
"Add file `x` (a length-n symbol vector) into cluster `c`'s sufficient stats."
function add!(c::ClusterStats, x::AbstractVector{<:Integer})
@inbounds for i in eachindex(x)
c.counts[i, x[i]] += 1
end
c.members += 1
return c
end
"Remove file `x` from cluster `c`'s sufficient stats (inverse of `add!`)."
function remove!(c::ClusterStats, x::AbstractVector{<:Integer})
@inbounds for i in eachindex(x)
c.counts[i, x[i]] -= 1
end
c.members -= 1
return c
end
"""
log_predictive(c, x, β) -> Float64
Log probability that file `x` was produced by cluster `c` under the collapsed
Dirichlet-Categorical predictive, given `c`'s current counts: at each position
`i`, `p(x_i | c) = (counts[i, x_i] + β) / (members + ALPHABET·β)`, summed in log
space over positions. Call with `c` NOT containing `x` (Gibbs excludes the point
being resampled), so an emptied cluster reduces to the uniform prior `1/ALPHABET`
per position, identical to a brand-new cluster, as it should be.
"""
function log_predictive(c::ClusterStats, x::AbstractVector{<:Integer}, β::Float64)
denom = log(c.members + ALPHABET * β)
s = 0.0
@inbounds for i in eachindex(x)
s += log(c.counts[i, x[i]] + β) - denom
end
return s
end
"Log likelihood of `x` under the fixed uniform component (each position uniform
over the 257 symbols): `n · log(1/ALPHABET)`. Used for both the never-adaptive
background 'junk drawer' and the prior predictive of a fresh cluster (DESIGN §4a)."
log_uniform(n::Integer) = -n * log(ALPHABET)
# Lanczos approximation to log Γ(x) for x > 0, so partition scoring (below) needs
# the Dirichlet-multinomial marginal's gamma terms without pulling in
# SpecialFunctions, keeping this stage dependency-flat (no Manifest churn).
# g = 7, standard coefficients; accurate to ~1e-14 over the range we use.
const _LANCZOS_G = 7
const _LANCZOS_C = (0.99999999999980993, 676.5203681218851, -1259.1392167224028,
771.32342877765313, -176.61502916214059, 12.507343278686905,
-0.13857109526572012, 9.9843695780195716e-6, 1.5056327351493116e-7)
function loggamma(x::Float64)
x < 0.5 && return log(π / sin(π * x)) - loggamma(1.0 - x) # reflection
x -= 1.0
a = _LANCZOS_C[1]
t = x + _LANCZOS_G + 0.5
@inbounds for i in 1:_LANCZOS_G + 1
a += _LANCZOS_C[i + 1] / (x + i)
end
return 0.5 * log(2π) + (x + 0.5) * log(t) - t + log(a)
end
# ---------------------------------------------------------------------------
# Phase A: collapsed Gibbs sampler (offline; the science)
# ---------------------------------------------------------------------------
"""
GibbsResult
Output of `gibbs_cluster`: `assignments` (one per input file; `0` = absorbed by
the background junk drawer, positive ints = cluster id), the surviving
`clusters` keyed by id, and `score` (the partition's collapsed pseudo-likelihood,
used to rank restarts).
"""
struct GibbsResult
assignments::Vector{Int}
clusters::Dict{Int,ClusterStats}
score::Float64
end
"""
gibbs_cluster(X; α, β, bg_mass, sweeps, restarts, rng) -> GibbsResult
Cluster the columns of `X` (an `n × N` header-symbol matrix) with a collapsed
Gibbs sampler over a CRP/Dirichlet-Categorical mixture plus a fixed uniform
background (DESIGN §5). Unknown *k* falls out of the CRP natively.
Per point, per sweep, the point is removed from its cluster and reassigned by
sampling from the CRP-predictive weights:
* existing cluster `k`: `members_k · exp(log_predictive)`
* background: `bg_mass · (1/ALPHABET)^n` (never adapts)
* a fresh cluster: `α · (1/ALPHABET)^n`
A uniform/high-entropy blob matches no structured cluster, and background vs.
fresh is then decided by `bg_mass` vs. `α`; with `bg_mass ≥ α` such blobs are
absorbed rather than minting singletons. `restarts` independent runs are made
from different seeds and the highest-scoring partition is returned (a cheap,
base-only stand-in for the offline Binder/VI point-summary the design defers).
"""
function gibbs_cluster(X::AbstractMatrix{<:Integer};
α::Float64=1.0, β::Float64=0.5, bg_mass::Float64=5.0,
sweeps::Integer=80, restarts::Integer=4,
rng::AbstractRNG=Random.default_rng())
best = nothing
for _ in 1:restarts
r = _gibbs_once(X; α=α, β=β, bg_mass=bg_mass, sweeps=sweeps, rng=rng)
if best === nothing || r.score > best.score
best = r
end
end
return best
end
function _gibbs_once(X::AbstractMatrix{<:Integer};
α::Float64, β::Float64, bg_mass::Float64,
sweeps::Integer, rng::AbstractRNG)
n, N = size(X)
# Seed every file in its own singleton (NOT the background). Cold-starting
# from the background deadlocks: at a format's first file, a fresh cluster
# and the background are equally uniform, so with bg_mass ≥ α the background
# always wins and no real cluster is ever seeded. Singleton init sidesteps
# this: same-format singletons merge and snowball, while a lone
# random-blob singleton dissolves on resample and is reclaimed by the
# (stickier) background. See DESIGN §4a.
z = collect(1:N)
clusters = Dict{Int,ClusterStats}()
for j in 1:N
c = ClusterStats(n)
add!(c, view(X, :, j))
clusters[j] = c
end
next_id = N + 1
log_u = log_uniform(n)
log_bg = log(bg_mass) + log_u
log_new = log(α) + log_u
# Reused across every point-visit so the sampler's hot loop allocates nothing
# per step (2M+ visits per run): `idbuf[t]` is the cluster id whose weight is
# `logw[t+1]` (logw[1] = background, logw[end] = fresh). Rebuilding these with
# fresh `Vector`/`collect(keys(...))` each step was both slow and enough GC
# churn to trip a Julia GC segfault on long grid runs.
idbuf = Int[]
logw = Float64[]
for _ in 1:sweeps
for j in 1:N
x = view(X, :, j)
# Remove point j from its current component.
zj = z[j]
if zj > 0
c = clusters[zj]
remove!(c, x)
if c.members == 0
delete!(clusters, zj) # prune emptied cluster; id retired
end
end
# Candidate log-weights: background, each live cluster, fresh.
empty!(idbuf); empty!(logw)
push!(logw, log_bg)
for (k, c) in clusters
push!(idbuf, k)
push!(logw, log(c.members) + log_predictive(c, x, β))
end
push!(logw, log_new)
# Gumbel-max sample from the categorical over components.
pick = _gumbel_argmax(logw, rng)
if pick == 1
z[j] = 0 # background
elseif pick == length(logw)
id = next_id; next_id += 1 # fresh cluster
c = ClusterStats(n)
add!(c, x)
clusters[id] = c
z[j] = id
else
id = idbuf[pick - 1]
add!(clusters[id], x)
z[j] = id
end
end
end
return GibbsResult(z, clusters, partition_logmarginal(X, z, clusters, α, β))
end
"Argmax of `logw .+ Gumbel noise`: an exact draw from softmax(logw) without
normalizing (numerically safe for the tiny header-likelihood magnitudes)."
function _gumbel_argmax(logw::AbstractVector{Float64}, rng::AbstractRNG)
best_i = 1
best_v = -Inf
@inbounds for i in eachindex(logw)
g = logw[i] - log(-log(rand(rng)))
if g > best_v
best_v = g
best_i = i
end
end
return best_i
end
"""
partition_logmarginal(X, z, clusters, α, β) -> Float64
The joint log-evidence `log p(z, X)` of a partition under the CRP prior and the
collapsed Dirichlet-Categorical likelihood. This is the principled score for ranking
Gibbs restarts (higher = better). It is the sum of:
* the Dirichlet-multinomial **marginal** of each cluster's per-position counts,
`lΓ(Aβ) − lΓ(mₖ+Aβ) + Σ_v [lΓ(c_v+β) − lΓ(β)]`, whose normalizer supplies the
Occam penalty that a plain product-of-predictives lacks. It is what makes a
*merged, heterogeneous* cluster score **worse** than two clean ones (an
earlier pseudo-likelihood scorer omitted this and wrongly rewarded merging);
* the CRP prior over the clustered points, `K·log α + Σₖ lΓ(mₖ) + lΓ(α) −
lΓ(α+N_clustered)`, penalizing gratuitous extra clusters; and
* the fixed uniform term for background-assigned files.
"""
function partition_logmarginal(X::AbstractMatrix{<:Integer}, z::AbstractVector{<:Integer},
clusters::Dict{Int,ClusterStats}, α::Float64, β::Float64)
n, N = size(X)
Aβ = ALPHABET * β
lg_Aβ = loggamma(Aβ)
lg_β = loggamma(β)
s = 0.0
# Dirichlet-Categorical marginal likelihood, per cluster × position.
for (_, c) in clusters
lg_denom = loggamma(c.members + Aβ)
@inbounds for i in 1:n, v in 1:ALPHABET
cv = c.counts[i, v]
cv > 0 && (s += loggamma(cv + β) - lg_β)
end
s += n * (lg_Aβ - lg_denom)
end
# CRP prior over the partition of the clustered points.
n_bg = count(==(0), z)
n_clustered = N - n_bg
K = length(clusters)
s += K * log(α) + loggamma(α) - loggamma(α + n_clustered)
for (_, c) in clusters
s += loggamma(float(c.members))
end
# Background files: independent, uniform.
s += n_bg * log_uniform(n)
return s
end
# ---------------------------------------------------------------------------
# Phase B: sequential CRP-predictive assignment (online; the catalog)
# ---------------------------------------------------------------------------
"""
assign_file(x, clusters, ids; α, β, bg_mass) -> Int
Deterministically assign a single file `x` against an existing catalog: the
same CRP-predictive rule as Gibbs but at the **argmax** (no sampling) with the
current assignments held fixed (DESIGN §5B). Returns the id of the chosen
cluster, `0` for the background, or `-1` to signal "mint a new cluster". `ids`
is the caller's stable ordering of `keys(clusters)`.
A new cluster is minted (`-1`) only when the fresh-cluster weight strictly wins.
Fresh and background share the same `(1/ALPHABET)^n` likelihood (one file, however
structured, is indistinguishable from a uniform blob until a *second* like file
appears), so this reduces to `α > bg_mass`. Under the calibrated `bg_mass > α`,
minting is therefore effectively off on the live path **by design**: a novel file
that matches nothing parks in the background, and genuinely new formats are
discovered by the periodic **offline Gibbs compaction** re-clustering that
residue (DESIGN §5), not by single-file minting. Because ids are frozen at birth
by the caller, there is no label switching.
"""
function assign_file(x::AbstractVector{<:Integer}, clusters::Dict{Int,ClusterStats},
ids::AbstractVector{<:Integer};
α::Float64=1.0, β::Float64=0.5, bg_mass::Float64=5.0)
n = length(x)
log_u = log_uniform(n)
best_kind = :bg # :bg, :existing, :new
best_id = 0
best = log(bg_mass) + log_u
new_w = log(α) + log_u
if new_w > best
best = new_w; best_kind = :new
end
for k in ids
c = clusters[k]
w = log(c.members) + log_predictive(c, x, β)
if w > best
best = w; best_kind = :existing; best_id = k
end
end
return best_kind === :bg ? 0 : best_kind === :new ? -1 : best_id
end
# ---------------------------------------------------------------------------
# Signatures and promotion (closing the loop to the classifier; DESIGN §6)
# ---------------------------------------------------------------------------
"""
signature(c; peak_threshold=0.9, β=0.5) -> Vector{Union{Int,Nothing}}
Turn a cluster's counts into a libmagic-style template: at each position, if the
modal symbol's posterior probability exceeds `peak_threshold`, that byte is
*required* (returned as the 0–255 byte value, or `PAST_EOF`); otherwise the
position is a wildcard (`nothing`). The vector of required bytes IS the
magic-number template, the whole point of the categorical model (DESIGN §4).
Peakedness is a **Bernoulli** question ("is this position fixed to byte `v`, or
not?"), so it uses a 2-way posterior mean `(count + β)/(members + 2β)`, NOT the
257-way mixture predictive. The alphabet-wide version would smear the estimate
across 257 symbols (`members + 257β` in the denominator), pulling even a
unanimous position below any sane threshold once β is small, which would make
promotion impossible. This decouples signature detection from the clustering
pseudocount and the alphabet size.
"""
function signature(c::ClusterStats; peak_threshold::Float64=0.9, β::Float64=0.5)
n = size(c.counts, 1)
sig = Vector{Union{Int,Nothing}}(nothing, n)
c.members == 0 && return sig
denom = c.members + 2β
@inbounds for i in 1:n
v = argmax(view(c.counts, i, :))
p = (c.counts[i, v] + β) / denom
if p > peak_threshold
sig[i] = v == PAST_EOF ? PAST_EOF : v - 1 # back to raw byte value
end
end
return sig
end
"Number of fixed (non-wildcard) positions in a signature: its 'magic length'."
magic_positions(sig::AbstractVector) = count(!isnothing, sig)
"""
is_promotable(c, sig; min_members=20, min_magic=3) -> Bool
A cluster qualifies for *nomination* (still human-gated, DESIGN §6) when it has
at least `min_members` files AND at least `min_magic` fixed signature positions.
The background (id 0) is never passed here; it is never promotable by design.
"""
function is_promotable(c::ClusterStats, sig::AbstractVector;
min_members::Integer=20, min_magic::Integer=3)
return c.members >= min_members && magic_positions(sig) >= min_magic
end
# ---------------------------------------------------------------------------
# Calibration metrics (DESIGN §7): agreement of recovered clusters vs. truth
# ---------------------------------------------------------------------------
"Map a label vector to consecutive integer ids and a group→indices table."
function _groups(labels::AbstractVector)
g = Dict{Any,Vector{Int}}()
for (i, l) in enumerate(labels)
push!(get!(g, l, Int[]), i)
end
return g
end
"""
adjusted_rand_index(a, b) -> Float64
Adjusted Rand Index between two labelings of the same items: 1.0 = identical
partitions (up to relabeling), ~0.0 = chance agreement, can go negative. This is
the §7 calibration objective: grid-tuning maximizes ARI of recovered-vs-truth
(magic-collapsed) labels. Hand-rolled to keep the dependency footprint flat;
matches `Clustering.randindex`.
"""
function adjusted_rand_index(a::AbstractVector, b::AbstractVector)
length(a) == length(b) || throw(DimensionMismatch("label vectors differ in length"))
n = length(a)
n < 2 && return 1.0
ga = collect(values(_groups(a)))
gb = collect(values(_groups(b)))
# Contingency-table sum of C(n_ij, 2).
comb2(x) = x * (x - 1) / 2
sa = Set.(ga)
index = 0.0
for A in sa, B in gb
nij = count(in(A), B)
index += comb2(nij)
end
sum_a = sum(comb2(length(g)) for g in ga)
sum_b = sum(comb2(length(g)) for g in gb)
total = comb2(n)
expected = sum_a * sum_b / total
maxi = (sum_a + sum_b) / 2
denom = maxi - expected
return denom == 0 ? 1.0 : (index - expected) / denom
end
"""
v_measure(truth, pred; β=1.0) -> (v, homogeneity, completeness)
Entropy-based cluster agreement (Rosenberg & Hirschberg): homogeneity (each
predicted cluster holds one true class), completeness (each true class stays in
one predicted cluster), and their weighted harmonic mean `v`. Reported alongside
ARI in §7 calibration as a second, differently-biased view.
"""
function v_measure(truth::AbstractVector, pred::AbstractVector; β::Float64=1.0)
n = length(truth)
n == 0 && return (1.0, 1.0, 1.0)
gt = _groups(truth)
gp = _groups(pred)
entropy(g) = -sum((length(v) / n) * log(length(v) / n) for v in values(g))
H_C = entropy(gt)
H_K = entropy(gp)
# Conditional entropies via the contingency table.
H_CK = 0.0 # H(truth | pred)
H_KC = 0.0 # H(pred | truth)
for (_, P) in gp
Ps = Set(P)
for (_, C) in gt
nij = count(in(Ps), C)
nij == 0 && continue
H_CK -= (nij / n) * log(nij / length(P))
end
end
for (_, C) in gt
Cs = Set(C)
for (_, P) in gp
nij = count(in(Cs), P)
nij == 0 && continue
H_KC -= (nij / n) * log(nij / length(C))
end
end
homogeneity = H_C == 0 ? 1.0 : 1 - H_CK / H_C
completeness = H_K == 0 ? 1.0 : 1 - H_KC / H_K
v = (homogeneity + completeness == 0) ? 0.0 :
(1 + β) * homogeneity * completeness / (β * homogeneity + completeness)
return (v, homogeneity, completeness)
end

127
src/config.jl
View File

@@ -13,7 +13,7 @@ Base.@kwdef struct Config
known_worker_count::Int = Threads.nthreads() known_worker_count::Int = Threads.nthreads()
known_queue_capacity::Int = 1000 known_queue_capacity::Int = 1000
# Stage 3 (content triage) also gets its own pool + queue: sorting an # Stage 3 (content triage) also gets its own pool + queue: sorting an
# unrecognized file into binary/ vs text/ is cheap I/O, tuned independently # unrecognized file into binary vs text is cheap I/O, tuned independently
# of the classify and enrich pools. # of the classify and enrich pools.
unknown_worker_count::Int = Threads.nthreads() unknown_worker_count::Int = Threads.nthreads()
unknown_queue_capacity::Int = 1000 unknown_queue_capacity::Int = 1000
@@ -22,17 +22,85 @@ Base.@kwdef struct Config
# github-linguist) is a mix of CPU and process-spawn work, tuned independently. # github-linguist) is a mix of CPU and process-spawn work, tuned independently.
text_worker_count::Int = Threads.nthreads() text_worker_count::Int = Threads.nthreads()
text_queue_capacity::Int = 1000 text_queue_capacity::Int = 1000
spool_dir::String = "data/spool" # files land here on intake (pending classification) # Which `JobQueue` implementation backs the four stage queues (src/queue.jl):
known_dir::String = "data/known" # classified-known, awaiting enrichment (stage 2) # `:channel` (in-process, the default, nothing external to run) or
unknown_dir::String = "data/unknown" # classified-unknown, awaiting content triage (stage 3) # `:rabbitmq` (durable broker queues, src/rabbit.jl). The queue *capacities*
# above apply to both, but mean different things: a hard limit in-process, an
# advisory one against a polled depth on the broker (see `RabbitQueue`).
queue_backend::Symbol = :channel
amqp_url::String = "amqp://guest:guest@localhost:5672/"
amqp_prefix::String = "darkstruct" # queue names are "<prefix>.<stage>"
# Unacked messages the broker will hand one stage at a time. Defaults to that
# stage's worker count (0 below means "use it"), so a worker holds at most the
# one it is working on and a restart redelivers the minimum.
amqp_prefetch::Int = 0
# Wait for a publisher confirm before intake reports a file accepted, so a
# 202 means the broker has it rather than only that our socket took it. Costs
# a round trip per file on the intake path *only*: inter-stage publishes stay
# fire-and-forget, because a lost one leaves its source message unacked and
# redelivery repairs it. See "Queue backends" in the README.
amqp_confirms::Bool = true
# Broker-backed startup does not re-drive `spool/`: the broker is the record
# of what is in flight, and re-driving would duplicate the whole backlog at
# stage 1 on every restart. Set this when the broker was purged or recreated
# and the spooled files are the only surviving record. Ignored (always on)
# for the in-process backend, whose queues never survive a restart.
recover_spool::Bool = false
# Every in-flight file lives here, at every stage, from intake until it is
# committed to a terminal sink below. Stages route by enqueueing a reference,
# not by moving bytes (src/worker.jl header), so there are no per-stage
# directories; the queues are what know how far a file has got.
spool_dir::String = "data/spool"
binary_dir::String = "data/binary" # stage-3 sink: unknown files that look like binary data binary_dir::String = "data/binary" # stage-3 sink: unknown files that look like binary data
text_dir::String = "data/text" # classified-text, awaiting language enrichment (stage 4)
done_dir::String = "data/done" # fully enriched known files (+ .meta.json sidecars) done_dir::String = "data/done" # fully enriched known files (+ .meta.json sidecars)
text_done_dir::String = "data/text_done" # fully enriched text files (+ .meta.json sidecars) text_done_dir::String = "data/text_done" # fully enriched text files (+ .meta.json sidecars)
failed_dir::String = "data/failed" # files move here if a worker throws failed_dir::String = "data/failed" # files move here if a worker throws
model_path::String = "model/classifier.jld2" # committed classifier artifact, loaded at startup model_path::String = "model/classifier.jld2" # committed classifier artifact, loaded at startup
# Intake reads each upload off the socket in chunks of this size and streams
# them straight to the spool file, so this, not the file size, is what
# bounds intake memory per in-flight upload (see src/multipart.jl).
upload_chunk_bytes::Int = UPLOAD_CHUNK_BYTES
exiftool_timeout::Int = 30 # seconds before a stuck exiftool is killed → degraded sidecar exiftool_timeout::Int = 30 # seconds before a stuck exiftool is killed → degraded sidecar
linguist_timeout::Int = 30 # seconds before a stuck github-linguist is killed → no programming language linguist_timeout::Int = 30 # seconds before a stuck github-linguist is killed → no programming language
# Stage 5 (unknown-format discovery). A separate single-owner batch process
# sweeps binary/ and clusters headers; these are its knobs (see
# model/DESIGN_clustering.md §9). Values are the calibrated defaults from
# bin/cluster_calibrate.jl on the training corpus.
cluster_dir::String = "data/binary" # stage-5 input: the :unknown/binary sink to sweep
cluster_n::Int = 32 # header bytes modeled per file (HEADER_N)
cluster_alpha::Float64 = 1.0 # CRP concentration: propensity to spawn new formats
cluster_pseudocount::Float64 = 0.1 # Dirichlet pseudocount β; calibrated on the training corpus
cluster_bg_mass::Float64 = 5.0 # fixed mass of the uniform background 'junk drawer'
promote_min_members::Int = 20 # cluster size threshold for promotion nomination
promote_min_magic::Int = 3 # required fixed signature positions for nomination
# The stage-5 catalog is a single durable file mutated by the one sweep
# process (never a worker), and nominations are written as one file per
# promotable cluster for a human to glance at before promoting (DESIGN §9/§6).
cluster_catalog_path::String = "data/catalog.json" # durable phase-B catalog (sufficient stats + processed set)
nominated_dir::String = "data/nominated" # one JSON per self-nominated cluster, awaiting a human promote
end
"""
parse_backend(s) -> Symbol
Turn `FS_QUEUE_BACKEND` into a `Config.queue_backend`. Unknown values are a
startup error rather than a silent fallback: a typo that quietly left you on the
in-process queue would look exactly like durability working right up until a
crash proved it wasn't.
"""
function parse_backend(s::AbstractString)
b = Symbol(lowercase(strip(s)))
b in (:channel, :rabbitmq) ||
throw(ArgumentError("FS_QUEUE_BACKEND must be \"channel\" or \"rabbitmq\", got \"$s\""))
return b
end
"Parse a boolean env var. Accepts the usual spellings; anything else is an error."
function parse_bool(s::AbstractString)
v = lowercase(strip(s))
v in ("1", "true", "yes", "on") && return true
v in ("0", "false", "no", "off") && return false
throw(ArgumentError("expected a boolean (true/false), got \"$s\""))
end end
""" """
@@ -40,26 +108,38 @@ end
Build a `Config` from environment variables, falling back to the struct Build a `Config` from environment variables, falling back to the struct
defaults. Any keyword `overrides` win over the environment (useful for tests defaults. Any keyword `overrides` win over the environment (useful for tests
and for `FileServer.run(; port=...)`). and for `DarkStruct.run(; port=...)`).
Recognised variables: Recognised variables:
FS_HOST, FS_PORT, FS_WORKERS, FS_QUEUE_CAPACITY, FS_HOST, FS_PORT, FS_WORKERS, FS_QUEUE_CAPACITY,
FS_KNOWN_WORKERS, FS_KNOWN_QUEUE_CAPACITY, FS_KNOWN_WORKERS, FS_KNOWN_QUEUE_CAPACITY,
FS_UNKNOWN_WORKERS, FS_UNKNOWN_QUEUE_CAPACITY, FS_UNKNOWN_WORKERS, FS_UNKNOWN_QUEUE_CAPACITY,
FS_TEXT_WORKERS, FS_TEXT_QUEUE_CAPACITY, FS_TEXT_WORKERS, FS_TEXT_QUEUE_CAPACITY,
FS_SPOOL_DIR, FS_KNOWN_DIR, FS_UNKNOWN_DIR, FS_BINARY_DIR, FS_TEXT_DIR, FS_SPOOL_DIR, FS_BINARY_DIR,
FS_DONE_DIR, FS_TEXT_DONE_DIR, FS_FAILED_DIR, FS_MODEL_PATH, FS_DONE_DIR, FS_TEXT_DONE_DIR, FS_FAILED_DIR, FS_MODEL_PATH,
FS_EXIFTOOL_TIMEOUT, FS_LINGUIST_TIMEOUT FS_UPLOAD_CHUNK_BYTES, FS_EXIFTOOL_TIMEOUT, FS_LINGUIST_TIMEOUT,
FS_CLUSTER_DIR, FS_CLUSTER_N, FS_CLUSTER_ALPHA, FS_CLUSTER_PSEUDOCOUNT,
FS_CLUSTER_BG_MASS, FS_PROMOTE_MIN_MEMBERS, FS_PROMOTE_MIN_MAGIC,
FS_CLUSTER_CATALOG, FS_NOMINATED_DIR,
FS_QUEUE_BACKEND, FS_AMQP_URL, FS_AMQP_PREFIX, FS_AMQP_PREFETCH,
FS_AMQP_CONFIRMS, FS_RECOVER_SPOOL
""" """
function config_from_env(; host=nothing, port=nothing, worker_count=nothing, function config_from_env(; host=nothing, port=nothing, worker_count=nothing,
queue_capacity=nothing, known_worker_count=nothing, queue_capacity=nothing, known_worker_count=nothing,
known_queue_capacity=nothing, unknown_worker_count=nothing, known_queue_capacity=nothing, unknown_worker_count=nothing,
unknown_queue_capacity=nothing, text_worker_count=nothing, unknown_queue_capacity=nothing, text_worker_count=nothing,
text_queue_capacity=nothing, spool_dir=nothing, text_queue_capacity=nothing, spool_dir=nothing,
known_dir=nothing, unknown_dir=nothing, binary_dir=nothing, binary_dir=nothing, done_dir=nothing, text_done_dir=nothing,
text_dir=nothing, done_dir=nothing, text_done_dir=nothing, failed_dir=nothing, model_path=nothing,
failed_dir=nothing, model_path=nothing, exiftool_timeout=nothing, upload_chunk_bytes=nothing, exiftool_timeout=nothing,
linguist_timeout=nothing) linguist_timeout=nothing, cluster_dir=nothing, cluster_n=nothing,
cluster_alpha=nothing, cluster_pseudocount=nothing,
cluster_bg_mass=nothing, promote_min_members=nothing,
promote_min_magic=nothing, cluster_catalog_path=nothing,
nominated_dir=nothing, queue_backend=nothing,
amqp_url=nothing, amqp_prefix=nothing,
amqp_prefetch=nothing, amqp_confirms=nothing,
recover_spool=nothing)
Config( Config(
host = something(host, get(ENV, "FS_HOST", "127.0.0.1")), host = something(host, get(ENV, "FS_HOST", "127.0.0.1")),
port = something(port, parse(Int, get(ENV, "FS_PORT", "8080"))), port = something(port, parse(Int, get(ENV, "FS_PORT", "8080"))),
@@ -72,23 +152,36 @@ function config_from_env(; host=nothing, port=nothing, worker_count=nothing,
text_worker_count = something(text_worker_count, parse(Int, get(ENV, "FS_TEXT_WORKERS", string(Threads.nthreads())))), text_worker_count = something(text_worker_count, parse(Int, get(ENV, "FS_TEXT_WORKERS", string(Threads.nthreads())))),
text_queue_capacity = something(text_queue_capacity, parse(Int, get(ENV, "FS_TEXT_QUEUE_CAPACITY", "1000"))), text_queue_capacity = something(text_queue_capacity, parse(Int, get(ENV, "FS_TEXT_QUEUE_CAPACITY", "1000"))),
spool_dir = something(spool_dir, get(ENV, "FS_SPOOL_DIR", "data/spool")), spool_dir = something(spool_dir, get(ENV, "FS_SPOOL_DIR", "data/spool")),
known_dir = something(known_dir, get(ENV, "FS_KNOWN_DIR", "data/known")),
unknown_dir = something(unknown_dir, get(ENV, "FS_UNKNOWN_DIR", "data/unknown")),
binary_dir = something(binary_dir, get(ENV, "FS_BINARY_DIR", "data/binary")), binary_dir = something(binary_dir, get(ENV, "FS_BINARY_DIR", "data/binary")),
text_dir = something(text_dir, get(ENV, "FS_TEXT_DIR", "data/text")),
done_dir = something(done_dir, get(ENV, "FS_DONE_DIR", "data/done")), done_dir = something(done_dir, get(ENV, "FS_DONE_DIR", "data/done")),
text_done_dir = something(text_done_dir, get(ENV, "FS_TEXT_DONE_DIR", "data/text_done")), text_done_dir = something(text_done_dir, get(ENV, "FS_TEXT_DONE_DIR", "data/text_done")),
failed_dir = something(failed_dir, get(ENV, "FS_FAILED_DIR", "data/failed")), failed_dir = something(failed_dir, get(ENV, "FS_FAILED_DIR", "data/failed")),
model_path = something(model_path, get(ENV, "FS_MODEL_PATH", "model/classifier.jld2")), model_path = something(model_path, get(ENV, "FS_MODEL_PATH", "model/classifier.jld2")),
upload_chunk_bytes = something(upload_chunk_bytes, parse(Int, get(ENV, "FS_UPLOAD_CHUNK_BYTES", string(UPLOAD_CHUNK_BYTES)))),
exiftool_timeout = something(exiftool_timeout, parse(Int, get(ENV, "FS_EXIFTOOL_TIMEOUT", "30"))), exiftool_timeout = something(exiftool_timeout, parse(Int, get(ENV, "FS_EXIFTOOL_TIMEOUT", "30"))),
linguist_timeout = something(linguist_timeout, parse(Int, get(ENV, "FS_LINGUIST_TIMEOUT", "30"))), linguist_timeout = something(linguist_timeout, parse(Int, get(ENV, "FS_LINGUIST_TIMEOUT", "30"))),
cluster_dir = something(cluster_dir, get(ENV, "FS_CLUSTER_DIR", "data/binary")),
cluster_n = something(cluster_n, parse(Int, get(ENV, "FS_CLUSTER_N", "32"))),
cluster_alpha = something(cluster_alpha, parse(Float64, get(ENV, "FS_CLUSTER_ALPHA", "1.0"))),
cluster_pseudocount = something(cluster_pseudocount, parse(Float64, get(ENV, "FS_CLUSTER_PSEUDOCOUNT", "0.1"))),
cluster_bg_mass = something(cluster_bg_mass, parse(Float64, get(ENV, "FS_CLUSTER_BG_MASS", "5.0"))),
promote_min_members = something(promote_min_members, parse(Int, get(ENV, "FS_PROMOTE_MIN_MEMBERS", "20"))),
promote_min_magic = something(promote_min_magic, parse(Int, get(ENV, "FS_PROMOTE_MIN_MAGIC", "3"))),
cluster_catalog_path = something(cluster_catalog_path, get(ENV, "FS_CLUSTER_CATALOG", "data/catalog.json")),
nominated_dir = something(nominated_dir, get(ENV, "FS_NOMINATED_DIR", "data/nominated")),
queue_backend = something(queue_backend, parse_backend(get(ENV, "FS_QUEUE_BACKEND", "channel"))),
amqp_url = something(amqp_url, get(ENV, "FS_AMQP_URL", "amqp://guest:guest@localhost:5672/")),
amqp_prefix = something(amqp_prefix, get(ENV, "FS_AMQP_PREFIX", "darkstruct")),
amqp_prefetch = something(amqp_prefetch, parse(Int, get(ENV, "FS_AMQP_PREFETCH", "0"))),
amqp_confirms = something(amqp_confirms, parse_bool(get(ENV, "FS_AMQP_CONFIRMS", "true"))),
recover_spool = something(recover_spool, parse_bool(get(ENV, "FS_RECOVER_SPOOL", "false"))),
) )
end end
"Create all the pipeline-stage directories if they don't already exist." "Create all the pipeline-stage directories if they don't already exist."
function ensure_dirs(cfg::Config) function ensure_dirs(cfg::Config)
for d in (cfg.spool_dir, cfg.known_dir, cfg.unknown_dir, cfg.binary_dir, for d in (cfg.spool_dir, cfg.binary_dir, cfg.done_dir, cfg.text_done_dir,
cfg.text_dir, cfg.done_dir, cfg.text_done_dir, cfg.failed_dir) cfg.failed_dir, cfg.nominated_dir)
mkpath(d) mkpath(d)
end end
return nothing return nothing

10
src/content.jl
View File

@@ -1,12 +1,14 @@
# Stage-3 content triage for unknown files. # Stage-3 content triage for unknown files.
# #
# A file that stage-1 couldn't recognize is still sorted into one of two coarse # A file that stage-1 couldn't recognize is still sorted into one of two coarse
# buckets so downstream tooling can treat them differently: `text/` for # buckets so downstream tooling can treat them differently: text (human-readable)
# human-readable content, `binary/` for everything else. We sniff only the first # and binary (everything else). Only binary is a directory, because it is terminal:
# the file is committed to `binary/`; a text file has stage 4 still to come, so it
# stays in `spool/` and only its queue reference moves on. We sniff only the first
# `CONTENT_SNIFF_BYTES` (no full read) and ask two questions: does the window # `CONTENT_SNIFF_BYTES` (no full read) and ask two questions: does the window
# decode as valid UTF-8, and are any of its control bytes ones that don't belong # decode as valid UTF-8, and are any of its control bytes ones that don't belong
# in text? This is the Unicode-aware successor to the classic "NUL byte" test — # in text? This is the Unicode-aware successor to the classic "NUL byte" test.
# it accepts non-ASCII text (accents, CJK, emoji) instead of misfiling it as # It accepts non-ASCII text (accents, CJK, emoji) instead of misfiling it as
# binary, while still rejecting binary formats, which almost never form valid # binary, while still rejecting binary formats, which almost never form valid
# UTF-8 near their start (and a NUL is never a valid UTF-8 scalar, so it still # UTF-8 near their start (and a NUL is never a valid UTF-8 scalar, so it still
# reads as binary for free). # reads as binary for free).

21
src/job.jl
View File

@@ -1,8 +1,11 @@
# A unit of work on the queue. Deliberately lightweight: the file *bytes* live # A unit of work on the queue. Deliberately lightweight: the file *bytes* live
# on disk in the spool directory, and only this small reference travels through # on disk in the spool directory, and only this small reference travels through
# the queue. This is what keeps intake fast and memory flat regardless of file # the queue. This is what keeps intake fast and memory flat regardless of file
# size, and it's the shape you'd publish to RabbitMQ later (the "claim check" # size, and it's the shape published to RabbitMQ (the "claim check" pattern:
# pattern — enqueue a reference, not the payload). # enqueue a reference, not the payload).
#
# `path` is a *local* filesystem path, which is why a broker-backed pipeline is
# still single-consumer-process: see "Queue backends" in the README.
struct Job struct Job
id::String # server-minted UUID; also the on-disk filename prefix id::String # server-minted UUID; also the on-disk filename prefix
@@ -10,4 +13,18 @@ struct Job
path::String # absolute-ish path to the spooled file path::String # absolute-ish path to the spooled file
size::Int # bytes size::Int # bytes
received_at::Float64 # time() at intake received_at::Float64 # time() at intake
# Broker delivery tag, and the one field that is *not* part of the job: it
# identifies this delivery on the channel that delivered it, so it is set on
# receipt and never serialized into the message body (a tag from a previous
# connection is meaningless, and a redelivery gets a fresh one). 0 means "no
# broker" — every job under `ChannelQueue`, and every job before publishing.
# Int64 rather than UInt64 to match AMQPClient's `TAMQPDeliveryTag`.
delivery_tag::Int64
end end
Job(id, original_name, path, size, received_at) =
Job(id, original_name, path, size, received_at, Int64(0))
"Copy of `job` carrying the delivery tag it arrived with."
with_delivery_tag(job::Job, tag::Integer) =
Job(job.id, job.original_name, job.path, job.size, job.received_at, Int64(tag))

18
src/language.jl
View File

@@ -1,13 +1,13 @@
# Stage-4 language enrichment for text files. # Stage-4 language enrichment for text files.
# #
# A file that stage-3 sorted into `text/` is human-readable, but we don't yet # A file that stage-3 sorted as text is human-readable, but we don't yet know
# know *what* it is. This stage answers two questions and records them in a # *what* it is. This stage answers two questions and records them in a
# `.meta.json` sidecar, exactly like the stage-2 known-file enrichment: # `.meta.json` sidecar, exactly like the stage-2 known-file enrichment:
# #
# * natural language — via Languages.jl's `LanguageDetector` (a Julia port of # * natural language, via Languages.jl's `LanguageDetector` (a Julia port of
# the `whatlang` n-gram model): English vs. French vs. Japanese, plus a # the `whatlang` n-gram model): English vs. French vs. Japanese, plus a
# confidence score. Pure Julia, no subprocess. # confidence score. Pure Julia, no subprocess.
# * programming language — via the `github-linguist` CLI, which recognizes # * programming language, via the `github-linguist` CLI, which recognizes
# source and markup by extension + content heuristics. There is no # source and markup by extension + content heuristics. There is no
# comparable native Julia library, so we shell out (mirroring stage-2's # comparable native Julia library, so we shell out (mirroring stage-2's
# exiftool dependency). # exiftool dependency).
@@ -18,7 +18,7 @@
# #
# The github-linguist quirk that shapes this code: run against a path *inside* a # The github-linguist quirk that shapes this code: run against a path *inside* a
# git repository, linguist reads the file's committed git blob, not the bytes on # git repository, linguist reads the file's committed git blob, not the bytes on
# disk — and an untracked file (which every file under `data/` is) has no blob, # disk, and an untracked file (which every file under `data/` is) has no blob,
# so it crashes. We sidestep this by copying the file to a fresh temp dir outside # so it crashes. We sidestep this by copying the file to a fresh temp dir outside
# any repo (preserving its name so linguist's extension heuristics still fire) # any repo (preserving its name so linguist's extension heuristics still fire)
# and pointing linguist there. # and pointing linguist there.
@@ -58,7 +58,7 @@ end
Run the `LanguageDetector` on `text`, returning the language's English name Run the `LanguageDetector` on `text`, returning the language's English name
(e.g. `"English"`), its ISO 639-3 code (e.g. `"eng"`), and the model's (e.g. `"English"`), its ISO 639-3 code (e.g. `"eng"`), and the model's
confidence in `[0,1]`. Returns `(nothing, nothing, nothing)` when there is no confidence in `[0,1]`. Returns `(nothing, nothing, nothing)` when there is no
usable text (empty/whitespace) or the detector errors — the caller records that usable text (empty/whitespace) or the detector errors; the caller records that
as a degraded result rather than failing the file. as a degraded result rather than failing the file.
""" """
function detect_natural_language(detector, text::AbstractString) function detect_natural_language(detector, text::AbstractString)
@@ -80,7 +80,7 @@ when linguist can't identify one. Plain prose reports as `"Text"` and
unrecognized content as JSON `null`; both collapse to `nothing` here, since only unrecognized content as JSON `null`; both collapse to `nothing` here, since only
a real programming/markup language is worth recording. a real programming/markup language is worth recording.
`path` MUST be outside any git repository — see the module header for why. `path` MUST be outside any git repository; see the module header for why.
""" """
function run_linguist(path::AbstractString, timeout::Integer) function run_linguist(path::AbstractString, timeout::Integer)
bytes = run_with_timeout(`github-linguist --json $path`, timeout) bytes = run_with_timeout(`github-linguist --json $path`, timeout)
@@ -104,8 +104,8 @@ end
detect_programming_language(job, cfg) -> Union{String,Nothing} detect_programming_language(job, cfg) -> Union{String,Nothing}
Programming/markup language of a text file, or `nothing`. Copies the file to a Programming/markup language of a text file, or `nothing`. Copies the file to a
throwaway temp dir *outside* the git repo — under its sanitized original name so throwaway temp dir *outside* the git repo (under its sanitized original name, so
linguist's extension heuristics still apply — runs linguist there, and cleans up. linguist's extension heuristics still apply), runs linguist there, and cleans up.
""" """
function detect_programming_language(job::Job, cfg::Config) function detect_programming_language(job::Job, cfg::Config)
lang = nothing lang = nothing

86
src/metadata.jl
View File

@@ -1,17 +1,17 @@
# Stage-2 metadata extraction and enrichment. # Stage-2 metadata extraction and enrichment.
# #
# Known files are enriched by shelling out to `exiftool -json -G` (the only tool # Known files are enriched by shelling out to `exiftool -json -G` (the only tool
# with broad, multi-format coverage — there is no comparable native Julia # with broad, multi-format coverage; there is no comparable native Julia
# library), then normalizing its output into a small, stable, documented schema # library), then normalizing its output into a small, stable, documented schema
# that downstream consumers can rely on, while preserving the full raw dump. # that downstream consumers can rely on, while preserving the full raw dump.
# #
# exiftool being installed is a hard startup prerequisite (see `assert_exiftool`, # exiftool being installed is a hard startup prerequisite (see `assert_exiftool`,
# called from `run`). A per-file extraction failure or hang does NOT quarantine # called from `run`). A per-file extraction failure or hang does NOT quarantine
# the file — it produces a *degraded* sidecar recording what we know plus the # the file. It produces a *degraded* sidecar recording what we know plus the
# error, because a file that passed classification is wanted regardless of # error, because a file that passed classification is wanted regardless of
# whether we could read its metadata. # whether we could read its metadata.
"Throw at startup if the `exiftool` binary isn't on PATH — fail fast rather than discover it per file." "Throw at startup if the `exiftool` binary isn't on PATH: fail fast rather than discover it per file."
function assert_exiftool() function assert_exiftool()
try try
Base.run(pipeline(`exiftool -ver`; stdout=devnull, stderr=devnull)) Base.run(pipeline(`exiftool -ver`; stdout=devnull, stderr=devnull))
@@ -27,14 +27,14 @@ end
# colon. Extend a field simply by appending tag names here. # colon. Extend a field simply by appending tag names here.
# #
# Note the documented `Creator` ambiguity: in PDF it's the authoring app, but # Note the documented `Creator` ambiguity: in PDF it's the authoring app, but
# elsewhere it's the person. We take the simple route — `Creator` feeds # elsewhere it's the person. We take the simple route: `Creator` feeds
# `created_by` only, and `author` relies on the person-specific tags. # `created_by` only, and `author` relies on the person-specific tags.
const CREATED_BY_TAGS = ["Producer", "CreatorTool", "Creator", "Software", "Application", "Encoder", "EncodingTool", "HostComputer"] const CREATED_BY_TAGS = ["Producer", "CreatorTool", "Creator", "Software", "Application", "Encoder", "EncodingTool", "HostComputer"]
const AUTHOR_TAGS = ["Author", "Artist", "By-line", "Owner", "Artist"] const AUTHOR_TAGS = ["Author", "Artist", "By-line", "Owner", "Artist"]
const CREATED_DATE_TAGS = ["DateTimeOriginal", "CreateDate", "MediaCreateDate", "CreationDate"] const CREATED_DATE_TAGS = ["DateTimeOriginal", "CreateDate", "MediaCreateDate", "CreationDate"]
const MODIFIED_DATE_TAGS = ["ModifyDate", "FileModifyDate"] const MODIFIED_DATE_TAGS = ["ModifyDate", "FileModifyDate"]
"fsync an open file descriptor, throwing on failure — used to make a write durable before a rename commits it." "fsync an open file descriptor, throwing on failure. Makes a write durable before a rename commits it."
function fsync_fd(fd) function fsync_fd(fd)
ccall(:fsync, Cint, (Cint,), fd) == 0 || error("fsync failed: $(Base.Libc.strerror())") ccall(:fsync, Cint, (Cint,), fd) == 0 || error("fsync failed: $(Base.Libc.strerror())")
return nothing return nothing
@@ -67,6 +67,19 @@ function coalesce_tag(bytag::Dict{String,Any}, tags)
return nothing return nothing
end end
"How long a child gets to honor SIGTERM before `run_with_timeout` escalates to SIGKILL."
const KILL_GRACE_SECONDS = 2.0
"""
Send `signum` to the whole process group `pgid` (a negative pid means "the group"
to `kill(2)`). Julia's `kill(::Process, sig)` signals only the child itself,
which is not enough to enforce a timeout; see `run_with_timeout`.
"""
function signal_group(pgid::Integer, signum::Integer)
ccall(:kill, Cint, (Cint, Cint), -pgid, signum)
return nothing
end
""" """
run_with_timeout(cmd, timeout) -> Union{Vector{UInt8},Nothing} run_with_timeout(cmd, timeout) -> Union{Vector{UInt8},Nothing}
@@ -75,33 +88,56 @@ Run `cmd`, capturing stdout, and return the captured bytes on clean exit, or
SIGKILL after a grace period) once it overruns `timeout` seconds, so one SIGKILL after a grace period) once it overruns `timeout` seconds, so one
pathological input can't wedge a worker forever. Shared by the exiftool (stage 2) pathological input can't wedge a worker forever. Shared by the exiftool (stage 2)
and github-linguist (stage 4) shells. and github-linguist (stage 4) shells.
The child runs in its own process group and the timeout signals the *group*, not
just the child. This is what makes the timeout enforceable: `wait` below returns
only once the captured stdout pipe closes, and any grandchild inherits that pipe,
so signalling the child alone leaves a `sh -c "...; sleep 30"`-shaped process
tree running to completion with the worker still blocked on it. The cost of the
process group is that a hard crash of the server orphans an in-flight child
rather than taking it down; these children are short-lived and timeout-bounded,
which is the cheaper side of that trade.
""" """
function run_with_timeout(cmd::Cmd, timeout::Integer) function run_with_timeout(cmd::Cmd, timeout::Integer)
out = IOBuffer() out = IOBuffer()
proc = Base.run(pipeline(cmd; stdout=out, stderr=devnull); wait=false) # `detach` puts the child in a fresh process group (it becomes the group
# leader, so the group id is its pid); see the docstring for why the group,
# and not the child, is what the timeout has to signal.
proc = Base.run(pipeline(detach(cmd); stdout=out, stderr=devnull); wait=false)
pgid = Base.getpid(proc)
# Kill the process if it overruns the timeout. `t` polls rather than blocking # A one-shot timer, cancelled the moment the child exits, rather than a
# so we can `kill` a hung child; the poll interval bounds shutdown latency. # polling loop the caller has to join. The polling version charged every
# call the remainder of its in-flight `sleep(0.1)` *after* the child had
# already exited: ~50 ms on average, and a measured 101 ms on a process
# that exits instantly. That is pure latency on the hot path of two stages
# (exiftool here, github-linguist in stage 4), and it dwarfed the work on
# anything but a slow file. Waiting on the process directly costs nothing
# when the child exits normally, which is the overwhelmingly common case.
killed = Ref(false) killed = Ref(false)
t = Threads.@spawn begin timer = Timer(timeout) do _
waited = 0.0 process_running(proc) || return
while process_running(proc) && waited < timeout killed[] = true
sleep(0.1); waited += 0.1 signal_group(pgid, Base.SIGTERM)
end # Escalate: a process that ignores/defers SIGTERM would otherwise pin the
if process_running(proc) # worker forever on the wait(proc) below, defeating the timeout. This
killed[] = true # runs off the timer's task so the event loop isn't held during the
kill(proc, Base.SIGTERM) # grace period, and it is not joined: by the time it wakes, `wait(proc)`
# Escalate: a process that ignores/defers SIGTERM would otherwise pin # has long since returned and `process_running` settles it.
# the worker forever on the wait(proc) below, defeating the timeout. Threads.@spawn begin
grace = 0.0 deadline = time() + KILL_GRACE_SECONDS
while process_running(proc) && grace < 2.0 while process_running(proc) && time() < deadline
sleep(0.1); grace += 0.1 sleep(0.05)
end end
process_running(proc) && kill(proc, Base.SIGKILL) process_running(proc) && signal_group(pgid, Base.SIGKILL)
end end
end end
wait(proc)
wait(t) try
wait(proc)
finally
close(timer) # cancel the pending kill; a no-op if it already fired
end
(killed[] || !success(proc)) && return nothing (killed[] || !success(proc)) && return nothing
return take!(out) return take!(out)
@@ -199,7 +235,7 @@ Sequence: write `<name>.meta.json` to a temp name, fsync its bytes, rename it
into place, fsync `dest_dir` so the rename itself is durable, THEN move the file into place, fsync `dest_dir` so the rename itself is durable, THEN move the file
into `dest_dir`. A crash between the two leaves only a harmless orphan sidecar in into `dest_dir`. A crash between the two leaves only a harmless orphan sidecar in
`dest_dir` while the file stays in its stage dir, so stage-aware recovery `dest_dir` while the file stays in its stage dir, so stage-aware recovery
re-drives it and overwrites the sidecar — idempotent. The fsyncs make the re-drives it and overwrites the sidecar, idempotently. The fsyncs make the
ordering hold across power loss, not just process crashes. ordering hold across power loss, not just process crashes.
""" """
function commit_enriched!(dest_dir::AbstractString, job::Job, meta) function commit_enriched!(dest_dir::AbstractString, job::Job, meta)

4
src/model.jl
View File

@@ -4,8 +4,8 @@
# drift out of sync (same input layout, same scaling, same net shape). # drift out of sync (same input layout, same scaling, same net shape).
# #
# The including scope is expected to have already done `using Lux` and # The including scope is expected to have already done `using Lux` and
# `using JLD2` (FileServer.jl and bin/train.jl both do), so this file adds no # `using JLD2` (DarkStruct.jl and bin/train.jl both do), so this file adds no
# top-level `using` of its own and works whether it's pulled into the FileServer # top-level `using` of its own and works whether it's pulled into the DarkStruct
# module or a bare script. # module or a bare script.
"Number of input features: first 16 bytes + last 16 bytes of a file." "Number of input features: first 16 bytes + last 16 bytes of a file."

296
src/multipart.jl Normal file
View File

@@ -0,0 +1,296 @@
# Streaming multipart/form-data reader.
#
# Why this exists: HTTP.jl's `parse_multipart_form` takes the *complete* request
# body as a byte vector, so using it means every file in the request sits in
# memory at once, and is then copied again per part. That contradicts the whole
# point of this service: file bytes belong on disk, and only a small reference
# travels through the queue. So intake needs a parser that never holds a file.
#
# This reader walks the body incrementally: it pulls fixed-size chunks off the
# socket and hands each part's bytes straight to a sink (the spool file). Peak
# memory per connection is `chunk_bytes` + the boundary length, regardless of how
# large, or how many, the uploaded files are.
#
# Interface: two calls in a loop, so the caller keeps ordinary control flow
# rather than inverting into callbacks.
#
# r = MultipartReader(io, boundary)
# while (part = next_part!(r)) !== nothing
# part.filename === nothing ? skip_part_body!(r) : write_part_body!(sink, r)
# end
#
# The grammar it implements (RFC 2046 §5.1, RFC 7578):
#
# [preamble] "--" boundary CRLF
# part-headers CRLF CRLF part-body
# CRLF "--" boundary CRLF ... another part ...
# CRLF "--" boundary "--" CRLF ... end of form, [epilogue]
#
# So the delimiter that *closes* a body is CRLF + "--" + boundary, and the two
# bytes after it say whether another part follows (CRLF) or the form is over
# ("--"). Every read is bounded, and the buffer retains only the last
# `length(delimiter)-1` bytes when no delimiter is found; that tail is what
# makes a delimiter split across two chunks parse correctly.
"Default socket read size, and therefore the memory bound per in-flight upload."
const UPLOAD_CHUNK_BYTES = 64 * 1024
"""
A part header block bigger than this is abuse, not a filename. Bounding it keeps
the one genuinely unbounded-looking read (headers, which must be buffered whole
to be parsed) from being a memory hole.
"""
const MAX_PART_HEADER_BYTES = 16 * 1024
const CRLF = UInt8[0x0d, 0x0a]
const CRLFCRLF = UInt8[0x0d, 0x0a, 0x0d, 0x0a]
const DASHDASH = UInt8[0x2d, 0x2d]
"A malformed (or truncated) multipart body. Callers turn this into a 400."
struct MultipartError <: Exception
msg::String
end
Base.showerror(io::IO, e::MultipartError) = print(io, "MultipartError: ", e.msg)
"What a part's headers said about it. `filename === nothing` means a plain form field, not a file."
struct MultipartPart
name::Union{String,Nothing}
filename::Union{String,Nothing}
content_type::Union{String,Nothing}
end
"""
MultipartReader(io, boundary; chunk_bytes = UPLOAD_CHUNK_BYTES)
An incremental reader over the multipart body arriving on `io`. `boundary` is the
value from the request's `Content-Type` header (see [`multipart_boundary`](@ref)).
"""
mutable struct MultipartReader{I<:IO}
io::I
dash_boundary::Vector{UInt8} # "--" boundary: opens the first part
delimiter::Vector{UInt8} # CRLF "--" boundary: closes every part
buf::Vector{UInt8} # rolling window; bounded by chunk_bytes + delimiter
pos::Int # next unconsumed index in buf
scratch::Vector{UInt8} # reused socket read target, so chunks don't churn the GC
chunk_bytes::Int
state::Symbol # :preamble | :at_delimiter | :body | :done
end
function MultipartReader(io::IO, boundary::AbstractString;
chunk_bytes::Int = UPLOAD_CHUNK_BYTES)
chunk_bytes > 0 || throw(ArgumentError("chunk_bytes must be positive"))
isempty(boundary) && throw(MultipartError("empty multipart boundary"))
dash_boundary = Vector{UInt8}(codeunits(string("--", boundary)))
return MultipartReader(io, dash_boundary, vcat(CRLF, dash_boundary),
UInt8[], 1, Vector{UInt8}(undef, chunk_bytes),
chunk_bytes, :preamble)
end
"""
multipart_boundary(content_type) -> String | nothing
Pull the boundary out of a `multipart/form-data` Content-Type header. Returns
`nothing` if the header is missing, is some other media type, or has no boundary.
All of those are the same 400 to a caller.
"""
function multipart_boundary(content_type::Union{AbstractString,Nothing})
content_type === nothing && return nothing
occursin(r"^\s*multipart/form-data"i, content_type) || return nothing
m = match(r"(?i:\bboundary)=(?:\"([^\"]+)\"|([^\s;]+))", content_type)
m === nothing && return nothing
return String(something(m[1], m[2]))
end
# ------------------------------------------------------------------ buffer plumbing
"Unconsumed bytes currently buffered."
navail(r::MultipartReader) = length(r.buf) - r.pos + 1
"Drop already-consumed bytes so the buffer stays bounded across a long body."
function compact!(r::MultipartReader)
r.pos == 1 && return nothing
n = navail(r)
n > 0 && copyto!(r.buf, 1, r.buf, r.pos, n)
resize!(r.buf, max(n, 0))
r.pos = 1
return nothing
end
"""
Pull one more chunk off the wire, returning `false` at end of body.
`readbytes!` on an `HTTP.Stream` returns at most what remains of the current
content-length or chunk, so this is bounded by `chunk_bytes`; `eof` is what
advances a chunked-encoded body to its next chunk, hence the guard.
"""
function fill_more!(r::MultipartReader)
eof(r.io) && return false
n = readbytes!(r.io, r.scratch, r.chunk_bytes)
n == 0 && return false
append!(r.buf, view(r.scratch, 1:n))
return true
end
"Buffer until `needle` is found, returning its range, or `nothing` at end of body."
function seek_needle!(r::MultipartReader, needle::Vector{UInt8}; limit::Int = 0)
while true
idx = findnext(needle, r.buf, r.pos)
idx === nothing || return idx
# Only the last length(needle)-1 bytes can still be part of a match, but
# the caller may need the skipped bytes (a part body), so trimming is the
# caller's job; we only enforce the optional limit.
limit > 0 && navail(r) > limit &&
throw(MultipartError("no delimiter within $limit bytes"))
compact!(r)
fill_more!(r) || return nothing
end
end
"Ensure at least `n` bytes are buffered; `false` if the body ended first."
function ensure!(r::MultipartReader, n::Int)
while navail(r) < n
compact!(r)
fill_more!(r) || return false
end
return true
end
# Write `r.buf[range]` to `sink`. Goes through `unsafe_write` because
# `write(io, ::SubArray{UInt8})` falls back to a byte-at-a-time loop in Base,
# which would dominate the cost of a large upload.
function emit!(sink::IO, r::MultipartReader, from::Int, to::Int)
n = to - from + 1
n <= 0 && return 0
buf = r.buf # GC.@preserve needs a plain symbol, not a field access
GC.@preserve buf unsafe_write(sink, pointer(buf, from), UInt(n))
return n
end
# ------------------------------------------------------------------- parts
"""
next_part!(r) -> MultipartPart | nothing
Advance to the next part and return its headers, or `nothing` at the end of the
form. The previous part's body must have been consumed first (with
[`write_part_body!`](@ref) or [`skip_part_body!`](@ref)). The reader cannot skip
a body it hasn't been told to, because the body is only bounded by finding the
next delimiter.
"""
function next_part!(r::MultipartReader)
r.state === :done && return nothing
r.state === :body &&
throw(MultipartError("the current part's body must be consumed before the next part"))
if r.state === :preamble
# Discard the preamble (RFC says ignore it) and consume the opening
# delimiter. Bounded: real clients send no preamble at all, and an
# unbounded scan here would be a way to make us buffer a whole body.
idx = seek_needle!(r, r.dash_boundary; limit = MAX_PART_HEADER_BYTES)
idx === nothing && throw(MultipartError("no multipart boundary found in body"))
r.pos = last(idx) + 1
r.state = :at_delimiter
end
# Just after a delimiter: "--" ends the form, CRLF introduces another part.
ensure!(r, 2) || throw(MultipartError("truncated body after a boundary delimiter"))
if view(r.buf, r.pos:r.pos+1) == DASHDASH
r.pos += 2
r.state = :done
return nothing
end
skip_linear_whitespace!(r)
ensure!(r, 2) || throw(MultipartError("truncated body after a boundary delimiter"))
view(r.buf, r.pos:r.pos+1) == CRLF ||
throw(MultipartError("boundary delimiter is not followed by a line ending"))
r.pos += 2
part = read_part_headers!(r)
r.state = :body
return part
end
"RFC 2046 allows spaces/tabs between the delimiter and its line ending."
function skip_linear_whitespace!(r::MultipartReader)
while ensure!(r, 1) && (r.buf[r.pos] == 0x20 || r.buf[r.pos] == 0x09)
r.pos += 1
end
return nothing
end
function read_part_headers!(r::MultipartReader)
# A part with no headers at all is `CRLF CRLF body`: the empty line comes
# immediately, so searching for CRLFCRLF would run past it into the body.
if ensure!(r, 2) && view(r.buf, r.pos:r.pos+1) == CRLF
r.pos += 2
return MultipartPart(nothing, nothing, nothing)
end
# The `limit` here bounds *buffering*, and only fires when the headers span
# chunks. The explicit length check below is the actual policy, so the rule
# doesn't depend on how the body happened to be chunked on the wire.
idx = seek_needle!(r, CRLFCRLF; limit = MAX_PART_HEADER_BYTES)
idx === nothing && throw(MultipartError("truncated body inside a part's headers"))
first(idx) - r.pos > MAX_PART_HEADER_BYTES &&
throw(MultipartError("part headers exceed $MAX_PART_HEADER_BYTES bytes"))
# Copying is fine: the check above bounds this block.
block = String(r.buf[r.pos:first(idx)-1])
r.pos = last(idx) + 1
return parse_part_headers(block)
end
"Unescape the backslash escapes RFC 2045 allows inside a quoted-string."
unquote(s::AbstractString) = replace(s, r"\\(.)" => s"\1")
function parse_part_headers(block::AbstractString)
name = filename = content_type = nothing
for line in eachsplit(block, "\r\n")
colon = findfirst(':', line)
colon === nothing && continue
key = lowercase(strip(line[1:colon-1]))
value = strip(line[colon+1:end])
if key == "content-disposition"
# `\b` matters: it keeps the `name=` pattern from matching inside `filename=`.
m = match(r"(?i:\bname)=(?:\"((?:[^\"\\]|\\.)*)\"|([^\s;]+))", value)
m === nothing || (name = unquote(String(something(m[1], m[2]))))
m = match(r"(?i:\bfilename)=(?:\"((?:[^\"\\]|\\.)*)\"|([^\s;]+))", value)
m === nothing || (filename = unquote(String(something(m[1], m[2]))))
elseif key == "content-type"
content_type = String(value)
end
end
return MultipartPart(name, filename, content_type)
end
"""
write_part_body!(sink, r) -> Int
Stream the current part's body into `sink`, returning the number of bytes
written. Nothing larger than a chunk is ever held in memory.
"""
function write_part_body!(sink::IO, r::MultipartReader)
r.state === :body || throw(MultipartError("no part body is open"))
total = 0
keep = length(r.delimiter) - 1 # a delimiter may straddle two chunks
while true
idx = findnext(r.delimiter, r.buf, r.pos)
if idx !== nothing
total += emit!(sink, r, r.pos, first(idx) - 1)
r.pos = last(idx) + 1
r.state = :at_delimiter
return total
end
# Emit only what cannot be the start of a straddling delimiter, then
# keep that tail and read more.
emit_to = length(r.buf) - keep
if emit_to >= r.pos
total += emit!(sink, r, r.pos, emit_to)
r.pos = emit_to + 1
end
compact!(r)
fill_more!(r) || throw(MultipartError("truncated body inside a part"))
end
end
"Consume and discard the current part's body (a form field, or a file we can't take)."
skip_part_body!(r::MultipartReader) = write_part_body!(devnull, r)

74
src/queue.jl
View File

@@ -1,13 +1,36 @@
# The queue seam. # The queue seam.
# #
# The rest of the app only ever calls `enqueue!`, `dequeue!`, and `close!`. # The rest of the app only ever calls `enqueue!`, `dequeue!`, `ack!`, `nack!`
# Today those are backed by an in-process, bounded, thread-safe buffer # and `close!`. Two implementations sit behind those five methods:
# (the Go-channel / Julia-`Channel` model). To move to RabbitMQ (or any broker) #
# later, implement a new `JobQueue` subtype with these three methods and swap # ChannelQueue in-process, bounded, thread-safe (the Go-channel / Julia-
# the construction in `run` — no HTTP handler or worker code needs to change. # `Channel` model). Lost on crash: everything still in `spool/`
# is re-driven from stage 1 by `recover_dir!`.
# RabbitQueue durable RabbitMQ queues (src/rabbit.jl). Survives a crash: a
# job stays unacked until its handler commits, so a restart
# resumes each file at the stage it had actually reached.
#
# `FS_QUEUE_BACKEND` picks one; `run` constructs accordingly. No HTTP handler or
# worker code knows which it got.
#
# The ack pair is what makes the broker worth having. Without it a delivery is
# settled the moment it is handed to a worker, so a crash mid-handler loses the
# job exactly as the in-process queue does, and the network hop buys nothing.
# `ChannelQueue` implements both as no-ops, which is honest rather than lazy:
# an in-process queue has no delivery to settle, and its recovery story is
# `recover_dir!`.
abstract type JobQueue end abstract type JobQueue end
# How long a producer backs off before retrying an enqueue onto a full queue.
# Blocking backpressure: a file that is already on disk is never dropped, so the
# producer parks until the consumer makes room. Used by the stage-1 and stage-3
# routing handoffs and by startup recovery (see `enqueue_blocking!`). Intake is
# deliberately *not* a user: the HTTP path's `enqueue!` stays non-blocking and
# turns a full queue into a 503, so a saturated pipeline slows uploads down
# rather than holding request threads hostage.
const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05
""" """
enqueue!(q, job) -> Bool enqueue!(q, job) -> Bool
@@ -21,10 +44,35 @@ function enqueue! end
dequeue!(q) -> Union{Job,Nothing} dequeue!(q) -> Union{Job,Nothing}
Blocks until a job is available and returns it. Returns `nothing` only when the Blocks until a job is available and returns it. Returns `nothing` only when the
queue has been closed *and* fully drained — the signal for a worker to exit. queue has been closed *and* fully drained, which is a worker's signal to exit.
""" """
function dequeue! end function dequeue! end
"""
ack!(q, job)
Settle `job` as done: the broker may forget it. Called by `worker_loop` after
the handler returns, and after a quarantine, so the message outlives the process
for exactly as long as the work is unfinished.
No-op for `ChannelQueue`.
"""
function ack! end
"""
nack!(q, job)
Settle `job` as rejected, without requeueing. Requeueing is deliberately not
offered: a job that failed on its bytes will fail again, and redelivering it is
a poison-message loop. With no dead-letter exchange configured the broker simply
discards it, so this differs from `ack!` only in what it says, not in what
happens — which is the point at the one call site that uses it (see
`worker_loop`).
No-op for `ChannelQueue`.
"""
function nack! end
""" """
close!(q) close!(q)
@@ -70,6 +118,10 @@ function dequeue!(q::ChannelQueue)::Union{Job,Nothing}
end end
end end
# Nothing to settle: an in-process job was never a delivery.
ack!(::ChannelQueue, ::Job) = nothing
nack!(::ChannelQueue, ::Job) = nothing
function close!(q::ChannelQueue) function close!(q::ChannelQueue)
lock(q.cond) lock(q.cond)
try try
@@ -81,6 +133,16 @@ function close!(q::ChannelQueue)
return nothing return nothing
end end
"""
capacity(q) -> Int
How many jobs the queue can hold before `enqueue!` starts refusing. Part of the
introspection seam alongside `length`: `/stats` reports depth against capacity,
because a depth of 900 means nothing without knowing whether the limit is 1000
or 1_000_000.
"""
capacity(q::ChannelQueue) = q.capacity
"Number of jobs currently buffered (for logging/introspection)." "Number of jobs currently buffered (for logging/introspection)."
function Base.length(q::ChannelQueue) function Base.length(q::ChannelQueue)
lock(q.cond) lock(q.cond)

512
src/rabbit.jl Normal file
View File

@@ -0,0 +1,512 @@
# RabbitMQ-backed `JobQueue` (src/queue.jl defines the seam this implements).
#
# What it buys, and what it doesn't
# ---------------------------------
# The in-process `ChannelQueue` dies with the process, so a crash leaves no
# record of how far each file had got and `recover_dir!` can only re-drive
# everything in `spool/` from stage 1. Durable queues plus per-job acks replace
# that guess with a fact: a job stays unacked until its handler commits, so a
# restart resumes each file at the stage it had actually reached, and no file is
# processed from scratch just because the process died.
#
# It is *at-least-once*, not exactly-once. Stages 1 and 3 publish downstream and
# then ack upstream (`worker_loop`), so a crash in that window redelivers a job
# that was already routed, and the file gets processed twice. That is safe here
# for the same reason replay was: classification and the UTF-8 sniff are pure
# functions of the file's bytes, and every terminal commit renames with
# `force=true`. The one new wrinkle is that a duplicate can now run *concurrently*
# with the original, and the loser finds the file already moved; `worker_loop`
# recognises that (a vanished `job.path`) and settles it quietly instead of
# quarantining a file that in fact succeeded.
#
# It also does not buy horizontal scale. `Job.path` is a local filesystem path,
# so a second consumer process on another host would be handed jobs whose files
# it cannot see. See "Queue backends" in the README.
#
# Shape
# -----
# One connection for the whole process, and per stage queue:
#
# con_chan consumes, with `basic_qos` prefetch bounding unacked messages,
# and carries this stage's acks (a delivery tag is only meaningful
# on the channel that delivered it)
# pub_chan publishes *into* this queue, from whichever stage feeds it
# buffer a bounded `Channel{Job}` bridging AMQPClient's push consumer to
# the blocking `dequeue!` that `worker_loop` expects
#
# plus one shared admin channel for the depth polls. Publishing and acking from
# many worker tasks at once is safe: AMQPClient serializes multi-frame publishes
# under a per-channel send lock and queues frames onto a thread-safe channel.
using AMQPClient
# The stage queues, in `STAGE_KEYS` order. The suffix is what appears after
# `FS_AMQP_PREFIX`, so a broker's queue list reads like the pipeline.
const AMQP_QUEUE_SUFFIXES = (classify = "classify", enrich = "enrich",
triage = "triage", language = "language")
# Startup connect: retry a broker that isn't up yet (compose starts the app long
# before RabbitMQ is ready) but give up rather than hang, so a genuinely
# misconfigured URL is a startup failure and not a silent hang.
const AMQP_CONNECT_RETRY_SECONDS = 30.0
const AMQP_CONNECT_RETRY_INTERVAL = 2.0
# How often the depth poller asks the broker for each queue's ready-message
# count. This is what `/stats` reports and what the advisory capacity check
# compares against, so it trades staleness for one round trip per queue per
# second instead of one per file.
const AMQP_DEPTH_POLL_SECONDS = 1.0
# A publisher confirm that never arrives must not park an HTTP request thread
# forever. The poller doubles as the watchdog (see `poll_loop!`).
const AMQP_CONFIRM_TIMEOUT_SECONDS = 10.0
"Connection parameters parsed out of an `amqp://` URL."
struct AMQPTarget
host::String
port::Int
virtualhost::String
login::String
password::String
end
"""
parse_amqp_url(url) -> AMQPTarget
Parse `amqp://user:pass@host:port/vhost`. Every component is optional and falls
back to the AMQP defaults, except that we do *not* default the credentials to
`guest`/`guest`: those only authenticate over loopback in a stock RabbitMQ, so
inheriting them across a container network would fail at connect time with an
authentication error that says nothing about the real mistake.
The vhost is the path with its leading `/` removed and percent-escapes decoded,
so the conventional default vhost `/` can be written either as a bare trailing
slash or as the `%2F` form.
"""
function parse_amqp_url(url::AbstractString)
uri = HTTP.URI(url)
scheme = lowercase(uri.scheme)
scheme in ("amqp", "amqps") ||
throw(ArgumentError("FS_AMQP_URL must be an amqp:// or amqps:// URL, got \"$url\""))
scheme == "amqps" &&
throw(ArgumentError("amqps:// (TLS) is not supported yet; use amqp:// on a trusted network"))
userinfo = split(uri.userinfo, ':'; limit = 2)
login = isempty(userinfo[1]) ? "guest" : HTTP.unescapeuri(userinfo[1])
password = length(userinfo) < 2 ? "guest" : HTTP.unescapeuri(userinfo[2])
path = lstrip(uri.path, '/')
vhost = isempty(path) ? "/" : HTTP.unescapeuri(path)
return AMQPTarget(isempty(uri.host) ? "localhost" : uri.host,
isempty(uri.port) ? AMQPClient.AMQP_DEFAULT_PORT : parse(Int, uri.port),
vhost, login, password)
end
"The process's single broker connection, plus the shared channel depth polls use."
mutable struct RabbitBackend
conn::AMQPClient.Connection
admin_chan::AMQPClient.MessageChannel
const admin_lock::ReentrantLock # one synchronous RPC on admin_chan at a time
const prefix::String
end
"""
RabbitQueue(backend, name, capacity, prefetch; confirms = false)
One durable queue. `capacity` is *advisory*: `enqueue!` compares it against a
depth that is polled once a second and adjusted locally in between, so a burst
can overshoot by up to a poll interval before the queue starts refusing. The
alternative — `x-max-length` with `overflow: reject-publish` — needs a publisher
confirm per message to learn about the rejection, which would put a round trip
on an inter-stage handoff that currently costs 0.12 µs.
`confirms` puts this queue's *publish* channel into confirm mode, so `enqueue!`
returns only once the broker has accepted the message. It is set on the stage-1
queue alone, because intake is the only publisher that answers to someone: a
`202` tells a client its file is safe, and that client may delete its copy.
Inter-stage publishes stay fire-and-forget, where a lost message leaves its
source job unacked and redelivery repairs it for free.
"""
mutable struct RabbitQueue <: JobQueue
const backend::RabbitBackend
const name::String
const capacity::Int
const buffer::Channel{Job} # consumer callback → dequeue!
const pub_chan::AMQPClient.MessageChannel
const con_chan::AMQPClient.MessageChannel
const depth::Threads.Atomic{Int} # last polled ready count, locally adjusted
const closed::Threads.Atomic{Bool}
const confirms::Bool
const pub_lock::ReentrantLock # serializes publish+confirm on pub_chan
const confirm_ch::Channel{Bool} # broker's verdict on the outstanding publish
const pending_since::Threads.Atomic{Float64} # when it was published (0.0 = nothing pending)
const on_lost::Function # called once if the connection dies
consumer_tag::String
poller::Union{Task,Nothing}
end
capacity(q::RabbitQueue) = q.capacity
"""
Depth as of the last poll, adjusted by our own publishes and takes since.
The broker's number counts *ready* messages, so it excludes the ones already
handed to workers and not yet acked. That is the right number for `/stats`:
depth is meant to say how much work is piled up in front of a stage, and the
messages in its workers' hands are not piled up, they are being worked.
"""
Base.length(q::RabbitQueue) = max(0, q.depth[])
# --- wire format ------------------------------------------------------------
"""
job_json(job) -> String
Serialize a `Job` for the wire. `delivery_tag` is deliberately absent: it names a
delivery on one channel of one connection, not the job, and a redelivery gets a
fresh one.
"""
job_json(job::Job) = JSON3.write((; job.id, job.original_name, job.path,
job.size, job.received_at))
"Rebuild a `Job` from a message body, carrying the tag it was delivered under."
function job_from_message(msg::AMQPClient.Message)
o = JSON3.read(String(copy(msg.data)))
return Job(String(o.id), String(o.original_name), String(o.path),
Int(o.size), Float64(o.received_at), Int64(msg.delivery_tag))
end
# --- connect / declare ------------------------------------------------------
"""
connect_backend(cfg) -> RabbitBackend
Open the process's connection, retrying a broker that is not up yet for up to
`AMQP_CONNECT_RETRY_SECONDS`. Compose starts this container long before RabbitMQ
finishes booting even with a healthcheck gate, so a first refusal is routine; a
persistent one is a startup error, because falling back to the in-process queue
would silently deliver none of the durability that was asked for.
"""
function connect_backend(cfg::Config)
t = parse_amqp_url(cfg.amqp_url)
auth = Dict{String,Any}("MECHANISM" => "AMQPLAIN",
"LOGIN" => t.login, "PASSWORD" => t.password)
deadline = time() + AMQP_CONNECT_RETRY_SECONDS
attempt = 0
while true
attempt += 1
try
conn = AMQPClient.connection(; host = t.host, port = t.port,
virtualhost = t.virtualhost,
auth_params = auth)
admin = AMQPClient.channel(conn, AMQPClient.UNUSED_CHANNEL, true)
@info "connected to broker" host=t.host port=t.port vhost=t.virtualhost attempts=attempt
return RabbitBackend(conn, admin, ReentrantLock(), cfg.amqp_prefix)
catch e
time() < deadline || rethrow()
@info "broker not ready, retrying" host=t.host port=t.port attempt=attempt
sleep(AMQP_CONNECT_RETRY_INTERVAL)
end
end
end
"""
open_queue(backend, cfg, key, capacity, workers) -> RabbitQueue
Declare one stage's durable queue and start consuming from it.
Prefetch defaults to the stage's worker count: enough that every worker can hold
a job, and no more, so a crash redelivers the smallest possible set. The bridge
`Channel` is sized to match, since it is exactly the set of messages the broker
has handed us and we have not yet acked.
"""
function open_queue(backend::RabbitBackend, cfg::Config, key::Symbol,
capacity::Int, workers::Int; on_lost::Function = () -> nothing)
name = string(backend.prefix, ".", AMQP_QUEUE_SUFFIXES[key])
prefetch = cfg.amqp_prefetch > 0 ? cfg.amqp_prefetch : max(1, workers)
con_chan = AMQPClient.channel(backend.conn, AMQPClient.UNUSED_CHANNEL, true)
pub_chan = AMQPClient.channel(backend.conn, AMQPClient.UNUSED_CHANNEL, true)
# durable: the queue definition survives a broker restart. Combined with
# PERSISTENT messages below, so do the jobs in it.
AMQPClient.queue_declare(con_chan, name; durable = true)
AMQPClient.basic_qos(con_chan, 0, prefetch, false)
confirms = cfg.amqp_confirms && key === :classify
q = RabbitQueue(backend, name, capacity, Channel{Job}(prefetch),
pub_chan, con_chan, Threads.Atomic{Int}(0),
Threads.Atomic{Bool}(false), confirms, ReentrantLock(),
Channel{Bool}(1), Threads.Atomic{Float64}(0.0), on_lost, "", nothing)
if confirms
confirm_select!(pub_chan)
# The broker reports each publish's fate asynchronously by sequence
# number. We hold `pub_lock` across publish-and-wait, so at most one
# publish is ever outstanding and any verdict that arrives is this one's.
AMQPClient.handle(pub_chan, :Basic, :Ack, (_c, _m, _x) -> settle_confirm!(q, true))
AMQPClient.handle(pub_chan, :Basic, :Nack, (_c, _m, _x) -> settle_confirm!(q, false))
end
ok, tag = AMQPClient.basic_consume(con_chan, name, msg -> deliver!(q, msg))
ok || error("could not start consuming from queue $name")
q.consumer_tag = tag
q.poller = Threads.@spawn poll_loop!(q)
@info "queue ready" queue=name prefetch=prefetch capacity=capacity confirms=confirms
return q
end
"""
open_queues(backend, cfg) -> NamedTuple
The four stage queues, keyed by `STAGE_KEYS` so they line up with the per-stage
worker counts, capacities and metrics without a second mapping to keep in sync.
"""
function open_queues(backend::RabbitBackend, cfg::Config; on_lost::Function = () -> nothing)
caps = (classify = cfg.queue_capacity, enrich = cfg.known_queue_capacity,
triage = cfg.unknown_queue_capacity, language = cfg.text_queue_capacity)
workers = (classify = cfg.worker_count, enrich = cfg.known_worker_count,
triage = cfg.unknown_worker_count, language = cfg.text_worker_count)
return NamedTuple{STAGE_KEYS}(map(STAGE_KEYS) do key
open_queue(backend, cfg, key, caps[key], workers[key]; on_lost)
end)
end
"Close the shared connection. Safe to call on a connection that is already gone."
function close_backend!(backend::RabbitBackend)
try
close(backend.conn)
catch e
@debug "error closing broker connection" exception=e
end
return nothing
end
"""
confirm_select!(chan)
Put `chan` into publisher-confirm mode.
Not `AMQPClient.confirm_select`, which is broken in AMQPClient 0.5.1: it builds
the `Confirm.Select` payload with no fields, while the spec (and the library's
own `CLASS_MAP`) declares a `Nowait` bit, so it throws a `BoundsError` before
anything reaches the wire. This is the same call with that one field supplied.
Drop it for the library's version once the fix is released.
"""
function confirm_select!(chan::AMQPClient.MessageChannel)
AMQPClient._wait_resp(chan, true, false, AMQPClient.on_confirm_select_ok,
:Confirm, :SelectOk, false, AMQPClient.DEFAULT_TIMEOUT) do
AMQPClient.send(chan, AMQPClient.TAMQPMethodPayload(:Confirm, :Select, (false,)))
end
end
# --- consume ----------------------------------------------------------------
"""
Consumer callback: hand a delivery to the bridge channel.
Blocking here is the mechanism, not a problem. The bridge holds `prefetch` jobs,
so when workers fall behind this parks, the broker's prefetch window fills, and
it stops sending — backpressure all the way to the queue, with no unbounded
local buffer.
"""
function deliver!(q::RabbitQueue, msg::AMQPClient.Message)
try
put!(q.buffer, job_from_message(msg))
Threads.atomic_sub!(q.depth, 1)
catch e
# Either we are shutting down (buffer closed after `basic_cancel`, and a
# delivery raced it) or the body was unparseable. Neither may kill the
# consumer task. The message stays unacked, so the broker redelivers it.
q.closed[] || @error "could not accept delivery" queue=q.name exception=(e, catch_backtrace())
end
return nothing
end
function dequeue!(q::RabbitQueue)::Union{Job,Nothing}
try
return take!(q.buffer)
catch e
# Closed and drained: the shutdown signal `worker_loop` waits for.
e isa InvalidStateException && return nothing
rethrow()
end
end
"""
Acks and rejects go out on the channel that made the delivery, because a
delivery tag is scoped to its channel.
A failure here is logged, not thrown: an unacked job is redelivered, which is
the correct outcome, and killing the worker over it would shrink the pool.
"""
function ack!(q::RabbitQueue, job::Job)
job.delivery_tag == 0 && return nothing
try
AMQPClient.basic_ack(q.con_chan, job.delivery_tag)
catch e
@error "ack failed; job will be redelivered" queue=q.name id=job.id exception=e
end
return nothing
end
function nack!(q::RabbitQueue, job::Job)
job.delivery_tag == 0 && return nothing
try
AMQPClient.basic_reject(q.con_chan, job.delivery_tag; requeue = false)
catch e
@error "reject failed; job will be redelivered" queue=q.name id=job.id exception=e
end
return nothing
end
# --- publish ----------------------------------------------------------------
"""
enqueue!(q, job) -> Bool
Publish `job` as a persistent message. Returns `false` — which intake turns into
a `503` and the routing stages turn into park-and-retry — when the queue is
closed, when the polled depth says it is at capacity, or when a confirm-mode
publish is not confirmed.
`delivery_mode = PERSISTENT` against a durable queue is what makes a broker
restart survivable; without it the queue would come back empty and the whole
point of the backend would be gone.
"""
function enqueue!(q::RabbitQueue, job::Job)::Bool
q.closed[] && return false
q.depth[] >= q.capacity && return false # advisory: see the `RabbitQueue` docstring
msg = AMQPClient.Message(Vector{UInt8}(job_json(job));
content_type = "application/json",
delivery_mode = AMQPClient.PERSISTENT)
ok = q.confirms ? publish_confirmed!(q, msg) : publish!(q, msg)
ok && Threads.atomic_add!(q.depth, 1)
return ok
end
function publish!(q::RabbitQueue, msg::AMQPClient.Message)
try
AMQPClient.basic_publish(q.pub_chan, msg; exchange = "", routing_key = q.name)
return true
catch e
@error "publish failed" queue=q.name exception=(e, catch_backtrace())
return false
end
end
"""
Publish and wait for the broker's confirm.
Serialized on `pub_lock`, so exactly one publish is outstanding per queue and the
next verdict to arrive belongs to it. That costs intake a round trip per file and
caps its rate at roughly one file per broker round trip; it is why confirms are
scoped to intake and configurable (`FS_AMQP_CONFIRMS`).
"""
function publish_confirmed!(q::RabbitQueue, msg::AMQPClient.Message)
lock(q.pub_lock) do
while isready(q.confirm_ch) # discard a verdict left by a timed-out publish
take!(q.confirm_ch)
end
q.pending_since[] = time()
try
publish!(q, msg) || return false
return take!(q.confirm_ch) # poll_loop! guarantees this is answered
finally
q.pending_since[] = 0.0
end
end
end
"Record the broker's verdict for the one outstanding publish."
function settle_confirm!(q::RabbitQueue, ok::Bool)
q.pending_since[] == 0.0 && return nothing # nothing waiting; a late/duplicate frame
isready(q.confirm_ch) || put!(q.confirm_ch, ok)
return nothing
end
# --- poll -------------------------------------------------------------------
"""
poll_loop!(q)
One task per queue, doing the two things that must happen on a timer:
* refresh `depth` from the broker (a passive `queue.declare` returns the ready
count), which feeds `/stats` and the advisory capacity check, and
* answer a publisher confirm that has gone missing, so a broker that accepted
a publish and then went quiet can't park an HTTP request thread forever, and
* notice that the connection has died and call `on_lost`, which asks `run` to
shut down. This backend does not reconnect (every in-flight delivery tag
would be invalid, and the jobs holding them would be processed twice), so a
lost connection means the process is finished: it drains, exits, and the
restart policy brings it back to redelivered messages and an intact
`spool/`. Without this the pipeline would sit there looking healthy with
four queues that can never receive anything again.
Folding the watchdog in here keeps it free: it is a comparison on a task that was
going to wake up anyway.
"""
function poll_loop!(q::RabbitQueue)
while !q.closed[]
sleep(AMQP_DEPTH_POLL_SECONDS)
q.closed[] && break
pending = q.pending_since[]
if pending != 0.0 && time() - pending > AMQP_CONFIRM_TIMEOUT_SECONDS
@warn "publisher confirm timed out; reporting the file as not accepted" queue=q.name
settle_confirm!(q, false)
end
try
lock(q.backend.admin_lock) do
ok, _, count, _ = AMQPClient.queue_declare(q.backend.admin_chan, q.name;
passive = true)
ok && (q.depth[] = Int(count))
end
catch e
q.closed[] && break
# A dead connection is not something this backend tries to ride out:
# every in-flight delivery tag becomes invalid on reconnect, so the
# honest recovery is a restart, where unacked messages redeliver and
# `spool/` is intact. Report it and let the supervisor act.
if !isopen(q.backend.conn)
# Two causes, indistinguishable from here: the broker or the
# network went away, or a SIGINT was delivered to one of
# AMQPClient's tasks instead of the main loop (see `run`).
# The response is the same either way — drain and exit.
@error "broker connection closed (lost broker, or an interrupt delivered to a broker task); shutting down (unacked jobs will be redelivered)" queue=q.name
q.on_lost()
break
end
@warn "queue depth poll failed" queue=q.name exception=e
end
end
return nothing
end
# --- shutdown ---------------------------------------------------------------
"""
close!(q)
Stop consuming, then close the bridge so workers drain what is already in hand
and exit — the same drain-then-exit contract `ChannelQueue` offers.
Jobs the broker has delivered but that we never got to stay unacked and are
redelivered on the next start. That is the backend working as intended, and the
reason shutdown does not need to be careful here.
"""
function close!(q::RabbitQueue)
q.closed[] || try
AMQPClient.basic_cancel(q.con_chan, q.consumer_tag)
catch e
@debug "error cancelling consumer" queue=q.name exception=e
end
q.closed[] = true
isopen(q.buffer) && close(q.buffer)
return nothing
end

257
src/server.jl
View File

@@ -1,13 +1,21 @@
# HTTP layer: a single multipart upload endpoint plus a health check. # HTTP layer: a single multipart upload endpoint plus a health check.
# #
# The handler's whole job is to get files onto the queue fast and get out of the # The handler's whole job is to get files onto the queue fast and get out of the
# way: spool each uploaded file to disk, enqueue a reference, respond 202. It # way: stream each uploaded file to disk, enqueue a reference, respond 202. It
# never does real processing — that's the workers' job. # never does real processing; that's the workers' job.
#
# Intake is *streamed*, not buffered (see src/multipart.jl for the reader). This
# is what makes the service's memory story hold end to end: bytes go from the
# socket to the spool file a chunk at a time, so a 4 GB upload costs the same
# resident memory as a 4 KB one. It is also why /upload is served by its own
# stream handler rather than an Oxygen route; see `root_stream_handler`.
# #
# NOTE: routes are registered at runtime via `register_routes()` (called from # NOTE: routes are registered at runtime via `register_routes()` (called from
# `run`), NOT with top-level macros. In a precompiled package, top-level # `run`), NOT with top-level macros. In a precompiled package, top-level
# `@get`/`@post` would execute during precompilation and be lost before serving. # `@get`/`@post` would execute during precompilation and be lost before serving.
const UPLOAD_PATH = "/upload"
jsonresp(status::Int, data) = jsonresp(status::Int, data) =
HTTP.Response(status, ["Content-Type" => "application/json"], JSON3.write(data)) HTTP.Response(status, ["Content-Type" => "application/json"], JSON3.write(data))
@@ -15,48 +23,215 @@ function health_handler(_::HTTP.Request)
return jsonresp(200, (; status = "ok")) return jsonresp(200, (; status = "ok"))
end end
function upload_handler(req::HTTP.Request) """
cfg = CONFIG[] `GET /stats`: the pipeline's own counters (src/stats.jl), as JSON.
queue = QUEUE[]
parts = try Read-only and cheap: a few atomic loads and one `length` per queue, no pipeline
HTTP.parse_multipart_form(req) state touched. Two scrapes Δt apart give per-stage throughput and utilization;
catch see bin/bench.jl, which is the intended consumer.
nothing
end
parts === nothing &&
return jsonresp(400, (; error = "expected multipart/form-data"))
files = filter(p -> p.filename !== nothing && !isempty(p.filename), parts) Unlike `/upload` this is an ordinary Oxygen route: it has no body to stream, and
isempty(files) && being in Oxygen's middleware chain is a feature here.
return jsonresp(400, (; error = "no files found in request")) """
function stats_handler(_::HTTP.Request)
accepted = NamedTuple{(:id, :name),Tuple{String,String}}[] queues = (classify = QUEUE[], enrich = KNOWN_QUEUE[],
for p in files triage = UNKNOWN_QUEUE[], language = TEXT_QUEUE[])
bytes = read(p.data) return jsonresp(200, stats_snapshot(CONFIG[], queues))
job = try
spool_file(cfg, p.filename, bytes)
catch e
@error "spool failed" name=p.filename exception=(e, catch_backtrace())
return jsonresp(500, (; error = "failed to store file", accepted))
end
if !enqueue!(queue, job)
rm(job.path; force = true) # never queued → don't leave it in spool
return jsonresp(503, (; error = "queue full, retry later", accepted))
end
@info "accepted" id=job.id name=job.original_name size=job.size
push!(accepted, (; id = job.id, name = job.original_name))
end
return jsonresp(202, (; accepted))
end end
"Register HTTP routes on the Oxygen instance. Must run at runtime, before serve." "Write a JSON response onto a raw stream (the streaming handler's `jsonresp`)."
function register_routes() function stream_jsonresp(stream::HTTP.Stream, status::Int, data)
@get("/health", health_handler) body = JSON3.write(data)
@post("/upload", upload_handler) HTTP.setstatus(stream, status)
HTTP.setheader(stream, "Content-Type" => "application/json")
HTTP.setheader(stream, "Content-Length" => string(sizeof(body)))
HTTP.startwrite(stream)
write(stream, body)
return nothing
end
"""
Read and discard whatever is left of the request body.
HTTP.jl's server calls `closeread` after the handler and *errors* if the body was
only partly consumed, because a half-read body can't be followed by another request on a
keep-alive connection. So every exit path drains first. Discarding is bounded in
memory (one chunk) and costs nothing in the normal case, where the body is
already fully consumed and this returns immediately.
"""
function discard_body!(stream::HTTP.Stream, chunk_bytes::Int)
scratch = Vector{UInt8}(undef, chunk_bytes)
while !eof(stream)
readbytes!(stream, scratch, chunk_bytes) == 0 && break
end
return nothing
end
"""
Did this exception mean the client went away, rather than something being wrong
on our side?
A client that hangs up mid-upload (user cancels, network drops) is routine, and
must not be logged as a server error or reported as a failed write. But it looks
like an I/O failure from inside the parser, so the distinction has to be made
explicitly. `EOFError` is what `HTTP.Stream` raises when a connection dies with
bytes still promised by `Content-Length`.
"""
is_client_gone(e) =
e isa EOFError ||
(e isa Base.IOError && e.code in (Base.UV_EPIPE, Base.UV_ECONNRESET, Base.UV_ECONNABORTED))
"""
Stream a multipart upload to disk, one part at a time.
Each file part is spooled straight from the socket and its reference enqueued.
The response reports what was actually queued:
* `202`: every file in the request was spooled and queued
* `400`: not multipart/form-data, no files present, or a malformed body
* `503`: the intake queue filled up; `accepted` lists what got in first
* `500`: a file could not be written to disk
Unlike a buffered handler, this one cannot know up front how many files a request
holds or whether they will fit. So when the queue fills mid-request it does not
abandon the connection: it stops spooling (discarding the remaining parts rather
than writing files it can't queue), drains the body, and answers `503` with the
`accepted` list. Files already queued stay queued, so a client can retry the rest.
"""
function upload_stream_handler(stream::HTTP.Stream)
try
return serve_upload!(stream)
catch e
# The client vanished while we were reading its body, draining it, or
# answering. Nothing is wrong on our side, so log it as the routine event
# it is instead of a server error.
is_client_gone(e) || rethrow()
@info "upload aborted by client"
# HTTP.jl insists a handler write *some* response before returning. The
# client is probably already gone, so this write is best-effort: attempt
# it only if we haven't started a response, and let it fail silently.
if isopen(stream) && !iswritable(stream)
try
stream_jsonresp(stream, 400, (; error = "upload truncated"))
catch e2
is_client_gone(e2) || rethrow()
end
end
return nothing
end
end
function serve_upload!(stream::HTTP.Stream)
Threads.atomic_add!(METRICS.intake.requests, 1)
cfg = CONFIG[]
queue = QUEUE[]
chunk = cfg.upload_chunk_bytes
boundary = multipart_boundary(HTTP.header(stream.message, "Content-Type", nothing))
if boundary === nothing
discard_body!(stream, chunk)
return stream_jsonresp(stream, 400, (; error = "expected multipart/form-data"))
end
reader = MultipartReader(stream, boundary; chunk_bytes = chunk)
accepted = NamedTuple{(:id, :name),Tuple{String,String}}[]
n_files = 0
queue_full = false
failure = nothing # (status, message) from a fatal error mid-body
try
while (part = next_part!(reader)) !== nothing
# A part with no filename is an ordinary form field, not a file.
if part.filename === nothing || isempty(part.filename)
skip_part_body!(reader)
continue
end
n_files += 1
# Already backpressured: consume the part, but don't write a file we
# know we cannot enqueue.
if queue_full
Threads.atomic_add!(METRICS.intake.rejected, 1)
skip_part_body!(reader)
continue
end
job = try
spool_stream(cfg, part.filename) do io
write_part_body!(io, reader)
end
catch e
(e isa MultipartError || is_client_gone(e)) && rethrow()
# A disk error leaves the reader mid-part, so the rest of the body
# can no longer be parsed, so stop and report.
@error "spool failed" name=part.filename exception=(e, catch_backtrace())
failure = (500, "failed to store file")
break
end
if enqueue!(queue, job)
@info "accepted" id=job.id name=job.original_name size=job.size
push!(accepted, (; id = job.id, name = job.original_name))
# Counted at the point the job becomes stage 1's problem, so
# intake totals and stage-1 arrivals refer to the same files.
Threads.atomic_add!(METRICS.intake.files, 1)
Threads.atomic_add!(METRICS.intake.bytes, job.size)
else
rm(job.path; force = true) # never queued → don't leave it in spool
Threads.atomic_add!(METRICS.intake.rejected, 1)
queue_full = true
end
end
catch e
e isa MultipartError || rethrow()
@warn "malformed multipart upload" reason=e.msg accepted=length(accepted)
failure = (400, "malformed multipart body")
end
discard_body!(stream, chunk)
failure !== nothing &&
return stream_jsonresp(stream, failure[1], (; error = failure[2], accepted))
n_files == 0 &&
return stream_jsonresp(stream, 400, (; error = "no files found in request"))
queue_full &&
return stream_jsonresp(stream, 503, (; error = "queue full, retry later", accepted))
return stream_jsonresp(stream, 202, (; accepted))
end
"""
root_stream_handler(middleware) -> (stream -> nothing)
Oxygen's root handler wraps `HTTP.streamhandler`, which does
`request.body = read(stream)`, the entire body into memory, *before* any route
is dispatched. That happens even for an Oxygen `@stream` route, so no route can
stream an upload. We therefore intercept `POST /upload` at the stream level and
hand everything else to Oxygen unchanged.
Trade-off: `/upload` bypasses Oxygen's middleware chain, so it is absent from
Oxygen's built-in metrics and docs. Deliberate: flat intake memory is the point
of this service, and intake is covered by our own counters (`/stats`) and `@info`
records anyway.
"""
function root_stream_handler(middleware::Function)
oxygen_handler = Oxygen.Core.stream_handler(middleware)
return function (stream::HTTP.Stream)
req = stream.message
if req.method == "POST" && HTTP.URI(req.target).path == UPLOAD_PATH
return upload_stream_handler(stream)
end
return oxygen_handler(stream)
end
end
"""
Register HTTP routes on the Oxygen instance. Must run at runtime, before serve.
`POST /upload` is deliberately absent: it is served by `upload_stream_handler`
via `root_stream_handler`, ahead of Oxygen's router.
"""
function register_routes()
@get("/health", health_handler)
@get("/stats", stats_handler)
return nothing return nothing
end end

73
src/spool.jl
View File

@@ -19,18 +19,44 @@ function sanitize_filename(name::AbstractString)::String
return first(base, MAX_NAME_LEN) return first(base, MAX_NAME_LEN)
end end
"Write `bytes` to the spool dir under `<uuid>-<sanitized>` and return the Job." "Build the spool path for a client-supplied name: `<uuid>-<sanitized>`."
function spool_file(cfg::Config, original_name::AbstractString, bytes::Vector{UInt8})::Job function spool_path(cfg::Config, original_name::AbstractString)
id = string(uuid4()) id = string(uuid4())
safe = sanitize_filename(original_name) return id, joinpath(cfg.spool_dir, string(id, "-", sanitize_filename(original_name)))
path = joinpath(cfg.spool_dir, string(id, "-", safe))
open(path, "w") do io
write(io, bytes)
end
return Job(id, String(original_name), path, length(bytes), time())
end end
"Move a spooled file into `dir` (done/ or failed/), returning the destination." """
spool_stream(write_body!, cfg, original_name) -> Job
Create the spool file for `original_name`, hand the open `IO` to `write_body!`,
and build the `Job` from however many bytes it reports writing.
This is the streaming counterpart to `spool_file`: the caller pumps bytes in from
the network as they arrive, so a file never exists in memory in one piece. A
partial file left by a failed or abandoned write is removed: intake either
produces a complete spooled file or nothing at all, so recovery on restart never
picks up a truncated upload.
"""
function spool_stream(write_body!, cfg::Config, original_name::AbstractString)::Job
id, path = spool_path(cfg, original_name)
nbytes = try
open(write_body!, path, "w")
catch
rm(path; force = true)
rethrow()
end
return Job(id, String(original_name), path, nbytes, time())
end
"""
move_to(dir, job) -> String
Move a spooled file into `dir`, returning the destination.
This runs once per file, at the end: into a terminal sink (`done/`, `text_done/`,
`binary/`) or into `failed/` when a handler throws. Files are *not* moved between
stages; see the header of src/worker.jl.
"""
function move_to(dir::AbstractString, job::Job)::String function move_to(dir::AbstractString, job::Job)::String
dest = joinpath(dir, basename(job.path)) dest = joinpath(dir, basename(job.path))
mv(job.path, dest; force=true) mv(job.path, dest; force=true)
@@ -46,12 +72,24 @@ const UUID_LEN = 36
Re-enqueue any files sitting in `dir` (left by a crash, hard shutdown, or an Re-enqueue any files sitting in `dir` (left by a crash, hard shutdown, or an
intake that never finished) onto `queue`. This is the payoff of spooling to intake that never finished) onto `queue`. This is the payoff of spooling to
disk: a restart resumes work instead of stranding it. Stage-aware recovery uses disk: a restart resumes work instead of stranding it. Returns the number of
one call per stage — `spool/` → stage-1 queue, `known/` → known queue — so each files recovered.
file re-enters at the correct stage rather than being reclassified from scratch.
Returns the number of files recovered.
Skips `.meta.json` sidecars: those are stage-2 output, not work to redo. A file's stage is not recorded on disk. It lives in whichever in-process queue
holds its reference, and a crash loses that (src/worker.jl header). So everything
still in `spool/` re-enters at stage 1 and replays. Replay is safe rather than
merely tolerable: classification and the binary/text sniff are pure functions of
the file's bytes, and the terminal commits rename with `force=true`, so a
replayed file lands where it would have landed and overwrites its own sidecar.
**This blocks on a full queue**, and that is the whole point: `spool/` now holds
every in-flight file, so a crash under load can easily leave more leftovers than
one queue's capacity. Dropping the excess would mean recovery quietly losing the
work it exists to save. Because the caller spawns the worker pools *before*
recovering, the consumers are live and the queue drains as we fill it, so parking
here costs latency, never progress.
Skips `.meta.json` sidecars: those are enrichment output, not work to redo.
""" """
function recover_dir!(dir::AbstractString, queue::JobQueue)::Int function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
n = 0 n = 0
@@ -67,7 +105,10 @@ function recover_dir!(dir::AbstractString, queue::JobQueue)::Int
name = fname name = fname
end end
job = Job(id, name, path, filesize(path), time()) job = Job(id, name, path, filesize(path), time())
enqueue!(queue, job) || @warn "queue full during recovery; leaving file for next start" path # `nothing` stats: recovery is not a stage, so its wait must not be
# charged to any stage's blocked_ns (see `enqueue_blocking!`).
enqueue_blocking!(queue, job, nothing;
retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
n += 1 n += 1
end end
return n return n

220
src/stats.jl Normal file
View File

@@ -0,0 +1,220 @@
# Per-stage pipeline metrics.
#
# End-to-end throughput says how fast the pipeline is; it does not say which
# stage is the reason. The external harness can't answer that either, and now
# cannot even approximate it: every in-flight file sits in `spool/` regardless of
# how far it has got, because stages route by enqueueing a reference rather than
# by moving bytes (src/worker.jl header). A file's stage is a property of the
# queue holding it, so only the pipeline itself can see it. There is no
# directory to poll.
#
# So each stage keeps four counters, all incremented in `worker_loop` (the one
# place every stage's work passes through) and read by `GET /stats`:
#
# completed / failed jobs finished, jobs quarantined → throughput
# bytes job bytes processed → per-stage MiB/s
# busy_ns summed handler wall time → service time
# blocked_ns of that, time parked on a full downstream queue
# in_flight handlers running right now → saturation
#
# Rates fall out of a pair of scrapes taken Δt apart:
#
# throughput = Δcompleted / Δt
# utilization = (Δbusy_ns - Δblocked_ns) / (Δt * workers) ~1.0 ⇒ bottleneck
#
# Utilization is the useful one. Throughput alone can't distinguish a stage that
# is saturated from one that is merely starved by the stage ahead of it: both
# show the same files/s. Utilization separates them: the saturated stage
# sits near 1.0 with its queue backing up, the starved stage sits near 0.
#
# `blocked_ns` is what keeps that true downstream. Stages 1 and 3 apply blocking
# backpressure: when the next queue is full the handler parks and retries rather
# than dropping the file (see ROUTE_ENQUEUE_RETRY_SECONDS). That parked time is
# inside the handler, so counting it as busy would show stage 1 pinned at 1.0
# whenever stage 2 is the real bottleneck, and every stage upstream of the jam would
# look like the jam. Subtracting it leaves utilization meaning "doing its own
# work", and the blocked share becomes its own signal: a stage blocked 90% of the
# time is naming its successor as the bottleneck.
#
# Everything here is monotonic since process start (or since `reset_metrics!`),
# in the Prometheus style: counters, never rates. Rates are the reader's job, so
# a scrape is stateless and two readers can't disturb each other.
#
# Cost is a handful of atomic adds per file, against handlers that spawn
# exiftool: unmeasurable in practice, and the counters are never read on the
# hot path.
# Stage identity lives here, in declaration order, so the report, the JSON and
# the worker wiring can't drift apart: adding stage 5 to the live path means
# adding it here and passing the new `StageStats` to its `worker_loop`.
const STAGE_KEYS = (:classify, :enrich, :triage, :language)
const STAGE_TITLES = (classify = "classify", enrich = "enrich",
triage = "triage", language = "language")
"""
Counters for one pipeline stage. All fields are atomic and monotonic: workers
only ever add, readers only ever read, so no lock is needed between them.
`in_flight` is the exception to monotonic. It goes up and down, and is the one
counter that is a *level* rather than a total.
"""
struct StageStats
completed::Threads.Atomic{Int}
failed::Threads.Atomic{Int}
bytes::Threads.Atomic{Int}
busy_ns::Threads.Atomic{Int}
blocked_ns::Threads.Atomic{Int}
in_flight::Threads.Atomic{Int}
end
StageStats() = StageStats(Threads.Atomic{Int}(0), Threads.Atomic{Int}(0),
Threads.Atomic{Int}(0), Threads.Atomic{Int}(0),
Threads.Atomic{Int}(0), Threads.Atomic{Int}(0))
"""
Counters for HTTP intake, which has no worker loop to hang them off.
`files` counts what was actually spooled *and* queued, the same thing the 202
body reports as `accepted`, so it lines up with stage 1's arrivals. Files
dropped because the queue was full are counted separately in `rejected`, since a
run where intake outruns the pipeline should be visible as backpressure rather
than as slow intake.
"""
struct IntakeStats
requests::Threads.Atomic{Int}
files::Threads.Atomic{Int}
bytes::Threads.Atomic{Int}
rejected::Threads.Atomic{Int}
end
IntakeStats() = IntakeStats(Threads.Atomic{Int}(0), Threads.Atomic{Int}(0),
Threads.Atomic{Int}(0), Threads.Atomic{Int}(0))
"""
Every counter in the process, plus the wall-clock origin the totals are measured
from.
`since` matters to a reader computing rates from a single scrape: without it,
"1000 files completed" has no denominator. Readers that scrape twice should use
their own Δt instead, which excludes the time before they started watching.
"""
struct Metrics
intake::IntakeStats
stages::NamedTuple{STAGE_KEYS,NTuple{4,StageStats}}
since::Base.RefValue{Float64}
end
Metrics() = Metrics(IntakeStats(),
NamedTuple{STAGE_KEYS}(ntuple(_ -> StageStats(), 4)),
Ref(time()))
# Process-global, like the logger: workers on every thread add to it and the
# HTTP layer reads it, with no way to thread a handle through both paths that
# wouldn't just be this by another name.
const METRICS = Metrics()
"""
reset_metrics!()
Zero every counter and restart the measurement window. For tests, and for a
benchmark that wants totals covering only its own run, though a harness that
scrapes before and after can subtract instead, which is safer against a server
that is also serving someone else.
"""
function reset_metrics!(m::Metrics = METRICS)
for a in (m.intake.requests, m.intake.files, m.intake.bytes, m.intake.rejected)
a[] = 0
end
for s in m.stages
s.completed[] = 0
s.failed[] = 0
s.bytes[] = 0
s.busy_ns[] = 0
s.blocked_ns[] = 0
s.in_flight[] = 0
end
m.since[] = time()
return nothing
end
"Record one finished job: `ok` distinguishes a completion from a quarantine."
function record_job!(s::StageStats, ok::Bool, bytes::Int, elapsed_ns::Int)
Threads.atomic_add!(ok ? s.completed : s.failed, 1)
Threads.atomic_add!(s.bytes, bytes)
Threads.atomic_add!(s.busy_ns, elapsed_ns)
return nothing
end
"""
enqueue_blocking!(queue, job, stats; retry_seconds) -> nothing
Hand `job` to a downstream queue, parking and retrying until it fits, and charge
the parked time to `stats.blocked_ns`.
This is the routing half of stages 1 and 3: a classified file is never dropped,
so a full downstream queue means waiting, not failing. Wrapping the retry loop
here, rather than repeating `while !enqueue! sleep end` at each call site, is
what makes that wait measurable at all, and keeps the three routing paths from
drifting into three different backoff behaviours.
`stats` is `nothing` for callers that are not a pipeline stage. Startup
recovery (`recover_dir!`) uses the same never-drop retry, but it runs outside any
handler, so it has no `busy_ns` to charge against. Billing its wait to a stage's
`blocked_ns` would make `/stats` utilization, `(Δbusy_ns - Δblocked_ns) / …`,
go *negative* after a long recovery. Skipping the charge keeps `blocked_ns`
meaning exactly what its docs say: time a handler spent parked on its successor.
"""
function enqueue_blocking!(queue::JobQueue, job::Job, stats::Union{StageStats,Nothing};
retry_seconds::Real)
enqueue!(queue, job) && return nothing
t0 = time_ns()
while !enqueue!(queue, job)
sleep(retry_seconds)
end
stats === nothing || Threads.atomic_add!(stats.blocked_ns, Int(time_ns() - t0))
return nothing
end
# --- snapshot ---------------------------------------------------------------
"""
stats_snapshot(cfg, queues, m = METRICS) -> NamedTuple
Read every counter into a plain value tree for `GET /stats`.
`queues` is a NamedTuple of `JobQueue`s keyed by `STAGE_KEYS`, supplying each
stage's live depth and capacity. The counters above are about work *done*, and
a stage's depth is what says whether work is piling up in front of it.
The read is not atomic across stages: counters keep moving while we walk them,
so a snapshot can show a job counted as complete by stage 1 but not yet arrived
at stage 2. Over a benchmark window that skew is a job or two and does not move
a rate; a consistent snapshot would mean stopping the pipeline to read it.
"""
function stats_snapshot(cfg::Config, queues, m::Metrics = METRICS)
workers = (classify = cfg.worker_count, enrich = cfg.known_worker_count,
triage = cfg.unknown_worker_count, language = cfg.text_worker_count)
now = time()
stages = map(STAGE_KEYS, ntuple(identity, 4)) do key, i
s, q = m.stages[key], queues[key]
(; stage = i,
name = STAGE_TITLES[key],
workers = workers[key],
queue_depth = length(q),
queue_capacity = capacity(q),
completed = s.completed[],
failed = s.failed[],
bytes = s.bytes[],
busy_seconds = s.busy_ns[] / 1e9,
blocked_seconds = s.blocked_ns[] / 1e9,
in_flight = s.in_flight[])
end
return (; now,
since = m.since[],
uptime_seconds = now - m.since[],
intake = (; requests = m.intake.requests[],
files = m.intake.files[],
bytes = m.intake.bytes[],
rejected = m.intake.rejected[]),
stages = collect(stages))
end

187
src/worker.jl
View File

@@ -3,53 +3,88 @@
# stage, so `worker_loop` is parametrized with a `handler` and reused. Today # stage, so `worker_loop` is parametrized with a `handler` and reused. Today
# there are four stages: # there are four stages:
# #
# stage 1 handle_classify_job spool/ → classify → known/ (+known queue) | unknown/ (+unknown queue) # stage 1 handle_classify_job classify → known queue | unknown queue
# stage 2 handle_known_job known/ → exiftool enrich → done/ (+ .meta.json) # stage 2 handle_known_job exiftool enrich → done/ (+ .meta.json)
# stage 3 handle_unknown_job unknown/ → binary-vs-text sniff → binary/ | text/ (+text queue) # stage 3 handle_unknown_job binary-vs-text sniff → binary/ | text queue
# stage 4 handle_text_job text/ → language enrich → text_done/ (+ .meta.json) # stage 4 handle_text_job language enrich → text_done/ (+ .meta.json)
# #
# Adding a stage later is just another queue + pool + handler; the loop below # Adding a stage later is just another queue + pool + handler; the loop below
# doesn't change. # doesn't change.
#
# A file does not move between stages. It is written once into `spool/` at
# intake and stays there for its whole in-flight life; only the small `Job`
# reference travels, and `job.path` is therefore constant from intake until the
# file is committed. The stage a file has reached lives in the queue holding its
# reference, not in which directory the bytes sit, so the intermediate hops
# (spool→known, spool→unknown, unknown→text) are three renames per file that buy
# nothing on the live path. What remains is one move at the end: into a terminal
# sink (`done/`, `text_done/`, `binary/`) or into `failed/` on a throw.
#
# The cost is on restart, and which cost depends on the backend. Under the
# in-process `ChannelQueue` a crash loses the queues, and recovery can only
# re-drive everything in `spool/` from stage 1. That is safe, since
# classification and the UTF-8 sniff are pure functions of the file's bytes and
# every commit is idempotent, but it redoes work the old directory-per-stage
# layout could skip. Under `RabbitQueue` (src/rabbit.jl) it doesn't arise: a job
# stays unacked until its handler commits, so a restart resumes each file at the
# stage it had reached. The fix was the queue seam, not the directories.
#
# (`ROUTE_ENQUEUE_RETRY_SECONDS`, the backoff the routing handoffs below use,
# now lives in src/queue.jl, since startup recovery shares it.)
# How long a stage-1 worker backs off before retrying an enqueue onto a full # Per-file logging in stage 1 is `@debug`, not `@info`, because it is the
# downstream queue (known or unknown). Blocking backpressure: a classified file # stage's dominant cost. Measured by bin/bench_stage1.jl (2000 x 64 KiB files,
# is never dropped, so the stage-1 worker parks until the next stage makes room. # min of 5 trials): a formatted pair of log lines costs 71.2 µs per file, against
# Keeps intake decoupled — the HTTP path's `enqueue!` stays non-blocking; only # the 11.7 µs the whole handler takes with them switched off: six times the rest
# this worker-to-worker handoff blocks. # of the stage put together. Nearly all of it is `ConsoleLogger` formatting
const ROUTE_ENQUEUE_RETRY_SECONDS = 0.05 # (64.2 µs); the FlushLogger's per-message flush is only ~7 µs on top. Switching
# them back on with `JULIA_DEBUG=DarkStruct` takes the handler to 106.3 µs, i.e.
# from 85.4k files/s down to 9.4k on one worker.
#
# With logging off the stage is the classifier and nothing else: classify 10.30 µs
# (of which read_features is 7.82 µs), the queue handoff 0.12 µs, the disabled
# `@debug` lines 0.29 µs, for 10.7 µs of the 11.7 µs total. Removing the inter-stage
# rename is what left it that way: that rename was 11.6 µs per file, so it was
# the equal of the classifier, and dropping it took one worker from ~35k to
# ~85k files/s (16 workers: ~333k files/s).
#
# `@debug` is compiled to a min-level check that doesn't evaluate its arguments,
# so a disabled line costs ~0.15 µs rather than ~36 µs. The messages are still
# there when wanted: run with `JULIA_DEBUG=DarkStruct` to get them back. Errors,
# quarantines and lifecycle events stay at `@error`/`@info`, since they are rare and
# their cost doesn't scale with throughput. `GET /stats` (src/stats.jl) is the
# per-file observability that survives, and it is counted, not formatted.
""" """
handle_classify_job(job, cfg, worker_id, known_queue, unknown_queue) handle_classify_job(job, cfg, worker_id, known_queue, unknown_queue)
Stage 1. Classify the spooled file and route it to the next stage's queue, Stage 1. Classify the spooled file and route it to the next stage's queue,
retrying on a full queue rather than dropping the file: retrying on a full queue rather than dropping the file:
* `:known` → move to `known/`, enqueue onto the known queue for stage 2. * `:known` → enqueue onto the known queue for stage 2.
* `:unknown` → move to `unknown/`, enqueue onto the unknown queue for stage 3. * `:unknown` → enqueue onto the unknown queue for stage 3.
In both cases move first so the file physically lives in its stage dir before the Routing is the enqueue and nothing else: the file stays where intake wrote it and
reference is visible downstream; the moved path becomes the routed job's location. the *same* `Job` is handed on, so `job.path` still points at it. Reaching stage 2
is a fact about which queue holds the reference, not about which directory holds
the bytes.
Sub-`MIN_FILE_BYTES` files short-circuit to `:unknown` inside `classify`. Sub-`MIN_FILE_BYTES` files short-circuit to `:unknown` inside `classify`.
""" """
function handle_classify_job(job::Job, cfg::Config, worker_id::Int, function handle_classify_job(job::Job, cfg::Config, worker_id::Int,
known_queue::JobQueue, unknown_queue::JobQueue) known_queue::JobQueue, unknown_queue::JobQueue,
stats::StageStats)
classification = classify(CLASSIFIER[], job.path) classification = classify(CLASSIFIER[], job.path)
@info "classified file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification @debug "classified file" worker=worker_id id=job.id name=job.original_name size=job.size classification=classification
if classification === :known if classification === :known
dest = move_to(cfg.known_dir, job) # known queue full → park and retry, don't drop (time charged to blocked_ns)
routed = Job(job.id, job.original_name, dest, job.size, job.received_at) enqueue_blocking!(known_queue, job, stats;
while !enqueue!(known_queue, routed) retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
sleep(ROUTE_ENQUEUE_RETRY_SECONDS) # known queue full → back off, don't drop @debug "routed to enrichment" worker=worker_id id=job.id path=job.path
end
@info "routed to enrichment" worker=worker_id id=job.id dest=dest
else else
dest = move_to(cfg.unknown_dir, job) enqueue_blocking!(unknown_queue, job, stats;
routed = Job(job.id, job.original_name, dest, job.size, job.received_at) retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
while !enqueue!(unknown_queue, routed) @debug "routed to content triage" worker=worker_id id=job.id path=job.path
sleep(ROUTE_ENQUEUE_RETRY_SECONDS) # unknown queue full → back off, don't drop
end
@info "routed to content triage" worker=worker_id id=job.id dest=dest
end end
return nothing return nothing
end end
@@ -58,10 +93,11 @@ end
handle_known_job(job, cfg, worker_id) handle_known_job(job, cfg, worker_id)
Stage 2. Extract metadata (exiftool, with timeout) and enrich: build the Stage 2. Extract metadata (exiftool, with timeout) and enrich: build the
normalized sidecar and commit both to `done/` sidecar-first. Extraction normalized sidecar and commit both to `done/` sidecar-first: the file's one and
only move, straight out of `spool/`. Extraction
failure/timeout yields a *degraded* sidecar (the file is still a wanted known failure/timeout yields a *degraded* sidecar (the file is still a wanted known
file), so the only way to land in `failed/` is a genuine I/O error writing the file), so the only way to land in `failed/` is a genuine I/O error writing the
sidecar or moving the file — handled by `worker_loop`'s quarantine. sidecar or moving the file, handled by `worker_loop`'s quarantine.
""" """
function handle_known_job(job::Job, cfg::Config, worker_id::Int) function handle_known_job(job::Job, cfg::Config, worker_id::Int)
meta = build_metadata(job, cfg) meta = build_metadata(job, cfg)
@@ -74,22 +110,25 @@ end
handle_unknown_job(job, cfg, worker_id, text_queue) handle_unknown_job(job, cfg, worker_id, text_queue)
Stage 3. Sort an unrecognized file into a coarse content bucket by sniffing its Stage 3. Sort an unrecognized file into a coarse content bucket by sniffing its
first bytes: `binary/` (terminal — no further stage) if it looks like binary first bytes.
data, `text/` otherwise. A text file is then routed onward to the stage-4
language-enrichment queue, retrying on a full queue rather than dropping the file The two outcomes are asymmetric, because one is terminal and one is not. Binary
(the same blocking backpressure stage 1 uses for its downstream queues). is the end of the live path, so the file is committed to `binary/`, which is
also where the offline stage-5 discovery sweep reads its corpus, so the move is
load-bearing, not bookkeeping. Text has a stage 4 still to come, so nothing moves:
the same `Job` goes onto the language-enrichment queue, retrying on a full queue
rather than dropping the file (the blocking backpressure stage 1 also uses).
""" """
function handle_unknown_job(job::Job, cfg::Config, worker_id::Int, text_queue::JobQueue) function handle_unknown_job(job::Job, cfg::Config, worker_id::Int,
text_queue::JobQueue, stats::StageStats)
if is_binary(job.path) if is_binary(job.path)
dest = move_to(cfg.binary_dir, job) dest = move_to(cfg.binary_dir, job)
@info "sorted unknown" worker=worker_id id=job.id name=job.original_name kind=:binary dest=dest @info "sorted unknown" worker=worker_id id=job.id name=job.original_name kind=:binary dest=dest
else else
dest = move_to(cfg.text_dir, job) # text queue full → park and retry, don't drop (time charged to blocked_ns)
routed = Job(job.id, job.original_name, dest, job.size, job.received_at) enqueue_blocking!(text_queue, job, stats;
while !enqueue!(text_queue, routed) retry_seconds = ROUTE_ENQUEUE_RETRY_SECONDS)
sleep(ROUTE_ENQUEUE_RETRY_SECONDS) # text queue full → back off, don't drop @info "routed to language enrichment" worker=worker_id id=job.id path=job.path
end
@info "routed to language enrichment" worker=worker_id id=job.id dest=dest
end end
return nothing return nothing
end end
@@ -99,9 +138,10 @@ end
Stage 4. Enrich a text file with its natural language (via `detector`) and Stage 4. Enrich a text file with its natural language (via `detector`) and
programming language (via github-linguist): build the sidecar and commit both to programming language (via github-linguist): build the sidecar and commit both to
`text_done/` sidecar-first. Detection failure yields a *degraded* sidecar (the `text_done/` sidecar-first: the file's one and only move, straight out of
`spool/`. Detection failure yields a *degraded* sidecar (the
file is still wanted text), so the only way to land in `failed/` is a genuine I/O file is still wanted text), so the only way to land in `failed/` is a genuine I/O
error committing — handled by `worker_loop`'s quarantine. error committing, handled by `worker_loop`'s quarantine.
""" """
function handle_text_job(job::Job, cfg::Config, worker_id::Int, detector) function handle_text_job(job::Job, cfg::Config, worker_id::Int, detector)
meta = build_text_metadata(detector, job, cfg) meta = build_text_metadata(detector, job, cfg)
@@ -111,26 +151,71 @@ function handle_text_job(job::Job, cfg::Config, worker_id::Int, detector)
end end
""" """
worker_loop(worker_id, cfg, queue, handler) worker_loop(worker_id, cfg, queue, handler, stats)
Consume jobs from `queue` until it is closed and drained, running `handler` on Consume jobs from `queue` until it is closed and drained, running `handler` on
each. A failure on one job is logged and the file quarantined in `failed/` — it each. A failure on one job is logged and the file quarantined in `failed/`. It
must never kill the worker, or the pool would silently shrink. must never kill the worker, or the pool would silently shrink.
This loop is also where per-stage metrics are recorded (`stats`, see
src/stats.jl). Instrumenting here rather than in each handler means every stage
is measured the same way, by construction, and a new stage is measured the
moment it is wired up, with no per-handler bookkeeping to forget.
The timed region is the handler alone, excluding the `dequeue!` above it: time
parked waiting for work is idleness, and counting it as service time would make
an idle stage look as busy as a saturated one. A quarantined job still counts
its time: the work was done, it just ended in `failed/`.
""" """
function worker_loop(worker_id::Int, cfg::Config, queue::JobQueue, handler) function worker_loop(worker_id::Int, cfg::Config, queue::JobQueue, handler,
stats::StageStats)
@info "worker started" worker=worker_id @info "worker started" worker=worker_id
while true while true
job = dequeue!(queue) job = dequeue!(queue)
job === nothing && break # queue closed and drained → exit job === nothing && break # queue closed and drained → exit
Threads.atomic_add!(stats.in_flight, 1)
t0 = time_ns()
outcome = :ok
try try
handler(job, cfg, worker_id) handler(job, cfg, worker_id)
ack!(queue, job)
catch e catch e
@error "processing failed" worker=worker_id id=job.id name=job.original_name exception=(e, catch_backtrace()) if !isfile(job.path)
try # The file is gone and we did not move it: another worker
move_to(cfg.failed_dir, job) # committed it, i.e. this is the duplicate half of an
catch e2 # at-least-once redelivery (src/rabbit.jl). The work is done, so
@error "could not quarantine failed file" worker=worker_id id=job.id path=job.path exception=(e2, catch_backtrace()) # settle the message and say nothing louder than @debug. It is
# emphatically not a failure: quarantining here would file a
# `failed/` entry against a file that succeeded, and counting it
# either way would double-count one file's work.
outcome = :duplicate
@debug "duplicate delivery; already committed by another worker" worker=worker_id id=job.id name=job.original_name
ack!(queue, job)
else
outcome = :failed
@error "processing failed" worker=worker_id id=job.id name=job.original_name exception=(e, catch_backtrace())
try
move_to(cfg.failed_dir, job)
ack!(queue, job) # quarantined: the file's story ends in failed/
catch e2
@error "could not quarantine failed file" worker=worker_id id=job.id path=job.path exception=(e2, catch_backtrace())
# Failed *and* unquarantinable: the file is still in spool/.
# Reject rather than ack, because we are not claiming this
# one is done. With no dead-letter exchange the broker
# discards it either way — `nack!` never requeues, so this
# cannot become a poison-message loop — but the distinction
# is free and the message is the honest one.
nack!(queue, job)
end
end end
finally
# In a `finally` so an InterruptException during shutdown can't leave
# in_flight permanently above zero, which would read as a stuck job.
# A duplicate records nothing: its work was already counted by the
# delivery that actually did it.
outcome === :duplicate ||
record_job!(stats, outcome === :ok, job.size, Int(time_ns() - t0))
Threads.atomic_sub!(stats.in_flight, 1)
end end
end end
@info "worker stopped" worker=worker_id @info "worker stopped" worker=worker_id

1010
test/runtests.jl
View File

File diff suppressed because it is too large Load Diff