Owning the codebase
A reading path through the code. The README says what footprint does and how to run it, and
.stuff/MODEL.md says why it is built this way. This page shows where each thing lives, how one
command moves through the crates, which rules hold the case together, and how to change it without breaking them.
Every surface stays brief. More depth is one click away: ▸ expandable blocks open in place, cards marked open ↗ open a panel, and any teal mono link opens that file's declarations from the reference atlas. Press / to filter chapters, Esc to close a panel.
01How to read this guide
Three sittings, with the repository open beside the page.
Chapters 02–10: the crates and which way they depend, the words the code uses, what one command does from main to the terminal, what runs in parallel, what the case database holds, and the CLI and TUI surfaces.
Chapters 11–25: how collectors discover and read evidence, the time model, browsers and system logs, availability and fallbacks, state and diff, safety. Then the case itself: identities, dedup, outside tools, verdicts, correlation, staging and export.
Chapters 26–37: options, errors, tests, recipes, invariants, the plan, then the atlas, the two maps and SQLite, the library the whole tool leans on. Chapter 38 asks twelve questions to check yourself.
Two functions map everything else. run_collect is the
composition root: it builds the environment, picks the collectors from registry,
discovers sources, runs them on workers and writes the case. main in the
CLI is the dispatcher: one match sends each subcommand to the function that answers it. Read those two, and every
other file is a detail of one of their lines.
02The shape of the thing
One Rust binary over one SQLite file, in nine crates that only depend downward. Click a layer for what lives there.
crates/cli · crates/tuifootprint binary: clap commands and the interactive terminal UI. Neither touches SQL.crates/queryrun_collect, the composition root.crates/collect · crates/correlate · crates/rendercrates/storecrates/core · test data — crates/fixturesmain parses Cli with clap, resolves the global
--tz, and sends each Command to one function in the same file.
Read commands open the case with Store::open, call fp_query, and print every evidence field through
clean.
The TUI is a library the CLI calls (fp_tui::run). Its state machine, App, has no terminal code, so tests drive it headless; draw renders it with ratatui.
May import: query, render, correlate (trace rendering), store (to open the case), core. Never: rusqlite directly, or a collector.
The crate doc says it plainly: frontends call this crate, and no SQL lives outside it and the store. It has two halves.
- Reads: timeline and timeline_page, event_detail, state_records, diff_runs, coverage, plus time parsing and format_time.
- The runner: run_collect and the signal handling in interrupt.rs. The CLI and the TUI both drive it; progress comes back as CollectEvent over a channel.
collect (the biggest crate) holds the Collector trait and one module per source family. A collector reads files and tools and fills a Sink; it never sees the store. Every read goes through guard and the helpers beside it.
correlate runs idempotent passes over the stored case (correlate) and builds
the trace tree. It reads and writes through Store.
render owns sanitize and the jsonl, csv and html exports with optional redaction.
One file, schema.rs: the DDL in SCHEMA,
open_or_create (mode 0600) and
open (never creates), the version check, ingest,
the filter SQL in build_events_query, and one typed row per table.
See chapter 07 for the tables and chapter 19 for the keys that make re-runs idempotent.
- core/lib.rs:
Event,StateRecord,Artifact,Link,EventKind,Availability,SourceInstance. No I/O. - core/time.rs: every raw timestamp to UTC µs (chapter 12).
- core/deny.rs: the secret file names nobody opens.
- fixtures: synthetic evidence in the real formats. A dev-dependency only; no real personal data ever enters the repository.
Where the lines are
Non-test lines per crate. Click a bar to open that crate in the atlas.
What each crate may depend on Cargo.toml
| Crate | Depends on (workspace) | Notable outside crates |
|---|---|---|
footprint (cli) | query, tui, render, correlate, store, core | clap |
fp-tui | query, render, store, core | ratatui, crossterm |
fp-query | collect, correlate, store, core | chrono, libc (signals), tempfile |
fp-collect | core | rusqlite, regex, flate2, quick-xml, sha2, libc |
fp-correlate | store, core | rusqlite, sha2 |
fp-render | correlate, store, core | sha2, chrono |
fp-store | core | rusqlite (bundled-full) |
fp-core | — | serde, chrono, thiserror |
cli → tui → query → collect, correlate, render → store → core. A collector that wanted the store would have to go through the runner; that is the point. The single writer (chapter 06) depends on it.
Unit tests sit at the bottom of each file in mod tests. Scenario tests that run the real binary live in crates/cli/tests; the read model and the TUI have their own tests/. Every test builds its evidence in a temp dir; the scenario tests use fp-fixtures for a whole synthetic machine.
03Vocabulary
The words the code uses, each with the symbol that carries it. Click one for what it is, where it is stored and the rule that matters.
04Boot, run, shut down
A command starts, does one thing and exits. Only collect has a real lifecycle, and run_collect is where it lives.
What run_collect does
Click a step for its detail.
Times are parsed by time::parse into UTC µs, and an inverted window is refused.
The registry is then filtered twice: opt-in collectors (live.state, scan.import) stay out unless their flag or an explicit
--sources names them, and selected applies --sources and --exclude by kind or family.
build_env starts from Env::current
(under sudo, the invoking user's home), sets the root first so homes are named through the image's own /etc/passwd,
then adds every --home, or all users for --all-users (root only).
Then, in this order: the case is opened or created (0600), handed to SUDO_UID by
hand_case_to_sudo_uid, the signal handlers are armed, the
runs row is inserted, and a footprint-* temp dir is made 0700.
Discovery is cheap and stays on the main thread. A per-user collector is discovered once per user target with
Env::for_user; a system collector once. Every instance gets its
sources row now, so its id exists before any worker needs it for a staging directory. Only Present instances become jobs;
the others keep their availability as their final status.
Up to eight worker threads (never more than there are jobs) take jobs from a shared channel and run
collect_only, which catches a panic and checks the interrupt flag.
The main thread receives each Outcome, stamps the owner with
stamp_owner, ingests the Sink in one transaction and sets the final status:
partial with warnings, empty when nothing came out, or the error. Chapter 06 has the picture.
When the outcome channel closes, the workers are joined, an interruption is recorded as a run error, and
correlate runs its passes over the whole case. A failing pass is a run
error, not a failed command: the events are already safe. finish_run stamps the end and the errors.
The temp dir is removed when temp_dir drops at the end of the function.
| Pass, in order | Adds | File |
|---|---|---|
| navigation chains | redirected_from, navigated_from from browser from_visit | chains.rs |
| downloads | stat and hash of downloaded files; download manager ↔ browser visit | downloads.rs |
| installs | file → install command → package log line | install.rs |
| anchoring | anchored_to, and a derived time on the shell line | anchoring.rs |
| site permissions | visited host → the permission its origin holds | permissions.rs |
| IOCs | ioc_match, rebuilt from scratch | ioc.rs |
Passes that run after every collect
footprint has no scheduler and never runs in the background. The nearest thing to periodic jobs is this fixed list, run once at the end of each collect (and the IOC pass again after ioc add and ioc import).
| Pass | When | Does | File |
|---|---|---|---|
chains | every collect | links redirects and navigations inside one browser profile | chains.rs |
downloads | every collect | stats and hashes files the downloads point at (deny-listed files never hashed) | downloads.rs |
install | every collect | reads a .deb's identity with the host's dpkg-deb -f | install.rs |
anchoring | every collect | gives untimed shell lines a derived time | anchoring.rs |
permissions | every collect | joins visits and site permissions | permissions.rs |
iocs | every collect, ioc add, ioc import | deletes and rebuilds every ioc_match link | ioc.rs |
Run and shutdown
The CLI's collect spawns run_collect on a thread and prints errors and warnings as CollectEvents arrive, then the one-line summary: new events, already known, states, artifacts, sources, failures. The TUI starts the same function from its Collect view.
Ctrl-C or SIGTERM sets a flag (on_signal). Workers skip every job not yet started, the main thread records them as not collected: interrupted, correlation still runs, and the temp dir is removed. The guard restores the previous handlers when the run ends.
Why the order matters collect
Each step produces what the next one needs. The source rows are inserted during discovery, before any worker exists,
so every instance already has the id its staging directory is named after (src-<id>), and workers never touch the store.
The signal handler is armed only after the case is open, so a Ctrl-C during argument parsing behaves like any other program.
Correlation runs after the last ingest because its passes join across sources: a download link needs the file's stat and the
install log at once. finish_run comes last so a run without a finish time means the process died mid-way.
A second signal calls _exit(130) from the handler, because nothing else is async-signal-safe. Destructors don't run, so the
footprint-* staging directory under the temp dir stays behind, with copies of browser history in it. It is 0700 and yours, but delete it by hand.
One Ctrl-C always cleans up.
05One command's run
footprint list --since 20:40 --kind web.*, from argv to the terminal. Click a step for its code.
--tzclap builds Cli. -c/--case and --tz are global, so they may come before or after the
subcommand. parse_tz accepts local, UTC/Z or an offset, and fails before
anything is opened.
Every read command calls Store::open, which refuses a missing file instead of creating an empty,
world-readable one, and opens read-only so an evidence-protected (0400) case can be queried. Then init checks PRAGMA user_version: version 2 is used, a writable open of an empty
file gets the schema, anything else is refused with "start a new case".
Writers — collect and ioc add — go through open_or_create and stay read-write. On a read-only handle init skips the links_uniq repair it may otherwise owe an older v2 file. The read functions themselves never write: trace only reads the links collect built, because only collect knew the run's --root.
time::parse tries @epoch, then a relative -24h/+2d, then yesterday,
then the date-time forms. A naive local time is converted with the zone's rules at that date, and a time that doesn't exist
(skipped by a DST jump) is an error, not a guess. collect parses its window through
parse_in instead: naive bounds resolve in --assume-tz
when one is given, while relative forms stay anchored to now. The CLI fills a TimelineFilter.
timeline maps the filter onto EventFilter and calls
list_events. build_events_query puts every
filter in SQL, so LIMIT and OFFSET count the right rows. Each row then gets its IOC hits from
match_event_iocs.
% and _ in user input are escaped by escape_like; only a trailing * on a kind becomes a wildcard (kind_pattern). Filtering after the limit once made pages come back short.
print_rows renders each row as a table line, a JSON object or a CSV record. The time goes through
format_time in the chosen zone ((untimed) for order-only events); the text goes through
sanitize, and CSV cells through escape_csv.
The dispatch table
| Subcommand | Handler | Reads or writes through |
|---|---|---|
collect | collect | run_collect on a thread, progress over a channel |
sources | sources | fp_query::sources, coverage |
list | list | timeline, then print_rows or print_grouped |
show | show | event_detail |
search | search | fp_query::search (FTS5) |
trace | trace | trace_tree, render_trace |
state, diff | state, diff | state_records, diff_runs |
ioc | ioc | Store::insert_ioc, correlate_iocs, check_iocs |
export | export | export_case |
explain | explain | fp_query::explain (no case needed) |
tui | fp_tui::run | all of the above through App |
A read command never creates a case. A case of another schema version is refused. Every evidence string reaches the terminal through
sanitize. Times are compared in UTC µs and only rendered in a zone. --group-by lifts the limit until after grouping, so
--limit 10 means ten groups.
Read the real dispatch, trimmed main.rs
06Concurrency model
Readers in parallel, one writer. Collectors never share state and never see the store.
Only the main thread holds the Store. Workers return a Sink; the main thread ingests it in one transaction.
An AtomicBool the signal handler sets; workers read it before each job.
catch_unwind around every collect: hostile evidence can crash a parser, not the run.
Two processes on one case wait up to five seconds for each other instead of failing.
Jobs go out on one channel, shared by the workers behind a Mutex; outcomes come back on another. When the job sender and the
main thread's outcome sender are dropped, the workers hold the only senders left, so outcome_rx.iter() ends exactly when the last worker exits.
Ingest order follows completion order, not discovery order. That's harmless: events are keyed by content, not by insertion order, and the reads sort by time.
A collector that is already reading finishes its instance; nothing interrupts a parser midway. The flag is reset by arm at the start of each run, which matters for the TUI, where several collects run in one process.
The source is stored as error: collector panicked: … and the run goes on. The mutation smoke tests (chapter 28) exist to keep panics rare in the first place.
The connection is NO_MUTEX because only one thread ever uses it. Isolation between processes is SQLite's file lock; each source's
ingest is one transaction, so a reader sees a source's events all or none.
What may run at the same time on one case? reasoning aid
| Activity | Serialised by |
|---|---|
| Two collectors reading two sources | nothing needed: separate Sinks, separate staging dirs |
| Two profiles of one browser | separate src-<id> dirs, so History-wal files never mix |
| Ingest of two sources | the main thread: one at a time |
| A TUI reading while a CLI collect writes | SQLite's lock, with a 5 s busy timeout |
| Two collects on one case | SQLite's lock per transaction; the dedup key keeps an event from being stored twice, and each run has its own rows |
| Correlation and a reader | SQLite's lock; each pass inserts its links in one transaction (anchoring's time updates are single statements) |
07State: the case and what lives in memory
Everything lasting is in one SQLite file. Everything else lives for one run and is thrown away.
The 8 tables
Seven tables and the FTS5 index over events. Click a group for every field, read from the code.
The same tables as an interactive map: chapter 33.
SCHEMA_VERSION is 2. A case of another version is refused with "start a new case". A change of shape bumps the number; there is no in-place upgrade.
Created 0600 before SQLite opens it; under sudo it is chowned to SUDO_UID. It holds history, so it never leaves the machine.
Event times are i64 UTC µs; file times in artifacts are ns; sizes are bytes, shown in IEC units (MiB). JSON columns end in _json.
What lives only for one run
| Family | Holds | File |
|---|---|---|
staging dir footprint-*/src-<id>/ | copies of live databases with their -wal/-journal, opened read-only | utils.rs |
Sink per instance | events, states, artifacts and warnings until the writer ingests them | collector.rs |
PkgVerifier caches | the root's dpkg file index (built once), one verify report per package | persist.rs |
INTERRUPTED | the signal flag workers read | interrupt.rs |
TUI App | the loaded page of the timeline, filters, the selected view, collect progress | app.rs |
Lose the staging dir entirely — what happens? failure drill
Nothing that lasts. The copies are rebuilt on the next run from the originals, which were never touched. A collector whose copy vanished
mid-read fails its instance with the I/O error; the source is stored as error: … and the run goes on. Lose the case file instead
and everything is gone except the evidence itself: a new collect rebuilds the timeline, but state history (and so diff) starts over.
08Patterns you'll reuse
Six shapes recur in every command and view. Copy them; don't invent a seventh.
Rule: evidence is hostile, so no field from the case reaches a terminal without sanitize. It drops ANSI escape sequences, control characters and the bidi overrides that make evilfdp.exe display as evilexe.pdf. The CLI wraps it as clean; the TUI truncates and sanitizes in truncate.
Rule: every listing offers --format table|jsonl|csv. A table is for eyes, JSON Lines for jq, CSV for spreadsheets, where each cell goes through escape_csv, which also defuses cells that start like a formula.
Rule: store and compare in UTC µs; render with format_time and the Tz from the global --tz. Never format a time any other way: an order_only event must print (untimed), not 1970.
Rule: long work reports through CollectEvent on a channel; the frontend decides what to show. The CLI prints errors and warnings to stderr and the summary to stdout; the TUI turns the same events into its Collect view.
Rule: a view of an unbounded table pages with timeline_page: ask for one row more than the page, and the extra row says whether there are more. Offsets are pushed into SQL, so a page is always full.
Rule: an action that fails in the TUI becomes the status line; it never ends the session. Every key handler returns Result, and the event loop catches it. A bad filter time is caught earlier still, in FilterDraft::apply, with the bad text quoted back.
09Commands, views and the registry
16 commands and subcommands, and eight TUI views over the same read model. Click a card for its flow and files.
discover and read sources into the case
status per source, and coverage
the merged timeline and FTS5
one event, its provenance and links
where a file or URL came from
how things were, and what changed
indicators, reports
the interactive UI; scanner false positives
collect fills a CollectParams from CollectArgs and runs run_collect on a thread (chapter 04). Flags: --since/--until, --sources/--exclude (kinds or families), --home (repeatable), --all-users, --root, --assume-tz, --with-live, --with-scanners.
Edge cases: a selection that matches no collector is refused; --all-users without root is refused; an inverted window is refused. A run with failed sources still exits 0 and says , N failed.
sources lists the last run's sources (--run N, --all-runs) with status, event count, path and note. --coverage groups them by evidence category through coverage: Covered, Partial or Gap, with the sources that answered and the fallbacks that were missing.
list takes --since/--until, --kind (families with web.*), --source, --host (subdomains match), --match, --limit, --reverse, --group-by dir|host|source|minute and --format. IOC hits show as [IOC: …].
search passes the query to FTS5 as is, so FTS5 syntax works (zoom AND deb, "exact phrase"); a syntax error comes back from SQLite.
show prints one EventDetailView: time in the zone and its precision, the raw time and its format, the assumed zone, source, actor, subject, summary, attributes, provenance (file, sha256, locator) and the links in both directions with their confidence and why.
trace takes a path, a URL or an event id. trace_tree finds the head (an artifact or an event), walks the redirect chain backwards, then the download and install links forwards; render_tree prints ◄/► edges with a confidence each. It only reads; links come from collect.
state shows the last run's state records (--kind persist.*, --match), with flags such as unowned or mismatch. diff compares two runs by (kind, key); runs can be ids, last or prev (resolve_run_id).
ioc has four subcommands: add TYPE VALUE (checked against IOC_TYPES), import FILE (the type is guessed by detect_ioc_type when a line has none), check, list. Adding re-runs the IOC pass at once. insert_ioc is idempotent over (type, value): re-adding returns the existing id, so a double import can't double the hits.
In an import file, # starts a comment only after whitespace (https://evil/#frag keeps its fragment), and a value with a scheme and query is a url, not a path glob (parse_ioc_line).
export writes jsonl, csv or a standalone html report with the list filters and --redact (chapter 25).
tui opens eight views, switched with 1–8: timeline, detail, trace, state, diff, ioc, sources, collect. t toggles local/UTC, ? shows the keys, q quits. It refuses to start without a case or a terminal.
explain needs no case: it looks a scanner finding up in EXPLANATIONS and prints why it is a known false positive and the read-only command that proves it.
The TUI's keys, per view app.rs
| Where | Keys |
|---|---|
| everywhere | ? help, q quit; 1–8 switch view, except from detail and trace (Esc first); none of these while a field is being edited |
| timeline | ↑ ↓ PgUp PgDn move (the next page loads at the end), Enter detail, r trace the row, i or / edit the filter bar, t local/UTC |
| detail | ↑ ↓ links, Enter jump to the linked event, r trace this event, Esc back |
| trace | ↑ ↓ rows, Enter open the event behind a row, Esc back |
| state, diff, ioc, collect | k kind filter (state), e run ids (diff), Enter opens a hit (ioc), the collect form takes the window and starts the run |
The exact bindings are in App::on_key and the per-view handlers beside it; draw_help lists them in the app.
The registry
Every routing name in the code, read from the atlas: subcommands (variants of a #[derive(Subcommand)] enum), collector kinds (the string each Collector::kind returns) and event kinds (EventKind::as_str). Filter it:
| Value | Family | Declared by | File | Doc or arguments |
|---|
10A first investigation
What a new user does, in order, and what each step needs from the code.
cargo build --release. The toolchain is pinned in rust-toolchain.toml; SQLite is bundled, so there is nothing else to install.footprint collect --since -2d. The case ./footprint.db is created 0600; a second run adds only what is new.footprint sources --coverage. A gap names what was missing and what would have filled it (journal groups, root for /root).footprint list --since 20:40 --until 21:30 --kind web.*,pkg.*,priv.sudo, then show ID for one row's provenance.footprint trace ~/Downloads/x.deb: the page, the redirects, the download, the install, each edge with its confidence.footprint ioc add domain evil.example; hits appear in every list. Then export --format html --redact to share.Input rules worth knowing flags
- Times (time::parse):
2026-10-16T20:40,2026-10-16 20:40:05,20:40(today),2026-10-16,-24h,+30m,-2w,yesterday,@1792172917. clap accepts a leading-for these flags. - Kinds: exact (
pkg.install) or a family with a trailing*(web.*); comma-separated or repeated. - Sources: in
collect, a kind or a family prefix (chromium,pkg,shell.history) orall; inlist, also an app orapp/profile(Brave/Default). - Hosts match their subdomains:
--host zoom.usincludescdn.zoom.us, notnotzoom.us. - Zones:
--tzrenders (local,UTC,+03:30);--assume-tzinterprets logs that store local time. They are different things. - The case is
-c PATH, global; read commands never create one.
11Collectors: discover, then collect
24 collectors behind one small trait. Discovery is cheap and says what exists; collection reads one instance and may fail alone.
Find instances and set their availability. No parsing, no copying.
Read one instance into the Sink; warn for what was skipped.
The one constructor: kind, subject, summary, stamp, provenance.
Tools by absolute path under the root, the distro, journal access.
Which collectors read one home, and how the owner is stamped.
Every collector, in the order discovery runs.
Discovery answers "what is there?" for an Env. It returns one SourceInstance per thing worth reading, including the ones that can't be read: a missing app is NotInstalled, a profile without its database Empty, an unverified format Unsupported. The runner stores them all, so coverage can say what was absent.
A collector reads one instance and pushes into the Sink. What it cannot read but can live without becomes sink.warn(…), which marks the source partial; what makes the instance useless is an Err. Chromium shows both: no visits is an error, no downloads a warning.
Collectors respect the window themselves with in_window, or push it into their SQL (webkit_window).
new_event fills the common fields; callers set actor, subject_host and attrs with struct update syntax. An untimed stamp makes the event derived, a timed one direct. Origin turns a locator into a Provenance.
Probe answers questions about the machine before anything is spawned. tool_path looks a tool up by absolute path under env.root, never through $PATH. can_read_system_journal checks the process's groups for adm or systemd-journal. get_xattr reads a download's origin URL without following a symlink.
A collector that reads one home returns true from per_user: Chromium, Firefox, XDM, the xattr and detect-only download collectors, shell history, recently-used, KDE, kactivitymanagerd and Zeitgeist. The runner discovers each once per UserTarget with Env::for_user, and stamp_owner puts that user's uid and name on every event that didn't name its own actor.
System collectors (packages, journal, filesystem, persistence) run once; the persistence snapshot walks every user target itself.
The last two are opt-in (chapter 04, step 1). The full list with each kind string is in the registry of chapter 09.
Which collector answers which question matrix
| Question | Primary | Fallbacks |
|---|---|---|
| What sites, and how did I get there? | chromium.history, firefox.history | — |
| Where did this file come from? | browser downloads, dlmgr.xdm | dlmgr.xattr, shell curl/wget, file times (fs.statx) |
| What was installed? | pkg.dpkg, pkg.apt | pkg.pacman, pkg.dnf, pkg.zypper, pkg.snap, pkg.flatpak |
| Who used root, who logged in? | auth.journald | auth.log, auth.wtmp |
| What was typed? | shell.history | — |
| What was opened? | desktop.recent | desktop.kde; desktop.kactivity and desktop.zeitgeist detect-only |
| What will run at boot or login? | persist.snapshot (state) | — |
12The time model
Every source counts time its own way. One function turns them all into UTC µs, keeps the raw value, and treats an overflow as an error.
TimeFormat | Counts | Used by |
|---|---|---|
WebKitMicros | µs since 1601-01-01 UTC | Chromium History, Preferences |
PRTime | µs since 1970 UTC | Firefox places, permissions |
UnixMillis | ms since 1970 | Firefox download endTime (modern) |
JournaldRealtime | µs since 1970 | journald __REALTIME_TIMESTAMP |
DotNetBinary | 100 ns ticks since year 1, kind in the top 2 bits | XDM downloads.db |
UnixSeconds, BashHashEpoch, ZshExtended | seconds since 1970 | wtmp, bash #epoch, zsh : epoch:0;, fish |
FileTime | ns since 1970 | statx, xattr downloads (UnixNanos is the same unit, unused today) |
LocalNaive(fmt) | wall-clock text, no zone | dpkg.log, apt history, auth.log, flatpak |
Rfc3339 | text with an offset | pacman, snap --abs-time, xbel, flatpak (some versions) |
None | no time: order_only | bash without timestamps, REPLs |
A log that writes local wall time (2026-10-16 21:08:00) is converted with the offset the zone had at that date, not today's: resolve_local asks chrono's local zone for that instant. A time skipped by a DST jump has no instant and is an error; one repeated by a fall-back resolves to the earlier instant.
--assume-tz +03:30 (or UTC) overrides the host zone for a fixed offset, which is what you want for a disk image from another machine. The event's ts_tz_assumed says which was used.
TimePrecision says how far to trust the instant: exact (sub-second from the source), second (whole seconds, or derived by anchoring), minute, day, or order_only (no instant; ts_utc is 0 and the attrs carry the position in the file).
Timestamps come from evidence, so a crafted one may be anything. scale multiplies with checked_mul, epochs are subtracted with checked_sub, and an overflow is TimeError::OutOfRange, never a panic in debug or a wrap in release. File times from statx saturate (file_time_ns): ext4 can store dates past what i64 ns holds.
Worked example: one Chromium visit WebKit → UTC → screen
visits.visit_timeis13436645597000000: µs since 1601-01-01.- Minus
WEBKIT_EPOCH_US(11 644 473 600 000 000) gives1792171997000000µs since 1970: 2026-10-16 17:33:17 UTC. That ists_utc;ts_rawkeeps the 1601 number andts_raw_formatsayswebkit_micros. footprint list --tz +03:30prints2026-10-16 21:03:17;--tz UTCprints17:33:17. The stored value never changes.- The window is pushed into the SQL the other way round: webkit_window adds the epoch to
--sincewithsaturating_add.
13Browsers and download managers
The main evidence: what was visited, how one page led to the next, what was downloaded, and what a site was allowed to do.
The copy is read; the original is named in the provenance, with the hash of what was read.
Chrome, Chromium, Brave, Edge, Vivaldi and Opera, with their beta and dev channels, native (~/.config), flatpak (~/.var/app/<id>/config) and snap roots (BROWSERS, SNAP_ROOTS). Every Default and Profile N is an instance (list_profiles).
- Visits (collect_visits): the transition's qualifier bits decide
web.redirect([3xx]server,[client]client) orweb.visit;from_visitis kept for chains. - Downloads (collect_downloads) with their URL chain; an empty target path means "not saved", and no artifact.
- Searches from keyword search terms.
- Preferences and Secure Preferences: real permission types only (PERMISSION_TYPES), changes and one-time uses as events, settings as state; extensions as state, with component extensions never reported as installs.
Firefox, Zen and LibreWolf, native, XDG (~/.config/mozilla), snap and flatpak (ROOTS). Profiles are named from profiles.ini (parse_profiles_ini): default-release, not the random directory prefix; a directory missing from the ini falls back to its prefix.
- Visits from
moz_historyvisits; visit types 5 and 6 are redirects. - Downloads are two
moz_annosrows per place (destination URI and metadata), keyed byplace_idso the locator is stable.endTimeis ms in modern Firefox and µs in old ones; PRTIME_FLOOR tells them apart. - Permissions from
permissions.sqlite, extensions fromextensions.json.cookies.sqlite,logins.jsonandkey4.dbare on the deny-list.
XDM keeps every download in ~/.xdman/downloads.db, including the ones a browser handed over and never recorded. The schema is checked first (table_has_columns), the old Java XDM is Unsupported, and only the columns in COLUMNS are selected, because the same table stores site and proxy passwords. Times are .NET binary.
The schema peek opens the live file with mode=ro&immutable=1 (open_immutable), with the path percent-encoded for the URI.
When no browser recorded a download, the file may have: browsers and some tools set user.xdg.origin.url and referrer.url. XattrDownloadCollector reads them in the download directories from XDG_DOWNLOAD_DIR (download_dirs) and ~/Downloads.
uGet, Motrix/aria2, KGet and JDownloader are detected only (KNOWN_MANAGERS) and reported unsupported until a parser is verified on real data.
Why redirects matter more than visits the lure
The scenario the tool was built for is a link in an email that bounced through two redirectors to a fake download page.
A redirect row carries both ends (from → to [3xx]) and its from_visit, so the chain pass (chapter 23) can link
each hop to the one before it. trace walks those links backwards from the download to the first page, which is how
"which email link started this?" gets an answer.
14System evidence: packages, privilege, shells
What the machine itself wrote down: what was installed, who used root, and what was typed.
The invoking user is before : , USER= is the target, and COMMAND= runs to the end of the line.
DpkgCollector reads dpkg.log and every rotation (.1, .2.gz, through with_rotations); one event per package action. AptCollector reads apt/history.log blocks: Start-Date (two spaces between date and time), Commandline, Requested-By, then one event per Install/Upgrade/Remove line, explicit packages first.
A line that won't parse is skipped and counted in one warning per file (warn_skipped).
pacman, dnf/rpm and zypper are logs, read like dpkg. snap and flatpak keep no log, so their own tools are run by absolute path with a fixed argv and the C locale (run_listing): snap changes --abs-time (the verb leads the summary, snap_change_kind) and flatpak history (syslog-style times without a year, parse_flatpak_time). A tool that fails is an error, never an empty result.
AuthCollector runs journalctl --output=json once with OR-ed matches (MATCHES): sudo, su, pkexec, the ssh daemon under both unit names, and systemd-logind. sudo lines become priv.sudo, pam_* lines auth.session, ssh logins auth.ssh, logind sessions auth.login (parse_logind_session, both message formats).
The window goes to journalctl as @epoch seconds and is then applied exactly; a non-UTF-8 message arrives as a byte array and is decoded lossily. Outside adm/systemd-journal you see only your own entries, and the source's note says so.
AuthLogCollector reads /var/log/auth.log* and secure* whenever they exist, with the same sudo and ssh parsers as the journal; it is what answers on systems without a journal. Where both exist, a sudo shows up twice, once per witness. Their timestamps have no year, so parse_log_timestamp rolls a date that would be in the future back a year.
WtmpCollector reads wtmp/btmp: fixed 384-byte glibc records, with the timestamp read as a 32- or 64-bit timeval (record_tv_sec).
ShellCollector knows bash, zsh, fish, and the python, psql, mysql, sqlite3 and node REPLs (HISTORIES), plus $HISTFILE and $ZDOTDIR.
- bash:
#epochlines time the entry after them, which may span lines (parse_bash); without them every line isorder_only. - zsh: read as bytes and unmetafied (unmetafy),
\-continued lines joined. - fish: entries start at
- cmd:, withwhen:andpaths:. - libedit REPLs escape bytes as octal (
\040).
Every command is redacted before it is stored (redact_secrets) and flagged when it looks dangerous (DANGEROUS: a download piped to a shell, base64 -d, eval, /dev/tcp, chmod +x, an inline interpreter, a typed curl/wget/git clone…). A typed download also names its URL: first_url fills subject_host, so --host and domain IOCs see it — the shell-history rung of the downloads fallback chain.
Desktop, filesystem and the rest smaller collectors
- recently-used.xbel: one
file.openedper application that opened a file. - KDE RecentDocuments:
.desktoplink files. - fs.statx: a native
statxwalk of each home,/tmp,/var/tmp,/dev/shmand/etc, keeping the files changed in the window; one filesystem only, no symlinks followed, noise directories skipped (is_noise_dir), and the run's own staging dir and case files skipped. - persist.snapshot, live.state and scan.import write state, not events: chapter 16.
15Availability, coverage and fallbacks
An app that isn't installed is a fact about the machine, not an error. Every source ends a run with one status, and coverage turns those statuses into an answer.
Discovery sets the first five; the runner sets partial, empty and error after collecting.
Six categories, each Covered, Partial or Gap, with the sources behind the answer.
When the primary source is missing, a weaker one answers, and says it is weaker.
| Status | Set by | Meaning |
|---|---|---|
present | discovery, then kept | read, and something came out |
not installed | discovery | the app, file or tool isn't there; a coverage note |
empty | discovery or the runner | there but holding nothing (a profile without a History, a run with nothing in the window) |
unreadable | discovery, or the runner on a permission denied | the OS refused, not the tool; needs_root says whether sudo would help — a coverage note, not an error |
unsupported | discovery | known format, no verified parser yet (old XDM, Zeitgeist, uGet…) |
partial | the runner | read, with warnings: skipped lines, an unreadable side file |
error: … | the runner | collect returned Err or panicked, or the run was interrupted first |
coverage walks CATEGORIES in priority order. A category is Covered when any of its sources has events, Partial when a source is present but produced nothing in the window, and a Gap otherwise. Each row lists the sources that answered (with counts) and a note for each one that didn't.
| Evidence | Chain, strongest first |
|---|---|
| Downloads | browser history → XDM → file xattrs in the download dir → shell curl/wget/git clone → file times from the walk |
| Packages | dpkg and apt logs → pacman, dnf/rpm, zypper logs → snap and flatpak through their tools |
| Privilege and logins | journald → auth.log/secure → wtmp/btmp |
| Desktop activity | recently-used.xbel → KDE RecentDocuments → (detected: kactivitymanagerd, Zeitgeist) |
| Times of untimed lines | bash #epoch → anchoring to sudo/apt (derived) → order only |
A weaker source never pretends to be a stronger one: an anchored time is derived, a download seen only through an xattr has no referrer chain, and coverage names the source that answered.
Reading a coverage report example
A Gap in Privilege with the note "journald: user entries only (not in adm or systemd-journal group)" means sudo lines exist
but weren't visible: run as root, or add the user to adm. A Partial in Browser history usually means the window
was empty for a browser that is installed. An unsupported note is a parser the project hasn't verified yet, not a failure.
16State, persistence and diff
Events say what happened; state says how things are. Each run keeps a snapshot, and diff shows what changed between two.
SystemPersistSnapshot records, under the root: ld.so.preload, uid-0 accounts and login shells, sudoers, root's SSH keys, cron (system and spool), systemd units, autostart entries, profile and rc files, PAM modules, udev and network hooks, apt hooks, and SUID/SGID binaries. UserPersistSnapshot records, per home: authorized keys, user units, autostart, rc files and git hooks in nested repositories.
Files are queued as Pending and resolved in one batch by flush_pending: owners first, then one state record each with size, times, sha256, owner, integrity and flags.
| Flag | Raised when |
|---|---|
unowned | no package owns a system file and the package database could be read |
hash_mismatch | the package manager's verify says mismatch or missing |
mtime_in_window | the file changed inside the collect window |
not_regular_file | a FIFO or device sits where a config file belongs |
world_writable_path | an rc file puts a world-writable directory on PATH |
uid0_non_root | an account other than root has uid 0 |
unowned_suid, unowned_pam_module | a SUID binary or a PAM module no package owns |
Only with --with-live, and only on the live host: processes from /proc with flags for binaries that are deleted, in /tmp-like dirs or hidden dirs (proc_flags); listening and connected sockets with their owner, kernel sockets labeled (socket_state follows tcp_states.h); kernel modules not owned by a package, found in the running kernel's tree; the tainted flag.
Only with --with-scanners: clamscan (homes and temp dirs), rkhunter and chkrootkit, each run by absolute path with a fixed argv and its output capped (MAX_SCAN_OUTPUT). Findings become scan.finding state.
Scanners have well-known false positives. explain matches a finding against EXPLANATIONS and prints why it is harmless and a read-only command that proves it, such as dpkg -S /usr/bin/lwp-request.
diff_runs loads both runs' state, keys it by (kind, key), and reports Added, Removed or Changed with a one-line detail (changed_summary). Keys are chosen so the same thing has the same key in every run: a unit file's path, an extension's profile|id, a socket's address and owner.
Why state is per run but events are not design
An event happened once; seeing it again in a later run is the same event, so the dedup key ignores the run. State is a
reading taken at a moment; the same unit file read twice is two readings, and the difference between them is the point.
That is why state is keyed (run_id, kind, key) and upserts within a run, while events is keyed by content.
17Safety: secrets, hostile evidence, output
The tool reads what an attacker may have written, on a machine that holds secrets, and prints to a terminal. Four guards, each in one place.
Cookies, saved logins, keys, wallets, .env: is_secret decides, guard refuses.
Only regular files, size caps, checked arithmetic, panics contained, nothing executed.
Before storage: PASSWORD '…', TOKEN=…, user:pass@, Authorization:.
Terminal escapes stripped, CSV formulas defused, the HTML report forbids fetches.
is_secret matches any path component in SECRET_COMPONENTS (Chromium Cookies, Login Data, Web Data, extension storage; Firefox cookies.sqlite, logins.json, key4.db; keyrings, wallets, .netrc, .pgpass…), private files under .ssh (public keys, authorized_keys, known_hosts and config allowed), .aws/credentials, Solana keypairs, .env*, and key and certificate bundles by extension. A -wal/-journal/-shm suffix is stripped first, so the sidecar of a denied database is denied too — and copy_db_set guards each side file it copies.
It lives in core so every crate agrees on it; guard enforces it in the helpers every read goes through, and the download correlation never hashes a target named like one.
- Only regular files. open_regular opens non-blocking and checks the type on the descriptor; a FIFO planted as a history file or a cron entry can't stall a run.
- Size caps. Text evidence over 256 MiB is refused (MAX_TEXT_BYTES), including after gunzip; SQLite evidence over 4 GiB (MAX_DB_BYTES).
- Checked arithmetic. Timestamps overflow into an error, file times saturate (chapter 12).
- Contained panics.
catch_unwindper instance (chapter 06), and mutation smoke tests per parser (chapter 28). - Nothing executed. External tools are the host's, by absolute path, with a fixed argv and no shell (chapter 20). A
.debis read withdpkg-deb -f, never installed or unpacked.
redact_secrets runs on every shell and REPL command and every journald message before it reaches the Sink. It returns a borrowed string when nothing matched, so the common case costs nothing, and the event's attrs record redacted: true when something did.
This is different from export --redact (chapter 25), which hashes rather than masks, and only in the exported copy.
sanitize strips ANSI sequences, control characters and bidi overrides from anything printed (chapter 08). escape_csv prefixes a cell starting with = + - @, tab or CR with ', so a spreadsheet never runs evidence as a formula. The HTML report carries a Content-Security-Policy of default-src 'none' and escapes every field (escape_html).
Refusals, as the user reads them table
| Situation | What is printed or stored |
|---|---|
| a deny-listed file | refusing to open a secret file: … (the source errors; nothing is read) |
| a FIFO or device as evidence | … is not a regular file; in the persistence snapshot, a not_regular_file flag |
| a file over the cap | over the 268435456-byte limit |
| a timestamp out of range | timestamp out of range: … for that line or row |
| a panicking parser | source status error: collector panicked: … |
| a case of another schema | case file schema vN is not supported (expected v2); start a new case |
18The map and its laws
The riskiest part is the path from a byte in someone's history file to a line in a report someone will act on. A bug here costs trust: a wrong time, a missing link, a duplicate, or a secret where it shouldn't be.
crates/collectStore::ingestcrates/correlatecrates/query · crates/renderBrowser databases, logs, history files, the journal through journalctl, package databases through their tools, /proc with --with-live. The tool runs as the user (under sudo it reads the invoking user's home) and records what it couldn't read instead of failing.
Chapters 11–17. What matters here: every Event leaves a collector with its raw time, its normalized time, a Provenance with a stable locator, and a confidence.
Events insert ON CONFLICT(dedup_key) DO NOTHING and count as new or known; state upserts per run; artifacts upsert by path and only ever fill in missing metadata (COALESCE).
Six passes (chapter 23), each idempotent: links are unique on (src, rel, dst), and the IOC pass deletes and rebuilds its own relation. A pass that changes an event (anchoring) only ever changes an order_only one, and marks it derived.
Chapters 05, 08 and 25. The read model never writes; exports are created 0600; --redact hashes URL secrets in the copy only.
The laws
Live databases are copied with their side files and opened read-only; XDM's schema peek is immutable=1.
File, sha256 of what was read, and a locator precise enough to find the same row again.
The dedup key is built from the source's identity and the event's content, never from run-local ids.
Times are compared in UTC µs; the raw value and its format are stored beside them.
Derived times and heuristic links say so; "unverified" is never printed as "ok".
Deny-listed files are never opened; typed passwords are masked before storage.
19Identities and keys
Two strings decide whether a re-run duplicates the case: the source's identity and the event's dedup key.
dpkg.log.1 and dpkg.log.2.gz have the identity of dpkg.log, so a line that moved into a rotation is still known.
A new app name or profile label (Firefox profiles named from profiles.ini) makes every event of that source new once. Such a change needs a new case.
Locators, per source what makes a row findable
| Source | Locator | Stable because |
|---|---|---|
| Chromium visits | visits.id=N | row ids never change in a history file |
| Firefox downloads | moz_annos.place_id=N | keyed by place, not by query order |
| XDM | downloads.id=N | the table's key |
| journald | the entry's __CURSOR | journald's own stable cursor |
| dpkg, apt, auth.log | line N of the file | with the rotation-stripped identity; a line moved by rotation keeps its time and subject |
| timed shell history | when=EPOCH | line numbers shift when the shell truncates its history |
| untimed shell history | line N | the best there is; a truncated history can re-number lines |
| snap, flatpak | change id=N, history CHANGE APP | the tools' own ids |
The insert itself:
source_id across runsAn event keeps the source_id of the run that first stored it. A pass that matches events by source_id misses everything a later run added; match by the history file (source_kind and prov_file) instead, as the chain and anchoring passes do.
20The outside tools it runs
Some evidence only a tool can read. Every tool is the host's, run by absolute path with a fixed argv, no shell, stdin closed, and a failure is an error, never an empty result.
/usr/bin/journalctl --output=json --no-pager --quiet [--directory DIR] [--since=@S] [--until=@S] MATCHES…. Streamed line by line; stderr read on a thread so a full pipe can't deadlock; a bad entry is skipped and counted. On an image, --directory points at the image's journal.
Ownership on a Debian root comes from the root's own var/lib/dpkg/info/*.list (no tool at all, so it works on images). rpm and pacman answer ownership by batches of OWNER_BATCH paths (query_owners). Integrity is one verify run per package, live host only (verify_report): dpkg --verify, rpm -V, pacman -Qkk.
snap changes --abs-time and flatpak history --columns=time,change,application, both under LC_ALL=C through run_listing. Their paths come from Probe::tool_path at discovery.
Correlation reads a downloaded package's name with DPKG_DEB -f FILE Package (inspect_deb): only a regular file named *.deb, never extracted, never installed.
SCANNERS: clamscan --infected --no-summary -r over homes and temp dirs, rkhunter --check --sk --rwo, chkrootkit -q. Output capped at 64 MiB.
Why never a shell, and never $PATH the threat
The tool runs on a machine that may be compromised. A $PATH lookup can be redirected by a planted ~/.local/bin/dpkg; a shell
would interpret a crafted file name. So every tool is found by absolute path under the root (Probe::tool_path) or is a
constant (/usr/bin/journalctl, /usr/bin/dpkg-deb), arguments are passed as a vector, and stdin is closed. Output is parsed as untrusted text, capped when it can be large.
| Tool | When | On an image | If it fails |
|---|---|---|---|
journalctl | auth.journald | reads the image's journal dir | source error with its stderr |
dpkg --verify, rpm -V, pacman -Qkk | persistence snapshot | not run: unverified | unverified, never ok |
rpm -qf, pacman -Qoq | persistence snapshot | not run: ownership unknown | owner unknown for that batch |
snap, flatpak | pkg.snap, pkg.flatpak | the image's tool path is probed | source error |
dpkg-deb | install correlation | the host's tool on the image's file | no package name; filename prefix used |
| scanners | only with --with-scanners | live host | source error |
21How it computes ownership and integrity
Five verdicts, and the rule that the tool never says ok when it didn't check.
| Verdict | Means |
|---|---|
ok | a verify ran for the owning package and named no problem for this file |
mismatch | the checksum or size differs (dpkg/rpm flag column has 5 or S) |
missing | the package lists the file and it is gone |
unverified | no verify could run (an image, no tool) or the tool couldn't read the file (?) |
unowned | the package database was readable and no package owns the path |
Worked example: the same SUID file, on the host and on an image both branches
Live host. The SUID scan walks /usr/bin (on merged-/usr systems /bin is a link to it) and finds /usr/bin/su. The root is /, so PkgVerifier::new finds dpkg and keeps its path.
The dpkg index (built once from /var/lib/dpkg/info/*.list) may list the file as /bin/su;
usr_merge_alias tries that alias and finds util-linux. dpkg --verify util-linux runs once, reports nothing for the file, and the verdict is ok.
Mounted image. The same file under --root /mnt/img. The index comes from /mnt/img/var/lib/dpkg/info, so the owner is still util-linux. No tool is kept for an image (the host's dpkg would check the host), so integrity returns unverified.
An image without a dpkg database can't resolve owners at all, and a file with no owner is then unverified, not unowned (resolves_owners).
The review found every SUID binary reported unowned (the /bin vs /usr/bin alias), sudoers reported as a hash mismatch when dpkg only lacked permission to read it (??????????), "ok" printed when no check ran, and images judged by the host's database. Each verdict now has exactly one way to be produced.
22One collect, start to finish
From the user's flags to a closed run, with every way it can refuse.
--all-users without root. Nothing is created yet.0600, chown to SUDO_UID, arm signals, insert the run, make the staging dir.sources row each.Every refusal, as the user reads it table
| Cause | Message | Where |
|---|---|---|
| unparsable time | unrecognized time: … | parse_datetime |
| a time skipped by DST | … does not exist in the local time zone | naive_to_utc_us |
| inverted window | --since is after --until | run_collect |
| nothing selected | no source kind matches --sources/--exclude | run_collect |
--all-users as a user | --all-users requires root privileges | build_env |
| a case of another schema | case file schema vN is not supported… | Store::init |
a bad --tz | unknown time zone `…` (use local, UTC, or ±HH:MM) | parse_tz |
What "new" and "already in the case" mean the summary line
run 2: 5 new events (26174 already in the case), 828 states, 13815 artifacts from 28 sources. New and known come from the
insert's row count (1 or 0) per event; states and artifacts are counted as written, because both upsert. A second run over the same
window should show almost nothing new: what remains is real activity since the first run.
23Correlation and the life of a source
A source moves through a few states in one run; correlation then runs over everything the case holds, old and new.
Which sudo apt update was this history line?
The download says /home/u/x.deb; the file is under --root.
sudo, apt and the shell all name the file. Which links?
Its links must go too.
Identical sudo and apt records within a minute are one run (SAME_RUN_US). Each history file is walked newest first; a line takes the latest unused run that lies between its timed or already-anchored neighbours, and stays untimed when none fits.
The link's why says run k of n identical; placed by history order, and the time is derived. A command typed in a shell whose history wasn't saved yet can shift the match by one run; that limit is documented.
Inside the root first (a mounted image), then as recorded (a live --home beside an image --root). The file is hashed only if it is a regular file and not deny-listed.
A file is linked to every sudo or apt record whose normalized command names it as a whole argument (mentions_file). A shell line is used only when no logged record names the file: it is the weakest witness. Each install command is then linked to the package log lines that follow it within two minutes (INSTALL_WINDOW_US), matched by the .deb's real package name when dpkg-deb can read it.
Every other pass only adds; this one owns its relation outright, so it can delete it. Hits link events, artifacts or state rows to the iocs row with LinkType::Ioc.
Why the passes run in this order dependencies
Chains first: they only read visits. Downloads next, because they create and fill the artifacts the install pass needs. Installs before anchoring, because both read sudo and apt records but anchoring changes shell lines' times. Permissions read visits only. IOCs last, so they see every artifact the download pass just created.
24Staging: copy, then read
A browser holds its database open and locked. footprint never touches it: it copies the file and its side files into a private directory, and reads the copy.
Copying only History loses what the browser hasn't checkpointed.
src-<id>, 0700, so two profiles' History-wal never mix.
The copy opens read-only; a schema peek at a live original uses immutable=1.
The whole staging dir goes when the run ends, or on the first Ctrl-C.
What can still go wrong with a copy limits
- The browser can write while the copy runs; the copy may then be torn. SQLite reads what it can, and a broken copy fails that source with SQLite's error, never the original.
- A database larger than 4 GiB is refused rather than copied.
- A side file swapped for a FIFO is refused (
regular_file); the main file is checked the same way. - The provenance hash is of the copy that was read, which is what a reader can check against.
Chromium keeps History in WAL mode. Recent visits live in History-wal until a checkpoint moves them into the main file. The copy takes both, named <full name>-wal so SQLite pairs them, and SQLite reads the WAL together with the main file. The fixture keeps a History open in WAL mode while the test copies it, which is how this is proven.
The id is the sources row's, assigned during discovery, so it is unique in the run and known before the worker starts.
open_snapshot opens the copy READ_ONLY. XDM's discovery checks the live original's schema without copying it, through a URI with mode=ro&immutable=1 (open_immutable): SQLite then takes no lock and writes no side file. The path is percent-encoded first, because ?, # and % mean something in a URI (percent_encode_path).
The staging root is a tempfile::TempDir owned by run_collect; it is removed when the function returns, normally or with an error. The filesystem walk skips it (and the case's own files) so the run never records its own copies. A second Ctrl-C exits at once and leaves it behind (chapter 04).
25Getting it out: export and redaction
The reverse of collecting: the case leaves as a file someone else will read. It goes out filtered, in a chosen zone, and with secrets hashed if asked.
--tz (ExportTime).0600 — an overwritten file's old, permissive mode is tightened, not kept: JSON Lines, CSV or a standalone HTML report.| Format | Holds | For |
|---|---|---|
| jsonl (render_jsonl) | every field of ExportEvent, attrs included | scripts, jq, another tool |
| csv (render_csv) | the columns a person scans, formulas defused | spreadsheets |
| html (render_html) | metrics, IOC detections, the runs, a filterable timeline; no network | a report to hand over |
Two kinds of redaction don't mix them up
| At collection | At export | |
|---|---|---|
| Function | redact_secrets | redact_url, redact_text, redact_json |
| Applies to | shell/REPL commands, journald messages | every exported URL and credential |
| Replaces with | *** (gone for good) | <sha256:16 hex> (comparable, not readable) |
| Changes the case | yes: the secret never enters it | no: only the exported copy |
26Options, environment and defaults
footprint has no config file. Everything is a flag, a few environment variables it reads, or a constant in the code.
| Flag | Default | Does |
|---|---|---|
-c, --case PATH | footprint.db | the case file; global |
--tz ZONE | local | render times in local, UTC or ±HH:MM; global, honoured by every read command, tui and export |
--since, --until | no bound | the window (chapter 10 lists the forms) |
--sources, --exclude | all, none | kinds or families to read or skip |
--home DIR | the invoking user's home | repeatable; each home is read as its owner |
--all-users | off | every passwd user with uid ≥ 1000 and a real shell; root only |
--root DIR | / | where system paths live: a mounted image |
--assume-tz ZONE | the host zone at each date | the offset of logs that store local time |
--with-live, --with-scanners | off | opt-in collectors |
All defined in Cli and CollectArgs.
| Variable | Read by | Effect |
|---|---|---|
SUDO_UID | sudo_invoker, hand_case_to_sudo_uid | under sudo, read the invoking user's home and give the case to them |
USER | Env::current, the run row | the user name when not under sudo |
HISTFILE, ZDOTDIR, SHELL, HOME | ShellCollector | where the current user's shell history lives and which format it is; only for the home of the running user |
NO_COLOR | fp_tui::run | any value turns colours off |
LC_ALL=C | set, not read: run_listing | package tools print English month names and verbs |
| Constant | Value | Why |
|---|---|---|
| SCHEMA_VERSION | 2 | other versions are refused |
| MAX_TEXT_BYTES | 256 MiB | text evidence read whole, gunzip included |
| MAX_DB_BYTES | 4 GiB | SQLite evidence copied |
| MAX_SCAN_OUTPUT | 64 MiB | scanner output kept |
| PAGE_SIZE | 500 | rows per TUI page |
| SAME_RUN_US | 60 s | identical sudo/apt records closer than this are one run |
| INSTALL_WINDOW_US | 120 s | a package log line follows its install command within this |
| PRTIME_FLOOR | 1014 | Firefox endTime above it is µs, below it ms |
| OWNER_BATCH | paths per rpm -qf | keeps the argv far below the kernel limit |
| workers | 1–8 | available_parallelism, capped, never more than the jobs |
27Errors, warnings and what the user sees
There is no log file and no metrics: a run's health is stored in the case itself, per source, and printed by sources.
sink.warn(…) for what was skipped (bad lines counted once per file, an unreadable side file). The joined warnings become the source's note, and the CLI prints them as warning: label: ….
An Err or a panic stores error: … on that source and adds a line to the run's errors_json. The run exits 0 and says , N failed.
Bad arguments, a missing or foreign case, no selected collector: anyhow errors with context, printed by main, exit 1, before anything is written.
No .unwrap() outside tests (CI greps for it). Errors carry the path or tool in their context (with_context). The TUI shows an error in its status line and keeps running. Comments say why, in lowercase.
28Tests
189 tests, all hermetic: every one builds its evidence in a temp dir and never reads the host.
| Tier | Where | What it proves |
|---|---|---|
| unit (147) | mod tests at the bottom of each source file | parsers on real formats, time conversion, verdicts, filters, sanitizing |
| mutation smoke | crates/collect/src/smoke.rs (test-only) | 400 corrupted variants per parser: events or an error, never a panic |
| scenario | crates/cli/tests | the real binary over a synthetic machine: fake Zoom, state and diff, IOC and export, Firefox, a git hook, several users |
| read model | crates/query/tests | timeline, paging, detail and coverage over a collected case |
| TUI | crates/tui/tests | App driven by keys, and frames rendered into a test backend at 80×24 |
| CI | .github/workflows/ci.yml | fmt, check, clippy -D warnings, the no-unwrap grep, all tests on the pinned toolchain |
A test that reads /proc, the host's wtmp, its dpkg database or $SHELL passes on one machine and fails on the next. Build a synthetic root and pass it as --root or Env::test. External tools are faked with a small script written into the temp dir (utils::script_tool, copied so exec never races a writer).
A test reads like an investigation
One function writes a whole synthetic home and root in the real formats.
Real formats, fixed zones, no host, a regression test per fix.
setup, collect, events, footprint.
Deterministic corruption of seed files, without nightly fuzzing.
fake_zoom builds the scenario the tool was made for: a Brave profile whose History is held open in WAL mode, Preferences with camera and microphone grants, an XDM database, dpkg and apt logs (one rotated and gzipped), bash/zsh/fish/psql histories, a recently-used file. Times are given in the scenario's zone (SCENARIO_TZ, +03:30) through local. firefox_profile adds a Firefox profile with a profiles.ini.
- Fixtures use the real format, found on a real machine; a format guessed from documentation once made the snap collector read nothing.
- Zones are fixed with
--assume-tzand--tz; no assertion depends on the host zone. - No real personal data in the repository, ever.
- Every fix comes with a test built from the case that broke.
- Each milestone also gets a smoke run on the real host, into a temp case that is deleted afterwards.
footprint runs the built binary (CARGO_BIN_EXE_footprint) and fails the test with stderr when the command fails. Assertions read JSON Lines, never table text.
A seed file in each real format is corrupted 400 times by a xorshift generator: bytes deleted, inserted (from a list of separators, escapes, NUL and multibyte starts), overwritten, duplicated or truncated. Each variant goes through the real collector. The test only asserts that nothing panics; a failure reproduces from its round number.
29Recipes
The common changes, file by file.
1 · Add a collector collect
- A module in
crates/collect/src/with a unit struct implementing Collector: akindlikefamily.name, adiscoverthat sets availability, acollectthat reads throughread_evidenceorcopy_db_set. - Build events with new_event, a
Stamp, and anOriginwith a stable locator. per_user() → trueif it reads one home.- Register it in registry, and add its kind to a category in CATEGORIES.
- Unit tests on the real format; a seed in the smoke tests; a fixture if a scenario should cover it.
2 · Add an event kind core
3 · Add a persistence location state
- A
collect_*method on SystemPersistSnapshot or UserPersistSnapshot that queues files withemit_file_state; call it fromrun. - Use a kind under
persist.and a key that is the same in every run (the path), sodiffworks. - Read contents only through
open_regular/read_evidence.
4 · Add a command or a flag cli
- A variant in Command with a doc comment (it becomes
--help) and an args struct. - One arm in main and one function. Read through
fp_query, open withStore::open, print throughclean, offer--format, take the globaltz. - If the TUI should have it, add it to the read model first, then to App.
- A scenario test in
crates/cli/tests; the README's command table.
5 · Add a correlation pass correlate
- A module in
crates/correlate/src/with acorrelate_*function returning the number of links inserted. - Idempotent: links are unique on (src, rel, dst); if the pass must forget old links, own a relation and
delete_linksit first. - Join events across runs by file (
source_kind,prov_file), never bysource_id. - Call it from correlate in the right order and count it in CorrelateStats; teach
tracethe relation if it belongs in a provenance tree.
6 · Change the schema store
- Edit SCHEMA, the row type and its
from_row, and the inserts. - Bump SCHEMA_VERSION. Old cases are refused; that is a breaking change for users and needs the owner's approval and a
!commit. - Extend the store's tests (re-run idempotency, refusal of the old version).
30Invariants and gotchas
What must never break, and the traps that broke it before.
Copy with side files, open the copy read-only. Never Connection::open a live original.
read_evidence, open_regular, copy_db_set. A plain File::open skips the deny-list and blocks on a FIFO.
A locator built from a row index of an unordered query duplicated every re-run once.
Collectors fill Sinks; the runner ingests. Never hand a Store to a worker.
Never apply today's offset to an old local time; never compare local strings in SQL.
Nothing from the evidence becomes an argument except as a path, and nothing is ever run through a shell.
Including errors that quote evidence, and every TUI cell.
The staging dir and the case files are skipped by the filesystem walk; otherwise a run records itself.
Gotchas that cost a day each from the history
| Symptom | Cause |
|---|---|
| snap source empty on a real machine | the collector parsed a format invented from docs; snap changes has no kind column |
| every SUID binary "unowned" | /bin vs /usr/bin under merged /usr |
| a 9-minute collect | one dpkg-query and one dpkg --verify per file instead of per batch and per package |
| duplicates on every re-run | a locator built from a row index of an unordered query |
| a redirect without its chain | a pass joined on source_id, which differs per run |
| logins recorded under a strange user name | logind changed its message format in systemd 255 |
Every line footprint prints may end up in a report someone acts on. When in doubt, say less and say how sure you are: unverified over ok, derived over direct, untimed over a guessed time.
31Where it stands; what's next
M0–M6 and the full review that followed are done; new work comes from the ideas list.
Candidates in .stuff/IDEAS.md: declarative collectors (TOML specs), time ranges for lines between two anchors, flagging packages installed from a local file, counting independent witnesses per event, Trash and thumbnail-cache evidence, network configuration as state.
A new source is a collector (recipe 1); ranges change correlate_anchoring and add a precision; witnesses are a correlation pass plus a column in the read model; network state is a persistence location (recipe 3).
32The reference atlas
Every folder, file, type and function, read straight from the code, with a purpose note per file and crate. Browse the tree, or search every declaration at once.
Sources: 55 non-test files under crates/*/src, parsed with syn. Line counts leave out
#[cfg(test)] modules and items; the “tests” figure is the number of #[test] functions in the file's own test module.
Test-only files (smoke.rs, crates/*/tests) aren't listed. Enum variants are consts; methods show their owner, with the trait for trait impls;
route entries are the registry of chapter 09. Counts include private names.
33The data model, as a map
Every table of the case, starting from runs and growing outward to what each run writes.
Each box shows the fields that matter; click it for all of them. Pick a flow to watch the rows a feature writes, step by step.
sources.run_id, events.run_id, events.source_id and artifacts.first_seen_run say REFERENCES, but SQLite only enforces that with PRAGMA foreign_keys, which footprint never sets. links point at rows by type and id, with no constraint at all.
Copies of browser databases live only in the run's temp dir and are gone when it ends. The case keeps what was read, with the hash of the copy.
Every links row is derived from events, artifacts and state by a correlation pass; delete them all and the next collect rebuilds them (except anchored times, which stay on the events).
Reading the map in two minutes orientation
- The middle column is a run's life:
runs, thesourcesit discovered, and theeventsthey produced, with the FTS index beside them. - To the right, what a run keeps besides events:
stateper run, andartifactsby path. - Bottom, what correlation adds:
linksbetween any two rows, and the user'siocs. - Left, the dashed boxes are not tables: the evidence a source points at, and the staging dir it was copied to.
34The type map
The Rust types that carry the work, about forty of them. Each arrow is a function turning one into the next: hover a box or follow a flow to see its name, and click the name to read it. Boxes marked ⛁ are shapes the case stores.
Collectors return plain records in a Sink; the store returns plain rows (EventRow, EventDetail). Nothing outside the store holds a connection.
A Stamp becomes an Event with both its raw time and its UTC time; neither is ever thrown away on the way to the case.
35Where the rest is written
This page is about the code. The other documents hold the why and the plan.
What footprint does, every command, safety, limitations, roadmap. The only public document.
The design: time model, every source and its format, correlation, trace, safety rules, lessons from real machines.
The plan, milestone by milestone, with commit references and every bug found on the way.
Code style, comments, tests, the safety rules for working on a forensic tool, and the commit format.
Ideas for later, with review notes. Nothing there is promised.
.github/workflows/ci.yml and rust-toolchain.toml (1.97.1). This guide's own sources are in .stuff/guide/tools/codebase.
36What SQLite is to footprint
SQLite is both what footprint reads and what it writes: most evidence is somebody else's SQLite file, and the case is footprint's own. It is not a server and holds no state outside its files. The project uses rusqlite 0.34 with SQLite 3.49.1 bundled and compiled in (bundled-full), so FTS5 is always there and the host's SQLite never matters.
The same SQLite on every machine, compiled into the binary.
How a database is read without being touched.
Why a copy needs its -wal, and what the case writes beside itself.
user_version and table_info before anything is trusted.
Every crate that touches SQLite asks for rusqlite = { version = "0.34", features = ["bundled-full"] }. The C library is compiled into the binary, so FTS5 (behind search), JSON functions and the URI syntax are always available, and an old system SQLite can't change behaviour. The case's events_fts is an external-content FTS5 table kept in step by three triggers.
SQLITE_OPEN_READ_ONLY on the staging copy means a bug can't write evidence even to the copy. For a live original, read-only isn't enough: SQLite would still take a shared lock and may create -shm. The URI parameter immutable=1 tells it the file can't change, so it takes no lock and creates nothing; footprint uses that only for XDM's schema peek, never for reading rows.
In WAL mode, committed rows sit in NAME-wal until a checkpoint copies them into the main file. A browser that is running rarely checkpoints, so the newest history is often only in the WAL; copy_db_set copies it under the same full name so SQLite pairs it with the copy. A rollback-journal database may have a NAME-journal, copied the same way.
The case uses the default rollback journal, so while a transaction is open there is a case.db-journal beside it. The filesystem walk skips all four of the case's names.
PRAGMA user_version is how the case knows its own shape (Store::init). pragma_table_info is how a collector checks someone else's: table_has_columns runs before a query that names columns, so a different app version is unsupported, not a crash.
37How footprint uses it, and inspecting a case
Four crates use rusqlite; only two of them may run SQL against the case.
Connection, opened NO_MUTEX with a 5 s busy timeout, every query in schema.rs.SELECTs on them; it never sees the case.Store, and inserts a placeholder artifact through store.conn() in the download pass.Connection (one per database), Statement from prepare or prepare_cached, Rows/Row with typed get, Transaction from conn.transaction(), OpenFlags, and params!/params_from_iter for bound values. The store maps rows into its own types (EventRow, EventDetail…) and never returns a Row.
chromium_history creates the tables Chromium has (HISTORY_SCHEMA) and, with hold_wal, switches to WAL, turns autocheckpoint off and returns the open connection, so the test copies a database whose rows are only in the WAL, as on a running browser. xdm_db and firefox_profile do the same for XDM and Firefox.
prepare_cachedfor statements run per row (ingest);preparefor one-off reads.- Bind every value with
?N; build SQL text only from constants (build_events_query adds clauses, never values). ON CONFLICT … DO NOTHINGfor idempotent inserts,DO UPDATE SET … COALESCEto fill gaps without erasing.LIKE … ESCAPE '\'with escape_like whenever user text meetsLIKE.
Use -readonly; a case is evidence of an investigation too. For anything a user will see, prefer the commands: they sanitize.
- Never open a live evidence database without staging it, and never read-write: a write, a lock or an automatic checkpoint changes evidence.
- Never
VACUUManything that isn't the case. - A query that names columns of someone else's database is preceded by
table_has_columns. - SQL outside
crates/storeand the collectors' evidence reads is a design break; the onestore.conn()use in correlation is the only exception today. - FTS5 query syntax reaches SQLite as typed in
search; a syntax error is SQLite's message.
The case's own SQL, in one place schema.rs
Every table is in SCHEMA; every insert is a constant beside it (INSERT_EVENT, INSERT_STATE, INSERT_ARTIFACT).
Never point Connection::open at the user's original evidence. Copy it with copy_db_set into the run's staging dir and open the copy with open_snapshot. Opening the original, even to read, can take a lock, write a -shm or trigger a checkpoint in the browser's file.
38Check yourself
Twelve questions. Each answer is a few lines of code you have now read.
A second collect over the same window reports “0 new events”. Which two strings made that possible?
events.dedup_key is UNIQUE and the insert is ON CONFLICT DO NOTHING.Why are source rows inserted during discovery, before any worker runs?
A parser panics on a crafted history file. What happens to the run?
catch_unwind; that source is stored as error: collector panicked: …, the run's errors list it, and every other source is collected and correlated as usual.A FIFO is planted as /etc/cron.d/backup. Why doesn't the snapshot hang, and what does the user see?
not_regular_file; content reads go through open_regular, which opens non-blocking and checks the type on the descriptor.footprint state says unverified for a file on a mounted image. Why not ok?
/, because the host's dpkg --verify would check the host. integrity returns unverified whenever no report could be produced.Where does --host zoom.us become SQL, and why doesn't it match notzoom.us?
subject_host = 'zoom.us' OR subject_host LIKE '%.zoom.us', with the host passed through escape_like. The dot before the suffix is what excludes notzoom.us.Five identical sudo apt update lines are in a bash history without timestamps; the journal holds two. What gets anchored?
run k of 2 identical; placed by history order.A redirect was collected today; the visit it came from was collected last week. Why is the chain still linked?
source_kind, prov_file) and visit id, not by the run-local source_id, which differs between the two runs.An IOC is deleted. When do its [IOC: …] badges and links disappear?
ioc_match link.What keeps an apt log line written at 21:08 local time from shifting by an hour when read in winter?
TimeFormat::LocalNaive goes through resolve_local, which asks the zone for the offset in effect at that date, not today's. --assume-tz replaces the zone with a fixed offset for images from elsewhere.Under sudo footprint collect, whose shell history is read, and who owns the case?
SUDO_UID and passwd to make the invoking user the target, so their home is read; hand_case_to_sudo_uid chowns the case to them.A CSV export has a summary starting with =HYPERLINK(…). What does the spreadsheet show?
= + - @, tab or CR with ', then quotes the cell.39What could be built next
The full list with rationale lives in .stuff/IDEAS.md; it moves into TASKS.md when work starts. These are the standouts, ★ marked there too.
Most tools report what they found. footprint verdict would report what was checked and found nothing, per attack stage, with the coverage behind each claim: "No execution evidence: shell history, sudo log, dpkg log and download dirs checked."
Map events onto stages — lure → delivery → execution → persistence → access/exfil — and show where a chain stops and why.
Offline: edit distance, homoglyphs and brand-in-subdomain against a data-driven brand list, plus first-seen hosts and webmail-click redirect chains.
inspect: pre-flight checkfootprint inspect <repo|archive> — static only, never executes: live git hooks, lifecycle scripts, typosquats, curl|sh, obfuscated JS. A risk summary before you check out job material.
Per event, how many independent sources agree (sudo journal, apt block, dpkg lines, bash line, ctime) — and which expected witnesses are missing.
Flag every package installed from a file (apt install ./x.deb, dpkg -i) — exactly how the fake-Zoom SDK got in — and pair it with the trace.
Trash .trashinfo files, thumbnail-cache tEXt chunks, and carving SQLite free pages — on the staging copy only, so the original is never touched.
Declarative TOML collectors (same deny-list and guarded helpers enforced by the engine), an IOC exchange format, and footprint pack: a self-contained, verifiable evidence archive.
verify (re-hash sources against the case), baselines and drift, story mode (a plain-language narrative with every sentence linked to events), clock-skew and history-tampering hints, extension permission drift.
No telemetry, no cloud upload, no online reputation by default; secret contents are never read, not even "just to check".
.stuff/guide/tools/codebase. Rebuild at every docs update.