Configuring compaction#

The logd overview says what compaction is: a logarithmic retention schedule that keeps every delta within a recent cutoff and thins older history down to snapshots. This page is how it is turned on, what each knob does, and — the part that surprises people — what cannot be asked of it.

It is off unless the config asks for it#

Compaction is the one section that does not default on. A config file with no compaction: section, or no config file at all, means compaction never runs and the full delta history is retained forever.

That is the opposite of snapshot:, which defaults to maxCommits: 1000 and maxBytes: 4194304 (4 MiB) even when nothing is configured.

An empty section is not nothing:

compaction: {}          # compaction ON, every knob at its default

Every field defaults when it is absent or zero, so compaction: {} and a section naming all five defaults are the same config.

It runs when a snapshot runs, and never otherwise#

Compaction is not scheduled. It is the last step of taking a snapshot:

a commit lands
  → thresholds checked (maxCommits, maxBytes)
    → active log switched, so writes keep landing in the new one
      → baseline snapshot written
        → the now-inactive log compacted

Four consequences follow from that, and they are the ones worth knowing before tuning anything:

• A store that stops taking writes never compacts. No commits means no threshold crossing, means no snapshot, means no compaction — however old its history gets. There is no timer and no idle sweep.
• There is no way to run it by hand. No o subcommand, no request in the session protocol. If it has not run, the only lever is writes.
• It can never run more often than snapshots do. Lowering cutoff below the snapshot interval does not make compaction happen sooner; it only changes what the next compaction keeps.
• Turning snapshots off turns compaction off, whatever compaction: says. A deliberate snapshot: {} disables both thresholds, so nothing switches logs and nothing is ever compacted.

Only the inactive log is compacted. The log currently being written to is never touched, so recent history is bounded by the snapshot policy rather than by anything here.

The knobs#

compaction:
  cutoff: 1h
  baseInterval: 1h
  slotsPerTier: 8
  multiplier: 2
  gracePeriod: 5s
  horizon: 2y

Every field may be left out, and a field left out takes its default. The server logs the policy it resolved, whole, so what a store says it is compacting with is what it is compacting with:

level=INFO msg="configured compaction" cutoff=2h baseInterval=1h slotsPerTier=8 multiplier=2 gracePeriod=5s horizon=0s

The horizon#

Without one, the tiers keep some snapshot from every era indefinitely, ever more sparsely. That is the right default for history, and the wrong answer for a record that retention has deleted: the delete removes it from the current state, but every older snapshot that survives still carries it, so it never leaves the disk. horizon is where history ends. It must be at least the cutoff, since the cutoff keeps every record; the newest snapshot in a file is kept whatever its age, because it is the state.

Durations#

cutoff, baseInterval, gracePeriod and horizon are written the way a duration is written — 1h, 90m, 30s, 500ms — which is what time.ParseDuration reads, plus d, w and y for 24h, 7d and 365d, calendar-blind (2y, 1y6w). A bare number is refused rather than guessed at, so there is no unit to remember and no 3600000000000 to get wrong.

What a compaction removes#

Not state — memory. Compaction does not undo the effect of the records it removes. Immediately before compacting, SwitchDLog writes a full baseline snapshot of the state at the switch commit, and that snapshot is moments old, so it always falls inside the cutoff window and always survives. Everything the removed records contributed is in it. A record that should leave the state — a job long done, an event a year old — is not compaction's to remove; that is retention, which deletes it as a write.

What is removed is the account of how the state got there. The store keeps knowing what it holds; it stops knowing which deltas built it. Concretely:

• reading the current state is exact, always, at any retention setting;
• reading a historical commit below the cutoff is approximate — it lands on the first surviving snapshot at or after the commit rather than being replayed to it, and the answer's commit names that snapshot's, so the client can tell (see Session);
• a client resuming a watch from a commit below the retained delta history is refused with replay_compacted rather than served a gap, and one resuming above it from a state that is no longer exact starts at the snapshot that stands for it.

Delta records are kept if the record's timestamp is newer than now - cutoff. A record whose timestamp will not parse is kept: the doubtful case keeps history rather than losing it.

Scope records are kept regardless of cutoff, until the scope itself is deleted, and here the reason is a stronger one — for a scope, the records are the state. Baseline snapshots are written for the root only; scope snapshots are deliberately not created, because a materialized scope layer resolves !key away and is unsound to re-apply onto a changed baseline, so a scoped read replays the raw op-preserving records in full. There is no coarser thing to fall back to, and dropping one would lose state rather than history. Bounding this is tracked in 5hmq80f3h12krh1mbsn0; until then a long-lived scope's record set grows without limit and no setting here changes that. What it costs a read is bounded separately: a read that names a path replays only the records bearing on that path, not the scope's whole history.

Snapshots are bucketed by age into tiers past the cutoff. With the defaults, measuring from the end of the cutoff window:

Within a tier the survivors are the most recent ones, not a spread across the tier, so a tier thins from its old end first.

The schema needs no snapshot kept for its sake. A schema change is a commit of its own, which compaction never drops, and every root snapshot records the schema in force at its commit; whichever survive carry it.

What cannot be configured#

• Zero does not mean zero. Every field treats an absent or zero value as "use the default", so there is no way to say keep no delta history (cutoff: 0s is one hour) or no grace period at all.
• A misspelled field is ignored. cutof: is not an error; it leaves cutoff at its default. Same class as a misspelled request field, filed as k0d4y1m6h12kr7cdgdn0. A value the policy refuses, on the other hand — multiplier: 1, a horizon inside the cutoff — is refused when the file loads, not inside every compaction attempt.
• There is no per-path or per-scope policy for history. One schedule covers the whole store. Per-path ageing of state is a different thing, and is a request rather than a setting: retention.
• It cannot be undone. The delta history a compaction removes is gone; the state it described is not, and never was at risk. What cannot be recovered afterwards is the ability to read or replay a commit below the cutoff exactly.
• A reader can be cut off. After gracePeriod the pre-compaction file is deleted; a read still holding it errors.

A worked example#

Keep a day of exact history, then thin fast, on a store that commits often enough to snapshot several times an hour:

snapshot:
  maxBytes: 4194304     # 4 MiB of delta since the last snapshot; naming the
                        # section leaves maxCommits at 0, which turns the
                        # commit threshold OFF
compaction:
  cutoff: 24h           # a day of exact history
  baseInterval: 24h     # then 24h-wide tiers
  slotsPerTier: 4
  multiplier: 4
  gracePeriod: 30s      # for slow readers

Point logd at it with -config:

o system logd serve -data ./data -config ./logd.tony

o system compose -config takes the same file.

Writing a section is how a threshold is turned off deliberately: a snapshot: that names only maxBytes leaves maxCommits at zero, and zero is disabled. Only a section left out entirely gets the defaults.

Verify what was actually read by the numbers logd prints on the way up:

INFO snapshot policy maxCommits=0 maxBytes=4194304