Each DAO operation on the same object generates a journal entry containing just the changed fields. Over time a single object may have hundreds of entries, slowing replay since replay time is proportional to journal line count. Compaction writes each object to a new journal file in its entirety, reducing many entries to one and dramatically improving replay time.
Compaction does not decide what data should exist – it writes whatever the MDAO currently holds. Archiving and TTL are separate, and are expressed as a removeAll() on their own schedule, or by dropping a partition directory. See Deciding what survives.
Compaction follows a five-step process:
+--------------------------------------------------------------------+
| CompactionCmd -> JDAO.cmd_ -> Compactor |
| |
| 1. ROLL foo renamed to the next generation, foo.N |
| | A fresh foo takes live traffic immediately |
| | |
| 2. SNAPSHOT Read all objects from MDAO (a functional |
| | snapshot -- later writes cannot disturb it) |
| | Write full copies to |
| | foo.N.snap.gz.tmp, beside live traffic |
| | |
| 3. COMMIT Close (writes the gzip trailer), then rename |
| | into place. This is the only commit point. |
| | |
| 4. CLEAN UP Delete generations <= N, now superseded |
| | |
| 5. COMPLETE Log statistics, append report to the command |
+--------------------------------------------------------------------+
Nothing is blocked at any step. Reads and writes continue throughout. See JournalFiles.md for the naming, the ordering guarantee and the crash windows.
Journal files after compaction:
Before: After first compaction:
users (large, many users (new traffic only)
delta entries) users.1.snap.gz (compacted state)
The roll step is an atomic rename, run as a job on the journal’s own assembly line so that it is ordered against every put and remove without blocking them. A rename moves no data, so the cutover costs the same at any journal size. (It used to be a copy + truncate, because a rename leaves an open writer pointing at the renamed file; the line lets the writer be closed first and reopened after.)
Key point: live traffic never stops. It writes to the fresh users from the instant of the rename, while the snapshot is built separately.
Compaction is aware of .0 (deployment) journals. During step 2, objects that are identical to their .0 version are skipped – they don’t need to be in the runtime journal since .0 is replayed on every startup.
This significantly reduces the compacted journal size when most data comes from deployment:
Before compaction:
users.0 (22,000 objects from deployment)
users (50,000 delta entries for 926 modified objects)
After compaction:
users.0 (22,000 objects, unchanged)
users.1.snap.gz (926 entries -- only modified/new objects)
users (new traffic only)
.0 into a temporary in-memory MDAO.0 version.0 provides it on startup)remove entry to prevent resurrection on next startup. That covers both a row removed outright and a LifecycleAware row soft-deleted to DELETED: the latter is still in the MDAO, so find_ sees it, while discardLifecycleDeleted keeps it out of the snapshot. Without the remove, .0 would bring it back – a snapshot supersedes the generations, never .0This optimization is automatic when a .0 file exists. It works with both filesystem and JAR-embedded .0 files. If no .0 file is found, compaction proceeds normally (all objects written).
| Setting | Default | Meaning |
|---|---|---|
compactible |
true |
All DAOs are compacted by default |
discardLifecycleDeleted |
true |
Objects marked DELETED are not written to the new journal |
keepSupersededGenerations |
true |
Keep the generations a snapshot supersedes. True while compaction is being proven |
If no Compaction record exists for a DAO, default settings are used: the DAO is compacted with lifecycle-deleted objects discarded.
FOAM’s root pom.js builds the compaction code (src/foam/dao/compaction/pom), and a CompactionCmd works without any configuration: when no compactionDAO is present, default settings are used. The DAOCompaction script, its parameters and per-DAO configuration come from the deployment/compaction journals. To include them, add them to the -J flag (from an application that has foam3 as a sub-directory):
./build.sh -J../foam3/deployment/compaction,...
The compaction deployment provides:
| File | Purpose |
|---|---|
services.jrl |
Registers the compactionDAO service for storing per-DAO configuration |
compactions.jrl |
Per-DAO compaction configuration entries |
scripts.jrl |
The DAOCompaction BeanShell script |
scriptparameters.jrl |
Script parameter with the daos list |
Without these journals, the compaction script and compactionDAO service will not be available.
Compaction is configured per-DAO in deployment/compaction/compactions.jrl.
A DAO is compactible by default. To explicitly configure it:
p({
class: "foam.dao.compaction.Compaction",
cSpec: "userDAO"
})
Set compactible: false. The DAO is skipped entirely: the journal is not rolled and nothing is rewritten.
p({
class: "foam.dao.compaction.Compaction",
cSpec: "alarmDAO",
compactible: false
})
| Property | Type | Default | Description |
|---|---|---|---|
cSpec |
Reference | (required) | CSpec ID of the DAO to configure |
compactible |
Boolean | true |
If false, compaction skips this DAO: no roll, no snapshot |
discardLifecycleDeleted |
Boolean | true |
Discard objects with lifecycleState=DELETED |
keepSupersededGenerations |
Boolean | true |
Keep superseded generations instead of deleting them. Defaults true while compaction is being proven, so a bad snapshot never destroys what it was built from. Costs disk only – replay skips them by name, so startup time is unaffected and they can be gzipped in place |
journalName |
String | auto | Journal filename. Defaults to CSpec name with “DAO” removed + “s” |
Compaction is a command sent through the DAO stack, so it reaches whichever JDAOs are underneath: one for an ordinary DAO, one per partition for a partitioned one. Each appends its own report to the command.
Compaction is controlled by the DAOCompaction Script and its ScriptParameter.
DAOCompaction ScriptParameter:
parameters.daos to a comma-separated list of DAO service namesRun the DAOCompaction script from the Scripts admin menu
FixedSizedDAO, only retained records are compactedimport foam.dao.DAO;
import foam.dao.compaction.CompactionCmd;
CompactionCmd cmd = new CompactionCmd();
cmd.setServiceName("myDAO");
((DAO) x.get("myDAO")).cmd(cmd);
// cmd returns once the rewrite is queued; wait for the report.
cmd.awaitCompletion(60L * 60L * 1000L);
print(cmd.getReport());
If you only need to back up the current journal and start a fresh one:
From a BeanShell script, where x is already in scope, this is a one-liner:
x.get("somethingDAO").cmd(new foam.dao.FileRollCmd());
Or, keeping the result to read the archive name back:
import foam.dao.FileRollCmd;
DAO dao = (DAO) x.get("myDAO");
FileRollCmd cmd = new FileRollCmd();
cmd = (FileRollCmd) dao.cmd_(x, cmd);
// cmd.getRolledFilename() = "mymodel.1"
Warning: Roll-only does NOT compact. The frozen generation retains all delta entries, and is replayed on every startup. The new journal only contains writes after the roll.
Roll-only is safe on a live system: the cutover is ordered against journal writes by the assembly line. See JournalFiles.md.
Compaction writes whatever the MDAO currently holds. Deciding what should be in it is a separate concern – run it on its own schedule and compaction will reflect it the next time it runs.
| policy | how |
|---|---|
| date TTL | drop the partition’s directory (see JournalFiles.md) |
| age, status, anything else | a CRON calling removeAll() with the predicate you want |
| lifecycle-deleted tombstones | discardLifecycleDeleted (below) |
Compaction used to take a predicate, createdSince or lastModifiedSince and
drop non-matching objects as it rewrote. Those are gone, along with the
per-model {ModelId}CompactionSink faceted lookup. Two reasons:
removeAll() deletes when the policy says to, writes
r() entries, fires events, and is visible to readers immediately.removeAll_ already
does; the filters carried a second copy plus config plumbing.The cost of a removeAll() is one journal entry per removed row, and those land
in a generation the next compaction supersedes and deletes – transient disk,
not permanent.
discardLifecycleDeleted stays, because it is not substitutable: for a
LifecycleAware object, LifecycleAwareDAO.remove_ converts a remove into a
setLifecycleState(DELETED) put (LifecycleAwareDAO.js:144-147). A CRON
removeAll against the served DAO therefore re-marks tombstones rather than
purging them; doing it properly means aiming below that decorator.
There is no rollback procedure, because a failed compaction is not a state to recover from. The snapshot only becomes visible at the rename; until then the frozen generations are still the record, and live traffic has been landing in the current journal the whole time. A failure leaves a .tmp that the next boot ignores and anything may delete.
If compaction throws, check the command’s getError() (printed by the script) or the log for the cause and run it again. Nothing is lost and nothing needs restoring.
A committed snapshot supersedes the generations it covers, and cleanup deletes them, so there is nothing to revert to afterwards. To keep that option, take a copy of the generation files before running compaction – they are ordinary files and can be restored by putting them back and deleting the snapshot.
Each journal compacted appends a human-readable report and a CSV line to the CompactionCmd (getReport(), getCsv()). The CSV is also written to the log.
The human-readable report printed by the DAOCompaction script:
Compaction Report
Instance: localhost
Date: Mon Feb 13 16:07:51 GMT 2026
Duration: 5s
Objects processed: 22926
Objects compacted: 926
Objects filtered: 95.96%
Journal entries: 160000 -> 926 (99.42% reduction)
Journal size: 48.0 MB -> 1.2 MB (97.50% reduction)
Removed (.0): 3 (deleted at runtime)
| Field | Description |
|---|---|
| processed | Total objects read from MDAO |
| compacted | Objects written to new journal |
| filtered | Percentage of processed objects not written: rows identical to .0, plus lifecycle-deleted tombstones when discardLifecycleDeleted is set |
| journal entries | Line count before vs after compaction |
| journal size | File size before vs after compaction |
| removed (.0) | .0 objects deleted at runtime (remove entries written); shown only when non-zero |
Roll-only does NOT compact – FileRollCmd only freezes the journal as a generation. Send a CompactionCmd for full compaction.
Compaction is async – The snapshot runs on the thread pool and cmd returns once it is queued. Call CompactionCmd.awaitCompletion(timeoutMs) to wait for the report.
One compaction per journal at a time – a CompactionCmd for a journal that is already compacting is skipped and reported on the command (getSkippedCount(), getSkipped()), not queued and not an error. Overlapping triggers are expected; the next run picks it up.
Live traffic during compaction – Writes go to the fresh journal from the moment of the cutover and are never at risk; the snapshot is a separate file that only becomes visible once complete.
MDAO is the source – Compaction reads from MDAO, not the journal file. The in-memory state is what gets written.
FixedSizedDAO interaction – If the DAO stack includes a FixedSizedDAO, only objects retained by it are compacted.
| File | Purpose |
|---|---|
foam/dao/compaction/Compaction.js |
Configuration model |
foam/dao/compaction/CompactionCmd.js |
Command sent through the DAO stack; config in, report out |
foam/dao/compaction/Compactor.js |
Roll + snapshot for one JDAO’s journal |
foam/dao/compaction/LifecycleDeletedCompactionSink.js |
Discard lifecycle-deleted tombstones |
foam/dao/FileRollCmd.js |
Command to trigger journal roll |
foam/dao/AbstractF3FileJournal.js |
Journal roll implementation |
foam/dao/JournalGenerations.java |
Derives the generation set from the filesystem |
doc/guides/JournalFiles.md |
Journal naming, numbering and rolling scheme |
deployment/compaction/services.jrl |
compactionDAO service definition |
deployment/compaction/compactions.jrl |
Per-DAO compaction configuration |
deployment/compaction/scripts.jrl |
DAOCompaction script |
deployment/compaction/scriptparameters.jrl |
Script parameters |