Scale budget

This page turns the scale targets into reproducible experiments. All time and memory figures are acceptance targets; this page contains no executed results for a million files, 1 TB or 100 GB. Existing short regression results cannot be extrapolated into a pass of this page.

Fixed inputs and capacity

• Use reference devices with recorded commit, release build command/features, configuration, CPU model/frequency, OS/filesystem, physical RAM, disk capacity/free bytes and container/VM limits. First measure sustained read/write speed and link throughput/RTT without CipherFS and confirm they meet the requirements; a test container limited to 3 CPUs/6 GiB is not the 8 CPU/16 GiB Full reference device. The mobile-class configuration must be measured on its own; runs on a powerful host do not prove the phone budget.
• Fixed million-file fixture: keys are d/{i // 1000:03}/f{i:06}.bin, i=0..999999; current bodies total 900,000,000,000 bytes, file 0 is 100,000,000,000 bytes, and the remaining 800,000,000,000 bytes are split evenly with divmod(800000000000, 999999), the first remainder files getting 1 extra byte each. File 1 has 769 older versions and files 2..99232 one older version each, each older version 1,000,000 bytes, for 100,000 unretired older versions; bodies total exactly 1,000,000,000,000 bytes. Older versions are written before current versions, mtime uses fixed integer milliseconds; commit order, seeds, FileId/VersionId, heads, frontier and each version's size/digest go into the manifest. Fixed 10 members (2 Full, 8 Partial); each Partial's pinned-directory bodies are at most 8 GiB, with read_cache_max_bytes=8589934592. Initially no retirement and no GC; sparse files, all-zero data, duplicate blocks or hard links may not stand in for data volume.
• Fixed mtimes are injected through Core's Clock: create the vault with Vault::create_with_clock, and use Vault::set_clock for existing/reopened vaults. After member creation/joining completes, take the maximum time_ms of those records plus 1 as base B and save it; before the n-th body write set now_ms()=B+n and keep it unchanged from that reader write until commit. prepare_write records mtime directly from that value and commits follow the manifest order; set_mtime may not publish another version with the same body, which would change the unretired version total above. Fixture clocks on every participating node are no earlier than accepted records and stay at the last value until the end, so that retirement/GC cannot change the input; wall time is still measured with a real monotonic clock. The three 100 GB files also get distinct, increasing mtimes this way.
• Content comes from a replayable deterministic pseudo-random stream, with a different seed per file/version. A standard SHA-256 counter stream suffices: the seed is the UTF-8 encoding of cipherfs-scale-v1:{i}:{version}, where version is the file's content version index starting at 0; output SHA256(seed || counter.to_be_bytes(8)) in turn, with counter increasing from 0, truncating the last block to the declared length. The generator buffers at most 64 KiB and computes the whole-file SHA-256 incrementally at the same time. The manifest records the seed algorithm and expected digests; incompressibility is checked with streaming compression and the compressed byte count recorded separately; compressed/deduplicated sizes may not stand in for body size.
• Reserve actual space as 1.2 × (CAS of all devices + store packs + local index/logs + retained plaintext input/export + maximum concurrent staging), adding measured/estimated bytes item by item and rounding up. A Full device's CAS covers all unretired versions, not just current files; history, ciphertext tags, metadata and uploaded objects awaiting GC all count. If the million-file fixture's 2 Full devices and 1 store share a disk, the body floor is 3 TB, plus the cache/pinned overhead of the 8 Partial devices, then 20% headroom. A 256 GB Partial keeps only the complete index and bodies within its budget.
• The standalone 100 GB fixture uses the streaming source below and keeps no fourth plaintext copy: the three body copies — source CAS, store packs, Full target CAS — have a floor of 300,000,000,000 bytes; prepare at least 360,000,000,000 free bytes and satisfy the overhead formula above. Digests are computed through reader/writer streams; if the 100 GB plaintext or an export file is stored as well, the body floor becomes 400,000,000,000 bytes, with overhead and 20% headroom still counted separately. The three root directories may not nest, share CAS, hard-link or reuse the target cache; a same-disk Directory run is labelled as a storage baseline and is not a 1 Gbit/s network result.

Timing and measurement

Pre-generated initialization information/member approvals and input manifests are not included in sync timing; the source's publish to the store is timed in the 100 GB experiment. The password is read from secure input and unsealing cost counts toward startup; time spent typing a password does not. Generating the million inputs during timing, or folding vault creation cost into the first index sync, is not allowed, but these preparation phases must also be timed separately. Every run records UTC start/end and monotonic wall time; multi-phase runs may not reset the total timer to hide an overrun.

Each independent scenario runs 3 times cold and all runs must meet the budget; warm reopen/query runs are done 3 more times and reported separately; the faster run may not stand in for the cold one. A cold first sync must use a new target directory; the OS caches of the source and store are cleared as well, and the method, sampling interval and whether caches were actually cold go into the evidence; when that is not possible the result can only be labelled warm.

Each device's main process and all worker/child processes form a process tree, and all threads and concurrent body work count: sample simultaneous RSS at intervals of at most 100 ms and sum it, supplemented by OS high-water/peak counters; when a child exits, keep its final value, and report short peaks that sampling may have missed. For 10 concurrent devices, report the sum of simultaneous per-tree RSS, capped at 5 GiB (5,368,709,120 bytes). Shared pages may be counted twice in RSS and are not subtracted. Store/S3 services, the load driver and the OS page cache report their actual peaks separately and do not count toward the 512 MiB client tree, but the machine as a whole may not swap or OOM.

Live heap uses allocator counters covering all threads or equivalent instrumentation, recording the overlapping peak of live allocations/reallocations and the tool's overhead; runtime, database caches, index/unit references/selection metadata are all included. RSS and live heap must each stay within 536,870,912 bytes and cannot substitute for each other; without a heap measurement the memory budget cannot be declared passed. Raw samples are kept for start/stop phases, page-scan CPU and periods when index and body background work run together.

100 GB through a store: steps

Run a clean round each for Directory (Range), S3 loopback (normal Range, and forced Range-ignoring with HTTP 200), and real S3 (normal Range; plus the equivalent fallback proxy when available). Loopback only proves the transfer path; time budgets are accepted only in a network/shaping environment that meets the reference link. When real accounts or long-running scenarios have not been executed, say so as not accepted; do not merge them into a single pass.

1. Source import, ≤1,800 s. In an independent source Full vault, write large.bin with put_from_reader/Core write_from_reader, generating exactly 100,000,000,000 bytes from the stream above with i=0, version=0, hashing incrementally with a fixed mtime; timing runs from the first generated byte to a successful write commit. This step connects to no store/peer and runs no publish background task, so early uploads cannot be subtracted from step 2. The source CAS must really be on disk; close, reopen and stream-read it back to check size/digest. That reopen check is timed separately and does not replace the final target verification of the whole run.
2. Store publish, ≤3,600 s. First write small/a.bin and small/b.bin, 65,536 bytes each, from the streams with i=1 and i=2, version=0, with increasing mtimes, committed after the large file, so step 4 can verify newest-first fetching; this preparation counts toward the whole run. The publish step runs from the source first starting store sync until every body pack, locator, small tail pack, related index/authority dependency and commit head has arrived and can be authenticated; it does not end at the first head upload. Save the object list, pack lengths and actual upload order, and confirm that the small pack and its dependencies are fully available before the receiver accepts them.
3. Empty Full target index, ≤300 s. The target has only initialization information and an approved identity, no database/CAS/body cache; the source device/P2P sources are shut down and the target syncs through the store only. Timing runs from sync start to files/versions/sizes/mtimes/heads/exact frontier all matching. When the index becomes visible, record the locally cached bodies and the units still missing, verifying index first and bodies after; copying the source database directly is not allowed.
4. Bodies, ≤3,600 s. From index convergence until all unretired bodies are held and missing_units=0. Record progress per batch/version and verify newest-first fetching. A separate interruption round stops the target after at least 1 GB has been persisted and before the bodies are complete (covering both explicit cancellation and process termination), saves missing_units (including units not yet attempted), and reopens the same target to resume; units are deduplicated by authenticated unit hash, completed units are not written/counted twice, and no unit may be under-reported, duplicated or published corrupt. Network retries/whole-pack fallback may resend wire bytes, counted separately. Resumed active time is still bound by this step's budget; planned downtime is listed separately and wall time including downtime is reported too; the whole-run ceiling applies to a round without injected interruption.
5. Target verification, ≤1,800 s. Close and reopen the target and use get_to_writer/Core read_to_writer to send the bodies into a counting + SHA-256 writer that outputs to a digest sink rather than a fourth 100 GB file; verify the full 100,000,000,000 bytes, SHA-256, mtime, each version's heads and the frontier, and check that the source and target selected the same versions. The middle/end ranges of the large file are also checked against the expected stream with get_range_to_writer; when verification fails, the authenticated prefix already output is discarded and not treated as a complete file. Record every byte of this step and the body missing count.

The small versions and required metadata take extra capacity/time; the whole run from step 1 to step 5 takes at most 14,400 s. The step ceilings add up to 11,100 s; the rest of the margin covers close/reopen, source verification and similar operations, whose time is not hidden. Interrupted and uninterrupted rounds are reported separately; the time budget is called passed only after all uninterrupted cold runs pass.

Each round reports source_plaintext_bytes, source_cas_bytes, store_ciphertext_bytes, wire_bytes (including HTTP/QUIC protocol, retries and fallback; state the basis when it cannot be captured), target_cas_bytes, temporary_peak_bytes and missing_units, plus AEAD tags, index/pack locators and filesystem allocated bytes. Record source/target and server concurrency, fetch batch configuration, page size and per-phase peaks; logical file length may not stand in for actual disk or transfer volume. Record authenticated misses and not-arrived retries; no missing units may remain at the end.

Window boundaries and verdict

Record the following distinct limits against the actual source code; changing a configuration requires rerunning the corresponding budget, and they are not added up as proof of whole-process peak:

Any of the following is a failure: wrong digest/size/mtime, missing or stale heads/frontier, silently accepted partial bodies, inaccurate missing counts, corrupt units/wrong packs published, capacity or reference environment not met, swapping/OOM, or any memory/time budget overrun. Failures keep their output and exact version; the fixture may not be deleted and scale claimed through a short regression. Hitting the authority proof/snapshot limits is also recorded as a scale failure and is not exempted because body streams are bounded.

Evidence includes at least the exact command argv (credentials redacted), environment/build and configuration, commit/fixture manifest hash, start/end/exit code of each phase, raw RSS/heap/CPU/network/disk samples, byte accounting, repeated runs and interruption/resumption records, the final content/metadata comparison, and an explicit passed/failed/not executed status. The current 512 MiB local stream and 96 MiB remote window regressions each prove only their own case; they do not prove the 100 GB, whole-process 512 MiB or million-file targets. The long-running experiments defined here belong to the scale acceptance in the backlog; a documentation check does not count as execution.

Source file in the repository