Rust integration

Use cipherfs_sync::SyncedVault for asynchronous object operations and background synchronization. Every device keeps the full shared index. Full is the default storage role: it fetches all unretired content in the background. Partial devices fetch pinned directories and cache other content on demand. Directory and S3 stores contain both encrypted indexes and content; connected peers are also content sources.

Dependencies and enrollment

The project is unreleased and used from source. Adjust paths for your checkout:

[dependencies]
cipherfs-core = { path = "../cipherfs/crates/cipherfs-core" }
cipherfs-sync = { path = "../cipherfs/crates/cipherfs-sync", default-features = false, features = ["iroh-p2p"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "io-util"] }

Add s3 to the Sync features to open S3-compatible stores. Core methods perform blocking database/filesystem I/O; call them through spawn_blocking in asynchronous applications. The streaming façade keeps its readers and writers on the caller's executor and runs Core work separately.

Create the first device with CLI init or Vault::create_with_recovery_key, and save the returned recovery secret plus independently trusted InitInfo. Enroll other devices through ticket/SAS pairing or recovery join. Creating two vaults independently, even with the same password, creates different vaults.

SyncedVault::open(...).await opens an enrolled device; SyncedVault::new(core) wraps an existing Core handle. SyncedVault::create is a convenience that does not return the recovery secret, so use the Core creation API when your application needs recovery setup. Low-level PendingDevice/enroll_device require an authenticated, approved request; the maintained crates/cipherfs-sync/examples/synced_vault.rs demonstrates enrollment before a real two-node exchange.

Open stores and start synchronization

This example opens an existing device and its persisted store configuration. It deliberately treats an unopened configured store as an error; applications may instead report and retry those failures.

use std::path::PathBuf;
use cipherfs_core::Vault;
use cipherfs_sync::{configured_stores, BackgroundSyncOptions, SyncedVault};

type AppResult<T> = Result<T, Box<dyn std::error::Error + Send + Sync>>;

async fn run(path: PathBuf, password: String) -> AppResult<()> {
    let (vault, stores) = tokio::task::spawn_blocking(move || -> AppResult<_> {
        let core = Vault::open(&path, &password)?;
        let (stores, failed) = configured_stores(&core)?;
        if !failed.is_empty() {
            return Err(std::io::Error::other(format!(
                "{} configured stores could not be opened", failed.len()
            )).into());
        }
        Ok((SyncedVault::new(core), stores))
    }).await??;

    vault.enable_sync(BackgroundSyncOptions {
        stores,
        ..BackgroundSyncOptions::default()
    }).await?;

    let input = tokio::fs::File::open("/data/input.bin").await?;
    vault.put_from_reader("docs/input.bin", input).await?;
    let mut output = tokio::fs::File::create("/data/output.bin").await?;
    let written_bytes = vault.get_to_writer("docs/input.bin", &mut output).await?;
    println!("written_bytes={written_bytes}");

    vault.close().await?;
    Ok(())
}

configured_stores returns (opened, failed); do not silently discard the second list. S3 credentials are read from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when stores are opened and are not persisted. enable_sync does not automatically open persisted stores: pass the opened StoreTarget values in BackgroundSyncOptions.stores.

For P2P, put enrolled peers' NodeAddr values in BackgroundSyncOptions.peers or call add_sync_peer; sync_addr supplies this device's address. The same listener serves incoming peers. RelayConfig::no_relay() is useful for directly reachable deployments/tests; default options use the default relay configuration. Options also specify bind port, polling and retry intervals. Port zero requests an available UDP port.

Each peer and store has an independent loop; writes wake synchronization, failures retry with backoff, and a separate content task fills Full/pinned content. A catching-up Full device advertises incomplete content separately from its complete index. A write succeeds after local persistence, before remote replication completes.

Object methods and memory

The adapter uses two-slot 64KiB channels. Writers may borrow caller-owned state and need neither Send nor Unpin. Reads keep their selected version despite concurrent writes or retirement, and flush on success. A missing unit, authentication error or sink error fails explicitly; a failed writer or HTTP stream may already have emitted an authenticated prefix. Discard that output or abort destination publication on failure. Buffered get/get_range return Err instead of a successful partial object. Use a temporary output file when your application requires atomic single-file publication.

Canceling an upload before EOF cannot publish partial content. After EOF, an admitted commit may finish and notify synchronization even if the caller stops waiting. disable_sync stops the service; close closes the whole façade and its clones, wakes blocked adapters and drains admitted work. Await close before reopening that device.

Body adapters bound payload buffers, not the entire application's memory. Byte Vec/Blob results cost O(object or requested range); Vec list/history/trash/conflict results cost O(returned entries). Page scans retain at most limit + 1 result entries plus the row being decrypted; per-file heads, path contenders and one version's content-unit metadata have separate costs. History still retains visited VersionIds and pending parent IDs. Complete membership/index state and remote source fetch/cache windows also retain their own costs. The local 512MiB allocation regression and its exact scope are recorded in docs/IMPLEMENTATION_AUDIT.md.

Core Vault::list_page(directory, limit, continuation.as_ref()) lists direct children, with directories before same-name files. Vault::list_file_entries_page and the async SyncedVault::list_page instead match a literal object-key prefix (leading slashes stripped); they include docs2/a for prefix docs. Core Vault::file_history_page and async SyncedVault::history_page preserve reachable cross-FileId parents from conflict resolution and traverse retired ancestors before excluding them from results.

limit must be 1..=MAX_PAGE_ENTRIES (1024). Start with None, process page.entries, then pass page.continuation to the same method/scope until None. Core takes borrowed continuations; async methods take owned values. EntryContinuation/HistoryContinuation are cloneable, serializable values exported by Core and Sync, hold no Vault/DB handle, and work across close/reopen. Scopes include the vault and canonical directory or literal prefix; history also checks the canonical path and selected FileId, rejecting path reuse by another identity.

Pages are current views, not a snapshot spanning calls: edits/deletions/retirement can change subsequent entries, and insertions before the boundary are omitted. On a quiescent vault, concatenating pages equals the convenience result. HMAC indexes may require a complete row scan for every page, so bounded results do not guarantee constant scan time. C exposes cipherfs_vault_write_stream/cipherfs_vault_read_stream callbacks, Go exposes WriteFrom/ReadTo, Python exposes write_from_reader/read_to_writer, and Web downloads use HTTP streams. Their byte/Blob and list convenience methods allocate complete results; those adapters do not currently expose these Rust list/history pages.

Shared versions and plaintext export

Core provides conflicts, read_head_to_writer, resolve_conflict, move_file_id, file_history, trash/restore and retirement operations. All devices see shared retained versions. A subsequent write parents all known heads; retention defaults to 365-day candidates and a cancellable 30-day grace period. Full/Partial controls local storage rather than access rights.

Export is a blocking Core operation, run outside the async executor:

fn export(vault: &cipherfs_core::Vault) -> cipherfs_core::Result<cipherfs_core::ExportReport> {
    vault.export_plaintext("/data/new-export", cipherfs_core::ExportOptions {
        include_history: true,
        include_conflicts: true,
    })
}

The two options are independent. Older non-head versions use original paths; additional heads use their FileId's materialized path. Files go under .cipherfs-history/<path>/<time_ms>-<full-device-hex>-<full-version-hex>. The report counts complete content files and lists target-relative missing destinations, including version filenames. Corruption/authentication/I/O errors abort rather than being listed as missing. mtime and owner executable are applied; local empty directories are excluded.

Complete offline export is guaranteed for a Full device that has finished content synchronization relative to its accepted index frontier and has an intact local cache, covering the selected current and unretired history/conflict content. It does not include changes the device has not yet accepted. A catching-up Full or Partial device can export locally available content offline and reports missing destinations; its role alone cannot guarantee completeness.

Export requires a nonexistent target, an existing parent without symlink components, and selected keys/archive paths losslessly representable by the native filesystem. File/directory shadows such as a and a/b, case/name/path collisions fail without renaming. Exclusive anchored publication is implemented for Linux/Android/Apple; other platforms return Unsupported. Selected versions and their content stay pinned during concurrent changes/retirement; body buffers stream, while selection metadata grows with the export.

Device and capacity limits

Recovery resets preserve DeviceId. Revocation and recovery rotation create new random key epochs; already disclosed secrets/content cannot be withdrawn. Recovery join can read/write immediately, but device management waits for an independent existing member's ratification. If every existing device is lost, a lone recovered device cannot revoke, rotate recovery or approve pairing until an independent member returns.

Cold pairing/recovery proofs and snapshots are single objects capped at 32MiB; multipart remains deferred. recovery_proof_deferred reports an oversized recovery publication while ordinary index/content sync and GC can continue. An initial upload with no acknowledged proof prefix can still hit the inventory-proof limit. Long-history/million-file scale, subtree grants, mobile clients and WebDAV stores remain outside current acceptance.

Source file in the repository