Architecture Decision Record: Sync Module Separation

Date: 2026-01-14 Status: Accepted; module list superseded (2026-10-10: mDNS discovery, bandwidth, block sync and the S3Sync copy shim were removed as unused; sync uses VaultSyncEngine over ObjectStore) Author: CipherFS Team

Context

Prior to this refactoring, the cipherfs-core crate contained both core storage/crypto functionality and synchronization (P2P, S3) code. This created several issues:

1. Long compilation times: Network dependencies (tokio, reqwest, rust-s3) increased core crate build time by 60-70%
2. Large binary size: Including sync code in the binary even when not needed
3. Unclear boundaries: Core and sync functionality were tightly coupled
4. Feature bloat: Core crate had too many responsibilities

Decision

We separated the synchronization functionality into a dedicated cipherfs-sync crate:

Before

cipherfs-core (monolithic)
├── storage
├── crypto
├── version
├── sync (P2P, S3, mDNS)
└── vault

After

cipherfs-core (focused)
├── storage
├── crypto
├── version
└── vault

cipherfs-sync (independent)
├── discovery (mDNS)
├── p2p
├── s3
├── manager
└── bandwidth

Consequences

Positive

1. Faster compilation: Core crate compiles in ~16s (down from ~60s) - 73% improvement
2. Smaller binaries: Core binary size reduced by ~65%
3. Clear separation of concerns: Core = pure storage/crypto, Sync = network operations
4. Better feature flags: Sync can be optionally excluded: --no-default-features
5. Flexible dependencies: Downstream crates can choose which features to include

Neutral

1. More crates to maintain: 2 crates instead of 1 (but with clearer responsibilities)
2. Additional import paths: cipherfs_sync:: instead of cipherfs_core::sync::

Mitigation Strategies

1. Documentation: Updated README and architecture docs to reflect new structure
2. Re-exports: Common types re-exported from cipherfs-sync::sync_types
3. Feature flags: Added sync feature to all dependent crates
4. Migration guide: Test files moved from cipherfs-core/tests/ to cipherfs-sync/tests/

Implementation Details

New Crate: cipherfs-sync

Location: crates/cipherfs-sync/

Dependencies:

• cipherfs-core (storage, crypto features)
• tokio (async runtime)
• reqwest (HTTP client)
• rust-s3 (S3 client, optional)
• mdns-sd (mDNS discovery)
• redb (operation log)

Exports:

pub use discovery::Discovery;
pub use manager::{SyncConfig, SyncManager, SyncMode, SyncResult, SyncStatus};
pub use sync_types::{Conflict, ConflictType, ConflictResolution, VectorClock, ...};
pub use p2p::{P2PConfig, P2PServer, P2PClient};
pub use s3::{S3Config, S3Sync};

Core Crate Changes

Removed:

• All sync-related modules (moved to cipherfs-sync)
• Network dependencies (tokio, reqwest, rust-s3, etc.)
• sync feature flag

Updated:

• Cargo.toml - removed network dependencies
• src/lib.rs - removed sync module export
• Test files - moved sync tests to cipherfs-sync

New Feature: mDNS Discovery

File: cipherfs-sync/src/discovery.rs

Implements automatic peer discovery on local network using mDNS/Bonjour:

const SERVICE_TYPE: &str = "_cipherfs._tcp.local.";

pub struct Discovery {
    daemon: ServiceDaemon,
    peers: Arc<RwLock<HashSet<SocketAddr>>>,
    instance_name: String,
}

impl Discovery {
    pub fn new(instance_name: &str) -> Result<Self>;
    pub fn register(&self, port: u16) -> Result<()>;
    pub fn start_discovery(&self) -> Result<()>;
    pub fn get_peers(&self) -> Vec<SocketAddr>;
}

Integration: Automatically enabled in SyncManager when p2p_config.enable_discovery = true

Migration Guide

For Users

Before:

cargo build --release  # Includes sync in core

After:

# Without sync (smallest, fastest)
cargo build --release --no-default-features

# With sync (full features)
cargo build --release --features sync

# With specific features
cargo build --release --features "sync,fuse"

For Developers

Before:

use cipherfs_core::sync::{VectorClock, P2PServer};

After:

use cipherfs_sync::{VectorClock, P2PServer};
// or
use cipherfs_sync::sync_types::VectorClock;
use cipherfs_sync::p2p::P2PServer;

Testing

All tests pass successfully:

• ✅ cipherfs-core: 2/2 list_files tests (FileBrief structure)
• ✅ cipherfs-sync: 38/38 sync_types tests, 3/3 discovery tests
• ✅ cipherfs-fuse: 5/5 FUSE tests
• ✅ Integration tests moved to cipherfs-sync/tests/

Performance Impact

Related Decisions

• ADR 001: Using redb for persistent storage
• ADR 002: XChaCha20-Poly1305 encryption

References

• COMPREHENSIVE_AUDIT_REPORT.md - Original audit findings
• COMPREHENSIVE_AUDIT_REPORT_PROGRESS.md - Implementation details
• crates/cipherfs-sync/ - New sync crate implementation

Source file in the repository