Building PhotoCraft Companion: Local AI for a WebAssembly Editor (Rust Architecture Deep Dive)

Posted on Oct 25 • Originally published at github.com

PhotoCraft is a modern, open-source desktop image editor with an internal plugin engine built around WebAssembly. Wasm delivers rock-solid security, sandbox isolation, and cross-platform portability. But when you want to run modern neural networks locally, it hits hard physical boundaries.

Three limits make local AI in Wasm painful:

For heavy creative workflows — studio-grade background removal, generative inpainting, and seam-blended super-resolution — users were trapped between cloud subscriptions with privacy concerns or awkward manual file-export roundtrips.

I wanted a local, offline, zero-telemetry alternative.

Instead of forking or patching the upstream editor codebase, I designed a standalone native companion in Rust. It runs as an independent OS process, interacts with PhotoCraft via its existing JSON-RPC control socket (TCP port 18492), and exchanges pixel buffers through the system clipboard.

The companion is structured as a Cargo workspace with 6 crates, ~25,000 lines of Rust, and governed by strict engineering constraints: a Definition of Done, resource guards, automated architectural linters, and bilingual localization enforced by CI.

┌─────────────────────────────────────────────────────────────┐ │ PhotoCraft (Host Editor) │ │ Canvas UI (Wasm Sandbox) ───► TCP Socket Server (18492) │ └───────────────────────────┬─────────────────────▲───────────┘ │ JSON-RPC Control │ │ (Layer copy/paste) │ ▼ │ ┌─────────────────────────────────────────────────┴───────────┐ │ PhotoCraft Companion (Rust Native) │ │ │ │ ┌──────────────────────┐ ┌────────────────────────┐ │ │ │ overlay_ui │ │ photocraft_bridge │ │ │ │ (egui floating panel)│ │ (transactional locks) │ │ │ └──────────┬───────────┘ └───────────┬────────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ core_contracts (Zero-Core API) │ │ │ │ plugin.json ·.pcpl packaging · i18n │ │ │ └──────────────────────────┬───────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ ai_engine (Local Inference) │ │ │ │ DirectML (GPU) ──► Transparent CPU Fallback │ │ │ │ Session LRU Cache · Single-Session Tiling Engine │ │ │ └──────────────────────────────────────────────────────┘ │ └──────────────────────────────┬──────────────────────────────┘ │ In-Memory Pixel Transport ▼ [ System OS Clipboard ] Enter fullscreen mode Exit fullscreen mode Key Architectural Principle — Zero-Core Hardcoding: No model filenames (.onnx ), neural input shapes, or plugin IDs are ever hardcoded in the Rust core. Everything is declared in dynamic JSON manifests. The runtime builds the UI, schedules the inference graph, and issues canvas commands purely based on the declared contracts.

The workspace is strictly partitioned by domain boundaries:

The dependency flow is strictly unidirectional: core_contracts sits at the root with zero internal deps, every other crate depends on it, and companion_app binds them together into a single, cohesive binary.

The bridge is the most delicate component because it crosses an inter-process boundary with an asynchronous peer. The editor might hitch during heavy operations, show a blocking modal dialog, or abruptly close. The companion must never hang.

The client uses Tokio's asynchronous TCP halves. Initially, writer and reader had independent mutexes. However, concurrent calls interleaved: Call A wrote, Call B wrote, Call A read — and received Call B's response!

The fix was a transactional lock that serializes the full request-response lifecycle:

pub async fn call ( & self, method: & str, params: Value ) -> Result < Value > { // Acquire a single lock that spans both write and read let _tx = self.call_lock.lock ().await; let id = self.next_id.fetch_add ( 1, Ordering:: SeqCst ); let req = JsonRpcRequest { id, method: method.to_string (), params, }; let mut writer_guard = self.writer.lock ().await; let writer = writer_guard.as_mut ().ok_or ( CompanionError:: Bridge ( "Disconnected".into ()))?; writer.write_all ( req.serialize ()?.as_bytes ()).await?; let mut reader_guard = self.reader.lock ().await; let reader = reader_guard.as_mut ().ok_or ( CompanionError:: Bridge ( "Disconnected".into ()))?; // Read with a strict deadline; drop handles on timeout to reset state let resp = tokio:: time:: timeout ( Duration:: from_secs ( 30 ), reader.read_response ( id )).await??; assert_eq! ( resp.id, id, "Mismatched JSON-RPC sequence ID" ); Ok ( resp.result ) } Enter fullscreen mode Exit fullscreen mode Contention Gotcha: The Color Poller When a plugin panel is open, a background task polls the active brush color every 180ms to keep overlay palettes in sync. Because it shares the bridge client, it created constant contention for the socket lock.

The solution: background polling uses a non-blocking try_lock(). If the bridge is busy running an inference command or layer operation, the poller silently skips that tick rather than queuing behind long-running operations.

The companion displays a two-tier floating toolstrip rendered with egui and eframe. It hovers at persistent screen coordinates above the canvas without creating clutter in the OS taskbar.

Default fonts in egui lack full Unicode Noto/Emoji coverage. Using emojis like ⏳, ⚠️, or ⚙ produces ugly "tofu" squares ( □ ).

The overlay enforces a strict Vector-First policy:

The overlay does not hardcode plugin forms. It reads the plugin.json manifest and instantiates sliders, dropdowns, and toggles on the fly:

// overlay_ui renders widgets dynamically from manifest schema for widget in & manifest.ui.widgets { match widget.kind { WidgetKind:: Slider { min, max, step } => { ui.add ( egui:: Slider:: new ( & mut state.get_mut ( & widget.id ), * min..=* max ).step_by ( * step as f64 ).text ( widget.label.localize ( & self.locale )) ); } WidgetKind:: Dropdown { options } => { egui:: ComboBox:: from_label ( widget.label.localize ( & self.locale )).selected_text ( & state.get_string ( & widget.id )).show_ui ( ui, | ui | { for opt in options { ui.selectable_value ( & mut state.get_mut ( & widget.id ), opt.value.clone (), & opt.label ); } }); } } } Enter fullscreen mode Exit fullscreen mode A third-party creator can introduce a new image processing filter with four sliders and two dropdowns entirely in JSON. The companion renders it immediately without recompiling Rust.

The AI engine uses ort (Rust bindings for ONNX Runtime C++ backend) with DirectML for GPU acceleration on Windows, falling back seamlessly to CPU on non-compatible hardware.

To see why the companion pattern is worth the architectural complexity, consider RMBG-1.4 inference on a 1024×1024 image:

Allocating an ONNX execution session and compiling model graphs to DirectML on every user action introduces 1–2 seconds of latency. The companion maintains an LRU session cache:

pub struct SessionCache { entries: HashMap < PathBuf, ( Arc < Session >, Instant ) >, max_capacity: usize, } Enter fullscreen mode Exit fullscreen mode Because Session::run takes &self in modern ort, an Arc<Session> can be safely shared across tasks and reused across sequential operations.

When users process 4K or 8K layers, running the full canvas through a model in one pass overflows VRAM. The engine slices the canvas into overlapping tiles, executes inference tile-by-tile within a single cached session, and blends edges using cosine feathering to avoid seam artifacts.

Plugins are distributed as.pcpl files — ZIP archives containing a manifest, optional scripts, icons, and model weights.

{ "id": "remove_bg", "name": { "en": "Background Removal", "ru": "Удаление фона" }, "version": "1.3.0", "type": "native_ai", "input": "active_layer", "output": "new_layer", "models": [ { "role": "segmentation_mask", "path": "models/rmbg-1.4.onnx", "sha256": "5c1f0d..." } ], "commands": [ "image.revealAll" ], "post_commands": [ "layer.renameActive" ] } Enter fullscreen mode Exit fullscreen mode Zip-Slip Mitigation Unpacking community archives is fraught with danger. A malicious archive containing filenames like../../appdata/startup.exe can overwrite arbitrary system files.

The archive extractor enforces three layers of validation:

for i in 0.. archive.len () { let mut file = archive.by_index ( i )?; // 1. Safe extraction path verification let enclosed = file.enclosed_name ().ok_or_else (|| CompanionError:: Security ( "Illegal path in archive".into ()))?; // 2. Reject absolute paths and parent traversals if enclosed.components ().any (| c | matches! ( c, Component:: ParentDir | Component:: RootDir )) { return Err ( CompanionError:: Security ( "Path traversal attempt detected".into ())); } // 3.

Reject symlinks if file.is_symlink () { return Err ( CompanionError:: Security ( "Symlinks forbidden in plugin packages".into ())); } let target = target_dir.join ( enclosed ); //... write file safely } Enter fullscreen mode Exit fullscreen mode 7. Engineering Discipline & Automated Governance When pairing with LLMs on an ambitious codebase, the primary danger is "architecture rot" — subtle hacks, keyword-sniffing shortcuts, and unprincipled workarounds.

To prevent this, the repository uses automated governance tests:

PhotoCraft Companion is open source and dual-licensed under MIT / Apache-2.0:

👉 github.com/kobaltgit/photocraft-companion

Contributions, architectural discussions, and new plugin manifests are welcome!