Local AI Is Not a Privacy Switch: Build a Workload Routing Contract

Posted on Oct 21 • Originally published at trust-vault.com

Running a language model on a laptop or an internal server can remove one important data transfer: the prompt no longer has to travel to a hosted inference provider. That is useful. It is not the same as making the workflow private, secure, compliant, reliable, or economically sensible.

The document may still arrive through a cloud inbox. The application may write prompts to a verbose log. A retrieval index may contain records the user should not see. A desktop extension may send telemetry. A model file may come from an unverified source. Generated text may be copied into a system that triggers an action without validation. “Local” describes where one computation happens; it does not describe the whole data path.

The source field report on running language models without cloud inference focuses on a real motivation: keeping confidential drafts, client documents, proprietary code, or sensitive records on controlled equipment. This guide develops the decision architecture around that motivation.

The practical answer is a workload routing contract. Before any prompt reaches any model, the contract classifies the data, evaluates the requested action, chooses an approved execution zone, applies controls, and records evidence. Some work goes to a local model. Some goes to an approved private cloud service. Some stays manual. Some is rejected.

This is a more demanding design than a checkbox labeled “use local AI.” It is also far more useful because it connects privacy, security, quality, availability, cost, and human responsibility to the actual unit of work.

A useful threat model follows information from origin to deletion. Draw every component that can read, transform, persist, or transmit the content:

If nine steps are local and one sends raw content to an external error tracker, the workflow is not local in the sense the user probably expects. If inference happens in a private data center but the retrieval layer has weak authorization, model placement does not solve the primary risk.

Create a data-flow inventory for each use case. For every edge, record:

Do not fill the table from product marketing. Validate configuration, network behavior, filesystem paths, contracts, and observed logs.

A binary local-versus-cloud policy is usually too crude. Use zones with explicit trust assumptions.

The task remains manual or uses deterministic, approved software. This zone covers workloads where the consequence, data sensitivity, missing authorization, or lack of reliable evaluation makes model use unacceptable.

Examples may include unreviewed final decisions about employment, credit, health, or legal rights; credentials and cryptographic secrets; or records whose use has not been approved. The exact list belongs to your organization's risk owners.

The model runs on managed equipment with outbound traffic blocked or tightly controlled. Inputs, model artifacts, logs, and outputs remain in an approved boundary. Tools are disabled unless separately approved. This is the strongest local zone, but it requires operational work: patching, access control, model provenance, monitoring, backup policy, and incident response.

The model runs on internal or dedicated infrastructure reachable over a controlled network. Centralization may improve patching, capacity, and observability, but it creates a shared service with authorization and multi-tenant risks. Treat it as a production platform, not a convenient server.

The workload goes to a contracted external service under defined data-handling, retention, training-use, region, identity, and logging controls. Hosted does not automatically mean unsafe, and local does not automatically mean safe. The zone is appropriate only when the workload classification and contract match.

The routing decision should select the least restrictive zone that still satisfies the requirements. Forcing every task onto local hardware can reduce quality, availability, or maintainability without creating meaningful additional protection. Sending every low-risk public task through isolated infrastructure can also consume scarce capacity needed for sensitive work.

Data classification is necessary but incomplete. Route on a vector of attributes:

A public text-generation task with authority to publish may require more control than a confidential summarization task whose output never leaves a review screen. Action often matters as much as content.

The CISA and UK NCSC announcement of their Guidelines for Secure AI System Development describes security across design, development, deployment, operation, and maintenance, and states that the guidance applies broadly to AI systems, including systems using external APIs. That lifecycle perspective is the right antidote to treating deployment location as the entire security design.

The contract should be readable by risk owners and executable by the application. Avoid a weighted score that lets a low-risk attribute compensate for a hard prohibition.

Only then optimize among eligible zones using cost, latency, quality, and availability.

flowchart TD A[Work request] --> B{Use case approved?} B -- No --> P[Reject or manual path] B -- Yes --> C{Classify data and action} C --> D{Hard prohibition?} D -- Yes --> P D -- No --> E{Restricted data?} E -- Yes --> F{Isolated local controls pass?} F -- Yes --> L[Zone 1 local inference] F -- No --> P E -- No --> G{Private service required?} G -- Yes --> V[Zone 2 managed private] G -- No --> H{Hosted contract matches?} H -- Yes --> C3[Zone 3 approved hosted] H -- No --> V L --> Q[Validate output and record evidence] V --> Q C3 --> Q Q --> R{Action permitted?} R -- Human review --> U[Review queue] R -- Reversible automation --> X[Controlled execution] R -- No --> P The route should fail closed when critical classification or configuration is missing. A fallback from local to hosted inference must never happen silently. If a confidential workload cannot run locally because memory is exhausted, the safe outcome may be delay or manual handling—not automatic transmission elsewhere.

const zones = { ISOLATED_LOCAL: " zone-1 ", PRIVATE_SERVICE: " zone-2 ", APPROVED_HOSTED: " zone-3 ", MANUAL: " manual ", REJECT: " reject " }; function route ( request, inventory ) { const reasons = []; if (! request. useCaseApproved ) { return { zone: zones. REJECT, reasons: [ " USE_CASE_NOT_APPROVED " ] }; } if (! request. userAuthorized ) { return { zone: zones. REJECT, reasons: [ " SOURCE_ACCESS_NOT_AUTHORIZED " ] }; } if ( request. containsCredentials ) { return { zone: zones.

MANUAL, reasons: [ " CREDENTIALS_PROHIBITED " ] }; } if ( request. action === " irreversible-decision " ) { return { zone: zones. MANUAL, reasons: [ " HUMAN_DECISION_REQUIRED " ] }; } const needsIsolation = [ " restricted ", " special-category " ]. includes ( request. classification ); if ( needsIsolation ) { if (! inventory. local?. healthy ) reasons. push ( " LOCAL_RUNTIME_UNHEALTHY " ); if (! inventory. local?. egressBlocked ) reasons. push ( " LOCAL_EGRESS_NOT_BLOCKED " ); if (! inventory. local?. modelApproved ) reasons. push ( " LOCAL_MODEL_NOT_APPROVED " ); if (! inventory. local?. encryptedStorage ) reasons. push ( " LOCAL_STORAGE_NOT_ENCRYPTED " ); return reasons. length? { zone: zones. MANUAL, reasons }: { zone: zones.

ISOLATED_LOCAL, reasons: [ " RESTRICTED_ROUTE " ] }; } if ( request. classification === " confidential " ) { if ( inventory. private?. healthy && inventory. private?. policyVersion ) { return { zone: zones. PRIVATE_SERVICE, reasons: [ " CONFIDENTIAL_ROUTE " ] }; } if ( inventory. local?. healthy && inventory. local?. modelApproved ) { return { zone: zones. ISOLATED_LOCAL, reasons: [ " PRIVATE_UNAVAILABLE " ] }; } return { zone: zones.

MANUAL, reasons: [ " NO_APPROVED_CONFIDENTIAL_ZONE " ] }; } if ( inventory. hosted?. approved && inventory. hosted?. retentionDays === 0 ) { return { zone: zones. APPROVED_HOSTED, reasons: [ " LOW_RISK_HOSTED_ROUTE " ] }; } if ( inventory. local?. healthy && inventory. local?. modelApproved ) { return { zone: zones. ISOLATED_LOCAL, reasons: [ " HOSTED_CONTRACT_MISMATCH " ] }; } return { zone: zones.

MANUAL, reasons: [ " NO_ELIGIBLE_ZONE " ] }; } const decision = route ( { useCaseApproved: true, userAuthorized: true, containsCredentials: false, action: " draft ", classification: " restricted " }, { local: { healthy: true, egressBlocked: true, modelApproved: true, encryptedStorage: true }, private: { healthy: true, policyVersion: " private-ai-4 " }, hosted: { approved: true, retentionDays: 0 } } ); console. log ( decision ); Enter fullscreen mode Exit fullscreen mode Change egressBlocked to false. The router chooses the manual path; it does not leak into another zone. Change the classification to public and the approved hosted zone becomes eligible.

In production, the policy engine should receive trusted classification attributes rather than accepting user assertions at face value.

Store the request identifier, policy version, decision, reasons, chosen runtime, model digest, and output-validation result. Do not log raw restricted prompts merely to prove that routing occurred.

An application can call itself local while still contacting licensing servers, update services, analytics endpoints, model repositories, fonts, crash reporters, or plugins. Some traffic is benign and necessary; some defeats the stated boundary. You need evidence.

Air-gapped is a strong claim. Use it only when the environment actually lacks a path to other networks and transfer procedures are controlled. “We did not configure an API key” is not an air gap.

Network isolation also creates maintenance questions. How will security patches, model updates, vulnerability intelligence, and revocations enter? How will compromised artifacts be removed? A frozen system may protect confidentiality while accumulating known vulnerabilities. The contract needs a controlled update path.

Local deployment increases your responsibility for artifacts. A downloaded model, runtime binary, native library, container image, tokenizer, template, or community adapter can alter behavior or execute code in the surrounding process.

Do not execute arbitrary model packages in the same security context as sensitive files. Use a dedicated identity, least privilege, read-only mounts where possible, resource limits, and a disposable test environment for new artifacts.

CISA's secure-by-design framing matters here: the user should not bear the full burden of compensating for unsafe defaults. Internally, platform owners should make the approved path the easy path—signed artifacts, known configurations, automatic isolation, and visible failure.

Retrieval-augmented generation often creates the largest data store in the workflow. Teams copy documents into a shared vector index, then celebrate that inference is local. If the index ignores source permissions, a user may retrieve content they could not open in the original system.

Maintain authorization through ingestion and retrieval:

Embedding vectors are not automatically anonymous. They are derived from source content and can expose information through retrieval behavior or associated metadata. Treat them according to the source risk unless a validated transformation justifies another classification.

A locally produced output can cause harm when it crosses into a trusted system. Generated shell commands, markup, database queries, configuration, or tool arguments are untrusted data until validated.

The OWASP Top 10 for Large Language Model Applications highlights risks including prompt injection, sensitive-information disclosure, supply-chain weaknesses, improper output handling, excessive agency, and overreliance. Local inference changes the exposure of some threats; it does not remove the threat classes.

The most secure local model can still generate a destructive command. Location is not authorization.

Organizations often compare local and hosted models on a public benchmark and select the fastest acceptable option. That misses the actual workload, the prompt template, retrieval quality, quantization, hardware, and acceptance criteria.

Build a representative, access-controlled evaluation set. Include ordinary cases, edge cases, adversarial inputs, long documents, ambiguous instructions, multilingual content, and cases where the correct action is abstention.

Quantization or a smaller context window may change error patterns rather than simply lowering an overall score. Keep results tied to a model digest, runtime, hardware class, and workflow version.

Local inference removes per-request vendor charges but introduces equipment, electricity, administration, monitoring, patching, capacity planning, and failure recovery. Hosted inference converts much of that into variable cost and provider dependence.

local cost = hardware amortization + energy + platform labor + security and evaluation labor + user review and rework + downtime impact hosted cost = inference and platform fees + integration labor + governance and evaluation labor + user review and rework + network and outage impact Enter fullscreen mode Exit fullscreen mode Do not assume the local machine is free because it was already purchased. If inference consumes memory that displaces other work or requires a new workstation, that is a real resource decision.

PwC's discussion of generative AI adoption in technology, media and telecommunications identifies potential edge-computing advantages including latency reduction, bandwidth optimization, offline operation, and storage efficiency. Those are workload-dependent advantages, not universal guarantees. A centralized system may still win when utilization, specialized accelerators, maintenance, or collaboration dominate.

A local runtime may fail because a laptop sleeps, a GPU runs out of memory, a model is updating, or concurrent requests exceed capacity. Define what happens before failure occurs.

Never let the application invent a fallback. The same routing contract should decide normal and degraded execution. Test failures by deliberately removing network, disk, memory, and model availability in a safe environment.

Track service objectives by zone: availability, queue delay, inference latency, acceptance, error severity, and policy-denial rate. A high denial rate may indicate users are attempting unsuitable workloads or that the approved platform does not cover legitimate needs.

Data may stay on the device while administrators lose inventory, patching visibility, incident evidence, or deletion control. Managed endpoints and attestation matter.

A shared platform supports consistent controls, but a permission error can expose information across teams. Isolation and authorization must be tested.

Limited capability may reduce some abuse paths, yet increase omissions or unsupported answers. Safety includes task quality, not only network placement.

Work continues without connectivity, but theft, disk failure, or local compromise becomes more consequential. Encryption, backup, and recovery remain necessary.

Record decisions and technical evidence, not unrestricted prompt content. Apply minimization and retention to observability.

Availability of weights or code can support review, but each distributed artifact, conversion, and dependency still needs verification.

The interface says “local,” but a plugin, telemetry client, or document parser sends data externally.

Control: enforce and test outbound network policy; maintain an approved endpoint inventory.

The local model fails and the application retries through a hosted service.

Control: make routing explicit, fail closed for sensitive classes, and log reason codes.

Users retrieve chunks from documents they cannot access.

Control: preserve source ACLs, filter before retrieval, and test cross-tenant queries.

A model, adapter, runtime, or container comes from an unverified source.

Control: use approved registries, immutable digests, manifests, scanning, and staged evaluation.

Generated commands or tool arguments cross directly into a privileged system.

Control: validate typed output, allowlist operations, minimize permission, and require approval.

Restricted content is protected during inference but copied into logs, histories, or backups.

Control: minimize logs, redact metadata, disable content capture, and verify retention behavior.

The offline system stops receiving security fixes and artifact revocations.

Control: establish a signed, controlled update process with rollback and evidence.

Every team downloads different runtimes and models without inventory.

Control: provide a managed catalog, endpoint policy, discovery, and removal workflow.

The team selects local inference but has no lawful purpose, minimization, retention, or user authorization.

Control: require the same data-governance review regardless of execution zone.

A routing contract is incomplete until the team can demonstrate what happens when one control fails. Run a tabletop exercise with a deliberately expired model approval, a blocked destination, a corrupted artifact digest, and a request containing restricted data. The expected result is not simply that inference stops. Operators should be able to identify the policy decision, the component that enforced it, the affected records, and the safe recovery path without opening raw prompts broadly.

Then test a less obvious scenario: a permitted local request completes, but the result is written to an unauthorized shared folder. This separates inference privacy from lifecycle privacy. Evidence should show the route selected, process identity, model and policy versions, destination decision, retention timer, and any human approval. It should not duplicate the sensitive payload merely to prove that controls ran.

Set measurable recovery criteria before the drill. Examples include revoking a compromised artifact, locating every deployment that used its digest, disabling a route, preserving a minimal audit trail, and restoring a known-good version. Record gaps as engineering work with an owner and deadline. If the team cannot contain, explain, and recover from a simulated failure, the workload is not ready for sensitive production data—even if the model never contacts the internet.

List intended tasks, data sources, actions, current tools, and destinations. Identify workloads that must remain prohibited or manual.

Choose a narrow read-only use case. Create managed hardware, model manifests, egress controls, encrypted storage, minimal logs, and an evaluation set.

Process approved copies without influencing real decisions. Test quality, leakage, resource exhaustion, retrieval authorization, updates, and deletion.

Expose results to trained users with mandatory review. Measure accepted outcomes, rework, incidents, latency, and denials.

Only after read-only controls hold, permit reversible, typed, least-privilege tool calls. Keep high-impact decisions human-controlled.

Reevaluate models, dependencies, routes, policies, and recovery after material changes. Remove unused artifacts and stale access.

Before calling a workflow “local and controlled,” verify:

Use local inference when the workload benefits materially from keeping processing inside a controlled boundary and your organization can operate that boundary responsibly. Use an approved hosted service when its contract, controls, quality, availability, and cost fit the workload. Use a private shared service when central governance outweighs endpoint autonomy. Keep the task manual when no zone satisfies the contract.

The strongest privacy claim is not “the model runs on my machine.” It is: “for this approved purpose, these data move through these verified components, under these identities and retention rules; no unapproved transmission occurs; the output cannot act beyond its authority; and we can produce evidence.”

Local AI is a useful execution option. The routing contract is what makes it an accountable system.