Off Grid AI Desktop verwandelt einen Windows-PC in einen lokalen KI-Server fürs ganze Team
Mit einer NVIDIA-Grafikkarte unter Windows lässt sich laut Anbieter ohne Cloud ein gemeinsamer KI-Server betreiben. Der Team-Modus des Gateways kann mehrere Anfragen gleichzeitig beantworten und stellt jedem Nutzer eine eigene API bereit.
Posted on Oct 27 • Originally published at getoffgridai.co
You have one Windows PC with a good NVIDIA card and a team that wants local AI, without sending your code and documents to a cloud API. Off Grid AI Desktop can turn that one PC into a shared AI server. Its gateway has a Team mode that answers several people at once, gives each person their own API key, and keeps the PC awake while it works. Every request stays on your network.
An NVIDIA GPU is what lets one PC serve many people. When the model runs on CUDA, Auto opens up to 4 requests at once on a card with less than 24 GB of VRAM, 8 from 24 GB (an RTX 3090 or 4090, for example), 16 from 48 GB and 32 from 96 GB. On a PC without CUDA, where the model runs on Vulkan or the CPU, the cap is 4. Inside that cap, the real number comes from the memory left after the model loads, with room for about 8,000 tokens per request. To check your card is in use, see how to use your NVIDIA GPU for local AI on Windows.
The gateway listens on port 7878 on every network interface. The first time the app opens it, Windows Defender Firewall may ask whether to allow Off Grid AI Desktop on your networks. Allow it on Private networks. Leave Public networks off, so the server is not open on a cafe or hotel Wi-Fi.
If someone is mid-answer when you change a setting, the app lets that answer finish before it restarts the engine.
When every slot is busy, new requests wait in line in the order they arrived. When the line is full, the gateway answers with HTTP 429 and a Retry-After header, so a client backs off and tries again instead of hanging.
By default the gateway does not ask for a key. On a shared network, turn keys on.
Each key shows whether it is Active, Expired or Revoked, and when it was last used. If a laptop is lost, click the revoke icon next to its key; nobody else needs a new one. The keys sit in the app's encrypted secret store, protected by Windows for your user account, not in a plain-text settings file.
from openai import OpenAI client = OpenAI ( base_url = " http://192.168.1.20:7878/v1 ", api_key = " THE_KEY_YOU_COPIED " ) resp = client. chat. completions. create ( model = " local ", messages = [{ " role ": " user ", " content ": " Summarise this changelog in two lines. " }], ) print ( resp. choices [ 0 ]. message. content ) Enter fullscreen mode Exit fullscreen mode A request that names local or no model runs on the chat model loaded on the PC. The Endpoints list on the Gateway screen shows what else the server offers: chat with images, text to image, speech to text, text to speech, embeddings and the model list.
To use the PC's models from the Off Grid AI phone app, follow the guide to use Off Grid AI Desktop from your phone and give the phone its own key.
Open Settings > AI activity. Every request is listed with its model and timing, and a request sent with a key shows that key's name.
For monitoring, GET /v1/gateway/status returns the mode, the requests-at-once setting, and how many requests are running, waiting and refused. It needs a key like every other route.
In Team mode the PC stays awake while it is answering, and Windows is allowed to sleep a minute after the last request ends. In Just me mode the app does not hold the PC awake.
To serve without the desktop window, start the app from PowerShell in server-only mode, with Team mode for that run only. With the default install location, that is:
& " $ env: LOCALAPPDATA \Programs\off-grid-ai\off-grid-ai.exe" --server-only --gateway-mode team --gateway-clients auto Enter fullscreen mode Exit fullscreen mode If you installed it elsewhere, right-click the desktop shortcut, choose Open file location, and use that path. --gateway-clients takes auto or a number. These flags apply to that launch and do not change the settings you chose in the app. The environment variables OFFGRID_GATEWAY_MODE and OFFGRID_GATEWAY_CLIENTS do the same.
Is Off Grid AI Desktop an alternative to Ollama for a team server on Windows? Yes, for teams whose tools speak the OpenAI API. It serves chat, vision, image generation, speech and embeddings from one address, with per-person API keys and a request log built in.
How many people can use one Windows PC at once? With Auto on an NVIDIA GPU, up to 4 requests at once below 24 GB of VRAM, 8 from 24 GB, 16 from 48 GB and 32 from 96 GB, limited further by the memory your model leaves free. Without CUDA the cap is 4. More people can send requests; the extra ones wait in line.
Does my team's data leave the office? No. Requests go from each computer to the PC on your network, and the models run on that PC. Nothing goes to a cloud service.
Why can't other computers reach the server? Check that they use the PC's IPv4 address, not 127.0.0.1, and port 7878. Then check that Windows Defender Firewall allows Off Grid AI Desktop on the network type you are on, which should be Private.
Can I revoke one person's access? Yes. Click the revoke icon next to their key under API keys. Their requests are refused and every other key keeps working.