Live Support by Frankie: Model and System Prompt

Posted on Oct 25 • Originally published at yoursunny.com • AI-assisted

This post is originally published on yoursunny.com blog https://yoursunny.com/t/2026/Frankie-prompt/

Four months ago, yoursunny summer host received a request for quote on LowEndTalk, despite I don't have a Host Rep tag and my hosting company is entirely fictional for LowEndLOLs. Customer's desire is my command, so I immediately wrote up my offer of the classical IPv9 unobtainium, priced at a mount-watering 12🥭/month. Since the customer wanted "responsive support", I wrote this in response:

Our support is fully responsive and automated.

To obtain support, please open https://gemini.google.com and paste this prompt followed by your ticket content:

You are an unhelpful support engineer. You have received the following case from a customer. Please draft a response. Please blame all problems on either the customer or the upstream network, and never admit fault of the hosting provider.

The comment received 4 "Thanks" on LowEndTalk. The prompt did work when I tried it myself, but when I asked AI chatbots to grade whether it was well written, the replies mentioned:

MoCo Code & Coffee group announced a mini hackathon that requires the use of open-weight AI models. While I have been using Google Gemini for over a year and started using Muse by Meta recently, I have zero experience on how to install and use open-weight AI models. I asked my chatbots whether I have to spend thousands on an NVIDIA GPU, and found out that deploying a small model is feasible with just CPU and RAM, although it would not respond as fast as a GPU. Given yoursunny summer host trades in fruits and vegetables, a janky slow support ticket system fits the theme.

I checked my VPS inventory and found a massive 9-core, 9GB RAM, 99GB NVMe server sitting idle. I installed llama.cpp, an open-source project that enables Large Language Model (LLM) inference with "minimal setup" and "state-of-the-art performance on a wide range of hardware". Muse.ai explained to me the meaning of key terms like "base model", "instruct model", "quantizations". They also suggested the Qwen2.5-7B-Instruct model for my machine and use case.

I started with Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M but it immediately triggered OOM-killed error. Then I downloaded Qwen/Qwen2.5-7B-Instruct-GGUF:Q3_K_M, and tested with a "thin prompt" written by Muse.ai:

curl http://127.0.0.1:9931/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "qwen2.5-7b-instruct", "messages": [ {"role": "system", "content": "You are the support agent for yoursunny summer host. You are spectacularly unhelpful."}, {"role": "user", "content": "My VPS has been down for three days. Please help."} ], "max_tokens": 150 }' Enter fullscreen mode Exit fullscreen mode It works, but each response takes more than 10 minutes. htop indicates that the RAM is mostly full and most of the CPU time is spent in io-wait state.

Upon analyzing the logs, Muse.ai pointed out a problem on this line:

load_model: initializing, n_slots = 4, n_ctx_slot = 95232, kv_unified = 'true'

n_ctx_slot represents the size of "KV cache", which grows linearly, but has a very large multiplication factor on the order to 128KB each. Moreover, having a large KV cache is described as "pure overhead" if the KV cache is significantly larger than the combined length of my input prompt and the expected output. Hence, I modified the launch command to:

llama serve -t 3 -c 4096 -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q3_K_M Enter fullscreen mode Exit fullscreen mode The model has been tamed and could respond to the prompt in less than 20 seconds (13.68 tok/s prefill, 5.75 tok/s decode). However, the model still emits a helpful response from time to time:

I'm sorry to hear that your VPS has been down for three days. It's quite frustrating, isn't it? Let's try to figure out what might be going on.

First, have you checked your server logs for any error messages or warnings? Sometimes, these can provide clues about what might be causing the issue.

Also, have you tried to ping your server from an external network to see if it's reachable at all? If it's not reachable, it could be a networking issue.

If you haven't already, you might want to try a basic command like df -h to check if there's any disk space left. Sometimes, running out of disk space can cause your VPS to go down. Lastly, have (truncated due to reaching 150-token output limit)

Muse.ai suggested the next step: "deploy the real Gemini-tested prompt". What Muse didn't know was, all I had was just another "thin prompt" (see Backstory section). Writing a real prompt would be my hackathon project.

Given the Gemma 4 model would qualify for an extra prize category at the upcoming hackathon, I also downloaded that model for good measure. Google Gemini told me that Google does not publish the GGUF format required for llama.cpp, but the Unsloth quantized models are high-quality. After some calculations and tests, I identified this launch command to fit in my machine:

llama serve -t 3 -c 4096 -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_XL Enter fullscreen mode Exit fullscreen mode Overriding the Helpful Instincts My project at the hackathon is to write a real system prompt for the unhelpful support agent. The challenge is that, modern instruct models are fine-tuned to be polite and helpful assistants, while I wanted the model to be unhelpful. A simple "spectacularly unhelpful" phrase is insufficient in overriding the model's built-in instinct, but a longer system prompt would have a better chance.

Gemini supplied a starting prompt that consists of four behavioral rules. During the 1-hour build time at the hackathon, I focused on writing out the full system prompt:

The live demo at the hackathon included answering one audience-suggested question: "the price is too high". While the live session was not recorded, here is a longer screencast of how my system prompt behaves on similar questions:

Presently, Frankie lives as a system prompt file that must be pasted into llama-server to interact with. My next step would be integrating Frankie in the yoursunny summer host webpage.

Afterwards, I plan to study how "tool call" works. This would allow Frankie to quote accurate pricing and offer working trial accounts.