Debugging LLM JSON Output: From Frustration to Function Calling

Alright, fellow builders. Let me tell you about a particular rabbit hole I found myself tumbling down for a good couple of weeks. You know that excitement when you get an LLM to do something genuinely useful? For me, it was turning natural language user prompts into perfectly formatted API request bodies. Sounds slick, right?

The Problem: JSON That Just Wouldn't Behave

I was building a tool that lets users describe an action, and the backend would translate it into an API call for various services. Think: "schedule a meeting with Bob next Tuesday about the Q4 report" turning into a POST request to a calendar API. The LLM (mostly GPT-3.5-turbo, then later GPT-4-turbo) was the brain of this operation.

My initial prompts were pretty straightforward: "Generate a JSON object for a POST request to /api/calendar with the following parameters: [user prompt]." For simple cases, it worked like a charm. But then, as the prompts got more complex, or just, you know, different, the JSON started to get… flaky.

I'd hit a wall with json.decoder.JSONDecodeError: Expecting ',' or '}' more times than I care to admit. Sometimes it was a trailing comma after the last item in an array. Other times, an unescaped double quote inside a string. Or a missing closing brace. Even phantom newlines. It was infuriating. I'd fix one prompt, only for another to break unexpectedly. My API calls were failing in about 15-20% of cases, which is just unusable in production.

"Just tell it to output valid JSON!" (The naive approach) My initial thought was that the LLM just needed a firm reminder. I added lines like: "The output must be valid JSON. Do not include any extra text." This helped slightly, but it felt like playing whack-a-mole.

Markdown Code Blocks Putting the expected JSON inside a... block in the prompt was the next step. This is standard advice, and it definitely improved consistency. The LLM understood I wanted code, not just prose. But still, the occasional rogue comma or malformed string crept in.

Providing a Schema / Example JSON I started including a JSON Schema definition or a very explicit example of the expected JSON structure in my system prompt. This helped tremendously with getting the structure right (e.g., ensuring start_time and end_time were present). But it didn't eliminate the subtle syntax errors like improper escaping or trailing commas.

Re-prompting on Failure My code would try-except the json.loads call. If it failed, I'd send the original prompt back to the LLM, maybe adding: "Your previous response was invalid JSON. Please correct it." This was a costly, slow, and often ineffective loop. The LLM would frequently just return the same invalid JSON or a slightly different, but still broken, version.

Pre-processing/Post-processing with Regex or String Hacks I even experimented with writing regex to strip trailing commas or fix common issues. This was a nightmare. It was brittle, prone to introducing new bugs, and felt fundamentally wrong. I quickly abandoned this path.

Switching Models Upgrading to GPT-4-turbo from GPT-3.5-turbo certainly reduced the frequency of issues. GPT-4 is just generally better at following instructions. But it wasn't a silver bullet. The problem persisted, albeit less often.

I'd easily lost a solid week of intermittent debugging, staring at error messages and tweaking prompts, feeling like I was missing something fundamental.

What Actually Fixed It: Stop Asking the LLM to Be a JSON Formatter

The breakthrough came when I realized I was asking the LLM to do something it wasn't optimally designed for: perfectly formatting a string according to a rigid syntax. Its strength is understanding intent and extracting semantic information.

The real fix, which finally brought my failure rate down to almost zero, was embracing structured output – specifically, using the LLM's function calling capabilities (available in models like OpenAI's GPT series).

Instead of asking the LLM to generate JSON, I started asking it to generate arguments for a function that my code would then use to construct the JSON. This shifts the burden of perfect syntax from the LLM to my well-tested, reliable Python json library.

Define a tool (or function) for your API call. This describes the function signature and expected parameters, including their types and descriptions. The LLM uses this definition to understand what kind of data it needs to extract.

functions = [ { "name": "schedule_event", "description": "Schedule a new event in the calendar", "parameters": { "type": "object", "properties": { "title": {"type": "string", "description": "Title of the event"}, "start_time": {"type": "string", "format": "date-time", "description": "ISO 8601 start time"}, "end_time": {"type": "string", "format": "date-time", "description": "ISO 8601 end time"}, "attendees": { "type": "array", "items": {"type": "string", "format": "email", "description": "Email of an attendee"} } }, "required": ["title", "start_time", "end_time"] } } ]

Send the user's prompt to the LLM with this tool definition.

response = client.chat.completions.create( model="gpt-4-turbo", messages=[{"role": "user", "content": "Schedule a meeting with Bob and Alice next Tuesday at 3 PM about Q4 report." }], tools=functions, tool_choice={"type": "function", "function": {"name": "schedule_event"}} )

The LLM returns tool_calls (structured arguments), not raw JSON.

The response tool_calls object contains function.name and function.arguments. The arguments are already a valid JSON string (because the LLM is good at generating these JSON strings to match the schema provided). My Python code then parses this:

python tool_call = response.choices.message.tool_calls if tool_call.function.name == "schedule_event": function_args = json.loads(tool_call.function.arguments) # This is usually perfectly valid now! # Use function_args to construct your API request body api_request_body = { "summary": function_args.get("title"), "startTime": function_args.get("start_time"), "endTime": function_args.get("end_time"), "attendees": [{ "email": email } for email in function_args.get("attendees", [])] } # Then, send api_request_body to your actual calendar API print(json.dumps(api_request_body, indent=2))

This fundamentally changed my approach. The LLM's job became to fill in the slots of a predefined function. My code's job became to call that function and then construct the final API body using those clean, parsed arguments. It's a subtle but powerful distinction.

So, if you're pulling your hair out over malformed JSON from an LLM, take a step back. Consider if you're asking it to do too much. Let it focus on understanding and extraction, and leave the perfect syntax to the tools designed for it. You'll save yourself a lot of headaches.