Your LLM returns JSON that crashes json.loads about 1 time in 20. Here is how to handle it

If you call an LLM and ask for JSON, it works in the demo and then fails in production at the worst moment. The usual culprits are a markdown code fence around the JSON, a sentence before it ("Sure! Here is the JSON:"), or a trailing comma.

Here is a small, step-by-step way to deal with all three in plain Python.

import json import re def extract_json ( text: str ): text = text. strip () # 1. remove ``` { % endraw % } json... { % raw % } `` ` fences fence = re. search ( r " ``` (?:json)?\s*(.*?) ``` ", text, re. DOTALL ) if fence: text = fence. group ( 1 ). strip () # 2. keep only the outermost {...} or starts = [ i for i in ( text. find ( " { " ), text. find ( " [ " )) if i!= - 1 ] if not starts: raise ValueError ( " no JSON found " ) start = min ( starts ) end = max ( text. rfind ( " } " ), text. rfind ( " ] " )) return text [ start: end + 1 ] Enter fullscreen mode Exit fullscreen mode Step 2: Parse, and repair the one common error def parse_llm_json ( text: str ): raw = extract_json ( text ) try: return json. loads ( raw ) except json. JSONDecodeError: # remove trailing commas before } or ] fixed = re. sub ( r ",\s*([}\]]) ", r " \1 ", raw ) return json. loads ( fixed ) # let it raise if still broken Enter fullscreen mode Exit fullscreen mode Step 3: Validate the shape, not just the syntax Valid JSON can still have the wrong keys.

Check the fields you rely on:

def require ( data: dict, schema: dict ): for key, typ in schema. items (): if key not in data: raise KeyError ( f " missing key: { key } " ) if not isinstance ( data [ key ], typ ): raise TypeError ( f " { key } should be { typ. __name__ } " ) return data result = require ( parse_llm_json ( reply ), { " title ": str, " score ": int }) Enter fullscreen mode Exit fullscreen mode Step 4: Retry once with the error message Feed the exact error back to the model. One retry fixes most failures, and more than two retries usually just burns tokens:

def ask_json ( call_llm, prompt, schema, retries = 1 ): reply = call_llm ( prompt ) for attempt in range ( retries + 1 ): try: return require ( parse_llm_json ( reply ), schema ) except ( ValueError, KeyError, TypeError ) as e: if attempt == retries: raise reply = call_llm ( f " { prompt } \n\n Your previous answer was invalid: { e }. " " Return only the corrected JSON, no extra text. " ) Enter fullscreen mode Exit fullscreen mode Better: ask the API to enforce it Parsing defensively is the fallback. Where the provider supports it, use structured outputs so the model is constrained to your schema in the first place. See OpenAI's Structured Outputs guide and Python's json module docs for the error types you can catch ( json.JSONDecodeError carries the line and column).

What is the weirdest broken-JSON output you have gotten from a model? I'd like to add it to the test cases.