Alright, fellow builders. You know the drill: shiny new idea, integrate an LLM, everything's gonna be super smart and dynamic. That was me last week, sketching out a new webhook handler for our internal tools. The goal was simple, or so I thought: feed some unstructured text (think user feedback, support tickets, internal notes) to an LLM, and get back a perfectly structured JSON object. This JSON would then drive subsequent actions – like creating a task, updating a record, or flagging an issue.

My initial mental model was this:

python

Imagine some LLM API call

llm_output = llm_client.generate(prompt=user_text, schema=expected_json_schema)
parsed_data = json.loads(llm_output)

... off to Pydantic validation and processing

Cute, right? Naive, more like. The reality hit faster than a compiler error after a typo. My LLM, bless its token-generating heart, was consistently spitting out JSON that was almost there, but not quite. Sometimes it was missing a comma, sometimes an extra newline, other times it wrapped the JSON in markdown fences ( ), or worse, added conversational filler like "Here's your data:".

This wasn't just a nuisance; it was a showstopper. If json.loads() failed, the whole webhook just… stopped. No task created, no record updated. It took a full day of hair-pulling and muttering to myself before I started seeing the light.

What I Tried (and What Didn't Quite Stick)

1. Aggressive Prompt Engineering: My first instinct was to just yell at the LLM harder. "RETURN ONLY THE JSON. NO OTHER TEXT. DO NOT EXPLAIN." I even included the full Pydantic model as a string in the prompt. This helped marginally. The LLM would usually return JSON, but the edge cases still tripped it up. An empty string somewhere, a missed closing brace. Annoying.

2. Basic String Cleanup: I tried some rudimentary regex to strip markdown fences ( ) or leading/trailing whitespace. This felt super hacky. It worked for some cases, but I knew it was a ticking time bomb. One slightly different output format from the LLM, and my regex would break or, worse, mangle valid JSON.

3. try-except with json.loads: This is, of course, essential. You must wrap your JSON parsing. But simply catching the JSONDecodeError just tells you it failed, not why or how to fix it. My webhook would just log the error and bail, which wasn't good enough for an automated system.

4. Pydantic Validation (after json.loads): Once I did get valid JSON, Pydantic v2 was awesome for schema validation. It ensured the types were correct, required fields were present, etc. But again, if json.loads failed, Pydantic never even got a chance.

I was stuck in a loop: LLM gives bad JSON -> json.loads fails -> I'm sad. I needed something more robust than just hoping the LLM would be perfect every single time.

The Combo That Finally Fixed It (Mostly)

After a lot of trial and error, here’s the stack of strategies that finally brought some sanity to my webhook handler:

1. The LLM's Own JSON Mode (If Available)

This was a huge win. For LLMs like OpenAI's gpt-4-turbo or gpt-3.5-turbo, there's a specific response_format parameter you can pass. Setting response_format={'type': 'json_object'} is a game-changer. It instructs the model to only return valid JSON. This significantly reduced the malformed output right out of the gate.

python

Example using OpenAI Python client (simplified)

from openai import OpenAI
import json

client = OpenAI()

def get_structured_output(prompt_text, system_message):
response = client.chat.completions.create(
model="gpt-4-turbo-preview",
messages=[
{"role": "system", "content": system_message},
{"role": "user", "content": prompt_text}
],
response_format={"type": "json_object"} # <<< The magic!
)
return response.choices[0].message.content

... then parse with json.loads and validate with Pydantic

Even with this, it's not 100% foolproof, especially with really complex or ambiguous prompts, but it gets you 95% of the way there.

2. A Resilient Retry Loop with Feedback

This was the real key to robustness. Instead of just failing on bad JSON, I implemented a retry mechanism that fed the error back to the LLM.

Here’s the basic flow:

1. Attempt 1: Send the prompt to the LLM (with json_object format if possible).
2. Parse and Validate: Try json.loads(). If it works, then pass the dict to your Pydantic model for validation. If either fails, catch the specific error.
3. Retry (with context): If parsing/validation fails, create a new prompt for the LLM. This prompt includes:
   • The original user input.
   • The malformed JSON the LLM previously produced.
   • The exact error message from json.loads or Pydantic.
   • A clear instruction: "The previous output was invalid. Here's the error: [...]. Please correct it."
4. Limit Retries: I set a strict limit, usually 2 or 3 retries. If it fails after that, then I log it as a legitimate failure and probably escalate it for human review.

This turns a hard failure into a soft loop, giving the LLM a chance to self-correct. It sounds simple, but it was surprisingly effective. The LLM, given its own error message, often nails it on the second try. My webhook went from failing on 10-15% of inputs to practically zero.

Wrapping Up

It turns out reliably getting structured data from an LLM isn't just about crafting the perfect initial prompt. It's about designing a system that acknowledges the LLM's inherent occasional weirdness and builds in mechanisms to recover. The combination of leveraging specific LLM features like json_object responses and implementing a smart retry loop with error feedback has been a lifesaver. My webhooks are happy, and so am I. Hopefully, this saves you some grey hairs on your next LLM project! 🚀