My AI Scrum Buddy Kept Missing Action Items: Here's How I Fixed It

We run a pretty small, nimble team, and like many, we've been trying to leverage AI wherever we can to shave off those repetitive, brain-drain tasks. One area I was particularly excited about was automating our daily scrum note processing.

Our team's daily scrum notes are, shall we say, free-form. Some days it's a bulleted list, other days a short paragraph in Slack. I built a little internal AI assistant using the OpenAI API (initially gpt-3.5-turbo) to digest these notes. The dream was simple: reliably pull out any clear action items (who, what, by when) and blockers (what's stuck, who's affected, what's needed).

Initially, it was... well, a bit of a disaster. It was like teaching a toddler to find specific toys in a messy room. Sometimes it'd grab the right thing, other times it'd bring me a sock, or insist there were no toys at all. Accuracy was hovering around 60% on a good day. It would frequently miss obvious action items like, "I'll push the new auth branch to staging by EOD Friday," or incorrectly tag something vague as a blocker. Worse, sometimes it would hallucinate an action item that wasn't even there!

"Spent most of yesterday debugging the payment gateway. Looks like a misconfiguration on the dev server. I'll need to coordinate with ops to get this sorted out by tomorrow if possible. Hopefully, we can get that migration done soon too."

{ "action_items": [ {"owner": "(unknown)", "task": "Debug payment gateway", "deadline": "yesterday"}, {"owner": "(unknown)", "task": "Migration", "deadline": "soon"} ], "blockers": [] }

Completely missing the actual action to coordinate with ops and failing to identify the payment gateway as a blocker, not an action I'd just done.

I spent a solid two weeks, probably 16-20 hours total, just tweaking things.

The real breakthrough wasn't a magic prompt or a new model. It was a combination of gently nudging our team to add minimal structure to their notes and then using a robust validation step.

Tiny Bit of User Guidance: We asked everyone to add a simple prefix for explicit actions and blockers. So instead of just writing something, they'd add:

This wasn't a strict template, just a keyword. People could still write their free-form updates around these. It felt like a small compromise for a huge gain.

Revised, Focused Prompt: The prompt now explicitly looked for these prefixes, essentially turning the AI's task from inference to parsing with intelligence:

markdown You are an AI assistant for a development team. Your task is to extract action items and blockers from daily scrum notes. Only extract items that are explicitly prefixed with "AI Note: Action -" or "AI Note: Blocker -".

If no such prefixes are found, return empty lists for both.

Output a JSON object with two top-level keys: "action_items" (a list of action item objects) and "blockers" (a list of blocker objects).

Pydantic Validation (The Strong Net): On the backend, after the LLM returned its JSON, I added a Python script that used Pydantic to strictly validate the structure. If the AI, despite its best efforts, messed up a key name or returned an unexpected type, Pydantic would catch it immediately. This allowed me to log the malformed outputs and flag them for manual review, ensuring that anything processed was always correctly structured.

This hybrid approach shot our reliability through the roof, hitting over 95% accuracy for correctly extracted and structured items. The LLM's job became much clearer, and the light structuring meant less guessing. It's not truly free-form anymore, but the compromise was so minor, and the productivity gain so significant, that our team adopted it without a second thought. Sometimes, the best AI solution involves a little bit of human-AI collaboration on the input side, too.