Event-Driven AI Orchestration with Kafka in .NET: A Production Blueprint

Posted on Oct 21 • Originally published at amiteshsurwar.com

Event-Driven AI Orchestration with Kafka in.NET: Learn how to build a production‑grade AI workflow in.NET using Kafka, covering outbox, transactional producers, deterministic keys, DLQ, and observability for high‑throughput, low‑cost orchestration.

In a multi‑tenant scheduling platform, a team glued Azure -ai-foundry-vs-azure-openai-service-observability-in-production-20261007" class="internal-link">Azure OpenAI calls into a handful of background services without a solid event backbone. The result was missed appointments, duplicated prompts, and a Kafka topic that grew until the broker crashed. The LLM was fine; the orchestration was broken. The core issue: an event‑driven architecture that ignored the plumbing that keeps AI pipelines reliable at scale.

Our production system processes ~8 k RPS of appointment requests. Each request triggers a language‑model inference, persists a summary, and pushes a notification. The flow looks like:

Queues (Service Bus, RabbitMQ) offer at‑most‑once delivery and limited replay. Kafka gives you a durable log, partitioned streams, and built‑in replayability. For AI, replaying a failed inference without re‑invoking the model saves tokens and cost. However, Kafka’s higher memory footprint and the need for a cluster add operational overhead.

Transactional producers (Kafka transactional.id ) guarantee atomicity across writes and publishes but require idempotent producers and a dedicated transaction manager. The outbox pattern keeps the transaction in the DB, which is simpler and works well with existing relational workloads. The trade‑off is a slight latency increase due to polling.

Blocking the consumer on a 200 ms to 2 s inference keeps the code simple but caps throughput. Offloading to a worker pool and emitting a model.completed event frees the consumer but adds complexity in state management.

Managed services give you elasticity, compliance, and reduced operational toil. Self‑hosted models provide predictable cost and full control over scaling, but you must manage GPUs, patching, and data residency. The choice hinges on data sensitivity, latency budgets, and cost models.

In production we adopt the following pattern:

Use Kafka transactional producers with a unique "transactional.id". In the consumer, call producer.sendOffsetsToTransaction after persisting the result. This guarantees that the offset commit and the result publication happen atomically, eliminating duplicate or lost events.

Choose deterministic keys that correlate with the natural partition key, such as appointmentId or a composite of userId+timestamp. Avoid random UUIDs or null keys, which scatter events across partitions and cause uneven load.

Maintain a processedRequests lookup (e.g., a table or Redis set) keyed by the message key. Before invoking the model, check this store; if the request has already been processed, skip the call. This prevents token waste and duplicate notifications.

Persist the outbox entity in the same DbContext transaction that creates the domain entity. After commit, a lightweight worker polls the outbox table, publishes the event to Kafka, and marks the row as dispatched. This keeps the write and publish atomic without a dedicated transaction manager.

Expose Prometheus counters like dlq_message_total and kafka_consumer_lag. Configure alerts for spikes in DLQ volume or lag thresholds, and surface metrics in Grafana dashboards to detect back‑pressure or processing failures early.

Event‑driven AI orchestration is not a “plug‑and‑play” feature of.NET or Kafka. It requires a disciplined approach to message design, idempotency, transactional guarantees, and observability. By following the patterns above – deterministic keys, outbox + DLQ, transactional producers, and a robust monitoring stack – you can build a production‑grade AI workflow that scales from a handful of requests to tens of thousands per second while keeping costs under control and ensuring data integrity.