# I Wanted to Understand AI Beyond the Prompt Box: My Journey Into DSPy

An exploration of open-source AI, the curiosity that started it, the errors that interrupted it, and the satisfaction of finally making something work.

There is something slightly inconvenient about being curious about too many things.

One day, I want to understand how language models work. Another day, I find myself exploring cybersecurity, quantitative finance, product management, or some entirely different corner of technology. Every field seems to have something interesting hiding underneath the surface, and I have this persistent feeling that I should at least try to understand it.

The problem is that trying everything and actually learning something are two very different things.

It is easy to watch a tutorial, understand the general idea, build something that works once, and move on to the next exciting thing. It is considerably harder to open someone else's codebase, navigate unfamiliar abstractions, figure out why something is failing, and stay long enough to understand what is happening beneath the interface.

For this exploration, I wanted to do the latter.

I decided to investigate DSPy, an open-source framework for programming language-model applications. I did not want to stop at understanding what the framework claimed to do. I wanted to see how it was structured, how its components interacted, and whether I could use what I learned to build a working demonstration of my own.

I started with questions. I ended up with source-code investigations, test runs, configuration errors, a locally running language model, and a much more concrete understanding of what it means to explore an open-source AI project.

When we talk about building with language models, the conversation often revolves around prompts.

We write instructions, adjust the wording, test different variations, and hope the model produces the desired output. When the application becomes more complicated, managing those prompts and their interactions can become a problem of its own.

That made me interested in a different question:

What if I could think about a language-model application as a programmable system rather than a collection of prompts?

DSPy provides abstractions for defining tasks, composing model-powered modules, and optimizing how those modules use language models. Instead of manually treating every prompt as an isolated piece of text, developers can describe the task they want to perform and build a structured program around it.

I found this interesting because it sits at the intersection of several things I wanted to understand: AI, software architecture, experimentation, and the process of turning an idea into a working system.

I could have chosen a framework simply because it was popular or built a small application using an API and stopped there. But I wanted to investigate the framework itself.

What happens when a prediction is called? What does a signature actually represent? How does a higher-level module build on a lower-level one? And what does optimization mean in a framework like this?

I did not have all the answers when I started.

My first step was to explore the repository and identify the main components I wanted to investigate.

Rather than trying to understand the entire codebase at once, I focused on a few concepts that appeared central to DSPy's programming model:

I treated these as connected pieces rather than independent features.

A DSPy signature describes the relationship between the inputs a task receives and the outputs it is expected to produce.

For example, a simple classification task might look like this:

import dspy class ClassifyIntent ( dspy. Signature ): """ Classify a customer message into a support category. """ query: str = dspy. InputField () intent: str = dspy. OutputField () Enter fullscreen mode Exit fullscreen mode This defines a task that accepts a customer query and produces an intent.

The interesting part is what the code does not contain: I am not manually writing out the complete prompt that the model must follow. Instead, I am defining the task's interface and letting DSPy handle the prediction workflow.

That distinction became a useful starting point for understanding the rest of the framework.

classifier = dspy. Predict ( ClassifyIntent ) result = classifier ( query = " I was charged twice for my order. " ) print ( result. intent ) Enter fullscreen mode Exit fullscreen mode

The signature specifies the task. The predictor executes it using the configured language model.

I wanted to understand how that relationship was implemented internally, not just how to use it in a script. That meant looking beyond the example and into DSPy's source code.

This was where the investigation started becoming more interesting. A framework's public interface tells you how developers are expected to use it. Its implementation begins to explain how that interface actually works.

I also investigated dspy.ChainOfThought.

Conceptually, it builds on the prediction machinery by extending a supplied signature with a reasoning field. It then delegates prediction to a standard dspy.Predict instance.

What interested me here was the design pattern.

Rather than creating an entirely separate prediction system, DSPy composes existing functionality into a higher-level module. That makes the framework's abstractions easier to connect conceptually: more sophisticated behavior can be built on top of the underlying prediction mechanism.

I explored DSPy's module structure to understand how these components fit into larger applications.

This mattered because real applications rarely consist of a single prediction. They combine multiple steps, pass outputs between components, and organize model calls into a workflow.

Thinking in terms of modules shifted my attention from individual prompts to the structure of the application around them.

The final major component I focused on was BootstrapFewShot.

DSPy's teleprompting and optimization machinery can use training examples, model-generated demonstrations, and an evaluation metric to construct a predictor with demonstrations selected through an optimization process.

This raised another question for me: how do you demonstrate optimization without confusing the existence of an optimizer with proof that it improves performance?

I would return to that question when I began testing the framework.

Understanding an architecture on paper is useful. Running the code gives you a different kind of understanding.

I created a small investigation demo covering the concepts I had been exploring. The goal was not to build an elaborate product. It was to make the framework's behavior observable through a few focused examples.

The demo covered: https://youtu.be/klimwICb1a4

For the initial experiments, I used DSPy's DummyLM.

A dummy language model is useful when you want to test a program's structure without depending on a real model response. It allows you to investigate how components are called and how data moves through the workflow.

However, I needed to be precise about what that proved.

A successful mock-based demonstration is evidence that the workflow executes. It is not evidence that a real language model will produce accurate outputs.

That meant moving beyond the initial tests and connecting DSPy to a real model.

Open-source exploration sounds particularly elegant when described after everything has worked.

At one point, Python raised a SyntaxError inside:

dspy/predict/chain_of_thought.py Enter fullscreen mode Exit fullscreen mode The error pointed to a line of explanatory prose that Python was interpreting as code.

I had managed to get ordinary article text into a Python source file while working between the investigation and its explanation. Naturally, Python was not impressed by my literary ambitions.

The immediate problem was not DSPy's architecture. The file itself was syntactically invalid.

I inspected the affected section, checked the repository's Git status, and verified the source file rather than immediately reinstalling packages or changing unrelated code.

Once the file was back to valid Python, I tested the import again.

dspy_investigation_demo/.venv/bin/python -c "import dspy; print('DSPy version:', dspy.__version__)" Enter fullscreen mode Exit fullscreen mode

DSPy version: 3.4.1 Enter fullscreen mode Exit fullscreen mode That was a small but important lesson in debugging: identify the layer where the failure occurs before attempting to fix it.

A syntax error in a source file, a missing dependency, and a model connection failure are different problems. Treating them as interchangeable usually creates more problems than it solves.

I had not learned anything profound about language-model optimization from this incident. I had, however, practiced a more useful habit: inspect the evidence before changing the system.

With DSPy importing successfully, I ran the investigation's test suite.

dspy_investigation_demo/.venv/bin/pytest dspy_investigation_demo/test_demo.py -v Enter fullscreen mode Exit fullscreen mode

These tests covered the basic prediction workflow, signature declarations, module composition, and the optimization demonstration.

At this stage, I had evidence that the examples executed successfully under the test conditions.

But I still wanted to see the framework communicate with a real model.

That became the next stage of the investigation.

I already had Ollama installed and the gemma4:latest model available locally.

My next goal was straightforward: configure DSPy to use Gemma and create a small classification task.

I started with a customer-support example.

I was charged twice for my order. Please refund the extra payment.

The classification task itself was simple. The integration was where the debugging began.

InvalidRequestError: 404 page not found (ollama, HTTP 404) Enter fullscreen mode Exit fullscreen mode This was confusing because the model was installed and Ollama was available.

I needed to distinguish between two questions:

I tested Ollama's API directly rather than assuming that the failure meant the model was broken.

I sent a direct request to the local chat endpoint, asking Gemma to reply with a fixed phrase.

My first attempt failed because the JSON contained an extra comma and a misplaced closing bracket.

invalid character ']' looking for beginning of value Enter fullscreen mode Exit fullscreen mode That was not a model failure or a DSPy failure. It was simply an invalid request body.

After correcting the JSON, the request succeeded:

{ "model": "gemma4:latest", "message": { "role": "assistant", "content": "CONNECTION WORKS" }, "done": true } Enter fullscreen mode Exit fullscreen mode The exact response structure returned by Ollama included additional metadata, but the important result was clear: Gemma responded successfully to a direct API call.

I now had evidence that Ollama was functioning independently of DSPy.

The remaining issue was the way DSPy routed the model request.

I adjusted the model configuration to use DSPy's LiteLLM compatibility engine explicitly, keeping the Ollama endpoint and model name consistent with the working direct request.

lm = dspy. LM ( " ollama_chat/gemma4:latest ", api_base = " http://localhost:11434 ", api_key = "", model_type = " chat ", engine = " litellm ", cache = False, ) dspy. configure ( lm = lm ) Enter fullscreen mode Exit fullscreen mode

With that configuration, the demo finally executed successfully.

The important part of this debugging process was not simply discovering which setting worked. It was separating the model server from the framework integration, testing them independently, and using the results to narrow down the failure.

I went from a generic HTTP 404 to a working local inference pipeline by breaking the problem into smaller, testable pieces.

With the integration working, I ran the classification script again.

--- DSPy + Gemma: Real Model Inference --- Input: I was charged twice for my order. Please refund the extra payment. Predicted intent: billing Inference completed successfully. Enter fullscreen mode Exit fullscreen mode It was a small result.

No complicated interface. No elaborate multi-agent system. No dramatic claim about changing the future of AI.

Just a customer message going into a program, a real local language model processing the request, and a category coming out.

But this result represented a different kind of progress from the earlier mock-based demonstration.

I had now verified that DSPy could execute the task using Gemma through Ollama on my machine.

I then extended the experiment into an interactive customer-support demo. Instead of supplying a single hard-coded input, I could enter different messages and have the program return a category and a suggested response.

The program used the same signature and prediction module for each request.

That made the demonstration more useful because I could observe how the same task definition behaved with different inputs.

It also introduced an important limitation: the categories and responses were model-generated, not guaranteed answers. A working inference pipeline does not automatically establish classification accuracy, response quality, or reliability across a wider dataset.

Those questions would require a separate evaluation.

For this investigation, the goal was narrower: understand the framework's structure and demonstrate a functioning real-model integration.

The technical results were useful, but the process taught me a few things that were not specific to DSPy.

It is easy to learn the syntax for creating a predictor.

It takes more work to understand how signatures define task interfaces, how modules compose, how prediction execution is organized, and where optimization fits into the design.

Reading source code alongside running small experiments helped connect these ideas.

I did not need to understand every file in the repository. I needed to identify the components relevant to my questions and follow their relationships.

That is a much more manageable way to approach a large codebase.

The five passing tests demonstrated that my mock-based workflows executed under the tested conditions.

The Gemma experiment demonstrated that real inference worked.

Neither result, on its own, proved that DSPy produced better outputs than a manually written prompt or that BootstrapFewShot improved task performance.

To establish that, I would need a defined dataset, an appropriate metric, a baseline, and a controlled comparison.

That distinction matters because AI projects can look impressive long before their actual performance has been measured.

I wanted my investigation to distinguish what I had demonstrated from what remained to be tested.

The syntax error, malformed JSON, and HTTP 404 were different failures at different layers.

Fixing them required different approaches.

I learned to check whether the source file was valid, whether the local model server responded, whether the request itself was well formed, and whether DSPy's provider configuration was correct.

Debugging became less about randomly changing settings and more about asking which assumption had failed.

I started this exploration because I wanted to understand something unfamiliar.

But curiosity alone can easily become endless browsing: another repository, another framework, another concept that seems interesting enough to investigate.

The useful part was choosing a specific set of questions and following them far enough to produce evidence.

I examined the architecture, built a small demo, tested the workflow, connected a real model, and recorded the result.

I did not master DSPy in a day. That was never a realistic goal.

I developed a more concrete understanding of a framework by working through a small, defined investigation.

That is a better outcome than simply collecting another technology name to put on a profile.

This investigation gave me a working foundation, but it also raised questions I have not yet answered.

The most interesting next step would be to evaluate optimization rather than merely demonstrate that an optimizer runs.

That would help answer a more substantive question: under what conditions does DSPy's optimization process improve the task, and how much improvement can be measured?

I would also want to investigate failure cases, including ambiguous messages, overlapping categories, and inputs that do not fit the predefined labels.

Those experiments would move the project from an architectural investigation and working demonstration toward a more rigorous evaluation.

For now, I have deliberately kept the scope small. A clear experiment with honest limitations is more useful than a complicated demo whose results I cannot explain.

I started this investigation with a familiar kind of curiosity: I wanted to understand another part of AI, and I did not want to settle for a superficial explanation.

I chose DSPy because its approach to programming language-model applications gave me a concrete architectural question to explore. I investigated its signatures, prediction modules, composition, and optimization workflow. I created a demonstration, ran five tests using a dummy language model, and then connected DSPy to Gemma through Ollama for real inference.

Along the way, I encountered errors that forced me to distinguish between source-code problems, malformed requests, and integration issues. I fixed them, verified the working configuration, and extended the experiment into an interactive demo.

The final output was modest. The learning process was more valuable.

I am still interested in too many things to pretend that I have settled on one narrow path. I want to explore AI, cybersecurity, quantitative systems, software engineering, and the ways technology becomes useful to people.

But I am beginning to understand that exploring everything does not mean trying to master everything simultaneously.

Sometimes it means choosing one question, following it beyond the documentation, making something work, understanding why it works, and being honest about what you have not yet proved.

I do want to become the kind of person who can approach something unfamiliar, investigate it seriously, and build a working understanding of it.

For this project, that meant going from What exactly is DSPy doing? to a working local inference pipeline.

For the next one, the question will probably be different.

Scope note: The mock-based tests verified the example workflows. The Gemma integration verified that local inference worked for the demonstrated task. A controlled benchmark comparing prediction quality before and after optimization was outside the scope of this investigation.