Spring AI: First Steps

Ensure that Ollama is running and the model is downloaded:

ollama pull qwen2.5:3b Enter fullscreen mode Exit fullscreen mode spring.ai.ollama.chat.options.model = qwen2.5:3b spring.ai.ollama.base-url = http://localhost:11434 # Optional: temperature (0.0 - deterministic, 1.0 - creative) spring.ai.ollama.chat.temperature = 0.7 Enter fullscreen mode Exit fullscreen mode You can use any other model — the name must exactly match the output of ollama list.

package com.example.demo; import org.springframework.ai.chat.client.ChatClient; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class AiController { private final ChatClient chatClient; public AiController ( ChatClient. Builder chatClientBuilder ) { this. chatClient = chatClientBuilder. build (); } @GetMapping ( "/ai/hello" ) public String aiHello () { // Formulate the prompt String prompt = "Say 'Hello, World!' and add one short, interesting fact about the Java language."; // Call the model and get the response return chatClient. prompt (). user ( prompt ). call (). content (); //.content() extracts just the response string from the ChatResponse object } } Enter fullscreen mode Exit fullscreen mode Open http://localhost:8080/ai/hello in your browser.

Spring AI can return not just plain text, but a ready-made Java object. Create a record:

package com.example.demo; public record RoutingIntent ( String targetService, String action, String userId, int priority ) {} Enter fullscreen mode Exit fullscreen mode Add an endpoint to AiController.java:

@GetMapping ( "/ai/route" ) public RoutingIntent routeRequest ( @RequestParam String text ) { String systemPrompt = """ You are a smart API gateway. Analyze the user's request and extract data for routing. If a field is unknown, leave it empty (null or 0). """; return chatClient. prompt (). system ( systemPrompt ). user ( text ). call (). entity ( RoutingIntent. class ); } Enter fullscreen mode Exit fullscreen mode Example: http://localhost:8080/ai/route?text=Disable notifications for user user-999

Response: A ready-made JSON, serialized by Spring. No manual parsing required.

Give the model "tools" — Java methods annotated with @Tool. If the user asks, "What is the status of the payment service?", the LLM won't hallucinate an answer. Instead, it tells Spring AI: "Hey, call the getServiceStatus("payment") method, and I will use its result."

Create the SystemTools.java class. The @Tool annotation makes the method visible to the LLM.

package com.example.demo; import org.springframework.stereotype.Service; import org.springframework.ai.tool.annotation.Tool; import org.springframework.ai.tool.annotation.ToolParam; @Service public class SystemTools { @Tool ( description = "Checks the current status and latency (in ms) of the specified microservice" ) public String getServiceStatus ( @ToolParam ( description = "Name of the service, e.g., 'payment', 'auth', 'database'" ) String serviceName ) { // A real call to Prometheus, ClickHouse, or your proxy logger could go here System. out. println ( ">>> LLM called Java method for service: " + serviceName ); return switch ( serviceName. toLowerCase ()) { case "payment" -> "Status: OK, average latency 45ms."; case "auth" -> "Status: DEGRADED, latency 300ms, experiencing timeouts."; default -> "Status: Unknown service or no data available."; }; } } Enter fullscreen mode Exit fullscreen mode Update AiController.java by adding a dependency on SystemTools:

package com.example.demo; import org.springframework.ai.chat.client.ChatClient; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; @RestController public class AiController { private final ChatClient chatClient; // Inject our tools public AiController ( ChatClient. Builder chatClientBuilder, SystemTools systemTools ) { this. chatClient = chatClientBuilder. defaultTools ( systemTools ) // <--- this is the key line. build (); } @GetMapping ( "/ai/tool" ) public String askSystem ( @RequestParam String question ) { return chatClient. prompt (). user ( question ). call (). content (); } } Enter fullscreen mode Exit fullscreen mode Request: http://localhost:8080/ai/tool?question=What is the current status of the payment and auth services?

The LLM will decide to call getServiceStatus("payment"), Spring AI will execute the method, and the model will formulate the final response based on the result.

You need to formulate prompts dynamically, substituting data from the context (user name, request parameters, database results). Hardcoding prompts is a bad practice. Spring AI allows you to create prompt templates with variables, similar to Thymeleaf or Mustache.

Add a new endpoint to AiController.java:

package com.example.demo; import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.prompt.PromptTemplate; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; import java.util.Map; @RestController public class AiController { private final ChatClient chatClient; private final SystemTools systemTools; public AiController ( ChatClient. Builder chatClientBuilder, SystemTools systemTools ) { this. chatClient = chatClientBuilder. defaultTools ( systemTools ). build (); } //... your previous /ai/hello, /ai/route, /ai/tool methods... @GetMapping ( "/ai/template" ) public String generateWithTemplate ( @RequestParam String userName, @RequestParam String serviceName, @RequestParam String action ) { // Create a prompt template with variables String templateText = """ You are a system administrator. User {userName} wants to perform the action '{action}' on the {serviceName} service.

Provide a brief instruction on how to do this safely. """; // Populate the variables PromptTemplate promptTemplate = new PromptTemplate ( templateText ); String finalPrompt = promptTemplate. render ( Map. of ( "userName", userName, "serviceName", serviceName, "action", action )); // Send the finalized prompt to the model return chatClient. prompt (). user ( finalPrompt ). call (). content (); } } Enter fullscreen mode Exit fullscreen mode Example: http://localhost:8080/ai/template?userName=John&serviceName=payment&action=restart

John, to safely restart the payment service, follow these steps: 1. Ensure there are no active transactions... 2. Execute the restart command... 3. Check the service logs... Enter fullscreen mode Exit fullscreen mode Templates can be stored in separate files (.txt,.prompt ) and versioned separately from the codebase.

If your proxy communicates with the user in real time (e.g., a chatbot or voice assistant), waiting 2–3 seconds for a full response is bad UX. Streaming allows delivering the response token by token as soon as they are generated. For chat interfaces, use.stream() and Flux<String>. Spring Boot automatically converts this to Server-Sent Events (SSE).

<dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-webflux </artifactId> </dependency> Enter fullscreen mode Exit fullscreen mode Add resources/static/stream-test.html:

<!DOCTYPE html> <html lang= "en" > <head> <meta charset= "UTF-8" > <meta name= "viewport" content= "width=device-width, initial-scale=1.0" > <title> AI Stream Test </title> <style> body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; } #question { width: 70%; padding: 10px; font-size: 16px; } button { padding: 10px 20px; font-size: 16px; cursor: pointer; } #response { margin-top: 20px; padding: 15px; border: 1px solid #ccc; border-radius: 8px; min-height: 150px; background: #f9f9f9; white-space: pre-wrap; font-size: 16px; line-height: 1.6; }.cursor { display: inline-block; width: 2px; height: 18px; background: black; animation: blink 0.7s infinite; vertical-align: text-bottom; } @keyframes blink { 0 %, 100 % { opacity: 1; } 50 % { opacity: 0; } } </style> </head> <body> <h2> 🤖 AI Streaming Chat </h2> <div> <input type= "text" id= "question" placeholder= "Enter a question..." value= "Tell me a short fact about Java" > <button onclick= "startStream()" > Send </button> </div> <div id= "response" ></div> <script> function startStream () { const question = document. getElementById ( ' question ' ). value; const responseDiv = document. getElementById ( ' response ' ); // Clear previous response responseDiv. innerHTML = ' <span class="cursor"></span> '; // Use fetch + ReadableStream (more reliable than EventSource) fetch ( ' /ai/stream?question= ' + encodeURIComponent ( question )). then ( response => { const reader = response. body. getReader (); const decoder = new TextDecoder ( ' utf-8 ' ); function read () { reader. read (). then (({ done, value }) => { if ( done ) { // Remove cursor when the stream ends const cursor = responseDiv. querySelector ( '.cursor ' ); if ( cursor ) cursor. remove (); return; } // Decode bytes to text const chunk = decoder. decode ( value, { stream: true }); // Parse SSE: each "data:..." line const lines = chunk. split ( ' \n ' ); let text = ''; for ( const line of lines ) { if ( line. startsWith ( ' data: ' )) { // Remove "data:" prefix and append to text text += line. substring ( 5 ); } } // Insert text before the cursor const cursor = responseDiv. querySelector ( '.cursor ' ); if ( cursor ) { cursor. insertAdjacentText ( ' beforebegin ', text ); } else { responseDiv. textContent += text; } // Read the next chunk read (); }); } read (); }). catch ( error => { responseDiv. textContent = ' Error: ' + error. message; }); } </script> </body> </html> Enter fullscreen mode Exit fullscreen mode Add the streaming endpoint:

import org.springframework.http.MediaType; import reactor.core.publisher.Flux; //... inside AiController... @GetMapping ( value = "/ai/stream", produces = MediaType. TEXT_EVENT_STREAM_VALUE + ";charset=UTF-8" ) public Flux < String > streamResponse ( @RequestParam String question ) { return chatClient. prompt (). user ( question ). stream (). content (); } Enter fullscreen mode Exit fullscreen mode Open in your browser: http://localhost:8080/stream-test.html

The response will appear gradually, word by word, just like in ChatGPT. This works via SSE (Server-Sent Events) — the browser receives a continuous data stream and renders it in real time.