AI agents make music, art and 3D with Infinite headless engine

Posted on Oct 24 • Originally published at infinitedevices.org

An AI agent that can run shell commands can make music, pictures and 3D with Infinite, because the engine runs headless. The executable inside the app is also a command-line tool. It reads a plain-text patch and writes stills, video and audio, with no window and no mouse. A patch is text, so an agent such as Claude can write it, check it, listen to it and fix it in a loop.

This post shows that loop, then three small patches: a sound, a picture and a 3D scene. It builds on the headless render walkthrough and the Headless User Guide.

Most creative tools hide their state behind a window. An agent can only guess at what a click did. An Infinite patch is different in four ways:

The output is also deterministic: the same patch gives the same frames and the same samples, so an agent can change one thing and trust that any difference came from that change.

The Headless User Guide gives the order, and it works as a prompt for an agent as it stands:

Infinite ships this loop as an agent skill, infinite-patch-authoring, written for AI coding agents. Point your agent at it and at the Headless User Guide, then describe what you want.

This is the smallest audio patch that ships with Infinite. A random note generator plays a Wavetable synth into an Audio Out.

infinite-patch 1 node 1 Notes Random Note Generator id notes end node 2 Synths Wavetable id synth end node 3 Utility Audio Out id speakers end note synth 0 notes aud speakers 0 synth Enter fullscreen mode Exit fullscreen mode An agent cannot hear, so it measures. --audio-summary writes loudness, peaks, clipping, silence, a spectrum and a tempo guess to a JSON file, and --wav keeps the audio.

Infinite --validate music.inf Infinite --audio-summary music.inf summary.json --wav music.wav --duration 8 Enter fullscreen mode Exit fullscreen mode Two warnings are worth teaching your agent to react to. W_SILENT means no sound reached an Audio Out, so check that notes reach the synth. W_CLIPPING means samples hit full scale, so lower a gain or add a Limiter. Add --stems id,id --stems-dir dir to write one WAV per named node, and --notes events.json to play your own note events into the patch. The agent can also ask --describe for the Synths and Audio Effects node types and build a fuller arrangement from them.

Pictures work the same way. The Field Pixel node draws an image from a formula, which suits an agent because it is short code with no hidden state. This patch ships as the Field starter template:

infinite-patch 1 node 1 Source FieldPixel id pix s code col = vec3(0.5 + 0.5 * sin(uv.x * 10.0 + t), 0.5 + 0.5 * sin(uv.y * 10.0 - t), 0.6); end node 2 Utility Output id out end cable out 0 pix Enter fullscreen mode Exit fullscreen mode The agent looks before it commits to a video. A contact sheet costs seconds:

Infinite --frame art.inf 0,1,2,3 pics/ --contact-sheet sheet.png Enter fullscreen mode Exit fullscreen mode The result line includes frame_stats with brightness, the share of black and white pixels, and a histogram per frame, so the agent can notice a black or washed-out frame without opening the image. Field is the built-in language for shaders, geometry and audio kernels; the Field Language Manual explains it.

Because sound and pictures live in one graph, an agent can wire them together. Audio Analyze listens to a synth and outputs level, low, mid, high, onset and eight frequency bands, and any of those can drive a modulatable control. The Music and Image starter does it with two mod lines:

aud hear 0 synth mod shape sizeX hear 0 0 1 0.5 0.12 0.4 mod shape sizeY hear 0 0 1 0.5 0.12 0.4 Enter fullscreen mode Exit fullscreen mode The last two numbers are absolute values the control sweeps between, not percentages. Put the Audio Analyze after your effects so the picture follows what you hear. The first steps tutorial builds this one by hand.

A 3D scene has four parts, and the starter template wires them like this:

infinite-patch 1 node 1 3D Torus Knot id knot end node 2 3D Camera id cam f orbitPerBeat 0.2 end node 3 3D Light id sun end node 4 3D Render 3D id view end node 5 Utility Output id out end geo view 0 knot geo view camera cam geo view light_1 sun cable out 0 view Enter fullscreen mode Exit fullscreen mode Render a few stills, then a clip:

Infinite --frame scene.inf 0,2,4 pics/ --contact-sheet sheet.png Infinite --render scene.inf scene.mp4 --duration 6 --fps 30 Enter fullscreen mode Exit fullscreen mode Give your agent the 3D rules from the guide. Primitives have fixed position, scale and rotation, so to animate a shape it should send it through a Transform node and modulate that. Infinite --explain scene.inf --all prints each piece's world bounding box, which tells the agent where things really are before it blames the lights. If the picture is black or faint, it should check the camera distance, the light intensities and the Render 3D ambient colour first.

Short, specific briefs work best. For example:

The agent loops through describe, write, validate, explain, look and listen, and hands back the patch plus the files. You can open the same.inf file in the app and keep working by hand, because it is the same engine.

Infinite is free and open source (MIT) for macOS, Windows and Linux. Download it from infinitedevices.org, read the Headless User Guide, and give your agent the patch-authoring skill. The source is at github.com/n1m21n/Infinite.

Originally published at https://infinitedevices.org/blog/ai-agents-make-music-art-3d-headless.html. Infinite is a free, open-source DAW for visuals, music and code: infinitedevices.org.