Claude als Regisseur: Ein Stack für komplette YouTube-Videos in Blender
Mit LLM-Agents, die Blender steuern, soll Claude eigenständig komplette YouTube-Videos erzeugen, von Skript über 3D-Szenen und Animationen bis zu Kamerafahrten und Rendering. Dafür braucht es einen Stack aus Python-Komponenten, die natürliche Sprache mit Blender verbinden.
Ich habe mit LLM-Agenten experimentiert, die Blender steuern. Das Ziel: Claude soll ein komplettes YouTube-Video erzeugen, inklusive Skript, 3D-Szenen, Animationen, Kamerafahrten und Rendering, ganz ohne menschliches Zutun. Das Ergebnis ist ein Stapel aus Python-Komponenten, die natürliche Sprache mit Blenders Python-API verbinden. Dieser Beitrag beschreibt die Architektur, den Code und die harten Grenzen, auf die ich gestoßen bin.
Blender ist eine GUI-Anwendung mit einer riesigen Python-API ( bpy ). Ein LLM kann Python-Code ausgeben, aber dessen direkte Ausführung ist gefährlich und fehleranfällig. Man braucht:
Wenn man Claude auf die einfache Weise bittet, ein "Blender-Skript zu schreiben", erhält man Code, der oft wegen API-Änderungen oder fehlendem Kontext scheitert.
Ich habe eine Agentenschleife mit Werkzeugnutzung gebaut (inspiriert von ReAct-artigem Prompting), bei der Claude eine Reihe von Python-Funktionen aufruft, die Blender über bpy manipulieren. Der Agent läuft innerhalb von Blenders Python-Umgebung (oder in einem Subprozess mit blender --background ). Die Schleife endet, wenn der Agent ein Finish-Tool aufruft oder ein Schrittlimit überschreitet.
So wird die Ausführung beliebigen Codes vermieden. Jedes Werkzeug ist eine spezifische, überprüfbare Funktion.
Führe dieses Skript im Scripting-Tab von Blender aus oder via blender --background --python bridge.py. Es lauscht auf einem lokalen Socket auf JSON-Befehle.
# bridge.py import bpy import json import socket import threading HOST = ' 127.0.0.1 ' PORT = 65432 # Whitelisted operations def op_create_cube ( name, location ): bpy. ops. mesh. primitive_cube_add ( location = location ) obj = bpy. context. active_object obj. name = name return { ' name ': obj. name, ' location ': list ( obj. location )} def op_set_camera ( location, rotation ): cam = bpy. context. scene. camera if not cam: bpy. ops. object. camera_add () cam = bpy. context. active_object bpy. context. scene. camera = cam cam. location = location cam. rotation_euler = rotation return { ' camera ': cam. name } def op_render ( output_path, frame_start = 1, frame_end = 1 ): bpy. context. scene. frame_start = frame_start bpy. context. scene. frame_end = frame_end bpy. context. scene. render. filepath = output_path bpy. ops. render. render ( animation = True ) return { ' rendered ': output_path } OPERATIONS = { ' create_cube ': op_create_cube, ' set_camera ': op_set_camera, ' render ': op_render, } def handle_client ( conn ): with conn: data = conn. recv ( 4096 ). decode ( ' utf-8 ' ) if not data: return try: msg = json. loads ( data ) op = msg. get ( ' operation ' ) args = msg. get ( ' args ', {}) if op not in OPERATIONS: response = { ' error ': f ' Unknown operation: { op } ' } else: result = OPERATIONS [ op ]( ** args ) response = { ' result ': result } except Exception as e: response = { ' error ': str ( e )} conn. sendall ( json. dumps ( response ). encode ( ' utf-8 ' )) def start_server (): with socket. socket ( socket. AF_INET, socket. SOCK_STREAM ) as s: s. bind (( HOST, PORT )) s. listen () print ( f ' Bridge listening on { HOST }: { PORT } ' ) while True: conn, _ = s. accept () threading.
Thread ( target = handle_client, args = ( conn,)). start () # Start server in a background thread so Blender's UI remains responsive threading. Thread ( target = start_server, daemon = True ). start () Enter fullscreen mode Exit fullscreen mode Sicherheitshinweis: Diese Bridge führt ausschließlich freigegebene Funktionen aus. Gebe sie niemals in nicht vertrauenswürdigen Netzwerken frei.
Binde sie an 127.0.0.1 und verwende bei Bedarf eine Authentifizierung.
Definiere die Werkzeuge in einem Format, das mit Anthropolics Tool-Use-API kompatibel ist.
# tools.py tools = [ { " name ": " create_cube ", " description ": " Create a cube mesh at a given location ", " input_schema ": { " type ": " object ", " properties ": { " name ": { " type ": " string " }, " location ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 } }, " required ": [ " name ", " location " ] } }, { " name ": " set_camera ", " description ": " Set the active camera ' s location and rotation (Euler angles in radians) ", " input_schema ": { " type ": " object ", " properties ": { " location ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 }, " rotation ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 } }, " required ": [ " location ", " rotation " ] } }, { " name ": " render ", " description ": " Render an animation to the given output path ", " input_schema ": { " type ": " object ", " properties ": { " output_path ": { " type ": " string " }, " frame_start ": { " type ": " integer " }, " frame_end ": { " type ": " integer " } }, " required ": [ " output_path " ] } } ] Enter fullscreen mode Exit fullscreen mode 3. Agentenschleife Diese Schleife sendet Nachrichten an Claude, führt Tool-Aufrufe über die Bridge aus und hängt die Ergebnisse an. Sie endet, wenn Claude ein Finish-Tool aufruft (nicht gezeigt) oder nach max_steps.
# agent.py import json import socket import anthropic client = anthropic. Anthropic ( api_key = " YOUR_API_KEY " ) def call_bridge ( operation, args ): with socket. socket ( socket. AF_INET, socket.
SOCK_STREAM ) as s: s. connect (( ' 127.0.0.1 ', 65432 )) s. sendall ( json. dumps ({ ' operation ': operation, ' args ': args }). encode ( ' utf-8 ' )) response = s. recv ( 4096 ). decode ( ' utf-8 ' ) return json. loads ( response ) def run_agent ( prompt, max_steps = 20 ): messages = [{ " role ": " user ", " content ": prompt }] for step in range ( max_steps ): response = client. messages. create ( model = " claude-3-5-sonnet-20241022 ", max_tokens = 1024, tools = tools, messages = messages ) # Append assistant's response messages. append ({ " role ": " assistant ", " content ": response. content }) tool_calls = [ block for block in response. content if block. type == " tool_use " ] if not tool_calls: print ( " No tool calls, ending. " ) break for tool_call in tool_calls: if tool_call. name == " finish ": print ( " Agent finished. " ) return result = call_bridge ( tool_call. name, tool_call. input ) messages. append ({ " role ": " user ", " content ": [{ " type ": " tool_result ", " tool_use_id ": tool_call. id, " content ": json. dumps ( result ) }] }) else: print ( f " Reached max steps ( { max_steps } ).
" ) Enter fullscreen mode Exit fullscreen mode 4. Orchestrierung Starte Blender mit der Bridge und führe dann den Agenten aus einem separaten Prozess aus.
blender --background --python bridge.py & python agent.py Enter fullscreen mode Exit fullscreen mode Wichtigste Erkenntnisse Eine Tool-Whitelist schlägt Codegenerierung: Statt Claude beliebigen bpy-Code schreiben zu lassen, sollte man eine kleine Menge sicherer Operationen bereitstellen. Das verhindert katastrophale Fehler und erleichtert das Debugging. Explizite Terminierung: Die Agentenschleife stoppt bei einem Finish-Tool-Aufruf oder einem Schrittlimit.
Ohne diese Maßnahme könnte sie endlos laufen. Sicherheit zuerst: Niemals die Ausgabe eines LLM per eval ausführen. Meine Bridge verwendet eine Whitelist und läuft lokal.
Wenn generierter Code zwingend ausgeführt werden muss, dann in einer Sandbox (zum Beispiel mit RestrictedPython oder in einem Container). Iterative Verfeinerung ist entscheidend: Der Agent kann Ergebnisse beobachten (zum Beispiel Objektpositionen) und nachjustieren. Für ein vollständiges Video würde man zusätzlich Werkzeuge für Keyframing, Text und Compositing ergänzen.
Das Rendering ist der Flaschenhals: Blender-Renderings können Stunden dauern. Der Agent sollte die gesamte Szene planen, bevor er ein Rendering auslöst. Dieser Stapel ist ein Ausgangspunkt.
Mit weiteren Werkzeugen (zum Beispiel add_keyframe, set_material ) kann Claude zunehmend komplexere Animationen steuern. Der Code ist auf GitHub verfügbar, der Link findet sich in den Kommentaren.
Automatisch übersetzt. Das Original auf Englisch ist unten aufklappbar.
Original (Englisch) anzeigen
I've been experimenting with LLM agents that control Blender. The goal: have Claude generate a complete YouTube video—script, 3D scenes, animations, camera moves, and render—without human intervention. The result is a stack of Python components that bridge natural language to Blender's Python API. This post outlines the architecture, the code, and the hard limits I hit.
Blender is a GUI application with a massive Python API ( bpy ). An LLM can emit Python code, but directly executing it is dangerous and brittle. You need:
Naively asking Claude to "write a Blender script" yields code that often fails due to API changes or missing context.
I built a tool-using agent loop (inspired by ReAct-style prompting) where Claude calls a set of Python functions that manipulate Blender via bpy. The agent runs inside Blender's Python environment (or a subprocess with blender --background ). The loop terminates when the agent calls a finish tool or exceeds a step limit.
This avoids eval of arbitrary code. Each tool is a specific, auditable function.
Run this script in Blender's scripting tab or via blender --background --python bridge.py. It listens on a local socket for JSON commands.
# bridge.py import bpy import json import socket import threading HOST = ' 127.0.0.1 ' PORT = 65432 # Whitelisted operations def op_create_cube ( name, location ): bpy. ops. mesh. primitive_cube_add ( location = location ) obj = bpy. context. active_object obj. name = name return { ' name ': obj. name, ' location ': list ( obj. location )} def op_set_camera ( location, rotation ): cam = bpy. context. scene. camera if not cam: bpy. ops. object. camera_add () cam = bpy. context. active_object bpy. context. scene. camera = cam cam. location = location cam. rotation_euler = rotation return { ' camera ': cam. name } def op_render ( output_path, frame_start = 1, frame_end = 1 ): bpy. context. scene. frame_start = frame_start bpy. context. scene. frame_end = frame_end bpy. context. scene. render. filepath = output_path bpy. ops. render. render ( animation = True ) return { ' rendered ': output_path } OPERATIONS = { ' create_cube ': op_create_cube, ' set_camera ': op_set_camera, ' render ': op_render, } def handle_client ( conn ): with conn: data = conn. recv ( 4096 ). decode ( ' utf-8 ' ) if not data: return try: msg = json. loads ( data ) op = msg. get ( ' operation ' ) args = msg. get ( ' args ', {}) if op not in OPERATIONS: response = { ' error ': f ' Unknown operation: { op } ' } else: result = OPERATIONS [ op ]( ** args ) response = { ' result ': result } except Exception as e: response = { ' error ': str ( e )} conn. sendall ( json. dumps ( response ). encode ( ' utf-8 ' )) def start_server (): with socket. socket ( socket. AF_INET, socket. SOCK_STREAM ) as s: s. bind (( HOST, PORT )) s. listen () print ( f ' Bridge listening on { HOST }: { PORT } ' ) while True: conn, _ = s. accept () threading.
Thread ( target = handle_client, args = ( conn,)). start () # Start server in a background thread so Blender's UI remains responsive threading. Thread ( target = start_server, daemon = True ). start () Enter fullscreen mode Exit fullscreen mode Security note: This bridge executes only whitelisted functions. Never expose it to untrusted networks.
Bind to 127.0.0.1 and use authentication if needed.
Define tools in a format compatible with Anthropic's tool-use API.
# tools.py tools = [ { " name ": " create_cube ", " description ": " Create a cube mesh at a given location ", " input_schema ": { " type ": " object ", " properties ": { " name ": { " type ": " string " }, " location ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 } }, " required ": [ " name ", " location " ] } }, { " name ": " set_camera ", " description ": " Set the active camera ' s location and rotation (Euler angles in radians) ", " input_schema ": { " type ": " object ", " properties ": { " location ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 }, " rotation ": { " type ": " array ", " items ": { " type ": " number " }, " minItems ": 3, " maxItems ": 3 } }, " required ": [ " location ", " rotation " ] } }, { " name ": " render ", " description ": " Render an animation to the given output path ", " input_schema ": { " type ": " object ", " properties ": { " output_path ": { " type ": " string " }, " frame_start ": { " type ": " integer " }, " frame_end ": { " type ": " integer " } }, " required ": [ " output_path " ] } } ] Enter fullscreen mode Exit fullscreen mode 3. Agent Loop This loop sends messages to Claude, executes tool calls via the bridge, and appends results. It terminates when Claude calls a finish tool (not shown) or after max_steps.
# agent.py import json import socket import anthropic client = anthropic. Anthropic ( api_key = " YOUR_API_KEY " ) def call_bridge ( operation, args ): with socket. socket ( socket. AF_INET, socket.
SOCK_STREAM ) as s: s. connect (( ' 127.0.0.1 ', 65432 )) s. sendall ( json. dumps ({ ' operation ': operation, ' args ': args }). encode ( ' utf-8 ' )) response = s. recv ( 4096 ). decode ( ' utf-8 ' ) return json. loads ( response ) def run_agent ( prompt, max_steps = 20 ): messages = [{ " role ": " user ", " content ": prompt }] for step in range ( max_steps ): response = client. messages. create ( model = " claude-3-5-sonnet-20241022 ", max_tokens = 1024, tools = tools, messages = messages ) # Append assistant's response messages. append ({ " role ": " assistant ", " content ": response. content }) tool_calls = [ block for block in response. content if block. type == " tool_use " ] if not tool_calls: print ( " No tool calls, ending. " ) break for tool_call in tool_calls: if tool_call. name == " finish ": print ( " Agent finished. " ) return result = call_bridge ( tool_call. name, tool_call. input ) messages. append ({ " role ": " user ", " content ": [{ " type ": " tool_result ", " tool_use_id ": tool_call. id, " content ": json. dumps ( result ) }] }) else: print ( f " Reached max steps ( { max_steps } ).
" ) Enter fullscreen mode Exit fullscreen mode 4. Orchestration Start Blender with the bridge, then run the agent from a separate process.
blender --background --python bridge.py & python agent.py Enter fullscreen mode Exit fullscreen mode Key Takeaways Tool whitelisting beats code generation: Instead of letting Claude write arbitrary bpy code, expose a small set of safe operations. This prevents catastrophic mistakes and makes debugging easier. Explicit termination: The agent loop stops on a finish tool call or a step limit.
Without this, it could run indefinitely. Security first: Never eval LLM output. My bridge uses a whitelist and runs locally.
If you must execute generated code, sandbox it (e.g., RestrictedPython or a container). Iterative refinement is key: The agent can observe results (e.g., object locations) and adjust. For a full video, you'd add tools for keyframing, text, and compositing.
Rendering is the bottleneck: Blender renders can take hours. The agent should plan the entire scene before triggering a render. This stack is a starting point.
With more tools (e.g., add_keyframe, set_material ), Claude can direct increasingly complex animations. The code is available on GitHub—link in comments.