From sandbox to infrastructure
The AI Workflow Sandbox grew into a dedicated local AI lab: inference, the application layer, and an agent sandbox have separate responsibilities. The aim is to make a system I can inspect and recover, with private data staying under my control.
Open WebUI provides the chat interface. LiteLLM is the gateway. Ollama handles local inference. The knowledge layer indexes selected lab documentation so answers can refer back to material I actually maintain.
Boundaries worth keeping
- GPU inference is separated from the chat and database services.
- The agent sandbox exposes read-only infrastructure tools.
- A separate, rules-based sanitization step prepares diagnostic text for external assistance.
- Traces and evaluation belong alongside the model, so a plausible answer is not mistaken for a verified one.
The unglamorous part
GPU passthrough has been the most physical failure mode. The guest can stop responding while the virtualization interface still appears to show a running machine. Recovering the model service is then a hardware problem, not a prompt problem.
The small local model also has limits. It can answer a narrow question well and still choose the wrong tool when offered several. Tool access, good retrieval, and good judgment are separate things.
Next on the bench
More repeatable evaluations, better model routing, and carefully scoped observability. This is a personal experiment under active development, not a claim of production readiness for someone else’s data.