Distributed Hardware & Cluster Architecture (Backend)
The foundation of this AI assistant is a bare-metal backend powered by a dual NVIDIA DGX Spark cluster. To ensure these machines function as a unified compute node rather than isolated servers, I integrated Ray for distributed workload management and NVIDIA’s Collective Communications Library (NCCL) for high-bandwidth, multi-GPU topology. This architecture allows for the deployment of massive, high-parameter models that would otherwise exceed the VRAM limits of a single system, ensuring rapid, scalable processing.
Containerized Inference & Manual Model Orchestration
To maintain a modular and conflict-free environment, the entire backend software stack is containerized via Docker. Rather than relying on a single monolithic AI, I engineered a highly modular deployment strategy using Ollama, LM Studio, and NVIDIA NIM. Depending on the exact requirements of my workload, I manually orchestrate the deployment of specific models: loading Qwen 3.5 (122B) or Llama 4 (128x17B MoE) for heavy coding, Nemotron 3 Super for writing, and Gemma 3 (27B) for math and science. Furthermore, I frequently run multiple models in parallel—such as simultaneously operating a vision model and a coding model—to tackle complex, cross-domain problems. This manual orchestration is the backbone of my active development pipeline, providing the isolated compute power necessary to write and debug scripts for my other core engineering projects. To maximize throughput during these intensive sessions, I utilize speculative decoding to accelerate token generation.
Edge Client Hardware & Dynamic Compute Fallback
True utility requires secure, low-latency accessibility from anywhere. I established a zero-trust mesh network using Tailscale, allowing remote devices to interface directly with the DGX backend. My primary edge client is an ASUS ROG Flow Z13 ACRNM (8GB VRAM, 32GB RAM), which operates with a dynamic compute fallback protocol. Depending on network availability, the tablet can run highly quantized, specialized models entirely locally—including Gemma 3 (4B Instruct QAT), Qwen 3 Coder (30B), Nemotron 3 Nano, and custom instruction-tuned ‘uncensored’ variants (e.g., Qwen 3.5 9B) for unrestricted logical processing. I have even successfully deployed heavily quantized 120B models locally for offline emergencies. When immediate, high-speed compute is required, the Tailscale network seamlessly offloads the inference request back to the DGX cluster.
UI, Multi-Modal Capabilities, & RAG Implementation
To interact with the models efficiently, I deployed Open WebUI as the primary user interface, providing a streamlined experience natively hosted on my hardware. Beyond text generation, the cluster is equipped with ComfyUI to handle complex, node-based visual generation workflows. Furthermore, I integrated Retrieval-Augmented Generation (RAG) pipelines, allowing the AI to query my local databases and personal documentation without ever exposing proprietary data to the public internet.
Ongoing Development & Custom Fine-Tuning
This infrastructure is an actively evolving ecosystem. Current development is focused on utilizing the DGX cluster’s compute power for custom model tuning. I am selectively fine-tuning LLaMA architectures using NVIDIA NeMo frameworks to train the assistant on highly specific domain knowledge and personal coding syntaxes, slowly transitioning the tool from a generalized AI into a highly specialized, personalized engineering assistant.
24/7 Autonomous Coding Agent
I engineered sloppilot — a fully autonomous coding agent that runs 24/7 on my DGX Spark, building features for my AI CAD program with zero human intervention. After designing the core architecture and initial primitives, the agent took over completely: 130+ parametric 3D shapes, 39+ self-authored helper functions, 60+ regression tests, and 10,916 lines of CAD engine code — all agent-built across 14 continuous days and still running.
The system runs Qwen 3.5 122B, or any model that is needed, locally through Ollama, or any address that is needed, at ~22 tok/s with zero cloud cost. Safety is the primary constraint — the agent can only read and edit whitelisted project files, vision verification toggles on/off, and the only outbound communication is email for status updates. No GitHub pushes, no client messages, no system file access. Every change passes automated regression testing with a 97.6% pass rate across 3,174 checks before acceptance.