DGX Spark MLOps Stack — Dual-Node LLM Pipeline

Built an end-to-end LLM cycle for internal use on two NVIDIA DGX Spark nodes (Production + R&D) — ClearML 10-step pipeline, NGC training containers, AWQ quantization, vLLM inference, an OpenAI-compatible router, and an operations agent that drives the stack from chat.

Architecture

InfoGrab MLOps

2026.03 - 2026.06

Tech Stack

DGX Spark ClearML vLLM NVIDIA NGC AWQ FP8 Quantization LLM Fine-tuning MLOps Hermes Agent Slack Gateway GPU

Highlights

Dual DGX Spark nodes — Production (inference SLA + training) and R&D (long-context, quantization experiments) cleanly separated
10-step ClearML pipeline — fetch → validate → train → convert → quantize → deploy → evaluate → report
Native vLLM with AWQ/FP8 quantization — measured on the same model, hardware, and prompt: BF16 at 8.96 tok/s versus FP8 at 26.05 tok/s
ig-router (FastAPI) — OpenAI-compatible /v1/* endpoints with dynamic routing across dual-serve models
Operations agent (Huginn) — performs deploys, health checks, and rollbacks from chat, with humans and agents sharing one router

Key Tasks

MLOps Pipeline (ClearML)

10-step DAG — fetch_dataset → validate_data → stop_vllm → start_ngc → train → convert → quantize_awq → deploy → evaluate → report_eval
ClearML Server (K8s) + Agent (systemd), experiments / artifacts / metrics centralized
LoRA fine-tuning on NGC 25.12 containers, AWQ W4A16 quantization via llmcompressor
Automated 3-way evaluation with promptfoo (base / base+prompt / fine-tuned)

Inference Stack

Native vLLM transition (Ollama retired); rebuilt gemma4 cu130 custom image
ig-router (FastAPI) — OpenAI-compatible dispatch + max_tokens clamp + SIGHUP routing reload
Dual-serve — gemma4-26B (FP8+AWQ) and domain fine-tuned model served concurrently
Open WebUI integration for chat interface

Infrastructure and Operations

DGX1 (Production) and DGX2 (R&D) duplicated, GitOps one-way sync (Mac SSOT → DGX1 push → DGX2 rebase)
Prometheus / Mimir for vLLM and GPU metrics, with seven alert rules in production
Mutual-exclusion control between vLLM and NGC for shared GPU memory
Teleport-based SSH access, Vault-managed ClearML / HF tokens

삶을 기록하는 두 축

Architecture