Prism: Fixing Foundry Local and Unifying ONNX & Ollama on Linux & WSL2

Source code — https://github.com/senssei/prism-local
Documentation — https://senssei.github.io/prism-local/
PyPI package — pip install "prism-local[cuda,pull]"

In my last post, I benchmarked Microsoft Foundry Local against Ollama using Ollama BenchRig. The benchmark showed something tantalizing: on an NVIDIA RTX 5070, Microsoft’s ONNX Runtime GenAI engine with CUDA acceleration reached 130 tok/s decode with sub-100ms Time to First Token (TTFT). For code generation and interactive chatting, that is blindingly fast.

Following up on Foundry Local Chat, my plan was simple: ditch the cloud tokens entirely, pull Microsoft’s Phi models, and make Foundry Local my default local inference daemon for development, Cursor, and agentic workflows on WSL2.

Then reality struck.

Trying to actually use Microsoft’s official Foundry Local CLI (cli-preview-0.10.3) as a daily driver on Linux and WSL2 proved to be an absolute minefield of missing libraries, silent CPU fallbacks, leaking VRAM, and broken catalogs.

To solve this, I built Prism (prism-local).

Photo by Evie S. / Unsplash

---

The Autopsy: What Went Wrong with Foundry Local on WSL2

Before explaining what Prism does, it is worth dissecting why the official Foundry Local distribution falls apart on Linux and WSL2. A deep dive into the microsoft/foundry-local repository revealed several major architectural friction points:

1. The Tale of Two Foundries: Modern C++ vs. Frozen .NET

The microsoft/foundry-local repository is in the middle of a massive generational rewrite:

microsoft/foundry-local/
├── sdk/                     # Legacy v1.x (.NET Core, powers CLI 0.10.3)
└── sdk_v2/                  # Modern v2.x (Native C++20 libfoundry_local.so, MIT)
    └── cpp/src/ep_detection/nvml_gpu_detector.cc  # Native NVML detector

The modern SDK (sdk_v2, v2.0.1, published September 2026 under MIT) is written in native C++20 and features an NvmlGpuDetector that dynamically opens libnvidia-ml.so.1 on Linux. It detects RTX GPUs inside WSL2 without breaking a sweat.

However, the standalone binary distributed via GitHub releases (cli-preview-0.10.3, distributed under a proprietary Microsoft license) is frozen on the legacy v1 .NET Core architecture. It does not use sdk_v2.

2. The WMI Trap: “System | GPU | Not detected”

Because the legacy CLI uses .NET 9 and relies on Windows Management Instrumentation (WMI: Win32_VideoController) and DirectX/DirectML interfaces, running foundry status inside WSL2 produces this depressing result:

System | GPU | Not detected

WMI does not exist in Linux user space. Because the CLI thinks you have no GPU, its Azure catalog client hardcodes a filter that restricts model downloads exclusively to CPU builds:

foundry model run qwen2.5-coder-7b
# Downloads: qwen2.5-coder-7b-instruct-generic-cpu
# Result: 9.1 tok/s, 7.74s TTFT

At 9 tokens/second, local AI feels like a sluggish chore rather than a superpower.

3. The Linux CUDA Packaging Gap

Inside Microsoft’s cuda_ep_manifest.cc, the dynamic execution provider (EP) bootstrapper reveals why GPU downloads fail on Linux:

Component	Windows Archive (WindowsX64Manifest)	Linux Archive (LinuxX64Manifest)
CUDA Toolkit Libraries	cublas64_12.dll, cudart64_12.dll (~640 MB)	❌ NOT BUNDLED
cuDNN Libraries	cudnn64_9.dll, cudnn_ops64_9.dll (~704 MB)	❌ NOT BUNDLED
ORT CUDA Provider	onnxruntime_providers_cuda.dll (~256 MB)	libonnxruntime_providers_cuda.so (~448 MB)

On Windows, Microsoft bundles the full CUDA 12 and cuDNN 9 runtime. On Linux, they assume the user already has matching cuBLAS and cuDNN shared libraries installed in system paths. On a clean Ubuntu 24.04 WSL2 installation, those libraries are missing, causing the CUDA provider registration to crash.

4. The Silent CPU Fallback (The “Ghost GPU”)

This was the most insidious issue of all. In ONNX Runtime GenAI, if the CUDA provider cannot resolve its dynamic dependencies, or if a model’s genai_config.json ships with an empty provider_options list (which many Hugging Face repositories do!), the engine silently falls back to CPUExecutionProvider.

It does not throw an error. It does not exit. It prints no warning. It simply runs inference on your CPU at 8 tokens/second. Because many models have folder names like phi-4-mini-cuda-gpu, developers assume their GPU is working when it is actually their CPU doing the heavy lifting.

5. Ephemeral Ports & Leaky VRAM

Even if you patched your way through with the Python wheel tricks I outlined in the previous post:

1. Random Ports: The Foundry daemon binds a random ephemeral port every time it launches, making it impossible to point Cursor, Cline, or local scripts to a static endpoint without reading ~/.foundry/daemon.json.
2. Catalog Bug (#1109): Active upstream issue #1109 confirms that even when the CUDA EP is installed, the Azure catalog service frequently returns 0 CUDA models.
3. VRAM Leak (#1079): Active upstream issue #1079 notes that calling foundry model unload retains ~99% of GPU device allocations. In our tests, foundrylocald.real held ~5.5 GB of VRAM hostage until the process was forcefully killed.

---

From foundry_wsl Bridge to Prism

My initial impulse was to write a lightweight bridge toolkit called foundry_wsl:

• A reverse proxy that read ~/.foundry/daemon.json and exposed the random daemon port at a predictable 127.0.0.1:5272.
• A configuration injector that patched genai_config.json with CUDA provider options.
• PowerShell scripts to run Foundry on the Windows host and connect from WSL2 over the virtual ethernet gateway.

The bridge worked, but it felt like duct-taping a broken car. Why wrestle with a closed-source, proprietary .NET binary and an uncooperative cloud catalog when we can talk to ONNX Runtime GenAI directly, and combine it with Ollama under one unified umbrella?

That realization gave birth to Prism.

---

What is Prism?

Prism (prism-local) is an open-source, multi-engine local AI CLI and OpenAI-compatible server specifically built for Linux and WSL2.

It puts ONNX Runtime GenAI (CUDA or CPU) and Ollama / llama.cpp (GGUF) behind:

• One unified CLI: prism status, doctor, list, pull, run, chat, serve, benchmark, mcp, connect.
• One static /v1/chat/completions endpoint on a fixed port (127.0.0.1:5272).
• Ready-made connectors for Cursor, Cline, Claude Desktop, and Antigravity MCP.

┌──────────────────────────────────────────────────────────────────────────────┐
│                             Prism Architecture                               │
└──────────────────────────────────────────────────────────────────────────────┘

  Clients:  Cursor │ Cline │ Claude Desktop │ Antigravity │ curl / Python
                              │
                              ▼
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ Prism Server (http://127.0.0.1:5272/v1)                                  │
  │ - Fixed Loopback Binding (127.0.0.1) & Host Header Rebinding Guard       │
  │ - API Key Authentication & CORS Origin Allowlist                         │
  │ - 10 MB Payload Guard & OpenAI-Compatible JSON Errors                    │
  └─────────────────────────────┬────────────────────────────────────────────┘
                                │
                                ▼
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ Model Catalog & Routing (prism/catalog.py)                               │
  │ - Curated Aliases (phi-4-mini -> GPU variant if present)                 │
  │ - Chat Template Resolution (Phi, ChatML/Qwen, Llama 3, DeepSeek)         │
  └──────────────────────┬─────────────────────────────┬─────────────────────┘
                         │                             │
       Ollama / GGUF Route                             │ ONNX GenAI Route
                         ▼                             ▼
  ┌───────────────────────────────┐     ┌────────────────────────────────────┐
  │ Ollama Bridge                 │     │ ActiveEngineManager (Thread-Locked)│
  │ - Talks to local Ollama daemon│     │ - Serialized single-model residency│
  │ - Streaming HTTP chunks       │     │ - Direct NVML GPU telemetry        │
  └───────────────────────────────┘     │ - In-process CUDA preloading       │
                                        │ - Explicit Execution Provider      │
                                        └──────────────────┬─────────────────┘
                                                           │
                                                           ▼
                                        ┌────────────────────────────────────┐
                                        │ ONNX Runtime GenAI                 │
                                        │ (CUDA 13 / libonnxruntime-genai)   │
                                        └────────────────────────────────────┘

---

Core Engineering Highlights

1. Direct NVML Hardware Probing

Prism does not touch WMI. It talks directly to NVIDIA’s Management Library (libnvidia-ml.so.1), which WSL2 maps into /usr/lib/wsl/lib/. It immediately detects your GPU name, total VRAM, and driver compute capability without requiring root or external utilities.

2. Explicit Execution Providers — No Silent Fallbacks

Where a model runs is explicitly controlled via --device auto|cuda|cpu (or $PRISM_DEVICE):

• --device auto (default): Tries CUDA. If CUDA libraries fail to load, it warns on stderr with the exact missing dependency and runs on CPU.
• --device cuda: Demands GPU execution. If CUDA fails, it aborts immediately with a clear error rather than secretly burning CPU cycles.
• --device cpu: Runs on CPU vector instructions.

Prism overrides the model’s genai_config.json decoder session options in memory at load time, ignoring whatever empty or broken provider lists came with the download. Furthermore, every response, benchmark, and the GET /health endpoint explicitly reports the active execution provider used.

3. Safe Diagnostics: prism doctor

Diagnosing CUDA issues usually involves running scripts that risk crashing Python with segmentation faults when a mismatched library is loaded via dlopen.

prism doctor takes a much smarter approach: it runs ldd against ONNX Runtime’s CUDA provider shared library (libonnxruntime_providers_cuda.so). This inspects ELF dynamic dependencies without executing foreign library code:

$ prism doctor
Hardware:
  GPU 0: NVIDIA GeForce RTX 5070 (12227 MB total, 5521 MB free)
  Driver compute capability: 12.0

Software:
  Python: 3.12.3 (/home/senssei/03-foundy-local/.venv/bin/python3)
  ONNX Runtime GenAI: Installed (0.16.0)
  CUDA execution provider: Can resolve all library dependencies
  Ollama daemon: Reachable at http://localhost:11434 (7 models installed)

If a library is missing (for instance, a cuBLAS or cuFFT version mismatch), prism doctor outputs:

❌ CUDA execution provider: cannot load: missing shared libraries: libcublas.so.13, libcublasLt.so.13
   Models will fall back to CPU. Install CUDA libraries matching your onnxruntime-genai build.

4. In-Process CUDA Preloading

Python pip wheels from NVIDIA (nvidia-cublas-cu13, nvidia-cudnn-cu13, etc.) place their .so files under site-packages/nvidia/<pkg>/lib.

However, setting os.environ["LD_LIBRARY_PATH"] inside a running Python script does not affect the running process’s dynamic linker. By the time Python executes, ld.so has already initialized.

Prism solves this cleanly: on startup, it locates the installed nvidia-* wheel directories and uses ctypes.CDLL with RTLD_GLOBAL to preload the necessary CUDA and cuDNN libraries into process memory before onnxruntime_genai is ever imported.

5. Single-Resident Engine Lock & Clean VRAM Release

ONNX Runtime GenAI model sessions are not safe to share across concurrent threads, and swapping models unloads the previous model.

Prism wraps the inference engine in an ActiveEngineManager with an acquisition lock:

• Concurrent requests queue cleanly instead of racing the engine or corrupting memory.
• When an ONNX model is unloaded or the server shuts down, resources are freed immediately. In our tests, VRAM dropped back down with 0 MB leaked.

---

Real-World Benchmarks on RTX 5070

To establish reproducible performance, I ran prism benchmark on the reference machine (NVIDIA GeForce RTX 5070 12 GB, WSL2 Ubuntu 24.04, driver 615.71, CUDA 13, onnxruntime-genai-cuda 0.16.0):

prism benchmark phi-4-mini --device cuda
prism benchmark phi-4-mini --device cpu

Execution Provider	Generation (Decode)	Time to First Token (TTFT)	VRAM Delta	Memory Lifecycle
CUDA (GPU)	79 – 98 tok/s	0.45 – 0.50 s	+4.5 GB	Completely freed on exit
CPU	7 – 9 tok/s	~0.60 s	0 MB	System RAM only

A Note on Reproducibility:
In my earlier BenchRig post, an isolated harness with a short synthetic prompt achieved 130 tok/s and 90 ms TTFT. On a live daily-driver desktop sharing ~6.7 GB of VRAM with browsers and editors, decode settled at a consistent 79–98 tok/s. What matters is that prism benchmark prints the provider it actually used, so you never fall into the silent CPU trap.

---

Cursor, Cline & Zero-Cost Agent Integration

One of the biggest friction points with local models is setting up IDEs and agents. Prism includes built-in connectors that configure everything in a single command.

1. Cursor

prism connect cursor --test --export-rules --export-mcp

This tests the server connection, prints Cursor’s OpenAI-compatible provider configuration (http://localhost:5272/v1), writes .cursorrules, and exports .cursor/mcp.json.

2. Cline

prism connect cline --test --export-mcp

Configures Cline to use Prism’s endpoint and exports approved tools to cline_mcp_settings.json.

3. Model Context Protocol (MCP) Server

Prism includes a native JSON-RPC 2.0 Model Context Protocol server (prism mcp) exposing 5 tools to agents like Antigravity, Claude Desktop, and Cursor:

• prism_ask_coder: Offload boilerplate generation, unit test writing, and bug fixes to your local GPU.
• prism_code_review: Perform automated security and edge-case code reviews.
• prism_list_models: Query available ONNX and Ollama models.
• prism_get_status: Live GPU telemetry, VRAM usage, and active execution provider.
• prism_benchmark: Real-time micro-benchmark execution.

You can wire it into your favorite agent with one command:

prism connect mcp --target antigravity --write  # ~/.gemini/config/mcp_config.json
prism connect mcp --target claude --write       # Claude Desktop
prism connect mcp --target cursor --write       # .cursor/mcp.json

---

Quickstart: Running Prism in 60 Seconds

1. Installation

Create a Python 3.11+ virtual environment and install Prism with the CUDA and Hugging Face extras:

python3 -m venv .venv
source .venv/bin/activate
pip install "prism-local[cuda,pull]"

(If you don’t have an NVIDIA GPU, pip install prism-local is sufficient for CPU and Ollama routing).

2. Verify Hardware with Doctor

prism doctor

3. Pull a Model

Pull either a Hugging Face ONNX model or an Ollama GGUF model:

# Pull Hugging Face ONNX model to ~/.prism/models:
prism pull phi-4-mini

# Or pull an Ollama model into your Ollama instance:
prism pull ollama:qwen2.5-coder:7b

4. Interactive Chat & One-Shot Execution

# Run one-shot completion via CLI:
prism run phi-4-mini "Write a Python context manager for timing code execution."

# Or launch interactive streaming chat in your terminal:
prism chat phi-4-mini

5. Start the OpenAI Server

prism serve

The server binds to 127.0.0.1:5272/v1. You can immediately test it with standard OpenAI client tooling or curl:

curl -s http://127.0.0.1:5272/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "phi-4-mini",
    "stream": true,
    "messages": [
      {"role": "user", "content": "Explain the difference between ONNX and GGUF."}
    ]
  }'

---

Conclusion & What’s Next

Microsoft’s ONNX Runtime GenAI is a formidable inference engine, but developer tools need to meet software engineers where they actually work: on Linux, inside WSL2, in terminals, and inside modern code editors without proprietary friction.

Prism bridges that gap. By combining ONNX GenAI and Ollama into a unified, zero-silent-fallback runtime with first-class IDE and MCP integrations, local AI on Linux is finally a turnkey, reliable experience.

What’s Next for Prism:

1. Tool Calling in /v1/chat/completions: Bringing full native OpenAI tool/function calling to local ONNX models.
2. Stop Sequences & Embedding Endpoints: Adding /v1/embeddings support for local RAG pipelines.
3. Multi-Model Queueing: Dynamic model hot-swapping based on request headers.

Prism is open-source under the Apache-2.0 license:

• GitHub: https://github.com/senssei/prism-local
• Docs: https://senssei.github.io/prism-local/
• PyPI: prism-local

Try it out with pip install "prism-local[cuda,pull]" and let me know how it performs on your rig!