Ollama vs MS Foundry Local: Benchmarking Local LLMs on an RTX 5070

Source code — https://github.com/senssei/benchrig

Update, 2026-10-17: the original comparison was flawed. A reader pointed out, correctly, that I used two different models: phi3:mini on Ollama against phi-4-mini on Foundry. Phi-3 and Phi-4 are different models, so the speed and coding gaps I reported mixed the model effect with the runtime effect. I re-ran everything with the same model, Phi-4-mini, on Ollama, direct ONNX Runtime GenAI and the Foundry Local daemon. The conclusions changed, and this post now shows the corrected numbers. The original table is kept further down for the record.

In my last post, I built Foundry Local Chat to bring Microsoft’s Foundry Local daemon into VS Code. That gave me private, zero-token-cost chat right in the editor.

The immediate follow-up question was obvious: how does Microsoft’s ONNX Runtime GenAI actually compare against the undisputed king of local LLM inference, Ollama (llama.cpp)?

Most LLM benchmarks rely on academic benchmarks (MMLU, GSM8K, or synthetic perplexity scores). While useful for foundation model researchers, they don’t answer what developers actually need to know:

1. How fast does it emit tokens during interactive generation?
2. How fast can it ingest a 4,000-token prompt or codebase snippet?
3. Does the generated code actually execute and pass unit tests?
4. What is it doing to VRAM, thermals, and GPU power draw?

To answer those questions with hard data, I built BenchRig.

Photo by Umut Yolac / Unsplash

What it does

BenchRig is an automated evaluation, hardware profiling, and benchmarking suite for local models. Instead of treating models like black boxes, it measures both sides of the pipeline:

• Multi-Runtime Engine Support: Evaluates models running on Ollama (llama.cpp) and Microsoft Foundry Local (ONNX Runtime GenAI) side-by-side.
• Automated Sandboxed Coding: Extracts code blocks from responses, wraps them in test harnesses, and executes them in isolated subprocesses against deterministic unit tests.
• High-Precision Timing: Measures decode tokens/sec, prompt prefill tokens/sec, and Time to First Token (TTFT) via nanosecond-precision streaming.
• Cross-Platform Telemetry: Samples dedicated VRAM, power, and temperatures on NVIDIA GPUs via NVML/nvidia-smi, and Metal buffer memory and GPU utilization on Apple Silicon without root.
• Deterministic 1:1 Alignment: Locks context length (4096), temperature (0.1), top-p (0.9), and seed (42) across runs to eliminate hyperparameter variance.
• Zero-Token-Cost Agent Pairing: BenchRig’s token-savings report shows what offloading coding tasks to a local GPU saves. The agent skills and Model Context Protocol (MCP) servers that do the offloading now live in senssei/local-coders.

The Test Setup

For this head-to-head run, I evaluated Microsoft’s lightweight Phi family on a modern desktop setup:

• GPU: NVIDIA GeForce RTX 5070 (12,227 MB VRAM)
• CPU: AMD Ryzen 7 5700G (16 vCPUs)
• RAM: 31.3 GB System RAM
• OS: WSL2 Linux (Ubuntu 24.04, kernel 6.6.87)
• Model: Phi-4-mini (3.8B) on all three runtimes: Ollama phi4-mini (GGUF Q4_K_M, llama.cpp), direct ONNX Runtime GenAI (int4, CUDA) and the Microsoft Foundry Local daemon (phi-4-mini, int4 ONNX weights with the CUDA provider)
• Method: 3 runs per runtime, one RTX 5070, same prompts and sampling settings
• Coding Suite: 4 deterministic algorithmic challenges executed in isolated Python sandboxes (Nested Dict Flattening, Merge Overlapping Intervals, Valid Balanced Brackets, and LRU Cache).

The Scorecard

Same model, three runtimes, mean of 3 runs (range in brackets):

Metric	Ollama phi4-mini [llama.cpp CUDA]	Direct ONNX Runtime GenAI	MS Foundry Local daemon
Decode Speed	149.1 tok/s (146.1-150.9)	123.7 tok/s (116.9-127.8)	118.3 tok/s (114.4-126.0)
Coding assertions passed	51/51	51/51	51/51
Composite Score	70.0 / 100	70.0 / 100	70.0 / 100
Time to First Token	0.04s	not measured	0.09s
Cold start	3.6s	0.8s	93.9s (daemon model load)

What the same-model run shows

1. Quality is identical

All three runtimes passed every assertion in all 4 scenarios, and the composite score is the same. The 17.6% vs 100% gap in my first version had nothing to do with the runtime. It was Phi-3 mini against Phi-4-mini. The runtime did not change whether the code worked.

2. Ollama decodes about 1.2x faster here

On this setup llama.cpp generated 149 tok/s against about 124 for direct ONNX and 118 for Foundry, and the three runs per runtime were consistent. I would not call this a verdict on the runtimes. The quantizations differ (GGUF Q4_K_M against ONNX int4), so part of the gap may come from the weights format. It is also one model on one GPU under WSL2.

3. Foundry’s cost is startup, not steady state

Once loaded, Foundry’s decode speed is close to direct ONNX. The visible difference is the 94 second model load in the daemon, against under 4 seconds for Ollama.

What I could not compare

• VRAM: another process held 5 GB of VRAM during the Foundry run, so its memory delta is not comparable.
• Prefill and TTFT: the direct ONNX path does not report engine prefill timing in this run.
• Foundry device: BenchRig reported Foundry’s device as unknown. GPU execution rests on the CUDA provider configuration, not on a measurement in this run.

Why the first numbers were misleading

For the record, the original run compared different models:

Metric	phi3:mini [Ollama]	phi-4-mini [MS Foundry]
Decode Speed	94.0 tok/s	130.2 tok/s
Prompt Prefill Speed	2,639.8 tok/s	1,006.8 tok/s
Coding Unit Test Pass Rate	17.6%	100.0%

Those differences are model differences. The NameError failures from phi3:mini (newcur_key, isner_is_valid_brackets) show that sandboxed unit tests catch broken code that fluent text hides, but they say something about Phi-3 mini, not about Ollama.

The lesson: when comparing runtimes, hold the model, its size and its quantization constant, and say which of those you could not match.

The Interesting Bit: Zero-Sudo CUDA 12 on WSL2

Getting Ollama running on WSL2 with CUDA is trivial because Ollama packages its own bundled CUDA binaries inside /usr/local/lib/ollama/cuda_v12/.

Microsoft Foundry Local is different. It is a .NET 9 application using native dynamic libraries (libonnxruntime_providers_cuda.so). It relies on Linux’s dynamic linker (LD_LIBRARY_PATH). Out of the box on a fresh Ubuntu 24.04 WSL2 instance, Foundry silently fell back to the CPU execution provider (CPUExecutionProvider) because 6 essential NVIDIA shared libraries were missing:

• libcudnn.so.9
• libcublasLt.so.12
• libcublas.so.12
• libcurand.so.10
• libcufft.so.11
• libcudart.so.12

Instead of messing with system-wide apt repositories or needing sudo permissions, you can install the official NVIDIA CUDA and cuDNN wheels into your user directory via PyPI:

pip install --break-system-packages --user \
  nvidia-cuda-runtime-cu12 \
  nvidia-cublas-cu12 \
  nvidia-curand-cu12 \
  nvidia-cufft-cu12 \
  nvidia-cudnn-cu12 \
  nvidia-cuda-nvrtc-cu12

Then expose them to the dynamic linker:

export LD_LIBRARY_PATH="$HOME/.local/lib/python3.12/site-packages/nvidia/cudnn/lib:$HOME/.local/lib/python3.12/site-packages/nvidia/cublas/lib:$HOME/.local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH"

Once exported, Foundry’s ONNX Runtime immediately bound to CUDAExecutionProvider, moving decode speed from single-digit tok/s on CPU to over 100 tok/s on the RTX 5070 (118 tok/s for Phi-4-mini).

Agent Pairing via MCP ($0 Token Bill)

The MCP servers and agent skills that came with the first version of this project have moved to their own repository, senssei/local-coders. Wired into an AI coding agent (Cursor, Claude Code and similar), they let you delegate high-volume tasks such as boilerplate, pytest suites and refactors to your local GPU.

BenchRig itself reports the effect: each run prints the prompt and completion tokens handled locally and the cloud API cost they would have incurred. In one benchmark run about 2,400 tokens were handled locally at no API cost.

Quick Start

The project is now called BenchRig and is on PyPI:

# Linux / WSL2
python3 -m venv .venv
source .venv/bin/activate
python -m pip install benchrig

# Run diagnostic check
benchrig --check

# Run a cross-engine comparison on the same model
benchrig --models ollama:phi4-mini,foundry:phi-4-mini --suite coding --runs 3

(On macOS, run ./setup_mac.sh for automated Apple Silicon Metal verification).

What’s next

1. Apple Silicon M-Series Shootout: Running the exact same suite on Unified Memory to compare Metal kernel throughput against desktop CUDA.
2. Context Scaling Curves: Pushing context from 512 up to 8,192 tokens to map the exact threshold where KV cache consumption forces layer offloading.
3. Direct TensorRT (libonnxruntime_providers_tensorrt.so): Compiling optimized .engine caches to see if graph layer fusion can beat raw CUDA kernels.

MIT licensed, not affiliated with Microsoft or Ollama.