Ollama vs MS Foundry Local: Benchmarking Local LLMs on an RTX 5070

Source code — https://github.com/senssei/ollama-benchrig

In my last post, I built Foundry Local Chat to bring Microsoft’s Foundry Local daemon into VS Code. That gave me private, zero-token-cost chat right in the editor.

The immediate follow-up question was obvious: how does Microsoft’s ONNX Runtime GenAI actually compare against the undisputed king of local LLM inference, Ollama (llama.cpp)?

Most LLM benchmarks rely on academic benchmarks (MMLU, GSM8K, or synthetic perplexity scores). While useful for foundation model researchers, they don’t answer what developers actually need to know:

1. How fast does it emit tokens during interactive generation?
2. How fast can it ingest a 4,000-token prompt or codebase snippet?
3. Does the generated code actually execute and pass unit tests?
4. What is it doing to VRAM, thermals, and GPU power draw?

To answer those questions with hard data, I built Ollama BenchRig.

Photo by Umut Yolac / Unsplash

What it does

Ollama BenchRig is an automated evaluation, hardware profiling, and benchmarking suite for local models. Instead of treating models like black boxes, it measures both sides of the pipeline:

• Multi-Runtime Engine Support: Evaluates models running on Ollama (llama.cpp) and Microsoft Foundry Local (ONNX Runtime GenAI) side-by-side.
• Automated Sandboxed Coding: Extracts code blocks from responses, wraps them in test harnesses, and executes them in isolated subprocesses against deterministic unit tests.
• High-Precision Timing: Measures decode tokens/sec, prompt prefill tokens/sec, and Time to First Token (TTFT) via nanosecond-precision streaming.
• Cross-Platform Telemetry: Samples dedicated VRAM, power, and temperatures on NVIDIA GPUs via NVML/nvidia-smi, and Metal buffer memory and GPU utilization on Apple Silicon without root.
• Deterministic 1:1 Alignment: Locks context length (4096), temperature (0.1), top-p (0.9), and seed (42) across runs to eliminate hyperparameter variance.
• Zero-Token-Cost Agent Pairing: Built-in Model Context Protocol (MCP) servers to offload coding tasks directly to local GPUs from agents like Antigravity, Cursor, or Claude Code.

The Test Setup

For this head-to-head run, I evaluated Microsoft’s lightweight Phi family on a modern desktop setup:

• GPU: NVIDIA GeForce RTX 5070 (12,227 MB VRAM)
• CPU: AMD Ryzen 7 5700G (16 vCPUs)
• RAM: 31.3 GB System RAM
• OS: WSL2 Linux (Ubuntu 24.04, kernel 6.6.87)
• Compared Pair: phi3:mini (Ollama / llama.cpp) vs phi-4-mini (MS Foundry / ONNX Runtime GenAI)
• Coding Suite: 4 deterministic algorithmic challenges executed in isolated Python sandboxes (Nested Dict Flattening, Merge Overlapping Intervals, Valid Balanced Brackets, and LRU Cache).

The Scorecard

Here is the direct 1:1 head-to-head comparison generated by BenchRig:

Metric	phi3:mini [Ollama / llama.cpp]	phi-4-mini [MS Foundry / ONNX Runtime GenAI]	Delta / Advantage
Decode Speed (Generation)	94.0 tok/s	130.2 tok/s	MS Foundry is 1.4x faster
Prompt Prefill Speed	2,639.8 tok/s	1,006.8 tok/s	Ollama is 2.6x faster
Time to First Token (TTFT)	0.21s	0.09s	MS Foundry is 2.3x lower latency
Coding Unit Test Pass Rate	17.6%	100.0%	MS Foundry leads by +82.4%
Peak Memory Footprint	11,135 MB (VRAM)	10,875 MB (RAM/VRAM)	Fits comfortably in 12GB VRAM
Composite Score	31.1 / 100	64.0 / 100	MS Foundry (+32.9)

The Surprising Findings

1. Decode Speed vs. Prefill Throughput

The most striking result is the stark split between token generation and prompt ingestion:

• Generation (Decode): ONNX Runtime GenAI with CUDA reached 130.2 tok/s with an astonishing 0.09s TTFT. The response begins almost instantaneously, making interactive chat feel faster and smoother than cloud APIs.
• Ingestion (Prefill): Ollama’s llama.cpp CUDA kernels crushed prompt ingestion at 2,639.8 tok/s — 2.6x faster than Foundry. When passing large context payloads (like entire source files or long system prompts), llama.cpp digests the prompt much more aggressively.

2. Sandboxes Don’t Lie: Perplexity vs. Unit Tests

Both models produced fluent explanations and markdown code blocks. But when BenchRig extracted the implementations and executed them against isolated assertions:

• phi-4-mini scored 100%, cleanly passing all test assertions across all 4 scenarios.
• phi3:mini collapsed to 17.6%. The failures weren’t subtle mathematical edge cases; the model hallucinated non-existent variables and helper functions:

  # Nested Dict Flattening:
  NameError: name 'newcur_key' is not defined. Did you mean: 'new_key'?
  
  # Valid Balanced Brackets:
  NameError: name 'isner_is_valid_brackets' is not defined.
  

Standard perplexity scoring would have treated these as near-perfect text. Only sandboxed unit testing exposes that the code won’t actually run.

The Interesting Bit: Zero-Sudo CUDA 12 on WSL2

Getting Ollama running on WSL2 with CUDA is trivial because Ollama packages its own bundled CUDA binaries inside /usr/local/lib/ollama/cuda_v12/.

Microsoft Foundry Local is different. It is a .NET 9 application using native dynamic libraries (libonnxruntime_providers_cuda.so). It relies on Linux’s dynamic linker (LD_LIBRARY_PATH). Out of the box on a fresh Ubuntu 24.04 WSL2 instance, Foundry silently fell back to the CPU execution provider (CPUExecutionProvider) because 6 essential NVIDIA shared libraries were missing:

• libcudnn.so.9
• libcublasLt.so.12
• libcublas.so.12
• libcurand.so.10
• libcufft.so.11
• libcudart.so.12

Instead of messing with system-wide apt repositories or needing sudo permissions, you can install the official NVIDIA CUDA and cuDNN wheels into your user directory via PyPI:

pip install --break-system-packages --user \
  nvidia-cuda-runtime-cu12 \
  nvidia-cublas-cu12 \
  nvidia-curand-cu12 \
  nvidia-cufft-cu12 \
  nvidia-cudnn-cu12 \
  nvidia-cuda-nvrtc-cu12

Then expose them to the dynamic linker:

export LD_LIBRARY_PATH="$HOME/.local/lib/python3.12/site-packages/nvidia/cudnn/lib:$HOME/.local/lib/python3.12/site-packages/nvidia/cublas/lib:$HOME/.local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH"

Once exported, Foundry’s ONNX Runtime immediately bound to CUDAExecutionProvider, boosting decode speed from ~14 tok/s on CPU to 130.2 tok/s on the RTX 5070.

Agent Pairing via MCP ($0 Token Bill)

BenchRig includes built-in Model Context Protocol servers (ollama_mcp_server.py and foundry_mcp_server.py).

By wiring these into AI coding agents (Antigravity, Cursor, Claude Code), you can delegate high-volume tasks — generating boilerplate, writing pytest suites, and performing AST refactoring — to your local GPU. In this single test run alone, 2,474 tokens were offloaded locally with zero cloud API spend.

# Example: Generate unit tests using local Qwen/Phi via CLI skill
python3 .agents/skills/foundry-coder/scripts/ask_foundry.py code \
  --task "Implement a thread-safe LRU cache with TTL expiration" \
  --output src/cache.py

Quick Start

git clone https://github.com/senssei/ollama-benchrig.git
cd ollama-benchrig

# Linux / WSL2
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Run diagnostic check
python3 benchmark.py --check

# Run cross-engine comparison
python3 benchmark.py --models ollama:phi3:mini,foundry:phi-4-mini --suite coding

(On macOS, run ./setup_mac.sh for automated Apple Silicon Metal verification).

What’s next

1. Apple Silicon M-Series Shootout: Running the exact same suite on Unified Memory to compare Metal kernel throughput against desktop CUDA.
2. Context Scaling Curves: Pushing context from 512 up to 8,192 tokens to map the exact threshold where KV cache consumption forces layer offloading.
3. Direct TensorRT (libonnxruntime_providers_tensorrt.so): Compiling optimized .engine caches to see if graph layer fusion can beat raw CUDA kernels.

MIT licensed, not affiliated with Microsoft or Ollama.