Home/Blogs/Local LLM Benchmark: DeepSeek R1 vs Llama 3.3 vs Qwen 2.5 on RTX 4090
Back to Blogs
AI & Agents
9 min readFebruary 24, 2026

Local LLM Benchmark: DeepSeek R1 vs Llama 3.3 vs Qwen 2.5 on RTX 4090

A deep empirical benchmark comparing throughput, memory footprints, reasoning fidelity, and tool-calling accuracy across open-weights LLMs.

A
Avernus Engineering Team
AI Systems Architecture
Key Takeaways
  • Qwen 2.5 14B achieves the highest balance of tool calling precision and tokens/sec on 24GB VRAM.
  • DeepSeek-R1-Distill-Qwen-14B excels in multi-step mathematical and code reasoning, but exhibits 35% higher latency due to verbose chain-of-thought tokens.
  • vLLM PagedAttention delivers up to 3.8x higher concurrency throughput compared to default Ollama runtime on multi-client workloads.

Benchmark Methodology & Test Environment

We evaluated the top open-weights models on a dedicated workstation equipped with an AMD Ryzen 9 7950X, 64GB DDR5 RAM, and a single NVIDIA GeForce RTX 4090 (24GB VRAM).

Each model was evaluated across three core criteria: raw generation throughput (tokens per second), JSON schema adherence accuracy, and complex multi-step reasoning.

ModelQuantizationVRAM UsageAvg Speed (t/s)Schema Accuracy
Llama 3.3 70BQ4_K_M (Offload)23.4 GB14.2 t/s96.4%
Qwen 2.5 14BFP16 / Q816.8 GB48.6 t/s98.1%
DeepSeek R1 Distill 14BQ8_017.2 GB39.4 t/s94.8%
Mistral NeMo 12BFP1613.5 GB54.1 t/s92.3%

Reasoning Tradeoffs: Chain-of-Thought Overhead

DeepSeek R1 distilled models produce superior logical deductions for code analysis, but their internal thinking block generates 300 to 800 additional tokens prior to delivering the final answer. In latency-sensitive production microservices, this translates to a 2 to 4 second time-to-first-meaningful-token delay.

For interactive customer-facing agents, Qwen 2.5 14B delivered near-instantaneous streaming with zero chain-of-thought latency penalty.

Concurrency: Ollama vs vLLM Serving

While Ollama is unparalleled for local desktop experimentation, production deployments requiring concurrent API requests must use vLLM. vLLM's PagedAttention dynamically allocates KV-cache memory, enabling 8 concurrent streams on an RTX 4090 without out-of-memory errors.

Topics Covered:
#Benchmarks#DeepSeek#Llama 3.3#vLLM#GPUs

More Engineering Guides

All Articles