Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU
Quickstart · Quantization & Memory Specs · Supported Models · Supported Hardware · Leaderboard · Technical Paper (paper/) · Architecture · FAQ
llama-modchip is a set of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools designed to run sparse Mixture-of-Experts (MoE) checkpoints entirely in system RAM on x86_64 (AVX-512 / AVX2) and ARM64 (NEON / SVE / Apple Metal) processors:
To answer the most common technical question upfront—exact bit-widths (bpw), base checkpoints, and what is compressed in each GGUF:
Note on MoE Expert Routing (Top-K): By default (v2.4.0 Lossless), DeepSeek-V4-Flash evaluates all 6 routed experts + 1 shared expert (Top-6) per layer with zero expert dropping (1.00000 logit cosine similarity vs the unpatched runtime on the same checkpoint). Setting LLAMA_MOE_TOP_K=5 or 4 is an optional runtime flag for users who want to trade a small perplexity delta for higher decode throughput.
Compatibility Note (GGML_TYPE_Q2_0 = 40): Fractal-64 models use custom block-64 BF16-scale tensors (GGML_TYPE_Q2_0, type ID 40) and deepseek4 / Stockfish-PVS kernels. Unpatched llama.cpp, Ollama, or LM Studio binaries will reject them with unknown tensor type 40. Use ./modchip above or the standalone pre-compiled binary (llama-server-modchip-linux-x86_64, zero external .so dependencies) included in the Hugging Face repositories and GitHub Release v2.4.0:
Open windows/ and run windows/AVVIA_MODCHIP.bat (or ./modchip start in Git Bash). It detects physical CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads llama.cpp CUDA 12.4 / Vulkan binaries, computes the —n-cpu-moe split, and launches the Web Chat UI (http://127.0.0.1:8080).
Submit your terminal scorecard via a GitHub Benchmark Issue or GitHub Discussions to add your CPU to the leaderboard below.
Each model family has a dedicated technical guide with mathematical derivations, operator-level latency profiles, and quality gate benchmarks:
Microarchitectural details and tuning flags for each processor and GPU hybrid are documented in docs/HARDWARE_OPTIMIZATIONS.md:
Operator-level profiling and mathematical derivations in docs/DEEPSEEK_V4_OPTIMIZATIONS.md (probabilities and KL-divergence measured relative to the unpatched Top-6 execution on the 56.56 GiB Fractal checkpoint):
Full ablation tables and 100/100 Quality Gate report in docs/NEMOTRON_OPTIMIZATIONS.md:
Run ./modchip bench (or windows\BENCHMARK_PC.bat on Windows) and Submit Your Scorecard (GitHub Issue).
Nemotron Deep-Dive (docs/NEMOTRON_OPTIMIZATIONS.md) · DeepSeek-V4 284B Deep-Dive (docs/DEEPSEEK_V4_OPTIMIZATIONS.md) · Hardware & Chip Matrix (docs/HARDWARE_OPTIMIZATIONS.md) · Full Paper (ARCHITECTURE.md) · FAQ (FAQ.md)
🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.