Apple Silicon for local LLMs

Apple Silicon Macs share one pool of memory between CPU and GPU. A 64 GB or 128 GB Mac can hold models that no single consumer GPU can, and memory bandwidth is high: about 100 GB/s on base chips, 200 to 400 GB/s on Pro and Max, and up to about 800 GB/s on Ultra. That combination (huge capacity, good bandwidth, low power) is why Macs punch above their weight for local inference.

#What this buys you

• Run 30B to 70B+ quantised models on a laptop or Mac Studio, and very large mixture-of-experts models on the highest-memory configurations.
• Excellent decode speed per watt, and near-silent operation.
• Simple setups with Ollama, LM Studio, llama.cpp (Metal) or MLX.
• Run models side by side with your normal apps, since there is no separate "GPU memory" to juggle.

#Bandwidth by chip family

Decode speed follows bandwidth (memory bandwidth). Approximate figures (check Apple's specifications for your exact configuration, since variants within a family differ):

Real speeds are about 60 to 80 percent of the ceilings. Two lessons: a Max chip is roughly twice as fast as a Pro at decode regardless of RAM, and RAM size decides which models fit, not how fast they run.

#How much RAM do you need?

Usable GPU memory is about two thirds to three quarters of total RAM by default. Plan for the model, its KV cache and about 4 to 6 GB left for macOS and your apps.

RAM cannot be upgraded after purchase on these machines, so buy for the models you will want in two years, not today (buying guide).

#What to watch for

• Prefill is slower than on high-end NVIDIA GPUs. Prompt processing is compute-bound, and a Mac's GPU has a fraction of a big discrete card's arithmetic throughput. Very long prompts show noticeable time to first token. Prompt caching helps a lot (prompt caching).
• Not all memory is usable by the GPU. macOS limits GPU-wired memory to roughly two thirds to three quarters of RAM by default (less on small-memory machines). Plan with the usable fraction in the hardware checker.
• Memory pressure. If the model plus your apps exceed RAM, macOS compresses and swaps, and speed collapses. Close browsers and heavy apps before loading big models, and watch the memory-pressure graph in Activity Monitor (green is fine, yellow is borderline, red is bad).
• Thermals. MacBooks throttle under sustained load, especially thin models; a Mac Studio or Mac mini with active cooling sustains speed much better. Plug in for serious work, since battery mode lowers performance.
• Battery. Long generations drain a laptop battery quickly.
• Concurrency. Runtimes designed for personal use do not batch many users the way vLLM does on a GPU (concurrency).

#Raising the limit (advanced)

Recent macOS versions let you raise the GPU wired limit until reboot:

# Allow ~56 GB for the GPU on a 64 GB Mac (value in MB). Resets on reboot.
sudo sysctl iogpu.wired_limit_mb=57344

Leave several GB for macOS: setting it too high can freeze the system. The setting does not persist across reboots, the exact key name has changed across macOS versions, and it requires administrator rights, so treat it as a power-user tweak and read Apple's and your runtime's current documentation first.

#Metal versus MLX

• llama.cpp and Ollama (Metal backend): GGUF models, the widest model compatibility, the largest ecosystem of quants, grammar and JSON-schema support.
• MLX: Apple's array framework; often faster on newer chips for small and medium models and has first-class quantization and fine-tuning (mlx-lm). See MLX on Apple Silicon for commands and a full comparison.
• LM Studio and similar apps wrap these runtimes with friendly interfaces.

If you are unsure, start with Ollama, then benchmark MLX for the models you use most.

#Practical tips

1. Check memory pressure before and during a run.
2. Choose the smallest quant that gives good quality to maximise speed and context (measuring quality).
3. Keep context modest. Long contexts cost memory and slow decoding.
4. Use a smaller "draft" model with speculative decoding where your runtime supports it (speculative decoding).
5. Prefer MoE models on high-memory Macs: they decode much faster than dense models of the same size.
6. Keep macOS and your runtime updated; Metal performance improves between releases.
7. Benchmark yourself with llama-bench or mlx_lm.generate and write the numbers down.

#When a Mac is not the best choice

• You need fast prompt processing over long documents all day: a discrete GPU is faster.
• You need to serve many concurrent users: use GPUs with vLLM.
• You need CUDA-only software.
• You want to upgrade piecemeal later: Macs cannot.

#Common mistakes

1. Buying by chip name only. Pro versus Max versus Ultra bandwidth differs by factors of two to four.
2. Buying too little RAM, the one thing you cannot add later.
3. Assuming all RAM is available to the GPU.
4. Judging speed from a short prompt, then being surprised by long-document latency.
5. Running a model that just fits and leaving no room for the OS.

#Exercise

You are choosing between a 36 GB Pro-class Mac (about 273 GB/s) and a 64 GB Max-class Mac (about 400 GB/s) to run a 32B model at Q4 (about 20 GB) with 16K context (about 4 GiB of KV).

1. Does the model fit on each, assuming 70 percent of RAM is usable by the GPU?
2. Estimate decode speed on each.
3. What does the 64 GB Mac allow that the 36 GB one does not?

Answers:

1. 36 GB × 0.7 ≈ 25 GB usable: weights 20 GB plus KV 4 GiB (about 4.3 GB) is about 24.3 GB: it barely fits, with no headroom. 64 GB × 0.7 ≈ 45 GB: fits easily.
2. Ceilings: 273 / 20 ≈ 14 tok/s (realistic 9 to 11) and 400 / 20 = 20 tok/s (realistic 13 to 16).
3. Headroom for longer context and concurrent apps, and the option of a 70B at Q4 (about 43 GB, which fits in the 45 GB usable), at roughly 6 to 8 tok/s. The 36 GB machine cannot run it at all.