Question 1

Can a NVIDIA RTX 6000 Ada (48GB) run Llama 4 Scout (MoE)?

Accepted Answer

Just about — Llama 4 Scout (MoE) runs at Q2_K (39.6 GB) at about 79–118 tok/s on a NVIDIA RTX 6000 Ada (48GB).

Question 2

How much memory does Llama 4 Scout (MoE) need?

Accepted Answer

Llama 4 Scout (MoE) needs about 41.2 GB at 4K context at its smallest quantization. A NVIDIA RTX 6000 Ada (48GB) has 48.0 GB.

Quant	Size	Speed	Fit
Q2_K	39.6 GB	79–118 tok/s	Tight / slow
Q3_K_M	51.8 GB	65–98 tok/s	Won't fit
Q4_K_M	65.4 GB	54–82 tok/s	Won't fit
Q5_K_M	76.6 GB	48–72 tok/s	Won't fit
Q6_K	88.4 GB	42–64 tok/s	Won't fit
Q8_0	114.5 GB	34–51 tok/s	Won't fit
FP16	215.6 GB	19–29 tok/s	Won't fit