Question 1

Can a NVIDIA RTX 4060 Ti (16GB) run Laguna XS 2.1 (MoE)?

Accepted Answer

Just about — Laguna XS 2.1 (MoE) runs at Q2_K (12.2 GB) at about 91–136 tok/s on a NVIDIA RTX 4060 Ti (16GB).

Question 2

How much memory does Laguna XS 2.1 (MoE) need?

Accepted Answer

Laguna XS 2.1 (MoE) needs about 13.6 GB at 4K context at its smallest quantization. A NVIDIA RTX 4060 Ti (16GB) has 16.0 GB.

Quant	Size	Speed	Fit
Q2_K	12.2 GB	91–136 tok/s	Tight / slow
Q3_K_M	15.6 GB	80–121 tok/s	Won't fit
Q4_K_M	20.6 GB	69–103 tok/s	Won't fit
Q5_K_M	24.0 GB	62–94 tok/s	Won't fit
Q6_K	29.0 GB	55–83 tok/s	Won't fit
Q8_0	35.6 GB	48–72 tok/s	Won't fit
FP16	66.9 GB	29–44 tok/s	Won't fit