Question 1

Can a MacBook Pro M2 Max (64GB) run Mistral Small 4 119B (MoE)?

Accepted Answer

Just about — Mistral Small 4 119B (MoE) runs at Q2_K (40.2 GB) at about 70–105 tok/s on a MacBook Pro M2 Max (64GB).

Question 2

How much memory does Mistral Small 4 119B (MoE) need?

Accepted Answer

Mistral Small 4 119B (MoE) needs about 41.1 GB at 4K context at its smallest quantization. A MacBook Pro M2 Max (64GB) has 64.0 GB.

Quant	Size	Speed	Fit
Q2_K	40.2 GB	70–105 tok/s	Tight / slow
Q3_K_M	54.4 GB	58–87 tok/s	Won't fit
Q4_K_M	73.8 GB	47–71 tok/s	Won't fit
Q5_K_M	89.2 GB	41–62 tok/s	Won't fit
Q6_K	99.4 GB	38–57 tok/s	Won't fit
Q8_0	126.5 GB	31–47 tok/s	Won't fit
FP16	238.0 GB	18–28 tok/s	Won't fit