John Leimgruber III PRO

Hoped I can make it work on my CPU... :P

If you are wanting to do pure CPU or hybrid CPU+GPU inferencing on a big AMD EPYC rig I generally advise going with https://github.com/ikawrakow/ik_llama.cpp/ which can run your standard Q8_0 llama.cpp quants as well as newer SOTA quants from ik which he's optimized for both AVX2 and the newer "real 512 bit" avx512_vnni instructions which make a big difference for prompt processing and work on both intel and amd CPUs.

But of course I'm biased! ;p

Oh interesting you're playing more with SGLang specific quantizations of the big ones!

No, I haven't had access to the big dual Intel Xeon 6980P rig in a while and do most everything on a big AMD Epyc now.

Unfortunately, AMX extensions are Intel Xeon only thing and the Intel pytorch team worked with SGLang to add support for the instructions on that project.

This is the intel pytorch guy who added some subset of AMX into llama.cpp: https://github.com/mingfeima who might have access to a big Intel rig?