Oh interesting you're playing more with SGLang specific quantizations of the big ones!

No, I haven't had access to the big dual Intel Xeon 6980P rig in a while and do most everything on a big AMD Epyc now.

Unfortunately, AMX extensions are Intel Xeon only thing and the Intel pytorch team worked with SGLang to add support for the instructions on that project.

This is the intel pytorch guy who added some subset of AMX into llama.cpp: https://github.com/mingfeima who might have access to a big Intel rig?

I've used this:
https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8/tree/main/inference

Hoped I can make it work on my CPU... :P

I've been using SGLang+KTransformers for a while. I happen to have access to a server with an EPYC 9004 CPU. Here are my takeaways:

The hardware setup is a single-socket 9V74 with 288GB RAM and a single RTX 3090. I usually run glm4.5air on it (GPU weights in bf16, CPU weights in int8), getting ~390 tokens/s prefill and ~34 tokens/s decode. For the AMD CPU to reach these speeds, you need to specifically install the BLIS library. The documentation for both AMD and ktransformers was seriously lacking, and the CPU weights also require a specific quantization method.

The above speeds were achieved by running with the --kt-max-deferred-experts-per-token 7 flag. Without it, the decode speed drops by about 1.4x, though the prefill slowdown is less dramatic.