Hello All,

First of all great thanks to the creator of this quant - it makes it possible to run this very capable model on a server-grade hardware (rather than farm-grade).

We are running this model, and wondering if we did the proper configuration or are we missing something. Our goal is to connect it to claude code and this actually works pretty decent, having several people testing it.

What we do not understand is the performance metrics cited, and especially how you benchmarked this 100 t/s for 2 H200 cards. In our evaluations we see following:

The above is with 36k prompt and 200 decode which is obviously big prompt, yet a typical one for claude code (or other cli tool) usage. Not sure how this affects the process, but lowering the prompt have minimal benefits, while lowering the expected output tokens is even worsening the result.

This is the command we use to start it:

vllm serve bullpoint/GLM-4.6-AWQ \
  --tensor-parallel-size 4 \ # or 2 for h200 setup
  --tool-call-parser glm45 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name glm-4.6-awq \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code \
  --port 8000 \
>> vllm-logs/vllm-stdout.log 2>> vllm-logs/vllm-stderr.log &

Our first goal is to understand if we are using the model properly within it's limits - therefore to adjust our testing strategy and understand how you achieve 50-60 or 100 t/s - so we can better think of hardware.