GLM-5.3 vLLM serving: 607.8 tok/s on two DGX Stations

Written by , Founder of Launch Llama · November 15, 2026

TL;DR
  • GLM-5.3 NVFP4 has 753B parameters (40B active per token) and needs about 464 GB, more than the 252 GB HBM3e on one GB300 GPU.
  • NVIDIA serves it across two GB300 DGX Stations over ConnectX-8 RDMA with vLLM and tensor parallelism (TP=2), behind one OpenAI-compatible endpoint.
  • On the SPEED-Bench Coding dataset, output peaked at 607.8 tok/s at concurrency 16 and fell to 385.0 tok/s at concurrency 32.

What happened

NVIDIA's technical blog published a tutorial on serving GLM-5.3 locally across two NVIDIA DGX Stations with vLLM. The model is NVIDIA's NVFP4 quantized version of Z.ai's mixture-of-experts GLM-5.3. It has 753 billion parameters with 40 billion activated per token. All weights must stay in memory because each token can route to different experts. The checkpoint is about 464 GB, so it does not fit in the 252 GB of HBM3e on a single GB300 GPU.

The setup splits the weights and computation across both GPUs with tensor parallelism (TP=2). Station A hosts the API and Station B runs only a worker. CPU weight offload is disabled. The vLLM config enables multi-token prediction (four drafted tokens per step), FP8 KV-cache storage and prefix caching. It sets a context ceiling of 98,304 tokens and gives vLLM 93% of GPU memory.

The post's summary says AIPerf benchmarks on the SPEED-Bench Coding dataset peaked at 607.8 output tokens per second at concurrency 16. Throughput then fell to 385.0 tokens per second at concurrency 32, with a mean request latency of 453.5 seconds. The page text I could read was cut off before the benchmark section, so I could not check those figures against the full results.

What this means for founders

  • This is a hardware-heavy path. The tutorial requires two GB300 DGX Stations, connected and validated, plus at least 464 GB of free disk on each. The source gives no hardware price, so I can't tell you the cost.
  • Concurrency matters more than the headline number. Throughput peaked at 16 concurrent requests and dropped at 32. If your product has bursty users, test your own load before promising anything.
  • Latency at high load looks bad. The summary lists a mean request latency of 453.5 seconds at concurrency 32. That is a benchmark on a coding dataset, not a product SLA, but it is a warning for interactive apps.
  • Licensing and access take extra steps. You must accept the NVIDIA Open Model Agreement and the original GLM-5.3 license terms. The benchmark step needs a Hugging Face token with access to the gated cais/hle dataset.
  • Security is on you. The launch script exposes an unauthenticated API on every Station A interface. NVIDIA says to firewall the port and add authentication and TLS before wider access.

How to try it today

  1. Connect and validate both Stations with NVIDIA's Connect Two DGX Stations playbook, including the GPU-memory RDMA test on both rails.
  2. Pick one rail (for example cx8r1), note its RDMA device and IPv4 address on each Station, and create ~/glm53.env on both with RANK=0 on Station A and RANK=1 on Station B.
  3. Pull the pinned vLLM image on both Stations (the full tag and digest are in the source's env file):
docker pull "$IMG"
  1. On Station A, download the pinned checkpoint with hf download (about 464 GB), then rsync it to the same path on Station B. Check that both copies match.
  2. Create ~/launch-glm53.sh on both Stations and run it. It starts one vLLM rank on each, and Station B runs headless.
  3. Send a request to the OpenAI-compatible endpoint on Station A, then rerun AIPerf with your own prompts to find the right concurrency for your workload.

For current recommended configs, NVIDIA points to its Local AI recipe repository.

Related tools on Launch Llama

Tools from the Launch Llama directory that fit with this news:

ToolWhat it doesWhy it matters here
LLM ReferenceFor engineers: LLM Reference compares 1,700+ models across 130+ providers so you pick the right LLM and lowest price in seconds.If you are weighing a self-hosted frontier model against hosted options, LLM Reference compares 1,700+ models across 130+ providers on price.
Router by RampFor engineers: Router by Ramp routes every LLM request to the lowest-cost model that meets your needs so you cut inference costs and simplify API management.Router by Ramp routes each LLM request to the lowest-cost model that meets your needs, which is an alternative to running your own hardware.
Hostnot GPUOn-demand GPU cloud for AI, inference and developmentHostnot GPU is an on-demand GPU cloud for inference, a possible route if you want GPUs without buying DGX Stations.

Browse all tools or list your own tool.

Llama's take

I would not read this as a recipe for most early-stage founders. It needs two GB300 DGX Stations and a lot of setup. If you already have that hardware and want a frontier-scale model on your own machines, this is a clear, pinned guide. Teams that care about keeping data local are the obvious audience.

If you don't own that hardware, wait. Nothing in the source says what the setup costs or how it compares with hosted APIs on price. I won't guess.

The one thing to watch is the drop from 607.8 to 385.0 tok/s when concurrency doubled to 32. Run your own benchmark before you size anything around the headline number.

FAQ

Why does GLM-5.3 need two DGX Stations?

The GLM-5.3 NVFP4 checkpoint is about 464 GB, and a single GB300 GPU has 252 GB of HBM3e. Two Stations split the weights and computation with tensor parallelism (TP=2), so each GPU holds roughly half.

How fast is GLM-5.3 on vLLM across two DGX Stations?

NVIDIA's summary reports overall output throughput peaking at 607.8 tokens per second at concurrency 16 on the SPEED-Bench Coding dataset. At concurrency 32 it fell to 385.0 tokens per second.

Does the GLM-5.3 vLLM setup expose an API?

Yes. Station A serves one OpenAI-compatible endpoint. The launch script leaves it unauthenticated on every Station A interface, so NVIDIA advises firewalling the port and adding authentication and TLS before wider access.

Sources

Related guides

Explore founder tools

Free calculators, plus newsletters and launch directories curated for founders.

Mini Apps →Newsletters →Launch Directories →

For bots & AI agents

Machine-readable versions of this article and Launch Llama. Fetch these instead of parsing this page.

This article (.md) llms.txt llms-full.txt ai.txt Full catalog (.md) Endpoint index API docs MCP server