devlog #17 MI300X vLLM serving
The old GPU box got destroyed, so I spun up a new MI300X with the same SSH key. ROCm 6.2 and Torch 2.5.1 were already set up, so getting the machine ready was easy.
The hard part was vLLM. The pip version is CUDA only, and the ROCm Docker image only supported the multimodal version of Qwen3.5. Our model is text-only, so I had to patch the image.
I added Qwen3_5ForCausalLM, fixed the hybrid KV-cache setup, added the missing Mamba state methods, and mapped the weight names so the text-only safetensors loaded correctly. I also patched the merged config to remove the M-RoPE and dtype fields.
After that, the model served normally.
I ran a one-task test with fix-git. The pipeline worked, but the model got stuck repeating git show until it ran out of turns.
##Shipped
The merged model is now public on Hugging Face as mateo0093/le-gros-chaton under Apache-2.0.
The 5×5 Terminal-Bench pilot finished at 3/25 (25%). All three successes were on fix-git, which shows that the SFT data was too focused on git tasks.
I also tested RLVR with GRPO and a novelty bonus. The reward had almost no variance, so the training signal was basically useless. I saved the step-10 adapter but didn’t merge it.
The GPU used about 12 hours of the 50 hour budget. All the vLLM patches and harness changes are in the repo.
Main lesson: Qwen3.5’s hybrid model is a bit annoying to serve as text-only on ROCm. The model works but i think this was more of a learning opportunity, i know what i’ve done wrong and what i would have done differently, I will be working on a new model in a few weeks once I do a bit more research.