B2CQuantizer

Quantizing a 24-billion-parameter model on a 48 GB card is tight to impossible, and shell scripts that chain quantizations one after another are hard-coded per run, abort everything on one failure and show no progress. B2CQuantizer turns the job into a disposable, self-service GPU pod driven from a browser: one source model, many formats, failures isolated per format, and live progress.

What it produces

A single job can emit any selection of 30 formats: GGUF K-quants and I-quants, Q8_0, F16 and BF16 repacks, a multimodal vision projector for Mistral3 models, and safetensors builds in GPTQ, AWQ, NVFP4 and two FP8 variants. Each safetensors format gets its own Hugging Face repo, and all GGUF files share one. When it finishes, the app writes a Quantizations table into the source model's README, replacing it on later runs.

How it works

• Lane A (GPU) loads the source onto the GPU once and runs each llm-compressor recipe against the resident model, uploading and deleting each result before the next.
• Lane B (CPU) converts to a BF16 GGUF once, computes an importance matrix once if any I-quant is selected, then runs llama-quantize per format. For multimodal models it patches two upstream converter bugs and asserts that a required tensor made it into the output.
• One calibration source feeds both lanes, so both see the same distribution. Calibration text is rendered through the same tokenizer path the model is served with, and a sample that fails to render stops the job instead of falling back quietly.
• Both lanes stream logs to the browser over Server-Sent Events, behind a FastAPI, HTMX and Alpine.js front end.

The Hugging Face token lives only in process memory, is never written or logged, and is wiped when the job ends.