Innodata  ·  2026

Inference Engineering

• LLM inference
• vLLM
• quantization
• FP8 / NVFP4
• speculative decoding
• LLM-as-a-judge
• GPU optimization
• model distillation

Exploring inference throughput improvements for an existing fine-tuned model: scheduler tuning, speculative decoding and quantization.

1. Background

1.1 Task

• The client has billions of product advertisements, each with a merchant supplied title and description.
• A/B tests showed a statistically significant CTR lift when the title and description were rewritten using generative AI.
• Billions of products need enhancement, with hundreds of millions arriving daily.
• The current pipeline uses the OpenAI Batch API and is becoming prohibitively expensive.

1.2 Goal

• Migrate the workflow to a small, cost effective open-source LLM with no regressions in generation quality.
• Article objective: maximize offline batch inference throughput with no measurable quality drift.

1.3 Problem overview

• Generated titles average 7 words.
• Generated descriptions average 4 sentences.

1.3.1 Models

A separate model was created for title and description enhancement. Both are Qwen3-4B, distilled via SFT from the production model that saw the CTR lift. Qwen3-4B is:

• small enough for fast inference
• large enough to avoid severe performance regressions.

1.3.2 Evaluation

Generated titles and descriptions are evaluated using a combination of a hand tuned LLM-as-a-judge and deterministic functions for verifiable qualities

• LLM-as-a-judge:
  • Assesses the factual adherence of the generated text to the original.
  • Tuned on human labelled data. The labelled dataset was mined for edge cases and made representative of the general production distribution.
  • Checks for unsupported claims, contradictory claims and omitted key facts.
  • Fails any generation that is not factually correct.
• Deterministic functions:
  • Check for undesirable generation artifacts, e.g. markdown bold characters, length violations, emojis, and n-gram checks for repetitive phrasing.

Each fine tuned model was distilled to match the production model's quality, with no regressions on auxiliary criteria like length and repetitive phrasing. The distilled BF16 baseline passes the judge 84.1% of the time on titles and 80.9% on descriptions (section 2), and every optimization below is measured against those numbers.

1.4 Specs

Benchmarks were run on a single RTX PRO 6000. Since deployment will eventually use data parallel inference, only single engine vLLM parameters need to be optimized.

1.5 Roofline

• GPU operations can be measured by their arithmetic intensity (the ratio of operations to bytes moved).
• Tasks with low arithmetic intensity are usually memory-bound; tasks with high arithmetic intensity are compute-bound.

• The sloped region has low arithmetic intensity and is memory-bound. The GPU cannot reach peak compute because bandwidth can't feed the tensor cores fast enough.
• At the ridge point, bandwidth meets compute demand. To the right, performance plateaus at peak compute

1.6 vLLM

1.6.1 Prefill and decode

During inference, the vLLM scheduler has 2 states:

Prefill

• When a prompt arrives, its KV cache must be computed (all prompt tokens in one forward pass) before the first output token.
• The title task is dominated by prefill: ~94% of tokens are prefill, with only ~16 decode steps.

Decode

• The scheduler generates one token per running sequence per step.
• The description task is long and dominated by decode.

1.6.2 Scheduler hyperparameters

The following hyperparameters are the most critical when tuning the vLLM scheduler:

• max_num_seqs
  • vLLM uses continuous batching. Finished sequences leave and queued ones join at every step. Tokens from all running sequences are flattened into one long batch with no padding, and masked attention ensures each token attends only to its own sequence.
  • max_num_seqs caps the number of concurrently running sequences per step (the decode batch size).
  • A higher value increases inter-token latency (ITL) per request, but increases aggregate throughput (req/s), which is our objective.
• batch_size
  • The number of prompts submitted to the engine per call.
  • Must be ≥ max_num_seqs, otherwise the scheduler drains at the tail of each batch and the GPU sits partly idle.
• max_num_batched_tokens
  • The per step token budget for chunked prefill, shared between decode tokens (one per running sequence) and prefill chunks.
  • A larger budget means more prefill per step and better utilization of the compute-bound phase, at the cost of higher ITL, but higher throughput

2. Baseline benchmarks

A baseline is established with the distilled Qwen3-4B in BF16 with vLLM defaults.

Default parameters

Baseline throughput and accuracy

Description throughput is lower than title, as expected (~8× more sequential decode steps per request).

2.1 Speculative decoding

• Generation is autoregressive and proceeds one token per sequence per decode step.
• Speculative decoding proposes k draft tokens cheaply, then the base model verifies all k in a single forward pass (one weight read instead of k). Tokens are accepted up to the first rejection, which is resampled.
• Generating multiple tokens per pass could improve throughput.

The following methods are candidates for proposing the additional tokens:

• Smaller draft model
  • Qwen3-4B is too small to benefit from a draft model.
  • Running inference on a smaller model like Qwen3-0.6B would incur more overhead than benefit. This strategy is only feasible for larger models.
• N-grams
  • Proposes n-grams from the existing prompt as subsequent tokens.
  • Only 1.13 tokens were accepted per step.
  • Most draft tokens were rejected, which resulted in slower inference.
• Multi-token prediction (MTP)
  • Newer models have multi-token prediction heads; Qwen3-4B does not.
  • This could be an area of improvement in future iterations, when the model inevitably needs retraining due to data drift.

No approach was feasible, so speculative decoding yielded no performance increase in this scenario.

2.2 Scheduler hyperparameter tuning

• A grid search over max_num_seqs, batch_size and max_num_batched_tokens determined the optimal scheduler parameters.
• A more granular search was repeated once it converged on the high-level optimal parameters.

• Title (prefill-bound):
  • Already near the compute roof during prefill, so there's little for the scheduler to recover: 1.07×.
  • The largest lever is max_num_batched_tokens, which vLLM autotunes per GPU by default, hence the minimal throughput gain.
• Description (decode-bound):
  • Throughput scales with concurrent decode: 1.51× (high max_num_seqs, maximum KV memory).

2.3 Quantization

Quantization approaches fall into two non mutually exclusive categories:

1. Weights
   • Quantize the model weights only.
   • Fewer bytes per weight read, which helps memory-bound decode.
2. Activations
   • Quantize the activations coming out of each layer so matmuls can run on low-precision tensor cores.
   • Speeds up both compute-bound prefill and weight reads, so the throughput gains are larger.
   • More likely to degrade quality.

The following approaches were considered:

FP8

• Matches judge accuracy, with no deterministic regressions.
• Titles gain more than descriptions relative to tuned:
  • Titles are prefill(compute)-bound, so 2× tensor core FLOPs helps directly.
  • Descriptions are partly KV-read-bound, which FP8 weights and activations do not shrink.

NVFP4

• Note: lm_head was ommited from quantization as this is known to cause severe performance regressions
• Both tasks see small judge regressions, and unacceptable switches into languages from pretraining that were not present in the input prompt.

3. Summary

Scheduler tuning plus FP8 quantization achieves the maximum throughput without regressing generation quality, according to the evaluation suite:

• Description: 2.14× throughput over baseline (41.7 to 89.2 req/s), with the judge pass rate unchanged (80.9% to 80.8%).
• Title: 1.67× throughput over baseline (115.2 to 192.1 req/s), with the judge pass rate unchanged (84.1% to 85.8%).