← All posts

Modelplane v0.6: making the most of your hardware

Modelplane v0.6 packs small models onto shared GPU nodes and serves on AMD GPUs and CPUs as well as NVIDIA. SGLang prefill/decode works too, behind a router you write.

Modelplane v0.6 is out, and most of it is about getting more from the hardware you have. Small models can now share a GPU node rather than each taking a whole one. A fleet can serve on AMD Instinct GPUs and on CPUs alongside NVIDIA GPUs. And a node pool can use GPU capacity you've reserved on GKE or EKS.

Disaggregated prefill/decode serving also works with SGLang now, because you write the router the same way you write your engines. Here's what's new.

Small models share a node

Modelplane's fleet scheduler picks a cluster and a node pool for each model replica. It used to measure a pool's room in nodes, so every pod that claimed a GPU took a whole node, however many of the node's GPUs it used. Qwen3-8B fits on one NVIDIA L4. A pool of two nodes with four L4s each has room for eight replicas of it, but the scheduler placed two.

The scheduler now counts free GPUs. Modelplane installs an agent on every cluster that reports how many of each pool's nodes have each number of GPUs free, and the InferenceCluster publishes those counts in its status. This pool has one node with one of its four L4s free, and can grow by two more nodes:

status:
  schedulablePools:
  - name: gpu-l4x4-a
    nodes: 3          # The most nodes the pool can have.
    observedNodes: 1  # The nodes the pool has now, ready or not.
    nodesByFreeDevices:
    - freeDevices:
        gpu.nvidia.com: 1
      nodes: 1
    devices:
    - name: gpu
      claim: DRA
      driver: gpu.nvidia.com
      deviceClassName: gpu.nvidia.com
      count: 4

The scheduler fits a replica's pods against those counts. Four one-GPU replicas fit one four-GPU node, and a replica with two four-GPU engines fits one eight-GPU node. The fleet scheduler still picks only the cluster and the pool, and each cluster's own scheduler picks the node.

There's nothing to configure. The agent reports on NVIDIA GPU pools today, so AMD and CPU pools still count in whole nodes. Modelplane doesn't request CPU or memory for an engine, so set requests on engines you expect to share a node.

AMD GPUs and CPUs

Each cluster now installs the device software its pools need, based on the DRA drivers its InferenceClasses use. A pool of AMD GPUs gets AMD's GPU operator, and a CPU pool gets dra-driver-cpu. A CPU-only cluster doesn't get NVIDIA's GPU operator or DRA driver, which every cluster used to.

Here's the class from the AMD recipe. It describes a pool of AMD Instinct MI325X nodes:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: amd-mi325x-8x
spec:
  description: "BYO 8x AMD Instinct MI325X 256GB"
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.amd.com
    deviceClassName: gpu.amd.com
    count: 8
    attributes:
      productName: { string: AMD_Instinct_MI325_OAM }
    capacity:
      memory: { value: "262128Mi" }

A ModelDeployment selects AMD GPUs the way it selects NVIDIA ones, with a CEL expression over the devices a class publishes, so one fleet can serve from both. Modelplane's telemetry collector reads AMD's metrics exporter too, and renames its series to the same modelplane_* names it uses for NVIDIA's. Modelplane installs AMD's software only on clusters you bring yourself, and doesn't provision AMD GPUs on any cloud yet.

A CPU pool lets you try Modelplane end to end without finding a GPU. An InferenceClass whose provisioning block omits the accelerator describes one, and the CPU recipe serves Qwen2.5-0.5B from it on Civo with vLLM's CPU image. A CPU pool needs Kubernetes 1.36, because dra-driver-cpu relies on a DRA feature that's on by default only from that release.

Prefill/decode with SGLang

Disaggregated serving splits a model across two engines. Prefill reads the prompt and builds the KV cache, and decode generates tokens from that cache. A router sends each request to both. On Modelplane that's llm-d's endpoint picker, which picks the pods, and a sidecar in each decode pod that hands the prompt to the prefill pod the picker chose.

The router has to match the engines. SGLang needs a different connector on the sidecar from vLLM. A router tuned for vLLM also lets short or cached prompts skip prefill, and SGLang's decode engine rejects those requests. Modelplane used to configure one router, for vLLM with the NIXL connector.

Now you write a PrefillDecode deployment's router the same way you write its engines. You pick the endpoint picker's image, flags, and configuration, and put the sidecar in the decode pod beside the engine. Modelplane runs the picker and wires it into the cluster's gateway, and passes your configuration through without interpreting it. Here's an SGLang deployment, trimmed from the routing docs:

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: laguna-pd
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: laguna
      serving:
        mode: PrefillDecode
        router: EndpointPicker
        endpointPicker:
          template:
            spec:
              containers:
              - name: epp
                image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0
                # Omitted for brevity.
          config:
            apiVersion: llm-d.ai/v1alpha1
            kind: EndpointPickerConfig
            plugins:
            # SGLang's decode engine can't serve a request alone, so every
            # request goes through prefill.
            - type: always-disagg-pd-decider
            # Omitted for brevity.
      engines:
      - name: prefill
        phase: Prefill
        # Omitted for brevity.
      - name: decode
        phase: Decode
        members:
        - role: Standalone
          # nodeSelector omitted for brevity.
          template:
            metadata:
              labels:
                llm-d.ai/engine-type: sglang
            spec:
              containers:
              - name: sidecar
                image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0
                args:
                - --secure-proxy=false
                - --kv-connector=sglang
                - --vllm-port=8001  # The decode engine's port, whatever the engine.
                ports:
                - containerPort: 8000
                # readinessProbe omitted for brevity.
              - name: engine
                image: lmsysorg/sglang:v0.5.12.post1-cu129
                command: ["python3", "-m", "sglang.launch_server"]
                args:
                - --model-path=poolside/Laguna-S-2.1-FP8
                - --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)
                - --tp-size=8
                - --port=8001
                - --disaggregation-mode=decode
                - --disaggregation-transfer-backend=nixl
                # Omitted for brevity.
                ports:
                - name: http
                  containerPort: 8001

Serving a new engine or KV transfer method no longer waits on a Modelplane release, only on a router that supports it. A Unified deployment doesn't need any of this. It keeps Modelplane's default picker unless it sets a router of its own.

Reserved capacity

The largest GPU machines can be hard to get without a reservation. A node pool on GKE can now use a Compute Engine reservation you've already bought. A node pool on EKS can use an On-Demand Capacity Reservation, as well as a Capacity Block. Register the reservation as a CapacityReservation, and reference it from the pool:

apiVersion: modelplane.ai/v1alpha1
kind: CapacityReservation
metadata:
  name: l4-reservation
spec:
  provider: GKE
  gke:
    zone: us-central1-a
    name: l4-reservation
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-reserved
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
  - name: l4-pool
    className: gke-l4-1x-g2
    nodeCount: 2
    maxNodeCount: 2
    zones: [us-central1-a]
    reservationRef:
      name: l4-reservation

The pool gets nodes only from its reservation, and never falls back to on-demand capacity.

Degraded replicas keep serving

An engine can run several copies within a replica. A replica used to take traffic only while every copy of every engine was ready, so one crashed copy withdrew the whole replica. A deployment with one replica went unreachable until the crashed copy came back.

Now a replica serves while at least one copy of each engine is ready, and the endpoint picker sends requests only to the copies that are. Ready still means full capacity. A degraded deployment reads as serving but not ready, and kubectl get shows the difference in a new SERVING column.

GPU metrics on Nebius

Modelplane maps NVIDIA's DCGM metrics onto modelplane_gpu_* names, but nothing exported DCGM metrics on Nebius. Modelplane now installs NVIDIA's DCGM exporter on Nebius clusters, and their GPUs report those series too.

Try it

The full release notes are on GitHub. The getting-started guide covers standing up a fleet. If you're a platform engineer new to inference, Annie Talvasto's field guide to inference engineering covers the concepts, production operations, and tooling you'll meet. Modelplane is Apache 2.0 and moving fast at github.com/modelplaneai/modelplane, and questions are welcome in Slack.

Nic Cope

Nic CopeSr Principal Engineer, Upbound

Nic is a Senior Principal Engineer at Upbound and a core maintainer of Modelplane and Crossplane. He's spent the last 15+ years working on cloud and infrastructure, including designing and building Crossplane from v0.1 through v2.0, and is now bringing that work to AI inference with Modelplane.

Modelplane v0.5: an AI gateway and fleet telemetry

Modelplane v0.5 turns the fleet gateway into an AI gateway and collects every engine's metrics under one vocabulary. Civo joins the clouds it can provision on.

Modelplane v0.4: NVIDIA Dynamo and AI Cluster Runtime

Modelplane v0.4: NVIDIA Dynamo and AI Cluster Runtime

Modelplane v0.4 composes NVIDIA's inference stack across a fleet: a new Dynamo serving stack, and cluster software built from NVIDIA AI Cluster Runtime.

Modelplane v0.3: Vultr, the Anthropic Messages API, and testing without a GPU

Modelplane v0.3: Vultr, the Anthropic Messages API, and testing without a GPU

Modelplane v0.3 adds Vultr VKE as an inference cluster provider, serves the Anthropic Messages API end to end so tools like Claude Code run against your own GPUs, improves multi-node scheduling, and adds local end-to-end testing that needs no cloud and no GPU.