Model deployment
This article refers to SDK version v0.5.0. The current SDK version is N/A.
Overview
The model catalog works the same way as the application catalog described in Deploying your own Docker apps: register a model, upload a version, deploy it. This mirrors the Panel flow in Deploy a TensorFlow model, just for any model format, not only TensorFlow SavedModel.
Model. A machine learning artifact an operator registers and deploys from Barbara Panel the same way as an application, so it runs inference at the edge instead of in a central cloud.
Learn moreA model workload has one extra constraint on top of a regular application workload: its compose_config must exactly match the service template declared by the model application version you're deploying.
Register the model
client.models.create(
"anomaly-detector", "Long description", "Barbara", model_type=0, engine=0
)
create(name, long_description, developer, *, model_type, engine, short_description=None, icon_path=None) on client.models mirrors the New Model form Panel opens after picking Model in the Select Library Item popup:
| Parameter | Type | What it is |
|---|---|---|
name | str | The model's name, for example half_plus_two in the Panel walkthrough |
long_description | str | The model's description |
developer | str | The publisher/author recorded against the model |
model_type | int | A numeric code for the kind of model (classification, detection, ...); the mapping isn't published in the SDK docstrings, so confirm the value you need against an existing model's raw payload or ask Barbara support |
engine | int | A numeric code for the serving engine (for example TensorFlow Serving); same caveat as model_type, cross-check against get(model_id).raw on a model you already have working |
short_description | Optional[str] | A shorter summary |
icon_path | Optional[str] | A local file path to a PNG, uploaded the same way as client.applications.create's icon_path |
The method returns None; use client.models.list() afterwards to find the new model's id, the same way client.applications.create doesn't hand back an id either.
Upload a version
client.models.create_version("<model-id>", "./model.onnx", "1.0.0", ["Initial release"])
create_version(model_id, artifact_path, name, release_notes) is positional, like client.applications.create_version, but with one fewer parameter: models don't take an architectures list, since a model artifact (an .onnx, .pb, or SavedModel zip) isn't compiled per CPU architecture the way a Docker image is.
| Parameter | Type | What to pass |
|---|---|---|
model_id | str | The model's id from client.models.list() |
artifact_path | str | A local file path to the model artifact: for a TensorFlow model, the zipped SavedModel, matching what you'd upload via the New Version dialog |
name | str | The version string, following the same x.y.z convention Panel suggests |
release_notes | List[str] | One string per release-note bullet, or [] |
sha256 and size for a model version are computed automatically from the artifact, you don't need to hash or measure the file yourself. This mirrors what happens behind the scenes when you upload through the New Version dialog in Panel: the checksum is derived from the file, never something you supply.
Deploy it
client.nodes.workloads.create_model_workload(
node.id,
app_version_id="<model-app-version-id>",
application_id="<model-application-id>",
name="my-model-workload",
compose_config=[{"name": "modelservice", "ports": {"PORT_NUMBER": "8501"}}],
)
create_model_workload on client.nodes.workloads takes the same core parameter shape as create_marketplace_workload covered in the Marketplace guide, minus App Config and App Secrets: node_id, app_version_id, application_id, name, run_docker, force_pull, enable_logs, gpu, compose_config. A model workload has Compose Config, but neither config/config_id (App Config) nor app_secrets, unlike Marketplace/Docker workloads. The parameter names in the SDK still read app_version_id/application_id even though they're pointing at a model rather than an application: pass the model's version id and model id in those same two slots.
The Panel walkthrough in Deploy a TensorFlow model sets a gRPC Port Number (default 9083) and a REST API Port Number (default 9084) as part of the model wizard's variables step; those two ports are exactly what belongs inside a compose_config entry's ports dict, keyed by whatever port-variable names the specific model application declares (check its README or Technical Notes for the exact keys, since they vary by serving engine).
compose_config must match the model application version's own service template exactly, every service name and each port/volume/env entry has to exist there, or the request is rejected. This is stricter than a regular Marketplace workload, where compose_config only needs to describe what you actually want configured. For a model, it has to describe everything the template declares, nothing more and nothing less.
If the node will serve more than one model, remember Panel's own warning from the model wizard: the gRPC and REST port pairs must be different for each model on the same node, or the second deployment's containers fail to bind their ports.
Manage versions
versions = client.models.list_versions("<model-id>")
client.models.rename_version("<model-id>", "<model-version-id>", "1.0.1")
client.models.delete_version("<model-id>", "<model-version-id>")
list_versions(model_id) returns a List[ModelVersion], each with an id and a model version name you can match against what you uploaded. rename_version(model_id, model_version_id, name) only changes the version label, not the underlying artifact. There's no "replace the file of an existing version" call, upload a new version instead. delete_version(model_id, model_version_id) removes a version; like client.applications.delete_version, it fails if that version is currently deployed on any node.
Deploy models at cluster scope
client.clusters.workloads.create_model_workload(
"<cluster-id>",
app_version_id="<model-app-version-id>",
application_id="<model-application-id>",
name="my-model-stack",
compose_config=[{"name": "modelservice", "ports": {"PORT_NUMBER": "8501"}}],
)
client.clusters.workloads.create_model_workload mirrors the node-level call, with the same compose_config-must-match-the-template constraint as above, just applied across every eligible node in the cluster in one call. At cluster scope, it also accepts deployment and placement_constraints, cluster-only parameters with no node-level equivalent; see Clusters & cluster workloads.
Run predictions
Once a model workload with a PORT_NUMBER-style REST port is running, query it the same way the Panel walkthrough does, over plain HTTP, unrelated to the SDK:
curl -d '{"instances": [1.0, 2.0, 5.0]}' \
-X POST http://<NODE_IP>:<REST_PORT>/v1/models/<MODEL_NAME>:predict
Substitute <NODE_IP> (from client.nodes.get_last_telemetry or the Network card), <REST_PORT> (the host port you set in compose_config), and <MODEL_NAME> (the name the serving engine was configured with). See Deploy a TensorFlow model for the full request/response walkthrough. The SDK's role stops at getting the model deployed and reachable.
Summary
You registered a model, published a version, and deployed it to run inference at the edge, respecting the service template the model itself declares.
The pattern mirrors Deploying your own Docker apps closely, models are just a separate catalog with one extra constraint on compose_config.
Continue to Workload lifecycle to manage a deployed workload day to day, or see client.models and client.nodes.workloads in the Reference for the full method list.