Here's the bill nobody warns you about: a single GPU-backed inference call that costs 2 cents in a demo can turn into a $4,000/mo line item once you hit real traffic β and most teams don't discover this until the finance alert lands. Deployment is where promising models go to die on cost, latency, and cold starts. This guide walks you through the deployment decision that separates a model that ships from a model that bankrupts the demo. I'm going to compare the three main paths β serverless, containers, and edge β then break down the cost math you must run before you pick one.
"It Worked in My Notebook" β The Sentence That Kills ML Projects
Every failed ML project I've debriefed shares the same death certificate: the model was great in research and collapsed in production. The gap between a notebook cell and a live endpoint is not a small jump β it's an entirely different engineering discipline with its own failure modes. Latency budgets, autoscaling, model versioning, retries, cold starts, and the unglamorous reality that production traffic looks nothing like your train split. A 2023 report from the Algorithmia/DBT era put the figure that most teams cannot even get a model to production in the first place, and the ones that do spend most of their engineering time on plumbing, not modeling. The tools you choose for deploying models determine whether the pipeline holds up at 3 a.m. during a traffic spike or silently serves yesterday's weights to everyone.
This guide is decision-oriented: it will help you pick a deployment path β serverless, container-on-Kubernetes, managed ML serving, or edge β then give you the concrete criteria and real pricing to commit. The right answer depends on your traffic shape, your team's Kubernetes appetite, and how fast your model retrains. There is no universal winner; there is a wrong tool for your specific traffic pattern.
The Four Deployment Models You're Actually Choosing Between
Strip away the branding and every deployment platform collapses into one of four architectures. Knowing which bucket you're in makes vendor selection trivially easier.
- Serverless inference (AWS Lambda, Google Cloud Run, Azure Functions): you write a handler, the cloud scales to zero and back. Ideal for spiky, low-QPS workloads and pay-per-call billing. Cold starts are the enemy β a 2-second first-call latency is unacceptable for real-time UX.
- Containerized serving on Kubernetes (Kubeflow, Seldon, KServe, plain deployment): full control, autoscaling, GPU scheduling. Best for sustained traffic and reproducible infra, but you own the operational burden β most teams underestimate the k8s ops cost by a lot.
- Managed ML serving (SageMaker, Vertex AI, Azure ML, Seldon Core Enterprise): the platform handles provisioning, scaling, and some of the monitoring. The pragmatic middle ground that removes k8s from your plate while keeping model control.
- Edge / on-device (ONNX Runtime, TensorFlow Lite, NVIDIA TensorRT): inference happens on the device. Non-negotiable for latency-critical or offline apps, but you inherit quantization, device fragmentation, and a harder update path.
Most teams end up running two of these at once β a serverless path for bursty or low-cost lanes and a managed/container path for the always-on critical endpoint. That's normal and fine. The mistake is forcing everything into one architecture to avoid "two stacks."
Comparison of Leading Model Deployment Platforms (2026)
| Platform / Tool | Key Features | Pricing |
|---|---|---|
| Amazon SageMaker | End-to-end MLOps, real-time + serverless endpoints, batch transform, auto-scaling, A/B routing | Pay-per-use; real-time instances from ~$0.05β0.35/hr (ml.t2.mediumβml.g4dn.xlarge); serverless pay per millisecond-invoke + GB-sec |
| Google Vertex AI | Vertex Endpoints, model registry, prediction explanation, tabular AutoML | Per node-hour for managed endpoints (Vertex online prediction billed by node-hour); serverless ~$0.106/hr/node |
| Azure Machine Learning | Managed endpoints, batch endpoints, MLOps in Azure ecosystem, AKS integration | Pay for compute; managed online endpoints billed per instance per hour (e.g., STANDARD_DS1v2 ~$0.08/hr) + per-provisioned concurrency |
| Seldon Core / Seldon Deploy | Open-source k8s ML serving, custom runtime, canary + shadow deployments, explainability | Seldon Core free (OSS); Seldon Deploy enterprise licensed (quote-based) |
| KServe (formerly KFServing) | Kubernetes-native inference, serverless-style autoscaling on k8s, multi-framework | Free OSS; you pay only for your k8s cluster compute |
| Ray Serve | Python-native model serving, scales from single-node to cluster, integrates with Ray ecosystem, good for ensembles | Free OSS; managed via Anyscale from ~$0.65/node/hr (compute billed separately) |
The cloud trio (SageMaker, Vertex, Azure ML) win on managed convenience and bracket pricing, and they're the safest default if you're already all-in on one cloud. Seldon and KServe give you freedom from vendor lock-in and are the choice of teams that live on k8s and want to own the serving layer β but they transfer that ownership cost directly onto your platform engineer. Ray Serve sits nicely in the middle for Python-heavy teams running models close to the training stack. Choose by honest answers to "how much k8s do we want to run," not by feature-list length.
Picking by Traffic Shape: The Question Most Guides Skip
Pricing and features matter, but the single most decisive factor is your request pattern. Three common shapes, three correct answers. Shape 1: spiky, low, bursty β think a referral form validated by an ML classifier that sees 50 requests normally and 5,000 on viral days. Serverless (Cloud Run, SageMaker Serverless, Lambda) wins because it scales to zero and spares you paying for idle GPUs; budget for cold-start mitigation (pinned concurrency or a small warm pool) so your real-time UX doesn't eat the delay. Shape 2: sustained, predictable β a live recommendation feed at 200 QPS around the clock. Managed endpoints or a small always-on k8s deployment win; you want warm instances and smooth autoscaling, and both cost roughly the same at steady state. Shape 3: latency-critical <50ms β voice assistants, fraud checks mid-transaction. Edge or co-located GPU inference is basically mandatory; a serverless cold start alone blows your budget. If your traffic is a mix, split lanes instead of forcing one platform to do everything.
There is a hidden cost beyond the unit prices: concurrency and autoscaling misconfiguration. Most overspend by standing up too many instances "to be safe," or under-provision and suffer painful scaling lag under spikes. Real traffic tests with predictable load (k6, Locust) against a staging config save you far more than any tool-choice micro-optimization. The last mile of deployment is almost always a scaling tuning problem, not a framework problem.
Versioning, Rollback, and the Registry You Can't Skip
Deployment is not just "put it online." Production ML demands that you know exactly which weights are live, when they shipped, and how to rewind the moment they misbehave. This is where the deployment tool's relationship to your other ML infrastructure shows its worth. A model that deploys cleanly but has no version-stamped artifact is untrustworthy, because you can't attribute a behavior change to the deploy itself.
Pair your serving platform with a model registry that records dataset hash, training code commit, hyperparameters, and evaluation metrics for every version. Then hook your deploy tool to that registry so that "promote v12 to production" is a tagged, auditable, revertible action β not a manual copy of a pickle file. And put a rollback plan in the runbook before you need it: shadow- or canary-deploy new versions into a fraction of live traffic, compare their predictions against the champion, and only cut over when agreement is high. The model monitoring guide explains how to watch for drift once it's live, which is the natural companion to a solid rollback loop.
Practical Pipeline Tips: Avoid These Five Production Blunders
Reflecting on production postmortems, five recurring failures are worth carrying into any deployment design. (1) Feature/computation mismatch: your offline training script and online serving code compute a feature differently (e.g., missing-value imputation order), and accuracy silently drops 5 points in production with no code change β fix by unit-testing feature parity between train and serve. (2) Dependency drift: your pinning is loose, numpy/scikit-learn bumps on deploy day, and the model behavior changes. Pin exact versions and rebuild from a locked environment (Docker image or lockfile). (3) No graceful degradation: when the model endpoint times out, does your app crash or fall back to a rule-based response? A 100ms fallback beats a 3s timeout. (4) Ignoring cold-start latency: serverless convenience deceives; measure p99 including cold start, not just the warm path. (5) Training/serving skew undiagnosed: always log a skew-detection metric so you notice when live inputs diverge from the train distribution.
These problems are not exotic β they are the daily reality of the field, and they're exactly why a platform with good observability, clean artifacts, and sane default retries outperforms one with the prettiest dashboard. If your deployment is one-off and manual, the tools can't save you. When you're ready to connect serving to the rest of your ops, the AI toolkit index organizes related infrastructure, and free productivity tooling covers cost-effective ways to run the surrounding monitoring and alert plumbing on a budget.
Cost Realities: What Your Deployment Bill Actually Looks Like
Let's be concrete about money. A single managed endpoint on a modest CPU instance (roughly ml.t3.medium / STANDARD_D1v2) held warm 24/7 costs roughly $40β100/month in compute β trivial for one model. Add a GPU instance (ml.g4dn.xlarge, ~$0.75/hr) for an LLM or vision model held warm and you're near $550/month before autoscaling multiplies it. Serverless eats the idle cost but adds per-call revenue risk: at high QPS, per-invocation billing plus GB-seconds can exceed a warm instance. The realistic full picture also includes storage for model artifacts, observability/log shipping, and the retraining compute you'll trigger on drift β often doubling the "deployment" line item. For teams estimating a rollout, budget a full pipeline cost per model, not just the "serving" number the vendor quotes.
Experienced teams contain cost with a tiered strategy: serve the hot, revenue-critical model on always-warm managed compute; route bursty or experimental lanes to serverless; keep GPU spending only where latency truly demands it; and explore CPU-quantized variants for low-risk workloads. Edge and quantization (ONNX, TensorRT) can cut a surprisingly large share of inference cost for mature batch tasks. The covers automation patterns that trim the surrounding effter, and for teams operating across languages the Chinese-language AI tools guide lists deployment-friendly AI tooling used by Chinese-speaking engineer teams.
For more, check out: .
Frequently Asked Questions
Should we use serverless or a container on Kubernetes for our first production model?
Start serverless unless you have a hard reason not to. For a first model with uncertain or spiky traffic, serverless (Cloud Run, SageMaker Serverless, or Lambda) removes the k8s ops burden, scales to zero so you don't pay for idle GPUs, and gets you live quickly. Move to Kubernetes or a warm managed endpoint only when you have sustained traffic, need GPU scheduling control, or hit cold-start latency ceilings. You can always migrate later β redeploying a well-versioned model is cheap; running a k8s cluster before you need it is not.
How do I actually do a safe canary rollout of a model?
Deploy the candidate model alongside the champion and route a small percentage (e.g., 5β10%) of live traffic to it via A/B routing or shadow serving. Compare its predictions, latency, and error rate against the champion on a rolling basis β typically for 24β72 hours or until you hit a statistically significant agreement threshold. Only then increase the share, with an automated rollback that reverts to the champion if the error rate or prediction-contrast breaches a threshold. Managed platforms (SageMaker A/B, Seldon Deploy) automate the traffic split and rollback; on raw k8s you'll wire it with ingress rules.
Why is my model slower in production than in my notebook, and how do I fix it?
Three culprits dominate. First, framework/input-preprocessing differences (the online path does more work per request than your notebook's tight loop). Second, cold starts and no warm concurrency. Third, suboptimal batching β real-time serving sends requests one-at-a-time, wasting the model's batch throughput. Profile with a real load test, not a notebook micro-benchmark; preprocess features the same way in train and serve; and experiment with opportunistic batching or a small warm concurrency pool if cold starts are the bottleneck.
How much does it realistically cost to serve a small LLM in production?
A small open-weights model (e.g., a 3β8B parameter model quantized to 4-bit) on a single GPU can run at roughly $0.10β0.30 per 1,000 tokens of GPU cost depending on the instance, but the more meaningful number is the fixed cost: an always-warm GPU endpoint starts around $0.50β0.90/hr ($400β650/month) before scaling. For low-traffic use, serverless or batched CPU inference on a quantized model can drop that to tens of dollars a month. The real lever is quantization and batching, not instance shopping β a 4-bit INT4 model often delivers 3β5Γ the throughput per GPU dollar.
Get one practical tool tip a week
Short, useful, no spam. Unsubscribe anytime.