MLA 014 Hosting and Deploying ML: Managed APIs, Serverless GPUs, Self-Hosting

Where the model behind your product should run in 2026: managed APIs vs open weights, AWS native vs Modal, RunPod and Cloud Run GPU, vLLM and SGLang, quantized CPU inference on Spot, batch pricing, prod evals, and infra as code. Links Try a walking desk - stay healthy & sharp while you learn & code More OCDevel shows - this one has siblings, each on its own subject and produced the same way

The infrastructure episode, standalone: where the model behind your product runs in 2026, and what each choice costs. Managed APIs versus self-hosted open weights, AWS native (Bedrock, SageMaker) versus serverless GPU platforms (Modal, RunPod, Cloud Run GPU) versus bare GPUs, the inference servers (vLLM, SGLang, llama.cpp), quantized CPU inference on Spot, queues and batch pricing, observability and evals, and a minute on infrastructure as code. The coding-agent workflow is covered in Vibe Coding in 2026 through Agentic Software Engineering; agent architecture in AI Agents in 2026. Calling a model vs running one The default for a product builder is a managed API: cost is linear in usage with no floor, while a self-hosted GPU costs the same idle as busy. Four things flip the default: sustained volume, privacy or data residency, a strict latency shape, or a model only available as open weights. Move one workload when a flip applies, not the whole stack. The cloud-hosted middle path: Amazon Bedrock's model catalog now spans Anthropic, OpenAI (including the open-weight gpt-oss models), Meta Llama, Mistral, DeepSeek, Qwen and more, and Custom Model Import serves your own safetensors weights on demand with no instance to manage. The framing on rent vs own is still a16z's "Navigating the High Cost of AI Compute": renting wins for almost everyone; owning pays only at large steady spend. AWS native Lambda and Fargate still have no GPUs; the Lambda quotas page has no accelerator dimension and the Fargate FAQ sends GPU work to EC2. Lambda offers 10 GB memory, 15-minute timeouts, 10 GB container images, and SnapStart for Python. SageMaker AI: real-time endpoints scale to zero only via inference components; asynchronous inference queues S3 payloads up to 1 GB and scales to zero; serverless inference is CPU-only; batch transform for offline scoring. AWS Batch GPU jobs on Spot compute environments are the cheapest bursty GPU on AWS, with no scheduler surcharge. Aurora PostgreSQL ships pgvector 0.8.2 with HNSW, and Aurora Serverless v2 auto-pause drops compute to zero when idle. Bedrock Knowledge Bases can use that Aurora as its store. Aurora DSQL has no extensions, so no pgvector. Serverless GPU platforms The category: ship a container or function, pay per GPU-second, scale to zero. Differentiators are cold start and warm-pool control. Modal: Python-native, per-second billing, T4 through B300; cold starts around a second with memory snapshots and minimum-container knobs; batch via Function.map. RunPod Serverless: queue-based or load-balanced endpoints, FlashBoot, async run and webhooks, wide GPU list from economy cards to B200. Cloud Run GPU: GA, NVIDIA L4 and RTX PRO 6000 Blackwell, scales to zero, instance-based billing required. Baseten with the open-source Truss packager: operated production deployments for custom models. Replicate joined Cloudflare in November 2025; Cloudflare Workers AI remains a curated open-model catalog, not a bring-your-own-weights host. fal: generative media inference first, serverless GPUs on the side. Status changes: Lambda's inference API is winding down in favor of GPU instances; Fly.io's GPU Machines appear withdrawn (docs removed, no GPU line on pricing). Bare GPUs, neoclouds, and Spot Hyperscaler families: AWS P6, Google Cloud GPUs, Azure ND/NC sizes. Neoclouds: CoreWeave, Nebius, Lambda, Crusoe; Vast.ai as a marketplace with interruptible pricing. A reserved box beats serverless when the GPU is busy most of the hour. EC2 Spot advertises up to 90 percent off on-demand, with a two-minute interr