Mlops llmops cloud

Automatically notifying people, through email, chat or paging, when a measurement crosses a threshold or a pattern appears. Good alerts are actionable and rare enough to be trusted; too many cause alert fatigue.

Example. An alert messages the on-call engineer when the error rate is above 5% for ten minutes.

Related: Monitoring, On-call, Incident response, Service level objective

Amazon Web Services' service for renting virtual servers of many sizes by the hour or second, including instance types with GPUs. Users choose the operating system and software, giving full control over the machine.

Example. A team rents a GPU instance for a day to fine-tune a model and shuts it down afterwards to stop charges.

Related: Virtual machine, GPU instance, Amazon S3, Spot instance

Amazon Web Services' managed registry for container images, integrated with its identity and access system and its container services. Teams push images from pipelines and deploy them to clusters without running their own registry.

Example. CI pushes a new model-server image to ECR, and the cluster pulls it using its IAM permissions.

Related: Docker registry, Amazon EKS, Docker image, IAM role

Amazon Web Services' managed Kubernetes service, in which AWS runs the control plane while customers manage worker nodes or use managed node options. It reduces the operational work of running Kubernetes clusters.

Example. A team creates an EKS cluster with a GPU node group for inference workloads.

Related: Kubernetes cluster, Google Kubernetes Engine, Amazon ECR, Managed service

Amazon Web Services' managed platform for machine learning, covering notebooks, training jobs, tuning, a model registry, pipelines and hosted endpoints. It lets teams run the whole lifecycle without managing servers directly.

Example. A team launches a training job on GPU instances and deploys the resulting model to an autoscaling SageMaker endpoint.

Related: Vertex AI, Azure Machine Learning, Model endpoint, Managed service

An open-source continuous-delivery tool for Kubernetes that follows the GitOps method. It watches a git repository, compares it with what runs in the cluster, and syncs any differences automatically or on approval.

Example. Argo CD shows an application as "out of sync" when the cluster differs from git, and one click reapplies the git version.

Related: GitOps, Kubernetes, Continuous delivery, Configuration drift

An isolated location within a cloud region, with its own power, cooling and networking, connected to other zones by fast links. Spreading a service across zones protects it from the failure of a single data centre.

Example. A database is replicated across three zones in one region, so a power cut in one leaves service running.

Related: Cloud region, High availability, Failover, Load balancer

Amazon Web Services' native infrastructure-as-code service. Templates written in YAML or JSON describe resources in a stack, and the service creates, updates and deletes them together, rolling back automatically if an update fails.

Example. A template creates a VPC, subnets and a database in one stack, and deleting the stack removes them all.

Related: Infrastructure as code, Terraform, Amazon EC2, Virtual private cloud

Amazon Web Services' serverless function service. It runs code in response to events such as an HTTP call or file upload, scales automatically and bills by execution time and memory, with limits on duration and package size.

Example. A Lambda function is triggered when a CSV lands in S3 and writes a cleaned copy to another bucket.

Related: Serverless, Amazon S3, Cold start, Event-driven architecture

Microsoft Azure's object storage service for large amounts of unstructured data such as images, documents and model files, organised in containers with access tiers that trade cost against retrieval speed.

Example. A company stores scanned invoices in Blob Storage with an archive tier for records older than a year.

Related: Object storage, Amazon S3, Google Cloud Storage, Data lake

Microsoft's offering that provides OpenAI models through Azure, with Azure's enterprise controls such as private networking, regional deployment and identity integration. Companies use it when they want model access inside their existing cloud governance.

Example. A bank calls a GPT model through its private Azure network so traffic never crosses the public internet.

Related: Amazon Bedrock, Vertex AI, Virtual private cloud, Managed service

The starting image named on the first line of a Dockerfile, on top of which everything else is layered. Choosing a small, maintained base reduces download size, build time and the number of known security flaws inherited by the final image.

Example. Switching a base image from a full Debian variant to a slim variant can cut an image from about 1 GB to about 200 MB.

Related: Dockerfile, Docker image, Multi-stage build, Image scanning

A release approach that keeps two identical environments: blue serves users while green receives the new version. After testing green, traffic is switched over at once, and switching back to blue gives an instant rollback path.

Example. A team deploys a new model API to the idle green environment, verifies it, and flips the load balancer so all users hit green.

Related: Canary deployment, Rolling update, Load balancer, Rollback

Releasing a new version to a small share of real traffic first, watching its errors, latency and quality, and widening the share only if it looks healthy. It limits the number of users affected by a faulty release.

Example. A new LLM prompt version serves 5% of chats for a day; when its thumbs-down rate matches the old one, it goes to 100%.

Related: Blue-green deployment, Shadow deployment, A/B test, Rollback

A method where the current production model, the champion, runs against one or more newer candidates, the challengers, on live or replayed traffic. A challenger replaces the champion only if it proves better on agreed measures.

Example. A bank keeps its scoring model as champion while a new gradient-boosted challenger scores the same applications in the background for a month.

Related: Shadow deployment, A/B test, Model promotion, Model monitoring

An automated, ordered series of jobs defined in a configuration file, such as install, lint, test, build and publish, that runs whenever code changes. Each stage must pass before the next runs, and results appear on the pull request.

Example. A YAML file defines four stages; a failed unit test in stage two stops the pipeline and marks the pull request red.

Related: CI/CD, GitHub Actions, Continuous integration, Deployment pipeline

Short for continuous integration and continuous delivery or deployment: the automated path that takes code from a commit through building, testing and releasing. In machine learning it is extended to also test data and models, not just code.

Example. Merging to main triggers tests, builds a Docker image, pushes it to a registry and updates the staging cluster automatically.

Related: Continuous integration, Continuous delivery, CI pipeline, MLOps

A geographic area where a cloud provider operates data centres, such as Mumbai or Hyderabad, made of several isolated availability zones. Choice of region affects latency, price, service availability and where data legally resides.

Example. A team picks the Mumbai region so Indian users get lower latency and data stays inside India.

Related: Availability zone, Data residency, Multi-region deployment, Cloud computing

A Kubernetes object that stores non-secret settings, such as feature flags or model names, as key-value pairs kept outside the container image. Pods read them as environment variables or files, so configuration changes without rebuilding the image.

Example. A ConfigMap holds `MODEL_NAME=llama-3-8b` and `MAX_TOKENS=512`, which the inference pods read at start-up.

Related: Kubernetes Secret, Environment variable, Twelve-factor app, Kubernetes deployment

The gradual divergence between what infrastructure code says should exist and what actually exists, usually caused by manual changes in a console. Drift causes surprises and outages, so teams detect it and reapply their declared setup.

Example. Someone opens a firewall port by hand; a scheduled Terraform plan detects the difference and flags it.

Related: Infrastructure as code, Terraform state, GitOps, Immutable infrastructure

Each instruction in a Dockerfile that changes files creates a layer, a stored set of file differences stacked on the layers below. Layers are cached and shared between images, so ordering instructions so that rarely changing steps come first speeds up rebuilds.

Example. Copying requirements.txt and installing packages before copying source code means editing a Python file reuses the cached install layer instead of reinstalling everything.

Related: Docker image, Dockerfile, Build cache, Multi-stage build

The low-level software that actually pulls images, creates isolated processes and runs containers on a machine. Kubernetes talks to it through a standard interface; well-known examples are containerd and CRI-O, and Docker Engine uses containerd underneath.

Example. A Kubernetes node running containerd starts a pod's containers without needing the full Docker command-line tool installed.

Related: Docker container, Kubernetes, Kubernetes node, OCI image

The practice of keeping software always in a releasable state by automatically building, testing and preparing every change for deployment, with a human approving the final push to production. Continuous deployment removes even that manual approval.

Example. After tests pass, a build is deployed to staging automatically, and a lead clicks one button to release it to production.

Related: Continuous integration, CI/CD, Deployment pipeline, Release management

Automatically retraining a model on a schedule or when triggers fire, such as new data arriving or measured performance dropping, and then validating and releasing it through the same pipeline. It keeps models current as the world changes.

Example. A recommender retrains every night on the day's clicks, and is promoted only if it beats the current model on held-out data.

Related: MLOps, Data drift, ML pipeline, Model registry

Tracking changes to datasets over time, like source control does for code, so any model can be linked to the exact data it was trained on. Tools such as DVC store lightweight pointers in git while the large files sit in object storage.

Example. A model card points to dataset version v12; checking out that version restores the same 2 million rows.

Related: DVC, Reproducibility, Data lineage, Experiment tracking

Fixing the exact versions of libraries a project uses, for example in a lock file, so builds and deployments install identical packages every time. It prevents surprise breakages when a library publishes a new release.

Example. A requirements file lists `numpy==1.26.4` rather than `numpy`, so a build next month still uses the same version.

Related: Reproducibility, Docker image, Semantic versioning, Software supply chain security

The automated sequence that moves a release from build through testing environments to production, with checks and approvals at each step. It makes releases repeatable, auditable and less dependent on individual engineers' manual steps.

Example. A change flows from build to staging, passes smoke tests, waits for approval, then rolls out to production in stages.

Related: CI/CD, Continuous delivery, Environment promotion, Rollback

Plans and technology for restoring systems and data after a major failure, such as a regional outage or data loss. Two key targets are recovery time objective, how long restoration may take, and recovery point objective, how much data loss is tolerable.

Example. A backup copy in another region and a tested restoration plan let a service return within an hour after a regional outage.

Related: High availability, Failover, Backup, Multi-region deployment

Following one request as it passes through many services, recording each step as a span with its start time and duration. The connected spans show where time is spent or failures happen in complex systems.

Example. A trace of one chat request shows 20 ms in the gateway, 300 ms in retrieval and 2.1 s in the language model call.

Related: Observability, OpenTelemetry, Span, Correlation ID

An open-source platform for packaging an application together with its libraries and settings into a portable unit that runs the same way on a laptop, a server or a cloud machine. It uses features of the Linux kernel to isolate programs from each other without the weight of a full virtual machine.

Example. A data scientist in Pune builds a model API on her laptop, packages it with Docker, and the ops team in Hyderabad runs the identical package on a cloud server with no "works on my machine" surprises.

Related: Docker image, Dockerfile, Docker registry, Kubernetes

A tool that describes a multi-container application in one YAML file and starts, connects and stops all its services with a single command. It is widely used for local development and testing rather than large production clusters.

Example. A compose file starts a model API, a Redis cache and a Postgres database together on a developer's laptop with `docker compose up`.

Related: Docker container, Container networking, Docker volume, Kubernetes

A running instance of an image: an isolated process with its own file system, network and process list, sharing the host machine's operating-system kernel. Containers start in seconds and are disposable, so they are the usual unit for deploying services and batch jobs.

Example. Running `docker run -p 8000:8000 sentiment-api:1.0` starts a container that answers requests on port 8000 until it is stopped.

Related: Docker image, Kubernetes pod, Container runtime, Docker Compose

A read-only, layered snapshot containing an application's code, runtime, libraries and file-system contents. It is the template from which running containers are started. Because images are built in layers, unchanged layers are reused, which makes builds and downloads faster.

Example. A 900 MB Python image for a fraud-scoring API is built once, tagged v1.4, and then started as ten identical running containers.

Related: Dockerfile, Docker registry, Container layer, Docker container

A server that stores and distributes container images, identified by name and tag. Teams push images after a build and pull them at deployment time. It can be public, such as Docker Hub, or private and run by a cloud provider or the company itself.

Example. CI pushes `registry.example.in/ml/churn-api:2.3.1`, and the production cluster pulls that exact tag when deploying.

Related: Docker image, Amazon ECR, Container runtime, Image tag

A plain-text recipe listing the steps Docker follows to build an image: choose a base image, copy files, install dependencies, set environment variables and define the start command. Keeping it in version control makes the build repeatable and reviewable.

Example. A Dockerfile starts with `FROM python:3.11-slim`, copies requirements.txt, runs `pip install -r requirements.txt`, and ends with `CMD ["uvicorn", "app:app"]`.

Related: Docker image, Multi-stage build, Base image, .dockerignore

An open-source tool that adds data and model versioning on top of git. Large files stay in remote storage while small text files recording their hashes are committed, and it can define reproducible pipelines linking data, code and results.

Example. `dvc add data/train.csv` stores the file remotely and creates a tiny pointer file that is committed to git.

Related: Data versioning, Reproducibility, Object storage, Experiment tracking

Running computation close to where data is produced or used, on devices or local servers, instead of in a distant central data centre. It cuts delay, saves bandwidth and can keep data local, and is used for on-device AI.

Example. A factory camera runs a defect-detection model on a small local box, avoiding sending video to the cloud.

Related: On-device inference, Latency, Content delivery network cache, Model serving

The amount of unreliability a service is allowed under its objective, such as 0.1% failed requests for a 99.9% target. While budget remains, teams can release quickly; when it is used up, they slow releases and focus on stability.

Example. A 99.9% monthly target allows about 43 minutes of downtime; after 40 minutes are used, feature releases pause.

Related: Service level objective, Service level indicator, Canary deployment, Incident response

A design in which components communicate by publishing and reacting to events, such as "file uploaded" or "order placed", rather than calling each other directly. It decouples services and suits asynchronous, scalable AI pipelines.

Example. Uploading a scanned form to storage emits an event that triggers an OCR service, which emits another that triggers a classifier.

Related: Message queue, Serverless, Webhook, Asynchronous inference

Recording, for every training run, its code version, data version, settings, metrics and outputs so that results can be compared and reproduced later. Without it, teams cannot explain why one model beat another.

Example. A run log shows that learning rate 3e-4 gave 0.87 F1 on commit a1b2c3, while 1e-3 gave 0.84.

Related: MLflow, Weights & Biases, Reproducibility, Hyperparameter sweep

Automatically switching to a standby component or site when the active one fails, so service continues with little interruption. It depends on health checks to detect failures and on standby capacity being ready.

Example. When the primary database becomes unreachable, the replica is promoted and applications reconnect within seconds.

Related: High availability, Health check, Disaster recovery, Load balancer

A secondary model or provider automatically used when the primary one fails, times out or is rate-limited. It improves availability of language-model applications, though answer style and quality may differ from the primary.

Example. When the main provider returns errors, the gateway retries the request on a backup model so users still get a reply.

Related: LLM gateway, Model routing, Circuit breaker, High availability

A switch in configuration that turns a capability on or off, or for selected users, without deploying new code. It separates releasing code from releasing behaviour, enabling gradual rollouts and quick shutdown of a faulty feature.

Example. An AI summary button is enabled for 10% of users through a flag, and disabled instantly if complaints rise.

Related: Canary deployment, A/B test, Rollback, CI/CD

A cycle in which a model's outputs influence the data it is later trained on or evaluated with. It can be useful, as when user ratings improve a system, or harmful, when a recommender keeps showing what it already showed and narrows its own data.

Example. A feed that promotes popular videos gets more clicks on them, so retraining strengthens the same bias.

Related: Delayed labels, Continuous training, Model monitoring, Human feedback collection

A practice that brings engineering, finance and business teams together to understand, allocate and optimise cloud spending, making cost a shared, measured responsibility. It is increasingly applied to GPU and language-model bills.

Example. A monthly review shows idle GPU instances cost ₹2.5 lakh, prompting an auto-shutdown policy.

Related: Cost allocation tags, Reserved instance, Spot instance, LLM cost management

An operating approach in which a git repository is the single source of truth for the desired state of infrastructure and applications, and an automated agent continually makes the running system match it. Changes happen through reviewed commits, giving history and easy rollbacks.

Example. Merging a pull request that changes replicas from 3 to 6 causes Argo CD to scale the live deployment accordingly.

Related: Infrastructure as code, Argo CD, Kubernetes deployment, Configuration drift

Google Cloud's object storage service, which stores files as objects in buckets with several storage classes for hot and cold data. It is commonly used for datasets, model checkpoints and pipeline outputs.

Example. A training job reads its dataset from a GCS bucket and writes checkpoints back to the same bucket.

Related: Object storage, Amazon S3, Azure Blob Storage, Data lake

A cloud virtual machine that includes one or more graphics processing units, used for training and running deep-learning models. They are expensive and often scarce, so teams watch utilisation and shut them down when idle.

Example. An instance with 8 GPUs is rented for a training run and billed per hour until it is stopped.

Related: Amazon EC2, GPU utilisation, Spot instance, Compute budget

An open-source tool for building dashboards and visualising metrics, logs and traces from many data sources such as Prometheus. Teams use it to display system health and share views during incidents.

Example. A Grafana dashboard shows GPU utilisation, request rate and error percentage for a model service on one screen.

Related: Prometheus, Metrics, Monitoring, Alerting

A lightweight request or command that reports whether a service is alive and able to do its job. Load balancers, orchestrators and monitors use it to stop sending traffic to unhealthy instances and to trigger restarts.

Example. A `/health` endpoint returns 200 OK when the model is loaded and 503 when it is not.

Related: Liveness and readiness probes, Load balancer, Uptime, Monitoring

Designing a system to keep working through failures of individual parts by running redundant copies, usually across several machines or data centres, with automatic failover. It is measured by uptime targets such as 99.9%.

Example. A service runs on three servers in two zones so that losing one machine or zone does not stop it.

Related: Uptime, Availability zone, Load balancer, Failover

A Kubernetes controller that changes the number of pod copies up or down based on measured load, such as average CPU use or a custom metric like requests per second. It matches capacity to demand and avoids paying for idle replicas.

Example. With a target of 60% CPU, traffic during an IPL final pushes a service from 3 to 12 pods, then back down afterwards.

Related: Autoscaling, Kubernetes deployment, Resource requests and limits, KEDA

A cloud identity with a set of permissions that trusted users, services or workloads can temporarily assume, instead of using long-lived passwords or keys. Attaching roles to machines and pods removes the need to embed credentials in code.

Example. A serving pod assumes a role that lets it read model files from one bucket, with no access key stored anywhere.

Related: Identity and access management, Least privilege, Secrets management, Amazon S3

The system a cloud or organisation uses to decide who or what is permitted to do which actions on which resources. It defines users, roles and policies, and its central rule is least privilege: grant only the access needed.

Example. A policy lets a training job read one bucket and write to another, but nothing else in the account.

Related: IAM role, Least privilege, Role-based access control, Secrets management

Automatically inspecting a container image's packages and libraries against databases of known security vulnerabilities before it is deployed. Pipelines can fail a build when serious findings appear, catching risky dependencies early.

Example. A scan of a model-serving image flags an outdated OpenSSL library with a critical flaw, and the pipeline blocks the release until the base image is updated.

Related: Base image, Software supply chain security, CI pipeline, Docker registry

A human-readable label attached to a specific image version in a registry, such as 1.4.2 or latest. Tags are movable pointers, so pinning a deployment to an exact version or to the image digest, a content hash, is safer than relying on latest.

Example. Deploying `churn-api:latest` can silently pull a different build tomorrow, while `churn-api@sha256:ab12...` always gives the same bytes.

Related: Docker registry, Docker image, Semantic versioning, Image scanning

A practice of never changing servers or containers after they are deployed; to update, you build a new image and replace the old instances. It removes hidden manual changes and makes environments consistent and easy to roll back.

Example. Instead of logging into servers to patch them, the team builds a new image and swaps out the old machines.

Related: Docker image, Configuration drift, Blue-green deployment, Infrastructure as code

Defining servers, networks, databases and permissions in version-controlled text files that tools apply automatically, instead of clicking in a console. Environments become repeatable, reviewable and easy to recreate or destroy.

Example. A repository holds files that create a GPU cluster, storage bucket and network in a new region with one command.

Related: Terraform, CloudFormation, Pulumi, GitOps

A Kubernetes object, working with an ingress controller, that routes external HTTP and HTTPS requests into the cluster by hostname or URL path. It centralises tasks such as TLS certificates and path-based routing behind one entry point.

Example. Requests to `api.example.in/chat` go to the chat service while `api.example.in/search` go to the search service, both through one ingress.

Related: Kubernetes service, Load balancer, API gateway, TLS termination

A test that checks several components working together, such as an API talking to a database and a model, rather than each alone. It finds interface and configuration mistakes that unit tests miss, but is slower and needs more setup.

Example. A test starts the API with a real test database, posts a document and checks that the stored record and the returned prediction match.

Related: Unit test, Smoke test, Continuous integration, End-to-end test

An open-source project for running machine learning workflows on Kubernetes. Its components cover pipelines, notebooks, distributed training and model serving, letting teams describe ML steps as containers and run them at scale.

Example. A team defines a Kubeflow pipeline of five containerised steps and runs it on its cluster with GPU nodes for training.

Related: Kubernetes, ML pipeline, Workflow orchestration, MLflow

An open-source system that runs and manages many containers across a group of machines. You declare what you want, such as five copies of a service, and it schedules them, restarts failed ones, balances traffic and rolls out updates. It is the standard platform for production workloads.

Example. A company tells Kubernetes to run five replicas of its recommendation API; when one machine fails, Kubernetes starts replacements elsewhere automatically.

Related: Kubernetes pod, Kubernetes deployment, Kubernetes node, Helm

The complete set of machines managed together by Kubernetes: a control plane that makes decisions and worker machines that run the containers. Teams often keep separate clusters for development, staging and production.

Example. A startup runs one small cluster of three worker machines for testing and a larger, separate one for customer traffic.

Related: Kubernetes, Kubernetes node, Control plane, Namespace

A Kubernetes object that declares how many identical copies of a pod should run and which image version they use. It handles rolling updates and rollbacks, replacing old pods gradually so a service keeps answering requests during a release.

Example. Setting replicas to 4 and changing the image tag from v1 to v2 makes the deployment swap pods one by one with no downtime.

Related: Kubernetes pod, ReplicaSet, Rolling update, Rollback

A single machine, physical or virtual, in a cluster that runs pods. Each node runs an agent that talks to the control plane, plus a container runtime. Nodes can have special hardware such as GPUs, which pods request when needed.

Example. A cluster has ten CPU nodes for web services and two GPU nodes reserved for model inference.

Related: Kubernetes cluster, Kubernetes pod, Node pool, Container runtime

The smallest deployable unit in Kubernetes: one or more tightly coupled containers that share a network address and storage and are always scheduled together on the same node. Pods are temporary, so they are normally created and replaced by higher-level controllers.

Example. A pod holds a model-server container plus a small logging sidecar container that share the same network address.

Related: Kubernetes deployment, Sidecar container, Kubernetes node, ReplicaSet

A Kubernetes object meant for sensitive values such as passwords and API keys, delivered to pods separately from images and code. By default the values are only base64-encoded, so encryption at rest and tight access rules are needed for real protection.

Example. An LLM gateway pod receives its provider API key from a Secret instead of having it written into the source code.

Related: ConfigMap, Secrets management, Role-based access control, Environment variable

A stable network name and address that sends traffic to a changing set of pods chosen by labels. Since pods come and go with new addresses, a service gives other applications one fixed endpoint and spreads requests across healthy pods.

Example. Other apps call `http://fraud-api` and the service distributes each request among the four running fraud-api pods.

Related: Kubernetes pod, Ingress, Load balancer, Service discovery

Health checks Kubernetes runs against a container. A liveness probe decides whether to restart a container that is stuck; a readiness probe decides whether it should receive traffic yet. Readiness is vital for model servers that take minutes to load weights.

Example. A GPU pod fails its readiness probe until its 20 GB model finishes loading, so no user requests reach it prematurely.

Related: Health check, Kubernetes pod, Rolling update, Cold start

Practices for keeping the spending of language-model applications under control: choosing smaller models where they suffice, caching, shortening prompts, batching, setting budgets and limits and tracking cost per feature or user.

Example. Routing simple questions to a cheaper model cuts a chatbot's monthly bill from ₹3 lakh to ₹1.4 lakh.

Related: Token usage tracking, Semantic caching, Model routing, FinOps

An automated process that runs a fixed set of test prompts and scoring methods against each new prompt, model or retrieval change, and reports quality before release. It is the LLM counterpart of regression testing.

Example. On each pull request, 200 saved questions are answered and scored, and the merge is blocked if accuracy falls below 85%.

Related: LLMOps, Regression test, Continuous integration, Prompt versioning

A service placed between applications and one or more model providers that offers one consistent API and central features such as authentication, rate limits, logging, cost tracking, caching, retries and fallback to other models.

Example. All apps call the company's gateway, which forwards to two providers and records tokens used by each team.

Related: API gateway, Model routing, Fallback model, Rate limiting

Monitoring and tracing applications built on language models by capturing each prompt, response, retrieved document, tool call, token count, latency and cost, linked into one trace. It lets teams debug bad answers and control spending.

Example. A trace shows that a wrong answer came from a retrieval step returning an outdated policy document.

Related: Observability, Distributed tracing, LLMOps, Token usage tracking

MLOps adapted to applications built on large language models. Alongside deployment and monitoring it manages prompts, retrieval data, evaluation sets, cost per request, safety filters and provider changes, since these often matter more than training a model from scratch.

Example. An LLMOps setup versions every prompt, runs an evaluation suite before release and tracks token spend per feature.

Related: MLOps, Prompt versioning, LLM observability, LLM evaluation pipeline

Recording timestamped messages about what a program does, including errors, warnings and key events, so problems can be investigated afterwards. Structured logs in JSON are easier to search and analyse than free text.

Example. A service logs `{"level":"error","msg":"model timeout","request_id":"a91"}` whenever an inference call takes too long.

Related: Observability, Structured logging, Distributed tracing, Log aggregation

A cloud offering where the provider handles installation, patching, scaling and backups of the underlying software, and the customer only uses it. It trades some control and higher unit prices for far less operational work.

Example. Using a managed database means the provider handles updates and backups instead of the team's own engineers.

Related: Serverless, Amazon SageMaker, Amazon EKS, Cloud computing

Numeric measurements collected over time, such as requests per second, error percentage, memory use and GPU utilisation. They are cheap to store and aggregate, and form the basis of dashboards and alerts.

Example. A time series records 120 requests per second at 10:00 and 340 at 10:05.

Related: Monitoring, Prometheus, Alerting, Observability

An open-source platform for managing the machine learning lifecycle. It records experiments with parameters and metrics, stores model artefacts, provides a model registry and offers a standard format for packaging and serving models.

Example. A data scientist logs 30 training runs with `mlflow.log_metric("auc", 0.91)` and compares them in a web interface.

Related: Experiment tracking, Model registry, Model artefact, Weights & Biases

A set of practices that combine machine learning, software engineering and operations to build, deploy, monitor and maintain models reliably in production. It covers versioning, automation, testing, deployment and monitoring so models keep working after the first launch.

Example. An MLOps team automates retraining a demand-forecasting model weekly, tests it, deploys it and alerts on accuracy drops.

Related: CI/CD, Model registry, Model monitoring, LLMOps

The saved output of training that is needed to use a model: the learned weights, plus configuration, tokenizer, preprocessing objects and metadata. It is stored as files, versioned and loaded by the serving system.

Example. A 14 GB artefact folder holds weights in safetensors format, the tokenizer files and a config.json.

Related: Model registry, Model serialisation, Object storage, Model versioning

Tracking a deployed model's behaviour over time: input data quality, prediction distributions, latency, errors and, where labels arrive, accuracy. It reveals when the world has changed or the model has stopped working well.

Example. A dashboard flags that the share of loan applicants under 25 doubled this month and that approval scores have shifted.

Related: Data drift, Concept drift, Model performance decay, Observability

Moving a model version from one lifecycle stage to the next, for example from development to staging to production, after it passes checks and approvals. Recording promotions gives an audit trail and stops untested models from reaching users.

Example. After beating the baseline by 2 points on a held-out set and passing fairness checks, model v9 is promoted to production.

Related: Model registry, Environment promotion, Model validation, Champion-challenger

A central catalogue of trained model versions with their metadata, metrics, lineage and lifecycle stage such as staging or production. It gives teams one controlled place to review, approve, promote and roll back models.

Example. Version 7 of a churn model is marked "staging" after tests, and a reviewer promotes it to "production" with one approval.

Related: Model artefact, Model versioning, MLflow, Model promotion

Checks run on a candidate model before release: accuracy against a baseline, performance on important slices of data, robustness, latency, fairness and format correctness. A candidate that fails a threshold is blocked from promotion.

Example. A pipeline rejects a new model whose recall for rural users is 9 points lower than for urban users, even though overall accuracy improved.

Related: Model promotion, Regression test, Model registry, Data validation

Giving every trained model a unique, ordered identifier so that any version can be found, compared, deployed or restored. It is normally tied to the code, data and settings that produced it.

Example. A registry lists churn-model v1 to v9, and production runs v8 while v9 is being tested.

Related: Model registry, Semantic versioning, Model lineage, Rollback

Continuously collecting and watching measurements about a system, such as error rate, latency and resource use, with alerts when they cross limits. It answers whether something is wrong, so problems are noticed by engineers before users complain.

Example. A dashboard shows API error rate; an alert fires when it exceeds 2% for five minutes.

Related: Observability, Alerting, Metrics, Prometheus

A Dockerfile technique that uses several FROM steps: an early stage installs compilers and build tools, and a later stage copies only the finished output into a small clean image. The result is a lean final image that excludes tools needed only during building.

Example. A team compiles a C++ tokenizer in a heavy build stage, then copies just the compiled library into a slim runtime stage, shrinking the image by hundreds of megabytes.

Related: Dockerfile, Base image, Docker image, Container layer

The ability to understand what is happening inside a system from the data it emits, mainly metrics, logs and traces. Where monitoring checks known signals, it helps engineers investigate new, unexpected problems by asking fresh questions of the data.

Example. When responses slow down, engineers follow a single request's trace to find that one database call takes 3 seconds.

Related: Metrics, Logging, Distributed tracing, OpenTelemetry

An arrangement in which engineers take turns being reachable to respond to alerts and outages outside normal working hours. Clear runbooks and fair rotations help keep services reliable without burning people out.

Example. This week's on-call engineer is paged at 2 a.m. when the payment model's endpoint stops responding.

Related: Alerting, Incident response, Runbook, Service level objective

An open, vendor-neutral standard and set of tools for generating and exporting telemetry: traces, metrics and logs. Instrumenting code once lets a team send data to many backends, and it now includes conventions for generative AI calls.

Example. An app instrumented with OpenTelemetry sends traces to Jaeger in testing and to a paid observability service in production.

Related: Observability, Distributed tracing, Metrics, LLM observability

A score comparing the share of data falling into each bin for two datasets, usually training and live data, to measure distribution shift. Practitioners commonly treat values under about 0.1 as stable and above about 0.25 as a significant shift. Formula: PSI = sum of (actual% - expected%) x ln(actual% / expected%).

Example. PSI on the applicant income feature reaches 0.31, so the team investigates and considers retraining.

Related: Data drift, Kolmogorov-Smirnov test, Model monitoring, Feature distribution

A written analysis after an incident that describes what happened, its impact, the timeline, root causes and follow-up actions. A blameless style focuses on fixing systems and processes rather than blaming individuals.

Example. A postmortem finds that a missing data check let a corrupted feed retrain the model, and adds a validation step.

Related: Incident response, On-call, Runbook, Root cause analysis

An open-source monitoring system that regularly pulls numeric metrics from services, stores them as time series and supports a query language, PromQL, and alert rules. It is a standard choice for Kubernetes monitoring.

Example. Prometheus scrapes a model server's `/metrics` page every 15 seconds and alerts if p95 latency exceeds 500 ms.

Related: Metrics, Grafana, Alerting, Monitoring

A Kubernetes controller that keeps a stated number of identical pods running at all times, creating new ones if any disappear. It is usually managed indirectly through a deployment, which creates a new one for each version.

Example. If a ReplicaSet is set to 3 pods and a node crashes taking one down, it launches a replacement pod on another node.

Related: Kubernetes deployment, Kubernetes pod, Kubernetes node, Horizontal Pod Autoscaler

Settings on a Kubernetes container stating how much CPU, memory or GPU it is guaranteed (request) and the most it may use (limit). Requests guide scheduling onto nodes; exceeding a memory limit gets the container killed, so values need careful tuning.

Example. A model-server container requests 4 CPUs and 16 GiB memory and is limited to 8 GiB more; going past the memory limit triggers an out-of-memory kill.

Related: Kubernetes pod, Resource quota, Kubernetes node, Horizontal Pod Autoscaler

A method of managing permissions by assigning them to roles, such as viewer or admin, and then assigning people or services to roles, rather than granting rights one by one. Kubernetes and clouds both use it.

Example. A data scientist role can view logs and run jobs in the `research` namespace but cannot change production deployments.

Related: Identity and access management, Least privilege, Namespace, Kubernetes cluster

A release strategy that replaces old copies of a service with new ones a few at a time, keeping enough healthy instances serving requests throughout. It avoids downtime and can be paused or reversed if the new version misbehaves.

Example. A deployment with 10 pods updates 2 at a time, so at least 8 keep serving users until all run the new image.

Related: Kubernetes deployment, Blue-green deployment, Canary deployment, Rollback

Automatically reducing a service to no running instances when it receives no traffic, and starting one again on the next request. It saves money for rarely used services, at the cost of a delay on the first request after idle time.

Example. A demo model endpoint costs nothing overnight, but the first morning request waits 30 seconds while a container starts.

Related: Cold start, Serverless, Autoscaling, KEDA

The practice of storing, distributing, rotating and auditing sensitive values such as passwords, tokens and API keys in a dedicated, access-controlled system, never in source code or images. Examples include HashiCorp Vault and cloud secret managers.

Example. An app fetches its database password at start-up from a secret manager, and the password is rotated every 30 days without a redeploy.

Related: Kubernetes Secret, Environment variable, IAM role, Software supply chain security

A virtual firewall in a cloud network that controls which traffic may reach a resource, by protocol, port and source. Rules default to denying everything and allow only what is listed, so opening ports narrowly limits exposure.

Example. A rule allows port 443 from anywhere and port 5432 only from the application servers.

Related: Virtual private cloud, IAM role, Network segmentation, Amazon EC2

A cloud model where the provider runs and scales the servers, and you supply only code or a container, paying per use. It removes server management and can scale to zero, but has limits on run time, memory and start-up delay.

Example. A function that resizes uploaded images runs only when a file arrives and is billed by the millisecond.

Related: AWS Lambda, Cold start, Scale to zero, Function as a service

A specific measurement of how well a service is performing from the user's point of view, such as the fraction of requests answered successfully or the share answered within 300 ms. It is the raw number that objectives are set against.

Example. The indicator is the proportion of prediction requests that return within 200 ms; this month it is 99.3%.

Related: Service level objective, Service level agreement, Error budget, Metrics

A helper container placed in the same pod as the main application to add a supporting capability such as log shipping, a network proxy or credentials refresh, without changing the application's own code.

Example. A proxy sidecar in each pod encrypts service-to-service traffic while the model server code stays unchanged.

Related: Kubernetes pod, Service mesh, Logging, Kubernetes deployment

A quick, shallow set of checks run right after a deployment to confirm the main functions work, such as the service starting and answering one basic request. If it fails, the release is stopped or rolled back before deeper testing.

Example. After deploying a model API, a script sends one sample request and expects a valid prediction within 2 seconds.

Related: Staging environment, Health check, Deployment pipeline, Integration test

A copy of production, as close as practical in configuration and infrastructure, used for final testing before a release. Problems that only appear with realistic setups can be found here without affecting real users.

Example. Before launch, a chatbot update is tested in staging with the same cluster settings and a masked copy of production data.

Related: Environment promotion, Deployment pipeline, Smoke test, Production

A service that keeps no user-specific data in its own memory between requests, so any instance can answer any request. This makes it easy to add, remove and replace copies, which is why state is kept in databases or caches.

Example. Any of the ten API replicas can handle a user's next message because the conversation history lives in a shared database.

Related: Load balancer, Stateful application, Autoscaling, Twelve-factor app

An open-source tool from HashiCorp for infrastructure as code. You describe desired cloud resources in a declarative language, it compares them with what exists, shows a plan of changes and then applies them across many cloud providers.

Example. Running `terraform plan` shows that one new S3 bucket and one IAM role will be created before `terraform apply` does it.

Related: Infrastructure as code, Terraform state, CloudFormation, Pulumi

Measuring how many input and output tokens each request, user or feature consumes, since language model providers charge per token. It supports budgeting, cost allocation, spotting runaway prompts and setting limits.

Example. A dashboard shows the summarisation feature uses 60% of monthly tokens, and one prompt is 4,000 tokens longer than needed.

Related: LLM observability, LLM cost management, Rate limiting, Cost allocation tags

A logically isolated private network inside a public cloud, where the customer defines address ranges, subnets, routing and firewall rules. It keeps databases and model services off the public internet unless deliberately exposed.

Example. Inference servers sit in private subnets of a VPC and are reachable only through a public load balancer.

Related: Availability zone, Security group, Amazon EC2, Network segmentation

Software that schedules and runs multi-step jobs in the right order, handling dependencies, retries, failures and alerts. It is the control layer that runs data and ML pipelines reliably. Airflow, Prefect and Dagster are common examples.

Example. An orchestrator runs extract, then feature building, then training, and retries the training step twice if a GPU node fails.

Related: ML pipeline, Directed acyclic graph, Apache Airflow, Kubeflow