Data engineering

Amazon Web Services' managed cloud data warehouse, which uses columnar storage and runs each query in parallel across many nodes. It integrates closely with S3 and other AWS services for loading and analysis.

Example. A company loads daily sales files from S3 into Redshift and queries them with SQL for dashboards.

Related: Snowflake, BigQuery, Amazon S3, Data warehouse

Amazon's object storage service, holding files as objects inside named buckets, with very high durability and pay-as-you-use pricing. It is the common foundation for data lakes built on Amazon Web Services.

Example. A team stores raw JSON logs at s3://acme-logs/2026/09/01/ and reads them with Spark.

Related: Object storage, Data lake, Parquet, Amazon Redshift

An open-source platform for authoring, scheduling and monitoring workflows written as Python code. Tasks and their dependencies are declared, and it retries failures and shows run history in a web interface.

Example. An Airflow workflow runs at 1 a.m.: extract sales, validate, transform, then refresh a dashboard table.

Related: DAG, Orchestration, Backfill, Sensor

An open-source programming model for defining batch and streaming pipelines once and running them on different execution engines, such as Flink, Spark or Google Dataflow. It offers unified concepts like windows and triggers.

Example. A pipeline written with the Beam Python SDK reads events, windows them and writes counts, then runs on a chosen runner.

Related: Apache Flink, Apache Spark, Windowing, Stream processing

An open-source engine for stateful computation over unbounded event streams, with precise control of event time, windows and fault-tolerant state. It also handles batch work by treating a finite dataset as a bounded stream.

Example. A Flink job keeps a running count of failed logins per user over the last five minutes and raises an alert above a threshold.

Related: Stream processing, Windowing, Watermark, Exactly-once semantics

An open table format and framework for lakes that supports record-level inserts, updates and deletes and incremental reading of only changed records. It is often used to keep lake tables close to real time.

Example. A lake table of ride bookings is updated as statuses change, and downstream jobs read only records changed in the last ten minutes.

Related: Delta Lake, Apache Iceberg, Upsert, Change data capture

An open table format for very large analytic tables that tracks files through metadata snapshots. It supports transactions, schema and partition changes without rewriting data, and is readable by several engines.

Example. Spark and Trino both query the same Iceberg table, and a partition scheme can change without rewriting old data.

Related: Delta Lake, Apache Hudi, Time travel, Data lakehouse

A distributed platform that stores streams of events in durable, ordered logs and lets many applications publish and read them at scale. It is widely used to connect systems and feed real-time pipelines.

Example. An e-commerce site publishes every click to Kafka, and separate services consume the same stream for search, fraud checks and analytics.

Related: Topic, Consumer group, Offset, Publish-subscribe

An open-source engine that processes large datasets in parallel across a cluster, keeping intermediate data in memory where possible. It offers APIs in Python, SQL, Scala and Java for batch, SQL and streaming work.

Example. A PySpark job groups 2 billion transaction rows by merchant across a cluster in minutes.

Related: Hadoop, MapReduce, Resilient Distributed Dataset, Shuffle

A guarantee that every message is delivered one or more times, so nothing is lost but duplicates may occur after retries or lost acknowledgments. Consumers must therefore be written to tolerate repeated messages safely.

Example. A broker resends a message because the acknowledgment was lost, so the consumer sees it twice and must ignore the second copy.

Related: Exactly-once semantics, Idempotency, Message broker, Deduplication

A compact, row-oriented binary format that stores the schema alongside the data and supports well-defined, controlled schema changes over time. It is widely used for messages on Kafka and for write-heavy ingestion.

Example. Each event published to a Kafka topic is serialised as Avro and validated against a registered schema.

Related: Parquet, Schema evolution, Apache Kafka, Data serialization

Running a pipeline over past time periods to fill in or correct historical data, for instance after a bug fix, a new column or a late source. Good pipelines are parameterised by date so this is simple.

Example. After fixing a tax calculation, the team re-runs the daily job for the last 90 days.

Related: Idempotency, Apache Airflow, Late-arriving data, Incremental load

A mechanism by which a slow consumer signals upstream stages to slow down, or a system limits intake, so buffers do not overflow when data arrives faster than it can be processed.

Example. When a database sink slows down, the stream job reads fewer messages from Kafka instead of running out of memory.

Related: Stream processing, Message queue, Apache Flink, Dead letter queue

Handling data in large, bounded groups collected over a period and processed together on a schedule, rather than record by record as it arrives. It is simple, efficient for big volumes and easy to re-run, but results arrive with a delay.

Example. A bank computes every customer's monthly statement in one job at 2 a.m. on the first of the month.

Related: Stream processing, Data pipeline, Apache Spark, Backfill

Google Cloud's serverless data warehouse, where users run SQL over very large tables without managing servers. It stores data in a columnar form and charges by data scanned or reserved capacity.

Example. An analyst runs a SQL query over billions of rows of app events in seconds without provisioning any machines.

Related: Snowflake, Amazon Redshift, Data warehouse, Columnar storage

Distributing rows into a fixed number of groups by hashing a column value, so rows with the same key land together. It can reduce shuffling for joins and aggregations on that key.

Example. A table bucketed into 64 groups by customer_id lets a join with another table bucketed the same way avoid a full redistribution.

Related: Partitioning, Shuffle, Apache Spark, Data skew

Detecting inserts, updates and deletes in a source database, typically by reading its transaction log, and streaming them to other systems. It keeps copies in sync with low load and without repeated full extracts.

Example. Every row change in a PostgreSQL orders table is read from its log and sent to Kafka, then applied to the warehouse.

Related: Incremental load, Apache Kafka, Upsert, Event sourcing

Periodically saving the state and progress of a long-running or streaming job to durable storage, so after a failure it can resume from the last saved point rather than start over.

Example. A Flink job saves its state every minute; after a crash it restarts from the most recent checkpoint.

Related: Apache Flink, Exactly-once semantics, Offset, Stream processing

Laying out a table so all values of one column are stored together instead of keeping each row's fields together. Similar values compress well and queries read only the needed columns, ideal for analytics but costly for frequent single-row updates.

Example. To average one price column across 100 million rows, the engine reads that column alone rather than every field of every row.

Related: Row-oriented storage, Parquet, OLAP, Predicate pushdown

Rewriting many small files or fragments of a table into fewer, larger ones, and dropping obsolete data. It improves read speed and cuts metadata load, and table formats provide commands to run it.

Example. A nightly job merges thousands of small Parquet files in a partition into a few files of about 256 MB each.

Related: Small files problem, Delta Lake, Apache Iceberg, Partitioning

A set of consumer instances that share the work of reading a topic, with each partition assigned to only one member of the group at a time. Different groups each receive the full stream independently.

Example. Three instances in one group split 12 partitions, four each, while a separate analytics group reads all messages again.

Related: Apache Kafka, Topic, Offset, Message broker

Comma-separated values, a plain-text tabular format where each line is a row and fields are split by a delimiter. It is universal and human-readable but carries no types, has quoting pitfalls and is slow and large at scale.

Example. A bank exports transactions.csv; a date like 03/04/2026 is ambiguous because the file does not say whether it is day-first.

Related: Parquet, JSON Lines, Data ingestion, Data serialization

A directed acyclic graph: a set of tasks joined by one-way dependencies with no loops, so tasks can be ordered. Orchestrators use it to decide what runs first, what runs in parallel and what waits.

Example. Task load_sales depends on extract_sales and clean_sales, but the two extracts run side by side.

Related: Apache Airflow, Orchestration, Data pipeline, Backfill

An organised inventory of an organisation's datasets holding descriptions, owners, schemas, tags and usage information, with search. It helps people find data, judge whether to trust it and understand where it comes from.

Example. An analyst searches the catalog for 'customer churn', finds a certified table, sees its owner and column descriptions, and avoids building a duplicate.

Related: Metadata, Data lineage, Data governance, Data mesh

Detecting and fixing or removing errors and inconsistencies in data, such as typos, wrong formats, missing values and duplicates, to make it fit for use. It is usually a large share of pipeline effort.

Example. Standardising 'Bengaluru', 'Bangalore' and 'BLR' into one city value and trimming spaces from names.

Related: Data quality, Deduplication, Data profiling, Data validation

A formal, versioned agreement between the producer of a dataset and its consumers that specifies schema, meaning, quality expectations and delivery terms. Enforcing it turns unexpected upstream changes into visible failures instead of silent breakage.

Example. The orders team promises a table with fixed columns and daily refresh by 6 a.m.; a change that drops a column fails the contract check in review.

Related: Schema evolution, Data validation, Data mesh, Data governance

Giving a single query interface over several separate data sources, so data is fetched from where it lives at query time instead of being copied into one store. It avoids duplication but depends on source performance.

Example. One SQL view combines sales in a warehouse and customers in an operational database without copying either.

Related: Trino, Data lake, Data warehouse, Metadata

How recently a dataset was updated relative to the real world, often measured as the delay between an event happening and it being available for use. Freshness targets are checked to keep reports current.

Example. A dashboard promises data no more than 30 minutes old, and an alert fires if the latest row is older than that.

Related: Data observability, Data SLA, Late-arriving data, Batch processing

The policies, roles and processes that control how an organisation's data is defined, accessed, protected and kept accurate. It sets who may see what, who is accountable for each dataset and how rules such as retention are enforced.

Example. A company defines that only the HR team may view salary columns, assigns a named owner to the employee table and logs every access.

Related: Data catalog, Data lineage, Data retention policy, Master data management

The step of collecting data from its origin, such as databases, files, APIs or event feeds, and bringing it into a storage or processing system. It can run in scheduled batches or continuously, and its reliability sets the ceiling for everything downstream.

Example. Ingesting GST invoice CSVs dropped in a folder every night into an S3 bucket, along with a tag recording when each file arrived.

Related: Data pipeline, Change data capture, Batch processing, Object storage

A large store that keeps raw data in its original format, structured, semi-structured or unstructured, cheaply on object storage. Structure is applied when the data is read, which gives flexibility for data science but risks turning into an unmanaged swamp without cataloguing.

Example. A lake holds JSON clickstream, PDF invoices and camera images side by side in an S3 bucket, all available for later analysis or model training.

Related: Data warehouse, Data lakehouse, Schema on read, Object storage

An architecture that puts warehouse-style features, such as transactions, schema enforcement and fast SQL, directly on open files in low-cost lake storage. It aims to serve both analytics and machine learning from one copy of the data.

Example. Tables stored as Parquet files with a Delta or Iceberg layer let analysts run SQL and data scientists read the same files in Python.

Related: Data lake, Delta Lake, Apache Iceberg, Data warehouse

A record of where data came from, which steps changed it and where it flows next, often shown as a graph from source to report. It supports debugging, impact analysis before changes and compliance audits.

Example. When a revenue figure looks wrong, lineage shows it derives from a join between two staging tables, one of which failed to refresh.

Related: Metadata, Data catalog, Data governance, Data observability

A focused subset of a warehouse prepared for one department or subject, such as marketing or finance. Restricting scope makes queries simpler and faster and lets the owning team shape it for their needs.

Example. A marketing mart contains only campaign, click and conversion tables, so the team does not need to navigate the whole warehouse.

Related: Data warehouse, Star schema, Fact table, Dimension table

An organisational and architectural approach where data is owned by the business domains that produce it, each publishing well-documented data products, supported by a shared self-serve platform and common governance rules. It counters bottlenecks of one central data team.

Example. The payments team owns and publishes a payments dataset with defined quality guarantees, and the risk team consumes it through the shared platform.

Related: Data governance, Data contract, Data catalog, Data lakehouse

Designing how data is structured and related in a system, at conceptual, logical and physical levels. In analytics it includes choosing tables, keys and relationships so that data is consistent and easy to query.

Example. A team decides that customers, orders and products become separate tables linked by keys, with a date dimension for reports.

Related: Star schema, Data vault, Grain, Fact table

Continuously monitoring the health of data and pipelines, using signals like freshness, volume, schema changes, distribution shifts and lineage, so problems are detected and traced before users notice. It resembles monitoring for software.

Example. An alert fires because today's orders table has 70 percent fewer rows than usual, and lineage shows the source extract failed.

Related: Data quality, Data freshness, Data lineage, Great Expectations

A chain of automated steps that moves data from one or more sources to a destination, applying checks and changes along the way. Pipelines replace manual copying, run on a schedule or on events, and are the basic building block of data engineering work.

Example. A pipeline reads app logs every hour, filters out bots, aggregates page views per city and writes the counts to a dashboard table.

Related: ETL, Orchestration, Batch processing, Stream processing

Examining a dataset to summarise its structure and content, such as data types, null counts, distinct values, ranges and common patterns. It reveals problems and informs the rules and transformations a pipeline needs.

Example. Profiling a loan file shows the age column has values up to 250 and 12 percent nulls, prompting validation rules.

Related: Data quality, Data validation, Data cleansing, Great Expectations

How well data fits its intended use, commonly judged on accuracy, completeness, consistency, uniqueness, validity and timeliness. Poor quality quietly corrupts dashboards and machine learning models, so it is measured and monitored.

Example. In a customer table, 4 percent of phone numbers are blank and 1 percent are duplicated, so the completeness and uniqueness checks flag it.

Related: Data validation, Data profiling, Data observability, Data cleansing

A rule stating how long different kinds of data are kept before being archived or deleted, driven by business need, cost and legal requirements. Enforcing it limits storage cost and exposure.

Example. Raw click logs are deleted after 90 days, while invoices are kept for the period required by tax law.

Related: Data governance, Data tiering, Data catalog, Data quality

Converting in-memory data structures into a sequence of bytes or text for storage or transmission, and rebuilding them later. Formats differ in size, speed, readability and how well they handle changing schemas.

Example. A Python dictionary is serialised to JSON to send over an API, or to Avro to place on a Kafka topic.

Related: Avro, Protocol Buffers, JSON Lines, Parquet

An uneven spread of data across partitions, where a few keys hold far more records than others, so a few workers do most of the work while the rest wait. It causes slow, straggling jobs and memory errors.

Example. If 40 percent of orders have city = 'Unknown', the one worker handling that key becomes the bottleneck of the job.

Related: Shuffle, Partitioning, Bucketing, Apache Spark

A service level agreement for data, stating measurable promises such as delivery time, freshness, completeness or availability that a data team makes to its users, along with what happens if they are missed.

Example. The sales table will be ready by 7 a.m. every day with at least 99 percent of orders loaded.

Related: Data freshness, Data contract, Data observability, Orchestration

Placing data on different storage classes according to how often it is used, with frequently read data on fast, costly storage and rarely used data on cheaper, slower storage. Rules can move data between tiers automatically.

Example. Last month's logs stay in standard storage, and anything older than a year moves to an archive tier at lower cost.

Related: Object storage, Data retention policy, Amazon S3, Data lake

Automatically checking that incoming or transformed data meets stated rules, such as required fields, allowed ranges, correct types and matching references, and reacting when it does not. It stops bad records from spreading downstream.

Example. A check rejects any order row where quantity is negative or the pincode is not six digits and routes it to a quarantine table.

Related: Data quality, Great Expectations, Data contract, Dead letter queue

A warehouse modelling method that separates business keys (hubs), relationships between them (links) and descriptive history (satellites). It is built for auditability, easy addition of new sources and loading tables in parallel.

Example. A customer hub holds only customer IDs, and each source system's attributes go into its own satellite table with load dates.

Related: Data modelling, Star schema, Data warehouse, Slowly changing dimension

A central database designed for analysis rather than day-to-day transactions. It stores cleaned, structured, historical data from many sources in a consistent model so analysts can run large aggregate queries quickly and trust the numbers.

Example. A company's finance team queries a warehouse holding five years of sales, returns and inventory from separate systems to build quarterly reports.

Related: Data lake, OLAP, Star schema, Data mart

An open-source tool that lets teams build warehouse transformations as SQL select statements, adding version control, tests, documentation and dependency ordering. It performs the transformation stage of ELT inside the warehouse.

Example. A dbt model called orders_clean selects from raw_orders, and a test checks that order_id is unique and not null.

Related: ELT, Data warehouse, Data lineage, Data validation

A separate queue or topic that receives messages that could not be processed successfully after retries, such as malformed records. It keeps the main flow moving and lets engineers inspect and replay the failures.

Example. A record with invalid JSON is moved to orders_dlq, and the pipeline continues with the next message.

Related: Message queue, Data validation, Message broker, Idempotency

Identifying and removing repeated records that describe the same entity or event, using exact keys or fuzzy matching, and keeping one authoritative copy. It prevents double counting in reports and models.

Example. Two rows with the same order_id from a retried upload are collapsed into one, keeping the latest.

Related: Data cleansing, At-least-once delivery, Master data management, Idempotency

An open table format that adds a transaction log over Parquet files in a lake, giving atomic writes, schema enforcement, updates and deletes, and the ability to view earlier versions. It underpins many lakehouse setups.

Example. A MERGE command updates 5,000 customer rows in a lake table safely, and a failed job leaves the table unchanged.

Related: Apache Iceberg, Apache Hudi, Time travel, Data lakehouse

A table in a dimensional model that holds descriptive attributes used to filter, group and label facts, such as product name, category or city. It is typically smaller and changes slowly.

Example. A customer dimension holds name, city and segment so revenue can be grouped by segment.

Related: Fact table, Star schema, Slowly changing dimension, Surrogate key

An open-source analytical database that runs inside the calling process, like SQLite for analytics, using columnar storage and vectorised execution. It can query Parquet and CSV files directly on a laptop.

Example. In Python, duckdb.sql("SELECT city, SUM(amt) FROM 'sales.parquet' GROUP BY city") runs without any server.

Related: Parquet, OLAP, Columnar storage, Data warehouse

A variation on classic integration where raw data is copied into the target platform first and reshaped there afterwards using the platform's own compute, usually SQL. It suits cloud warehouses with cheap storage and scalable processing, and keeps the raw copy for reprocessing.

Example. Raw Razorpay payment events land in BigQuery untouched; a scheduled SQL model then builds a cleaned payments table from them.

Related: ETL, dbt, Data warehouse, Medallion architecture

A data integration pattern with three stages: pull records out of source systems, reshape and clean them on a separate processing engine, then write the finished result into a target store such as a warehouse. Transforming before loading keeps bad or sensitive data out of the target.

Example. A retailer in Pune extracts daily orders from a MySQL database, converts currencies and removes test orders in a Python job, then loads the clean table into the warehouse.

Related: ELT, Data pipeline, Data warehouse, Data ingestion

A design where every change to a system's state is stored as an immutable event in order, and the current state is obtained by replaying those events. It gives a full audit trail and the ability to rebuild state at any moment.

Example. A bank account is stored as deposits and withdrawals; the balance is computed by summing them.

Related: Change data capture, Apache Kafka, Publish-subscribe, Time travel

A guarantee that each event affects the result one time, even when failures and retries occur. It is usually achieved by combining at-least-once delivery with idempotent or transactional writes rather than by delivering each message once.

Example. After a crash and retry, a payment event is not counted twice in the account balance.

Related: At-least-once delivery, Idempotency, Offset, Apache Flink

The central table of a dimensional model that records measurable business events, such as sales or clicks, as rows with numeric measures and keys pointing to dimensions. It is usually the largest table.

Example. Each row is one order line with quantity, price and keys for date, customer and product.

Related: Dimension table, Star schema, Grain, Surrogate key

Copying an entire dataset from source to target on every run, replacing whatever was there before. It is simple and self-correcting, because any earlier error disappears, but slow and costly for big tables.

Example. A 50,000-row product table is fully reloaded every night because it is small and changes unpredictably.

Related: Incremental load, ETL, Data ingestion, Batch processing

The precise level of detail that one row of a fact table represents, such as one order line or one daily total per store. Declaring it first prevents mixing levels and double counting.

Example. A sales fact at order-line grain has one row per product in an order, not one per order.

Related: Fact table, Star schema, Dimension table, Data modelling

An open-source Python framework for declaring data quality checks, called expectations, running them against datasets and producing readable reports. Teams embed it in pipelines to catch problems before data is consumed.

Example. An expectation states that the column order_id must never be null and must be unique; the pipeline fails if a new batch violates it.

Related: Data validation, Data quality, Data profiling, Orchestration

An open-source framework for storing and processing very large datasets across clusters of ordinary machines, made up of a distributed file system, a resource manager and the MapReduce engine. It defined early big data practice.

Example. An older enterprise keeps years of logs in a Hadoop cluster and queries them with Hive.

Related: HDFS, MapReduce, Apache Spark, Data lake

The Hadoop Distributed File System, which splits large files into blocks and replicates each across several machines so data survives hardware failures and can be processed where it is stored.

Example. A 1 GB file is split into 128 MB blocks, each copied to three different nodes.

Related: Hadoop, MapReduce, Object storage, Small files problem

A data warehouse layer over Hadoop that lets users query files in a distributed file system using SQL-like statements. Its metastore of table definitions is reused by many other engines.

Example. A Hive table defined over ORC files in HDFS is queried with SQL, and Spark reads the same table through the metastore.

Related: Hadoop, HDFS, ORC, Metadata

The property that running an operation more than once has the same final effect as running it once. Idempotent pipeline steps make retries and re-runs safe, since they cannot create duplicates or double counts.

Example. A job that overwrites the partition for 2026-09-01 gives the same table whether it runs once or three times.

Related: Exactly-once semantics, Backfill, Upsert, At-least-once delivery

Loading only the data that is new or changed since the last run, tracked using timestamps, IDs or change logs, rather than copying everything. It is faster and cheaper than repeated full loads.

Example. Each night the job copies rows with updated_at later than the previous run's high-water mark.

Related: Full load, Change data capture, Upsert, Data ingestion

A text format with one complete JSON object per line, so files can be read, appended to and split record by record without loading everything. It is popular for logs and streaming exports.

Example. Each line of events.jsonl is like {"user":42,"action":"click"}, so a job can process a billion lines one at a time.

Related: CSV, Data serialization, Data ingestion, Avro

A simplification where all processing, live and historical, is done by a single stream-processing path over an immutable log, with reprocessing done by replaying the log. It avoids duplicating logic across batch and speed layers.

Example. To fix a bug, the team replays the Kafka log from the start through the corrected stream job.

Related: Lambda architecture, Apache Kafka, Stream processing, Event sourcing

A design that runs a batch layer for accurate, complete results over all history and a speed layer for quick results on recent data, then merges both when queried. It gives low latency and accuracy but means maintaining two code paths.

Example. Yesterday's totals come from a nightly batch job, and today's running totals come from a stream, and dashboards combine them.

Related: Kappa architecture, Batch processing, Stream processing, Apache Kafka

Records that reach a pipeline after the time period they belong to has already been processed, often because of network delay or offline devices. Systems handle it with allowed lateness, reprocessing or corrections.

Example. A phone offline in a tunnel uploads a 9:02 event at 9:20, after the 9:00 window closed.

Related: Watermark, Windowing, Backfill, Stream processing

Delaying execution of operations until a result is actually requested, letting the engine see the whole plan and optimise it first. Spark transformations are lazy and run only when an action such as a count or write is called.

Example. In Spark, filter and select only build a plan; nothing runs until write.parquet() is called.

Related: Apache Spark, Resilient Distributed Dataset, Predicate pushdown, Batch processing

A programming model for processing big data in two phases: a map step turns input into key-value pairs in parallel, and a reduce step combines all values sharing a key. The framework handles distribution and failures.

Example. Counting words: mappers emit (word, 1) for every word, and reducers sum the counts per word.

Related: Hadoop, Apache Spark, Shuffle, Batch processing

Practices and tools for creating a single, trusted record of core business entities, such as customers or products, by reconciling versions held in different systems and sharing the result. It resolves conflicting identifiers and attributes.

Example. Three systems hold slightly different records for the same customer; a matching process merges them into one golden record.

Related: Deduplication, Data governance, Surrogate key, Data quality

A layered lakehouse pattern that improves data step by step: bronze holds raw ingested data, silver holds cleaned and conformed data, and gold holds business-ready aggregates. Each layer is a checkpoint for quality.

Example. Raw app events land in bronze, are deduplicated and typed in silver, and become daily active user counts in gold.

Related: Data lakehouse, ELT, Delta Lake, Data quality

Software that receives messages from senders and delivers them to receivers, decoupling the two so they need not run at the same time or know each other. It handles routing, buffering and delivery guarantees.

Example. An order service hands a message to a broker, which delivers it to the billing service even if billing was briefly offline.

Related: Message queue, Publish-subscribe, Apache Kafka, Dead letter queue

A buffer in which messages wait in order until a consumer takes and processes them, usually each message being handled by one consumer. It smooths bursts of traffic and lets producers and consumers work at different speeds.

Example. Image-resize tasks wait in a queue while a pool of workers picks them up one by one.

Related: Message broker, Publish-subscribe, Backpressure, Dead letter queue

A storage type that keeps data as whole objects, each with an identifier and metadata, in a flat namespace accessed over an HTTP interface, rather than as blocks or a folder hierarchy. It scales cheaply to petabytes.

Example. Photos and Parquet files sit in a bucket, each retrieved by its key, without a real folder tree.

Related: Amazon S3, Data lake, HDFS, Data tiering

A sequential number giving a message's position within a Kafka partition. Consumers record the number they have reached so they can resume after a restart or deliberately replay earlier messages.

Example. A consumer that crashed at offset 5,000 restarts and continues from 5,001.

Related: Apache Kafka, Consumer group, Exactly-once semantics, Topic

Online analytical processing: workloads that run complex queries summarising large volumes of historical data, such as totals by region and month. Systems for it favour read speed and columnar layouts over frequent single-row updates.

Example. An analyst asks for total revenue by state and quarter across three years of orders.

Related: OLTP, Data warehouse, Columnar storage, Star schema

Online transaction processing: systems that handle many small, fast reads and writes for day-to-day operations, such as placing an order or transferring money, with strong consistency. They are optimised for individual records, not big analysis.

Example. A bank's core database records each ATM withdrawal as a quick, isolated transaction.

Related: OLAP, Row-oriented storage, Data warehouse, Change data capture

Optimized Row Columnar, a columnar file format originating in the Hadoop ecosystem, storing data in stripes with indexes and statistics. It compresses well and is especially common with Hive workloads.

Example. A Hive table of telecom call records is stored as ORC so queries on one column skip most of the data.

Related: Parquet, Columnar storage, Hadoop, Predicate pushdown

Coordinating when and in what order pipeline tasks run, handling dependencies, schedules, retries, alerts and resources. It is separate from the tasks' own logic, which does the actual data work.

Example. An orchestrator starts the transformation only after all three source extracts finish and pages the team if one fails twice.

Related: Apache Airflow, DAG, Data pipeline, Sensor

An open columnar file format for analytics that stores values column by column with compression, encodings and built-in statistics. It makes scans of a few columns from huge tables fast and small, and is read by Spark, pandas, DuckDB and most warehouses.

Example. A 10 GB CSV of clickstream often shrinks to a fraction of its size as Parquet, and a query needing two columns reads only those.

Related: Columnar storage, ORC, Predicate pushdown, Avro

Splitting a large dataset into separate pieces by the value of a chosen column, commonly date, so each piece is stored and processed independently. Queries that filter on that column read only the relevant pieces.

Example. Sales are stored in folders like date=2026-09-01/, so a query for one week touches seven folders, not years of data.

Related: Bucketing, Predicate pushdown, Data skew, Small files problem

An optimisation where filter conditions are handed down to the storage layer so irrelevant data is skipped at read time, using file statistics or partitions, instead of loading everything and filtering afterwards.

Example. For WHERE order_date = '2026-08-15', the engine uses min/max statistics to skip Parquet row groups that cannot contain that date.

Related: Parquet, Partitioning, Columnar storage, Apache Spark

A language-neutral binary serialisation system from Google in which structures are declared in a schema file and compiled into code. Messages are small and fast to parse, and field numbers allow forward and backward compatible changes.

Example. A service defines message Order with numbered fields and sends it over gRPC; older readers ignore new fields they do not know.

Related: Avro, Data serialization, Schema evolution, Apache Kafka

A messaging pattern where senders publish messages to a topic without knowing who reads them, and every subscriber to that topic receives a copy. It decouples producers from consumers and allows new consumers to be added easily.

Example. A payment event is published once, and the notification, ledger and fraud services each subscribe and receive it.

Related: Topic, Message broker, Apache Kafka, Event sourcing

Spark's low-level collection of records split across machines, immutable and rebuilt from a recorded chain of operations if a partition is lost. It gives fault tolerance without replicating data, though DataFrames are usually preferred.

Example. An RDD of log lines is filtered and mapped; if a node fails, Spark recomputes only the lost partitions.

Related: Apache Spark, Lazy evaluation, Shuffle, MapReduce

Laying out a table so all fields of one record are stored together. It makes reading or writing a whole record fast, which suits transaction systems, but it reads unneeded columns during large analytical scans.

Example. An order-processing database fetches all details of order 1234 in one read because its fields sit together on disk.

Related: Columnar storage, OLTP, Avro, CSV

The ability of a data format or table to cope with changes to its structure over time, such as added, renamed or retyped columns, while old and new data remain readable. Rules decide which changes are compatible.

Example. A new nullable column loyalty_tier is added to an Avro schema, and old files still read fine with the value treated as null.

Related: Avro, Delta Lake, Data contract, Schema on write

An approach where data is stored without enforcing a structure and the structure is applied only when it is queried. It allows fast ingestion of varied data, but errors in shape surface later at query time.

Example. Raw JSON logs are dumped into a lake, and each analyst defines the fields they need in their own query.

Related: Schema on write, Data lake, Metadata, Data lakehouse

An approach where data must match a predefined structure before it is stored, with rejections or transformations at load time. It gives consistent, trustworthy tables but needs upfront design and slows ingestion of new formats.

Example. A warehouse load fails if the amount column contains text, since the table defines it as a decimal.

Related: Schema on read, Data warehouse, Data validation, ETL

In Airflow, a special kind of task that does no data work itself but waits until an external condition becomes true, such as a file appearing or another workflow finishing, before letting downstream tasks proceed.

Example. A sensor polls an S3 path every five minutes and releases the pipeline once the partner's daily file arrives.

Related: Apache Airflow, DAG, Orchestration, Data ingestion

The redistribution of data across machines so that records with the same key end up together, needed for joins, groupings and sorts. It involves disk and network transfer and is often the most expensive part of a distributed job.

Example. Grouping sales by city in Spark moves rows across the network so every row for Mumbai reaches the same worker.

Related: Data skew, Apache Spark, Bucketing, MapReduce

A technique for handling descriptive attributes that change infrequently, such as a customer's city. Type 1 overwrites the old value, while Type 2 adds a new row with validity dates so history is preserved.

Example. When a customer moves from Chennai to Bengaluru, a Type 2 approach keeps both rows with start and end dates so old sales remain tied to Chennai.

Related: Dimension table, Surrogate key, Star schema, Fact table

The slowdown and overhead caused by storing very many tiny files in a distributed file or object store, since each file adds metadata handling and open costs. Jobs are usually fixed by merging files into larger ones.

Example. A streaming job writing every minute produces 1,440 small files per day per partition; queries slow down until they are merged.

Related: Compaction, Partitioning, HDFS, Delta Lake

A cloud data platform that separates storage from compute, so several independent compute clusters can query the same data. Users work mainly in SQL and pay for storage and compute used.

Example. A finance team and a data science team run queries on separate Snowflake compute clusters over the same tables without slowing each other.

Related: BigQuery, Amazon Redshift, Data warehouse, Object storage

A variation of the star design in which dimension tables are broken into further related tables to remove repetition, for example splitting product into category and brand. It saves space but needs more joins.

Example. A product dimension links to a separate category table instead of repeating category names in every product row.

Related: Star schema, Dimension table, Fact table, Data warehouse

A warehouse design with a central fact table holding measurable events, linked directly to surrounding dimension tables that describe them, such as customer, product and date. It is simple and fast for analytical queries.

Example. A sales fact table with keys to date, store and product dimensions lets you sum revenue by month and city.

Related: Snowflake schema, Fact table, Dimension table, Data warehouse

Computing on data continuously as individual events arrive, producing results within seconds or less instead of waiting for a full batch. It powers use cases like fraud alerts and live metrics, and requires care over ordering, lateness and state.

Example. A UPI fraud system scores each transaction the moment it is published to a Kafka topic and blocks suspicious ones within a second.

Related: Batch processing, Apache Kafka, Apache Flink, Windowing

An artificial identifier, usually a meaningless number, assigned to a row in a warehouse table instead of relying on the source system's business key. It stays stable if source keys change and supports keeping history.

Example. Customer 'C-1023' from the CRM is stored under surrogate key 88,412 in the customer dimension.

Related: Dimension table, Slowly changing dimension, Fact table, Master data management

The ability of a versioned table to be queried as it was at an earlier time or version, kept through snapshots or a log. It helps audit, reproduce results and recover from bad writes.

Example. After a faulty job overwrote a table, an engineer queries the version from yesterday and restores it.

Related: Delta Lake, Apache Iceberg, Data lineage, Backfill

A named category or feed in a messaging system to which producers send messages and from which consumers read. In Kafka it is split into partitions that spread load and preserve order within each one.

Example. A topic called upi_payments receives every payment event, split into 12 partitions.

Related: Apache Kafka, Publish-subscribe, Offset, Consumer group

A distributed SQL query engine that runs interactive queries across many data sources, such as lakes, databases and warehouses, without moving the data first. It is the basis for federated querying.

Example. One Trino query joins a table in a data lake with a table in a PostgreSQL database.

Related: Data lakehouse, Data federation, Apache Iceberg, Data warehouse

A write that updates a record if its key already exists and inserts it if not, in one operation. It keeps target tables current without duplicates when changed data arrives.

Example. An incoming row for customer 501 replaces the old row, while a row for a new customer 9001 is added.

Related: Incremental load, Change data capture, Idempotency, Delta Lake

A marker in stream processing stating that the system believes no more events older than a given event time will arrive, so windows up to that time can be closed and their results emitted. It balances waiting for late data against latency.

Example. With a 2-minute watermark delay, the 10:00 to 10:05 window is finalised once events stamped 10:07 have been seen.

Related: Windowing, Late-arriving data, Apache Flink, Stream processing

Grouping an endless stream of events into finite chunks, by time or count, so aggregates can be computed. Common types are tumbling (fixed, non-overlapping), sliding (overlapping) and session windows (gaps of inactivity).

Example. A tumbling one-minute window counts orders per minute, while a 10-minute sliding window updated every minute smooths the trend.

Related: Stream processing, Watermark, Apache Flink, Late-arriving data