Generative media multimodal

A scene representation made of millions of coloured 3-D Gaussian blobs that are projected and blended to render images. It reaches high visual quality and real-time rendering speeds after fitting to a set of photographs.

Example. A phone video of a room is converted into a set of Gaussians that can be viewed from any angle at interactive frame rates.

Related: Neural radiance field, Text-to-3D generation, Volume rendering

The part of a speech system that links audio features to linguistic units such as phonemes or characters. In classic pipelines it works with a separate language model; in end-to-end systems it is folded into one network.

Example. Given a mel spectrogram frame, the acoustic model outputs probabilities over 40 phonemes.

Related: Phoneme, Automatic speech recognition, Connectionist temporal classification

A model that treats discrete audio tokens as a sequence and predicts them autoregressively, like a text language model, allowing generation of speech, music or sound continuations from a prompt.

Example. Given 3 seconds of a piano tune as tokens, the model generates the next 10 seconds in the same style.

Related: Neural audio codec, Zero-shot TTS, Autoregressive image generation

A VAE variant that multiplies the KL term in the objective by a factor beta greater than 1. This pressure encourages more independent, interpretable latent dimensions at the cost of blurrier reconstructions.

Example. With beta = 4, one latent dimension of a face model may control only smile while another controls only head angle.

Related: Variational autoencoder, Disentangled representation, Evidence lower bound

A family of vision-language models (BLIP and BLIP-2) for captioning, question answering and retrieval. BLIP bootstraps cleaner captions from noisy web data, and BLIP-2 links frozen image and language models via a Q-Former.

Example. BLIP captions a batch of product photos to build searchable alt text for an online catalogue.

Related: Q-Former, Image captioning, Vision-language model

A technique that makes diffusion models follow a text prompt more closely. The model is trained sometimes with and sometimes without the condition, and at sampling time the prediction is pushed away from the unconditional output toward the conditional one. It trades diversity for prompt adherence.

Example. output = uncond + w * (cond - uncond) with w = 7.5 is a typical setting in Stable Diffusion pipelines.

Related: Guidance scale, Classifier guidance, Text-to-image generation

A metric that measures how well a generated image matches its text prompt by taking the cosine similarity between CLIP embeddings of the image and the text. Higher values mean closer agreement, but it is only a proxy for human judgement.

Example. An image of 'a red bus' scoring 0.32 against that prompt but 0.18 against 'a blue car' shows the intended match.

Related: Contrastive language-image pretraining, Fréchet Inception Distance, Text-to-image generation

A learned table of vectors used in vector quantisation, where each continuous feature is replaced by the index of its nearest table entry. It turns images or audio into sequences of discrete tokens and is a key part of VQ-VAEs and neural audio codecs.

Example. A codebook of 1024 entries lets each audio frame be stored as a single integer from 0 to 1023.

Related: VQ-VAE, Residual vector quantisation, Image tokenizer

A GAN in which both generator and discriminator receive an extra input, such as a class label or another image, so the output can be controlled. It is the basis for many image-to-image and label-to-image systems.

Example. Feeding the label 'sari' to a conditional GAN produces a clothing image of a sari rather than a random garment.

Related: Generative adversarial network, Pix2Pix, Image-to-image translation

A generative model trained so that any point along a noising trajectory maps directly to the same clean output, allowing one or a few steps of generation. It can be distilled from a diffusion model to gain large speedups at some cost in quality.

Example. A distilled consistency model produces a usable 512 x 512 image in 1 to 4 network calls instead of 50.

Related: Latent consistency model, Diffusion sampler, Knowledge distillation

Metadata or cryptographic records that show where a piece of media came from and how it was edited. Standards such as C2PA content credentials attach signed history to files so viewers can check whether an image was camera-captured or AI-generated.

Example. A news photo carries a signed manifest listing the camera capture and each edit made afterwards.

Related: Deepfake, Watermarking generated content, Synthetic media

A neural network add-on that lets a pretrained diffusion model follow extra spatial guidance such as edge maps, depth maps or human pose skeletons. It copies the model's encoder blocks, trains the copy on the new condition, and adds its output back, leaving the original weights frozen.

Example. Feeding a pose skeleton to ControlNet makes the generated person hold exactly that pose while the prompt sets clothing and setting.

Related: Stable Diffusion, Text-to-image generation, LoRA fine-tuning for diffusion

The mechanism by which a text prompt steers an image generator. Image features act as queries and the text encoder's token vectors act as keys and values, so each image region can attend to the words that describe it at every denoising layer.

Example. When the prompt says 'a red kurta', attention maps from the word 'red' highlight the clothing region of the latent.

Related: Text-to-image generation, Latent diffusion model, Attention mechanism

Finding items in one modality using a query in another, such as retrieving images from a text description or captions from a picture. Models embed both modalities in a shared space where matching pairs lie close.

Example. Typing 'red silk saree with gold border' returns matching catalogue photos from an e-commerce store.

Related: Multimodal embedding, Contrastive language-image pretraining, Image-text retrieval

A GAN framework for image translation between two domains without paired examples. Two generators translate in opposite directions, and a cycle-consistency loss requires that translating there and back returns the original image.

Example. Photos of horses are turned into zebras, and translating back recovers the horse, without any matched horse-zebra pairs.

Related: Pix2Pix, Image-to-image translation, Generative adversarial network

Synthetic media, usually video or audio, made with generative models to convincingly show a real person doing or saying something they never did. It enables fraud and misinformation, so detection and provenance tools have become important.

Example. A fake video call using a cloned face and voice of a company executive is used to trick an employee into a payment.

Related: Voice cloning, Content provenance, Synthetic media

A deterministic sampling method (DDIM) for diffusion models trained in the usual way. It skips steps in the reverse process, so good images can be produced in tens of steps instead of hundreds, and the same starting noise always gives the same output, which enables smooth interpolation.

Example. A model trained with 1000 steps is sampled with DDIM in 50 steps, cutting generation time roughly twentyfold with slightly lower fidelity.

Related: Denoising diffusion probabilistic model, Diffusion sampler, Consistency model

The 2020 formulation of diffusion models (DDPM) that trains a network to predict the noise added at a random step and samples by running a long chain of small denoising steps. It made diffusion competitive with GANs for image quality, though sampling needs hundreds or thousands of steps.

Example. A DDPM trained with 1000 steps produces sharp faces but takes 1000 network evaluations per image, which motivated faster samplers.

Related: Diffusion model, Denoising diffusion implicit model, Noise schedule

A setting in image-to-image generation that decides how much noise is added to the input picture before denoising. Low strength keeps the original almost unchanged; high strength lets the model change it substantially, approaching generation from scratch.

Example. At strength 0.3 a portrait keeps its face and only gets a new texture, while at 0.9 only the rough colour layout survives.

Related: Image-to-image translation, Inpainting, Diffusion model

A generative model that learns to create data by reversing a gradual noising process. Training teaches a network to remove a little noise from corrupted samples; generation starts from pure random noise and repeatedly cleans it up until a realistic image, audio clip or video appears. It underlies most modern image generators.

Example. Starting from a 512 x 512 grid of random noise, a model runs 30 cleaning steps and ends with a picture of a Diwali rangoli that matches the text prompt.

Related: Forward diffusion process, Reverse diffusion process, Latent diffusion model, Noise schedule

The algorithm that turns a trained diffusion network into a sequence of denoising steps at generation time. Different samplers (such as DDIM, Euler or DPM-Solver variants) trade speed, determinism and image quality by using different numerical solvers and step counts.

Example. Switching from a 50-step Euler sampler to a 20-step DPM-Solver-style sampler can halve generation time with little visible quality loss.

Related: Denoising diffusion implicit model, Reverse diffusion process, Inference steps

A diffusion model whose denoising network is a transformer that processes patches of a latent image as tokens, instead of a convolutional U-Net. It scales predictably with model size and compute and is used in many recent image and video generators.

Example. A latent is cut into 2 x 2 patches, each treated as a token, and a stack of transformer blocks predicts the noise for all patches together.

Related: Latent diffusion model, Vision transformer, Text-to-video generation

In a GAN, the classifier that scores whether a sample is real or produced by the generator. Its judgement is the training signal for the generator, so an over-strong or over-weak discriminator can destabilise learning.

Example. The discriminator outputs 0.92 for a genuine photo and 0.15 for a generated one, and both scores drive gradient updates.

Related: Generator, Generative adversarial network, Mode collapse

Extracting meaning from scanned or digital documents by combining text, layout and visual cues such as tables, headings and stamps. Systems handle invoices, forms and reports and often use vision-language models.

Example. An invoice photo is parsed into vendor name, GST number, line items and total.

Related: Optical character recognition, Vision-language model, Chart understanding

A fine-tuning method that teaches a text-to-image model a specific subject from a few photos by binding it to a rare identifier token, while a preservation loss keeps the model's general knowledge from being overwritten.

Example. Uploading 15 photos of a pet dog and training with the identifier 'sks dog' lets prompts like 'sks dog on a beach' produce that exact dog.

Related: Textual inversion, LoRA fine-tuning for diffusion, Stable Diffusion

A multimodal design that combines modalities at the input level, with images, audio and text converted to tokens and processed together by one model from the start. It allows deep cross-modal interaction.

Example. Image patches and text tokens are placed in a single sequence and passed through the same transformer layers.

Related: Late fusion, Multimodal model, Omni model

The objective maximised when training a VAE: the expected reconstruction quality minus a KL divergence penalty pulling the encoder's distribution toward a prior. It is a tractable lower bound on the log probability of the data.

Example. ELBO = E[log p(x|z)] - KL(q(z|x) || p(z)); a rising ELBO means better reconstructions with a well-behaved latent space.

Related: Variational autoencoder, Posterior collapse, Beta-VAE

A non-autoregressive TTS model that predicts all mel spectrogram frames in parallel, using a duration predictor to decide how long each phoneme lasts. It is faster and more robust than attention-based models that can skip or repeat words.

Example. Because durations are predicted explicitly, FastSpeech does not repeat or drop words in long sentences.

Related: Tacotron, Text-to-speech, Neural vocoder

A DeepMind vision-language model that keeps a pretrained language model frozen and inserts gated cross-attention layers so it can attend to visual features, enabling few-shot learning from interleaved images and text.

Example. Given three example image-caption pairs in the prompt, Flamingo captions a fourth image in the same style.

Related: Perceiver resampler, Vision-language model, Few-shot learning

A training framework for generative models that learns a velocity field transporting simple noise to data along smooth paths, using a simple regression loss. It often gives straighter trajectories than classic diffusion, allowing high quality with fewer sampling steps.

Example. Instead of predicting noise, the network predicts the direction from a noisy point toward the target image, and sampling integrates that direction.

Related: Rectified flow, Diffusion model, Consistency model

The fixed, non-learned half of a diffusion model in which small amounts of Gaussian noise are added to a clean sample over many steps until only random noise remains. Because it is defined in closed form, any noisy step can be produced directly from the original, which makes training efficient.

Example. An image is corrupted over T = 1000 steps; by step 1000 it is indistinguishable from random static, and the network is trained on randomly chosen intermediate steps.

Related: Reverse diffusion process, Noise schedule, Diffusion model

A metric for generated images that compares the statistics (mean and covariance) of deep features from real and generated sets using a pretrained Inception network. Lower values mean the generated distribution is closer to the real one.

Example. A model with FID 8 on a face dataset matches real-image statistics more closely than a model scoring 25.

Related: Inception Score, CLIP score, Generative adversarial network

A generative model made of two networks trained against each other: a generator that fabricates samples and a discriminator that tries to tell them from real data. As each improves, the generator learns to produce outputs the discriminator can no longer separate from real ones.

Example. A GAN trained on 70,000 face photos learns to output new, realistic faces of people who do not exist.

Related: Generator, Discriminator, Mode collapse, StyleGAN

The step of converting written letters into the sounds used to pronounce them, by dictionary lookup or a trained model. It helps TTS systems pronounce names and unusual words correctly.

Example. A G2P model maps the spelling 'Bengaluru' to a phoneme sequence so a voice assistant says it properly.

Related: Phoneme, Text-to-speech, Text normalisation

The number that controls how strongly a diffusion model is pushed to follow the prompt during generation. Low values give varied but loosely related pictures; high values follow the text closely but can oversaturate colours and reduce variety.

Example. Setting the scale to 3 for 'a masala chai stall at dusk' gives soft, creative results, while 12 gives a literal but harsher image.

Related: Classifier-free guidance, Negative prompt, Text-to-image generation

Generating a natural-language sentence that describes the content of an image. It is used for accessibility alt text, photo search and dataset labelling, and is typically scored with metrics such as CIDEr or BLEU.

Example. Given a street photo, the model outputs 'A vendor selling flowers beside a busy road in the morning.'

Related: Vision-language model, Visual question answering, BLIP

A model, often a VQ-VAE or similar autoencoder, that converts an image into a grid of discrete tokens and can decode them back. It lets language-model-style transformers generate or read images as if they were text.

Example. A tokenizer turns a 256 x 256 image into 256 tokens, and a transformer predicts them one by one to generate a picture.

Related: VQ-VAE, Codebook, Autoregressive image generation

The task of converting an input image into a related output, such as a sketch to a photo, day to night, or a low-quality scan to a clean version. Methods include conditional GANs like Pix2Pix and diffusion pipelines that start from a partly noised input.

Example. A photo of a living room is turned into the same layout in a different interior style while keeping the furniture positions.

Related: Pix2Pix, CycleGAN, Denoising strength

A metric for generated images that rewards each image for being confidently classified by a pretrained Inception network and rewards the whole set for covering many classes. Higher is better, but it ignores comparison with real data.

Example. A generator producing sharp images of all 10 CIFAR classes scores higher than one stuck on a single class.

Related: Fréchet Inception Distance, CLIP score, Mode collapse

Filling in a masked region of an image so that the new content blends with its surroundings, optionally guided by a text prompt. It is used to remove objects, repair damage or change part of a picture while keeping the rest untouched.

Example. A shopkeeper masks a stray signboard in a street photo and the model fills the gap with brick wall that matches the lighting.

Related: Outpainting, Image-to-image translation, Diffusion model

A multimodal design in which each modality is processed by its own model and their outputs or predictions are combined at the end, for example by averaging scores. It is simple and modular but captures less cross-modal interaction.

Example. A video classifier averages the predictions of an image model and an audio model.

Related: Early fusion, Multimodal model, Cross-modal retrieval

A diffusion model that runs its noising and denoising in the compact latent space of a pretrained autoencoder instead of on raw pixels. Working on much smaller tensors cuts memory and compute sharply while keeping detail, which made high-resolution image generation practical.

Example. A 512 x 512 x 3 image is compressed to a 64 x 64 x 4 latent, about 48 times fewer values, and the denoiser only ever sees the latent.

Related: Stable Diffusion, Variational autoencoder, Latent space

The compressed, learned coordinate system where a model represents data as vectors. Points near each other correspond to similar content, so moving through it, or blending two points, produces smooth changes in the decoded output.

Example. Averaging the latent vectors of two face photos and decoding gives a face that visibly blends both.

Related: Variational autoencoder, Latent diffusion model, Disentangled representation

An open vision-language model that connects a pretrained vision encoder to a large language model through a projector and is fine-tuned on image-based instruction data. It popularised a simple recipe for chat about images.

Example. A researcher fine-tunes LLaVA on a few thousand annotated X-ray images to answer questions in a specific domain.

Related: Visual instruction tuning, Projector layer, Vision-language model

Adapting a diffusion model to a new style, character or product by training small low-rank weight updates on top of frozen weights. The resulting files are a few megabytes, can be swapped or combined at generation time, and need far less GPU memory than full training.

Example. A brand trains an adapter on 30 product photos so that its prompts produce the company's bottle design consistently.

Related: LoRA, Stable Diffusion, DreamBooth

A subjective quality rating of synthesised speech or audio, obtained by asking listeners to rate samples on a 1 to 5 scale and averaging. It is the standard human evaluation for TTS naturalness.

Example. 20 listeners rate a TTS sample and give an average of 4.2, close to the 4.5 they give real recordings.

Related: Text-to-speech, Word error rate, Neural vocoder

A spectrogram whose frequency axis is warped to the mel scale, which spaces frequencies the way human hearing perceives pitch, giving finer detail at low frequencies. It is the usual input for speech recognition and the usual target for acoustic models in TTS.

Example. Whisper-style models convert 16 kHz audio into an 80-channel mel spectrogram before processing.

Related: Spectrogram, Mel scale, MFCC, Neural vocoder

Mel-frequency cepstral coefficients, a compact set of numbers describing the spectral shape of a short audio frame, computed from a mel filter bank, a logarithm and a discrete cosine transform. They were the standard speech feature before deep learning.

Example. Each 25 ms frame of speech is summarised as 13 MFCC values, which a classic recogniser used to tell vowels apart.

Related: Mel spectrogram, Mel scale, Automatic speech recognition

A GAN failure where the generator produces only a few kinds of outputs, or a single one, that fool the discriminator, instead of covering the full variety of the training data. It lowers diversity even when individual samples look good.

Example. A digit GAN that outputs only convincing 1s and 7s, however different the noise input, has collapsed.

Related: Generative adversarial network, Wasserstein GAN, Discriminator

A vector representation in which text, images, audio or other inputs are mapped into one shared space so that semantically related items are close regardless of modality. It enables cross-modal search and zero-shot classification.

Example. The embedding of a photo of a tiger lies near the embedding of the text 'a tiger', but far from 'a teacup'.

Related: Cross-modal retrieval, Contrastive language-image pretraining, Image embedding

A model that processes or produces more than one type of data, such as text, images, audio and video, within a single system. It can answer questions about a photo or transcribe and reason about speech.

Example. A user uploads a photo of a handwritten Hindi note and asks the model to translate it into English.

Related: Vision-language model, Early fusion, Late fusion, Omni model

A text description of what the generator should avoid. In diffusion pipelines using guidance, the negative prompt replaces the empty unconditional input, so sampling is pushed away from those concepts, helping remove unwanted artefacts or styles.

Example. Adding 'blurry, extra fingers, watermark' as a negative prompt reduces malformed hands and stray logos.

Related: Classifier-free guidance, Guidance scale, Text-to-image generation

A learned compressor that encodes audio into a few discrete tokens per frame and decodes them back to sound. Its tokens let language-model-style networks generate speech and music. This compact discrete representation is what lets language-model-style networks generate speech and music efficiently.

Example. A codec turns 1 second of audio into 75 frames of 8 tokens each, and a transformer can predict those tokens.

Related: Residual vector quantisation, Codebook, Audio language model

A representation that trains a small neural network to output colour and density for any 3-D position and viewing direction, so new views of a scene can be rendered from a set of photos with known camera positions.

Example. With 60 photos of a temple statue, a radiance field renders it from viewpoints no photographer captured.

Related: 3D Gaussian splatting, Volume rendering, Text-to-3D generation

A neural network that converts an intermediate acoustic representation, usually a mel spectrogram, into a raw audio waveform. Its quality determines how natural and free of buzzing the final synthesised voice sounds.

Example. A TTS pipeline predicts a mel spectrogram from text, and a vocoder turns that into a 22,050 Hz waveform.

Related: Mel spectrogram, WaveNet, HiFi-GAN, Text-to-speech

The rule that sets how much noise is added at each step of the forward diffusion process. Common choices are linear and cosine schedules. The schedule controls how quickly the signal is destroyed and strongly affects training stability and the quality of generated images.

Example. A cosine schedule keeps more of the image signal in early steps than a linear one, which helped improve results at low resolutions.

Related: Forward diffusion process, Diffusion model, Signal-to-noise ratio

A generative model built from a chain of invertible transformations that turn a simple distribution such as a Gaussian into a complex data distribution. Because each step is reversible with a known determinant, it gives exact likelihoods and exact sampling.

Example. A flow of 16 invertible layers maps Gaussian noise to a plausible face and can map any face back to noise.

Related: Variational autoencoder, Energy-based model, Generative adversarial network

A failure of vision-language models in which the answer mentions objects, attributes or text that are not in the image. It arises from strong language priors overriding visual evidence and is measured with benchmarks such as POPE.

Example. Shown an empty desk, a model describes 'a laptop and a coffee mug' because desks usually have them.

Related: Vision-language model, Image captioning, Hallucination

A single model that accepts and generates several modalities such as text, images, audio and sometimes video, often with real-time voice interaction. It removes the need to chain separate recognition, language and synthesis models.

Example. A user speaks a question while showing their camera view, and the model answers aloud in a natural voice.

Related: Multimodal model, Early fusion, Realtime speech interaction

Extending an image beyond its original borders by generating new content that continues the scene consistently. The existing picture is kept as context and the model synthesises the missing surroundings, often used to change aspect ratios.

Example. A square 1080 x 1080 product shot is extended sideways to a 1920 x 1080 banner by generating matching background.

Related: Inpainting, Image-to-image translation, Diffusion model

A module that compresses a variable number of visual features into a fixed small set of tokens using cross-attention with learned latent queries, keeping the language model's input short. It is used in Flamingo-style models.

Example. Frames from a video produce thousands of patch features, which the resampler reduces to 64 visual tokens.

Related: Q-Former, Flamingo, Cross-attention conditioning

The smallest unit of sound in a language that can change a word's meaning. Speech systems often model phonemes instead of letters because pronunciation is more regular there. Speech models often work with phonemes rather than letters because pronunciation is more regular there, which helps both recognition and synthesis.

Example. The words 'pat' and 'bat' differ by one phoneme, /p/ versus /b/.

Related: Grapheme-to-phoneme, Acoustic model, Automatic speech recognition

A conditional GAN for paired image-to-image translation that learns a mapping from an input image to a target image, using an adversarial loss plus a pixel-wise reconstruction loss. It needs aligned training pairs.

Example. Trained on pairs of building outlines and photos, Pix2Pix turns a new outline into a realistic facade.

Related: Conditional GAN, CycleGAN, Image-to-image translation

The small connector, often a linear layer or a two-layer MLP, that maps vision encoder features into the language model's embedding space so they can be read like word tokens.

Example. A projector converts 1024-dimensional patch features into 4096-dimensional vectors matching the language model's token embeddings.

Related: Vision encoder, LLaVA, Vision-language model

The rhythm, stress, pitch and intonation of speech that carry emphasis, emotion and sentence type beyond the words themselves. Good TTS must model it or the voice sounds flat and robotic.

Example. Raising pitch at the end of 'You are coming?' marks it as a question, while the same words with falling pitch are a statement.

Related: Text-to-speech, Voice cloning, Speech emotion recognition

Peak signal-to-noise ratio, a measure of how close a reconstructed image is to a reference, expressed in decibels from the mean squared error. Higher is better, though it correlates only loosely with how good the image looks.

Example. PSNR = 10 * log10(MAX^2 / MSE); for 8-bit images MAX = 255, and an MSE of 25 gives about 34.2 dB.

Related: SSIM, LPIPS, Super-resolution

A lightweight transformer introduced with BLIP-2 that uses a fixed set of learnable query vectors to pull the most useful information out of a frozen image encoder and pass it to a language model.

Example. 32 learnable queries condense hundreds of patch features into 32 vectors for the language model.

Related: BLIP, Perceiver resampler, Vision encoder

Voice conversation with an AI system in which audio is streamed both ways with low delay, allowing interruption and natural turn-taking. It is implemented either by chaining recognition, language and synthesis or by a native speech model.

Example. A customer interrupts a voice agent mid-sentence and it stops speaking and responds to the new question.

Related: Omni model, Streaming ASR, Voice activity detection

A method that rewrites sampling from a learned Gaussian as a deterministic function of its mean and standard deviation plus independent noise, so gradients can flow through the sampling step. It makes VAEs trainable with ordinary backpropagation.

Example. z = mu + sigma * eps with eps drawn from a standard normal lets the loss gradient reach mu and sigma.

Related: Variational autoencoder, Evidence lower bound, Backpropagation

A compression method that applies several codebooks in sequence, each one encoding the error left by the previous one. More codebooks give higher fidelity, and the stack of indices per frame forms the tokens used by neural audio codecs.

Example. A codec uses 8 codebooks per frame: the first captures the coarse sound and later ones add fine detail, so dropping the last few lowers bitrate.

Related: Codebook, Neural audio codec, VQ-VAE

The learned half of a diffusion model that starts from pure noise and removes noise step by step to recover a clean sample. A neural network predicts the noise (or the clean data) at each step, and repeating this many times turns random static into a coherent output.

Example. To generate one image, the network is called 25 to 50 times, each call slightly sharpening the picture from noise toward a plausible photograph.

Related: Forward diffusion process, Denoising diffusion probabilistic model, Diffusion sampler

A sequence model for streaming speech recognition that combines an audio encoder, a prediction network over previous outputs and a joint network. It can emit tokens as audio arrives, so it suits on-device and live captioning.

Example. A phone keyboard's voice typing uses a transducer to show words as you speak rather than after you finish.

Related: Connectionist temporal classification, Streaming ASR, Automatic speech recognition

The number of audio measurements taken per second when sound is digitised, in hertz. By the Nyquist rule it must be at least twice the highest frequency to be captured, so speech systems typically use 16 kHz and music 44.1 or 48 kHz.

Example. Phone-call audio at 8 kHz holds frequencies up to 4 kHz, while 16 kHz audio keeps clearer consonants for speech recognition.

Related: Spectrogram, Short-time Fourier transform, Automatic speech recognition

A technique that uses a frozen text-to-image diffusion model as a critic to optimise another representation, such as a 3-D scene. Gradients from the diffusion model's noise prediction push renderings toward images that fit the prompt.

Example. Each optimisation step renders the 3-D model from a random angle, adds noise, and uses the diffusion model's prediction to update the 3-D parameters.

Related: Text-to-3D generation, Neural radiance field, Diffusion model

A generative approach that learns the score, the gradient of the log probability density of the data, and generates samples by following this gradient from noise with Langevin-style dynamics. It is mathematically linked to diffusion models, which can be seen as a discretised version of the same idea.

Example. A network estimates which direction makes a noisy pixel grid more like real photographs, and sampling repeatedly nudges the grid in that direction.

Related: Score matching, Diffusion model, Energy-based model

A method that cuts audio into short overlapping windows and applies a Fourier transform to each, giving the frequency content over time. It is the step that produces spectrograms from raw waveforms.

Example. With a 25 ms window and 10 ms hop, one second of audio yields about 100 frames of frequency values.

Related: Spectrogram, Mel spectrogram, Sample rate

A CLIP-style image-text model that trains with a pairwise sigmoid loss, treating every image-text pair as an independent yes/no decision instead of using a softmax over the whole batch. It scales more easily and is a common vision encoder in vision-language models.

Example. A vision-language model uses a SigLIP encoder to turn each input image into patch features for the language model.

Related: Contrastive language-image pretraining, Vision-language model, Multimodal embedding

The task of splitting an audio recording into segments and labelling who spoke when, without necessarily knowing the speakers' identities. It is combined with transcription to produce speaker-labelled meeting notes.

Example. A recorded team call is tagged as Speaker 1, Speaker 2 and Speaker 3, each with time ranges.

Related: Speaker embedding, Voice activity detection, Automatic speech recognition

A fixed-length vector that captures the characteristics of a person's voice, so recordings by the same speaker lie close together. Embeddings such as x-vectors serve speaker verification, diarisation and voice-cloning conditioning.

Example. Two clips of the same person give embeddings with cosine similarity 0.85, while clips of different people give about 0.2.

Related: Speaker diarisation, Voice cloning, Cosine similarity

A picture of sound that shows how energy at each frequency changes over time, with time on one axis and frequency on the other. It is the standard input to speech and audio models because it exposes patterns that raw waveforms hide.

Example. In a spectrogram of the word 'namaste', the vowel sounds appear as bright horizontal bands at particular frequencies.

Related: Short-time Fourier transform, Mel spectrogram, Automatic speech recognition

Translating spoken language in one language directly into spoken output in another, either by chaining recognition, translation and synthesis or with a single model. Direct models can preserve the speaker's voice and reduce delay.

Example. A Hindi speaker talks and a listener hears a Kannada rendering within a couple of seconds.

Related: Automatic speech recognition, Text-to-speech, Code-switching

Structural Similarity Index, an image-quality measure that compares local luminance, contrast and structure between two images. It ranges up to 1, where 1 means identical, and matches perceived quality better than raw pixel error.

Example. A compressed photo with SSIM 0.97 against the original is visually near-identical, while 0.70 shows clear degradation.

Related: PSNR, LPIPS, Super-resolution

A family of open-weights text-to-image models built on latent diffusion. It combines a VAE that compresses images, a U-Net or transformer that denoises in the compressed space, and a text encoder for conditioning, and runs on consumer GPUs, which spawned a large community of fine-tunes.

Example. A designer runs Stable Diffusion locally on a 12 GB GPU to generate product mock-ups without sending data to a cloud service.

Related: Latent diffusion model, Text-to-image generation, LoRA fine-tuning for diffusion, ControlNet

Speech recognition that outputs text while the person is still speaking, using limited look-ahead so latency stays low. It trades some accuracy for responsiveness compared with processing a complete recording.

Example. Live subtitles on a cricket commentary stream appear within about half a second of the speaker's words.

Related: RNN-Transducer, Automatic speech recognition, Voice activity detection

A family of GAN architectures that maps a latent vector through a mapping network and injects the resulting style at each layer of the generator. This gives fine control over coarse and fine attributes and produced highly realistic faces.

Example. Swapping the style vector at coarse layers changes head pose, while changing it at fine layers changes hair colour or skin texture.

Related: Generative adversarial network, Latent space, Disentangled representation

The task of increasing the resolution of an image or video while recovering plausible fine detail. Methods range from interpolation to CNNs, GANs and diffusion models that hallucinate realistic texture consistent with the low-resolution input.

Example. A 480p cricket clip is upscaled to 1080p, with the model synthesising sharper jersey text and grass texture.

Related: Generative adversarial network, Diffusion model, PSNR

A family of sequence-to-sequence text-to-speech models that use attention to predict mel spectrograms directly from characters or phonemes, which are then converted to audio by a vocoder. It showed that end-to-end TTS could be natural.

Example. Given the letters of a sentence, Tacotron outputs mel spectrogram frames that a vocoder turns into speech.

Related: Text-to-speech, Neural vocoder, FastSpeech

The property that objects, colours, identities and lighting remain stable from one video frame to the next. Generators lacking it produce flicker, morphing faces or objects that appear and vanish.

Example. In a generated clip of a cricketer, temporal consistency means his shirt number stays the same in all 120 frames.

Related: Video generation, Text-to-video generation, Optical flow

Producing a 3-D asset from a text prompt, often by optimising a 3-D representation so that its renderings look plausible to a pretrained text-to-image diffusion model, or by generating multiple consistent views first.

Example. The prompt 'a carved wooden elephant' yields a mesh that can be imported into a game engine.

Related: Score distillation sampling, Neural radiance field, Text-to-image generation

The task of producing an image from a written description. Modern systems encode the text into vectors and use them to condition a generative model, usually a diffusion model, so the result matches objects, style and layout described in the prompt.

Example. Typing 'a Kerala houseboat at sunrise, watercolour' returns four candidate paintings within seconds.

Related: Diffusion model, Cross-attention conditioning, Stable Diffusion, Negative prompt

Technology that turns written text into natural-sounding spoken audio. Current systems predict an acoustic representation such as a mel spectrogram from text, then convert it to a waveform with a neural vocoder, or generate audio tokens directly.

Example. A banking app reads account alerts aloud in Tamil using a synthetic voice.

Related: Neural vocoder, Prosody, Tacotron, Automatic speech recognition

Creating a short video clip from a text prompt, usually with diffusion models that extend image generation with a time dimension. The hard part is keeping objects, lighting and motion consistent across frames.

Example. The prompt 'auto-rickshaw driving through Bengaluru rain' yields a 5-second clip at 24 frames per second.

Related: Temporal consistency, Diffusion transformer, Video generation

A personalisation technique that learns a new word embedding representing a concept from a handful of example images, without changing the generator's weights. The learned token can then be used inside ordinary prompts.

Example. Learning a token for a particular illustration style from 5 images lets users write 'a temple in the style of <my-style>'.

Related: DreamBooth, Text-to-image generation, LoRA fine-tuning for diffusion

A generative model that pairs an encoder, which maps data to a probability distribution over a latent space, with a decoder that reconstructs data from samples of it. A regularising term keeps that space smooth, so sampling random latent points yields new, plausible outputs.

Example. A VAE trained on handwritten digits can decode a random 2-D latent point into a new, believable digit and interpolate smoothly between 3 and 8.

Related: Evidence lower bound, Reparameterisation trick, Latent space, Autoencoder

The broad task of synthesising video, from text, an image or another video, by modelling both appearance and motion. Approaches compress video into spatio-temporal latents and denoise them with diffusion transformers.

Example. An app animates a single product photo into a 4-second rotating shot for an online store.

Related: Text-to-video generation, Temporal consistency, Diffusion transformer

The image-processing component of a vision-language model that converts a picture into a sequence of feature vectors. It is commonly a vision transformer pretrained with a contrastive objective and often kept frozen.

Example. A 336 x 336 image passes through the encoder and emerges as 576 patch feature vectors.

Related: Vision transformer, Vision-language model, Projector layer

A model that takes images together with text and produces text, for tasks like captioning, visual question answering and document reading. It usually links a vision encoder to a language model through a connector layer.

Example. A shopkeeper photographs a shelf and asks, 'Which products are almost out of stock?', and the model lists them.

Related: Multimodal model, Image captioning, Visual question answering, LLaVA

Linking words or phrases in text to specific regions in an image, usually by outputting bounding boxes or masks. It lets a model point at 'the person in the blue kurta' instead of just describing the scene.

Example. Asked to find 'the red bicycle', the model returns box coordinates around that bicycle.

Related: Visual question answering, Vision-language model, Object detection

Fine-tuning a vision-language model on images paired with instructions and responses, so it follows user requests about pictures. The data are often generated with a strong language model to describe image content.

Example. Training examples show a chart image, the instruction 'Summarise the trend', and a written answer.

Related: LLaVA, Instruction tuning, Vision-language model

A task in which a model answers a free-form natural-language question about an image. It needs both perception of the picture and reasoning about the question. It is a common benchmark for measuring how well a vision-language model links perception with reasoning.

Example. Shown a photo of a train platform, the model is asked 'How many people are waiting?' and answers 'Five.'

Related: Vision-language model, Image captioning, Visual grounding

An end-to-end TTS model that combines a VAE, normalising flows and adversarial training to go from text directly to waveform in one network, without a separate vocoder stage. It is used in many open voice projects.

Example. A single VITS checkpoint turns phonemes into 22 kHz audio in one forward pass.

Related: Text-to-speech, Variational autoencoder, HiFi-GAN

Detecting which parts of an audio stream contain human speech and which are silence or background noise. It saves compute, trims recordings and decides when a user has stopped talking in voice assistants.

Example. A voice bot waits until the detector reports 700 ms of silence before treating the customer's turn as finished.

Related: Streaming ASR, Speaker diarisation, Automatic speech recognition

Generating speech that imitates a particular person's voice from a short sample or a recording, using a TTS model conditioned on that person's speaker representation. It has uses in dubbing and accessibility and risks in fraud.

Example. With 30 seconds of a narrator's audio, a system reads a new script in a similar voice, ideally with the narrator's consent.

Related: Speaker embedding, Zero-shot TTS, Deepfake

A VAE variant whose latent codes are discrete: the encoder output is snapped to the nearest entry in a learned codebook of vectors. This gives images or audio a token-like representation that autoregressive models can predict, and avoids posterior collapse.

Example. A 256 x 256 image becomes a 32 x 32 grid of indices into a codebook of 8192 vectors, which a transformer can then model as a sequence.

Related: Codebook, Image tokenizer, Variational autoencoder

A GAN variant that replaces the usual loss with an estimate of the Wasserstein (earth mover's) distance between real and generated distributions, using a critic constrained to be smooth. It gives more stable training and a loss that tracks sample quality.

Example. Training a WGAN, the critic's loss falls steadily as images improve, unlike the noisy loss of a standard GAN.

Related: Generative adversarial network, Mode collapse, Discriminator

Embedding a hidden but detectable signal in AI-generated images, audio or text so the origin can later be verified. Watermarks can survive some edits, but crops, re-encoding and adversarial changes can weaken them.

Example. An image generator adds an invisible pattern to every output, and a detector later flags a downloaded picture as machine-made.

Related: Content provenance, Deepfake, Synthetic media

A self-supervised speech model that learns representations from unlabelled audio by masking parts of the latent features and predicting them contrastively. Fine-tuned on a small amount of transcribed data it reaches strong recognition accuracy, which helps low-resource languages.

Example. Pretrained on thousands of hours of untranscribed audio, then fine-tuned on 10 hours of labelled Odia speech.

Related: Self-supervised learning, Automatic speech recognition, Connectionist temporal classification

A deep generative model from DeepMind that produces audio one sample at a time using stacked dilated causal convolutions. It set a new bar for natural synthetic speech, though sample-by-sample generation is slow.

Example. To generate one second at 16 kHz, WaveNet makes 16,000 sequential predictions.

Related: Neural vocoder, HiFi-GAN, Text-to-speech

An open-source speech recognition model from OpenAI trained on a large multilingual collection of audio with transcripts. It transcribes many languages, translates speech to English, and is widely used as a strong general-purpose baseline.

Example. A developer runs Whisper on a recorded interview in Hindi and gets a timestamped transcript.

Related: Automatic speech recognition, Word error rate, Mel spectrogram

Speech synthesis that imitates a voice never seen in training, using only a short reference clip supplied at run time and no fine-tuning. It generally relies on large models trained on many speakers.

Example. A 5-second recording of a new speaker is passed as a prompt and the model reads a fresh sentence in that voice.

Related: Voice cloning, Text-to-speech, Speaker embedding