Nlp embeddings

A language that forms long words by attaching many suffixes in a row, each carrying one meaning. This produces huge numbers of word forms, so word-level vocabularies fail and subword methods work much better. Tamil, Telugu, Kannada and Turkish are examples.

Example. A single Telugu word can combine a root with tense, person and case endings, so listing every form as its own vocabulary entry is impractical.

Related: Morphology, Subword tokenisation, Out-of-vocabulary word

A strategy for language models that falls back to a shorter context when the longer one has no counts, for example using a bigram when a trigram was never observed. Discounting keeps the final probabilities summing to one.

Example. If the trigram "very old chai" was never seen, the model uses the bigram "old chai" instead.

Related: Smoothing, N-gram language model, Trigram

A model that encodes two texts, such as a query and a document, independently into vectors and compares them afterwards. Document vectors can be pre-computed and indexed, so search is fast, though accuracy is often lower than a cross-encoder.

Example. A support bot encodes 1 lakh help articles ahead of time and only encodes each new question at query time.

Related: Cross-encoder, Sentence embedding, Semantic similarity

A sequence-labelling architecture combining a bidirectional LSTM, which reads text left-to-right and right-to-left to build a vector for each token, with a CRF layer that picks the best overall label sequence. It was a standard high-performing design for entity recognition before transformers.

Example. A BiLSTM-CRF tagger reads "Rahul joined Wipro" and outputs B-PER O B-ORG.

Related: Conditional random field, Named entity recognition, Sequence labelling

An n-gram of length two: a pair of neighbouring words or characters. Bigrams capture simple phrases and the probability of a word given the previous one, at the cost of many more distinct features than unigrams.

Example. The bigrams of "machine learning course" are "machine learning" and "learning course".

Related: N-gram, Trigram, N-gram language model

An extended span-labelling scheme with Beginning, Inside, Last, Outside and Unit tags, where U marks a one-token entity. The extra tags give clearer boundary information and often help sequence models slightly.

Example. "Mumbai" alone is U-LOC, while "New Delhi" is B-LOC L-LOC.

Related: BIO tagging, Named entity recognition, Sequence labelling

A labelling format for marking spans in text: B marks the Beginning of an entity, I marks tokens Inside it and O marks tokens Outside any entity. It lets a per-token classifier represent multi-word entities and their boundaries.

Example. "State Bank of India" is tagged B-ORG I-ORG I-ORG I-ORG, and other words get O.

Related: BILOU tagging, Named entity recognition, Sequence labelling

A ranking function for keyword search that scores how well a document matches a query. It builds on term frequency and inverse document frequency, but limits the gain from repeated terms and adjusts for document length. It remains a strong baseline for search.

Example. Searching "solar subsidy India" ranks a short government page that repeats these words above a long article mentioning them once.

Related: TF-IDF, Inverse document frequency, Term frequency

A variant of byte-pair encoding that works on the raw bytes of UTF-8 text instead of characters, so the starting vocabulary is only 256 byte values. Any string in any script can be encoded without an unknown token, at the cost of longer sequences for scripts that need several bytes per character.

Example. A Devanagari letter takes three bytes in UTF-8, so an unusual Hindi word may be split into many byte-level tokens if no merges cover it.

Related: Byte-pair encoding, Unicode, Tokenizer fertility

A subword method that starts with single characters and repeatedly merges the most frequent adjacent pair of symbols in the training corpus into a new symbol, until a chosen vocabulary size is reached. At use time, the learned merges are applied in order to split new text.

Example. If "l" and "o" appear together most often, the first merge creates "lo"; later merges may build "low" and "lower".

Related: Subword tokenisation, Vocabulary size, Byte-level BPE

Converting all letters to a single case, usually lower case, so that "Apple", "APPLE" and "apple" count as the same word. It shrinks the vocabulary but can erase useful signals, such as distinguishing a company name from a fruit.

Example. Folding "Delhi" and "delhi" gives one entry, but "US" and "us" also collapse into one.

Related: Text normalisation, Vocabulary, Named entity recognition

Treating every single character as a token. The vocabulary is tiny and no word is ever unknown, but sequences become very long and each token carries little meaning, so models need more computation to learn word-level patterns.

Example. "chai" becomes ["c", "h", "a", "i"], four tokens instead of one.

Related: Tokenisation, Word tokenisation, Subword tokenisation

Grouping consecutive words into flat phrases such as noun phrases or verb phrases, without building a full sentence tree. It sits between part-of-speech tagging and full parsing and is often done with BIO tags.

Example. "The old bridge" is grouped as one noun phrase (NP) in "The old bridge collapsed".

Related: Part-of-speech tagging, BIO tagging, Dependency parsing

A table counting how often each pair of words appears near each other, within a window of a few words or in the same document. Rows can act as simple word vectors and are the raw material for methods such as GloVe.

Example. If "chai" appears within two words of "hot" 340 times, the cell (chai, hot) contains 340.

Related: Distributional hypothesis, GloVe, Pointwise mutual information

Mixing words or phrases from two or more languages within one sentence or conversation. Common in Indian social media and chat, it breaks tools built for a single language and is a known challenge for tokenisation and classification.

Example. "Kal meeting cancel ho gayi, so let's catch up later" mixes Hindi and English.

Related: Transliteration, Tokenisation, Multilingual embedding

A combination of words that occurs together more often than chance would predict and often has a fixed meaning. Statistical measures such as pointwise mutual information help discover them from a corpus.

Example. "strong tea" is a collocation in English while "powerful tea" sounds wrong.

Related: Pointwise mutual information, N-gram, Bigram

A probabilistic model for labelling sequences that scores whole label sequences using features of the input and of neighbouring labels, then normalises over all sequences. It avoids impossible label combinations and is often placed on top of a neural encoder.

Example. A CRF layer learns that an I-PER tag cannot directly follow an O tag, and so prevents that invalid output.

Related: Sequence labelling, Viterbi algorithm, Hidden Markov model

A well-known benchmark dataset and shared task for named entity recognition in English and German newswire, labelling persons, organisations, locations and miscellaneous entities. It has long been a standard test for entity recognisers.

Example. A new tagger reports an entity-level F1 on the CoNLL-2003 English test set to compare with earlier work.

Related: Named entity recognition, Entity-level F1, BIO tagging

Analysing a sentence as nested phrases, such as noun phrases inside verb phrases, forming a tree whose leaves are the words. It describes grammatical structure by grouping, as opposed to linking words directly.

Example. "The cat sat on the mat" splits into a noun phrase "The cat" and a verb phrase "sat on the mat".

Related: Dependency parsing, Chunking, Part-of-speech tagging

A word vector computed from the whole sentence, so the same word gets different vectors in different contexts. Deep sequence models produce them and they capture word senses far better than static vectors.

Example. "bat" gets one vector in "he swung the bat" and a clearly different one in "the bat flew out".

Related: Static embedding, ELMo, Sentence embedding

A Word2Vec architecture that averages the vectors of the surrounding context words and uses that average to predict the centre word. It trains faster than skip-gram and works well for frequent words.

Example. Given context [we, drink, chai], CBOW learns to predict the missing word "hot".

Related: Word2Vec, Skip-gram, Bag of words

Deciding which words or phrases in a text refer to the same real-world thing, such as linking a pronoun to the noun it stands for. It is needed to follow who is being discussed across sentences.

Example. In "Priya opened the shop. She sold sweets", "She" refers to Priya.

Related: Named entity recognition, Entity linking, Dependency parsing

A large, organised collection of text used to study language or to train and test language models. Corpora may be raw web pages, books or news, or carefully annotated with labels such as parts of speech. The quality and coverage of a corpus strongly shape what a model learns.

Example. A corpus of 2 million Hindi news articles is used to train a tokeniser and a word-embedding model.

Related: Vocabulary, Tokenisation, Word embedding

A model that reads two texts together as one input and outputs a single relevance or similarity score. It captures fine interactions between them and is usually more accurate than a bi-encoder, but must be rerun for every pair, so it is slow at large scale.

Example. A cross-encoder rescored the top 20 search results for a query to pick the best 3.

Related: Bi-encoder, Sentence-BERT, Semantic textual similarity

Training a model on labelled data in one language, often English, and applying it to other languages with little or no labelled data there. It works when a multilingual model shares representations across languages.

Example. A sentiment classifier fine-tuned on English reviews is applied directly to Marathi reviews.

Related: Multilingual embedding, Sentence embedding, Code-mixing

Analysing a sentence by linking each word to the word it depends on, labelling links such as subject or object. The result is a tree that exposes who did what to whom, useful for relation extraction and grammar-aware tools.

Example. In "Asha bought mangoes", "Asha" is the subject of "bought" and "mangoes" is its object.

Related: Constituency parsing, Part-of-speech tagging, Semantic role labelling

The idea, often summarised as "a word is known by the company it keeps", that words appearing in similar contexts have similar meanings. It is the principle behind almost every method that learns word vectors from raw text.

Example. "doctor" and "physician" both appear near words like hospital, patient and prescribe, so they get similar vectors.

Related: Word embedding, Co-occurrence matrix, Word2Vec

A table with one row per document and one column per term, whose cells hold counts or weights such as tf-idf. It turns a text collection into a numeric matrix that most standard machine-learning algorithms can use, though it is mostly zeros.

Example. With 10,000 reviews and 5,000 terms, the matrix has 10,000 rows and 5,000 columns; each review fills only a few dozen cells.

Related: Bag of words, TF-IDF, Sparse vector

Scoring two vectors by multiplying matching entries and summing them: a . b = a1 b1 + a2 b2 + ... . For vectors of unit length it equals cosine similarity, but for other vectors it also grows with their lengths.

Example. [1, 2, 3] . [4, 0, 1] = 4 + 0 + 3 = 7.

Related: Cosine similarity, Sentence embedding, Semantic similarity

A contextual embedding method that computes word vectors from the internal states of a bidirectional LSTM language model trained on large text. Its word representations change with the sentence, and it was an early step towards pretrained language models.

Example. ELMo gives "play" different vectors in "a play at the theatre" and "children play cricket".

Related: Contextual embedding, Static embedding, Language model

Connecting a mention of an entity in text to the matching entry in a knowledge base, such as an encyclopaedia page, resolving ambiguity by using the surrounding context. It turns a text span into a precise, unique identity that other systems can use.

Example. "Jaguar" in a car review is linked to the car brand, and in a wildlife article to the animal.

Related: Named entity recognition, Word sense disambiguation, Coreference resolution

An evaluation score for entity recognition in which a prediction counts as correct only if both the span boundaries and the type exactly match the gold entity. Precision and recall are computed over entities, not over tokens.

Example. Predicting "Bank of India" as ORG when the gold entity is "State Bank of India" counts as one error, not partial credit.

Related: Named entity recognition, BIO tagging, Sequence labelling

A word-embedding library that represents each word as the sum of vectors for its character n-grams. It can produce a vector for words never seen in training and copes well with spelling variants and rich word forms.

Example. FastText builds a vector for the unseen word "chaiwala" from pieces like "cha", "hai" and "iwa".

Related: Word embedding, Out-of-vocabulary word, Morphology

Finding strings that are close to, but not exactly the same as, a target, using measures such as edit distance or n-gram overlap. It helps match misspelt names, addresses and product titles to their intended records.

Example. "Bengaluru" and "Bangalore" are matched as the same city by a fuzzy score above a chosen threshold.

Related: Jaccard similarity, Text normalisation

A list of known names, such as cities, companies or drug names, used to help find entities in text. Matching against it gives useful features to entity recognisers but cannot alone handle new or ambiguous names.

Example. A gazetteer of 700 Indian district names helps flag "Nashik" as a location.

Related: Named entity recognition, Named entity, Entity linking

A word-embedding method that trains vectors so that the dot product of two word vectors approximates the logarithm of how often the words co-occur across the whole corpus. It combines global count statistics with the vector-learning style of Word2Vec.

Example. GloVe vectors trained on a web corpus are downloaded and used to initialise the embedding layer of a sentiment classifier.

Related: Co-occurrence matrix, Word2Vec, Word embedding

Choosing negative examples that look similar to the positive but are wrong, instead of random ones. These challenging cases force the model to learn finer distinctions and usually improve embedding quality.

Example. For the question "capital of Karnataka", a passage about Karnataka's population is a hard negative, unlike a random passage about cricket.

Related: Contrastive learning, Triplet loss, In-batch negatives

A probabilistic model of a sequence in which hidden states, such as part-of-speech tags, follow a Markov chain and each state emits an observed item, such as a word. It is fitted from counts and decoded with the Viterbi algorithm.

Example. In a POS tagger, hidden states are tags, and P(noun to verb) and P(word | tag) are the model's probabilities.

Related: Viterbi algorithm, Part-of-speech tagging, Markov assumption

A way to compute output probabilities over a large vocabulary using a binary tree whose leaves are words, so a word's probability is a product of about log(V) binary decisions instead of a sum over V words. It was used to speed up Word2Vec training.

Example. With 1,000,000 words, about 20 binary decisions replace one million-way normalisation.

Related: Word2Vec, Negative sampling, Skip-gram

A cheap way to get negative examples in contrastive training: for each pair in a batch, treat the positives of all the other pairs in the same batch as its negatives. Larger batches give more negatives per pair at no extra encoding cost.

Example. A batch of 64 question-answer pairs gives each question 63 negative answers for free.

Related: InfoNCE loss, Contrastive learning, Hard negative mining

A contrastive loss that, for each anchor, treats the true positive as the correct class among many candidates and applies softmax cross-entropy over their similarity scores divided by a temperature. Lower loss means the positive scores higher than all negatives.

Example. With one positive and 7 negatives per query, the loss is the cross-entropy of picking the positive among 8 candidates.

Related: Contrastive learning, In-batch negatives, Cosine similarity

Assigning a user's message to one of a fixed set of goals, such as check_balance or cancel_order, so a system knows what action to take. It is a text classification task usually paired with slot filling.

Example. "Where is my parcel?" is classified as the intent track_order.

Related: Text classification, Slot filling, Natural language understanding

A measure of how rare a term is across a collection of documents: terms found in few documents get high values and terms found in nearly all get low values. A common form is idf = log(N / df), where N is the number of documents and df is how many contain the term.

Example. With N = 1000 documents and "the" in 990 of them, idf is tiny; "mitochondria" in 5 gives log(1000 / 5), a much larger weight.

Related: Term frequency, TF-IDF, BM25

A strong smoothing method for n-gram models that estimates a word's lower-order probability by how many different contexts it follows, not by its raw frequency. This stops words that are common only in fixed phrases from being over-predicted.

Example. "Kong" is frequent but almost only follows "Hong", so Kneser-Ney gives it a low chance after other words.

Related: Smoothing, Backoff, N-gram language model

A benchmark that evaluates text embedding models across many task types, such as retrieval, classification, clustering and similarity, in many datasets and some languages. It is widely used to compare embedding models and choose one for a job.

Example. A team compares two embedding models by their MTEB retrieval scores before picking one for a document search feature.

Related: Sentence embedding, Semantic textual similarity, Text classification

A training method that makes the first k dimensions of an embedding useful on their own, for several values of k. Vectors can then be cut to a shorter length to save storage and speed search, with only a modest loss in quality.

Example. A 1024-number embedding can be cut to its first 256 numbers for a cheaper index.

Related: Embedding dimension, Sentence embedding, Contrastive learning

Producing one vector for a text by averaging the vectors of all its tokens, usually ignoring padding positions. It is a simple, common way to turn token-level outputs into a sentence embedding.

Example. Averaging three token vectors [1, 2], [3, 4], [5, 6] gives [3, 4].

Related: Sentence embedding, Padding token, Word embedding

The study of how words are built from smaller meaningful parts such as roots, prefixes and suffixes. In NLP it guides splitting and analysing words, and it matters most in languages where a single word can carry the meaning of a whole English phrase.

Example. "unhelpfulness" contains the prefix un-, root help, and suffixes -ful and -ness.

Related: Stemming, Lemmatization, Agglutinative language

An embedding model trained so that texts with the same meaning in different languages have nearby vectors. It allows searching or comparing across languages without translating first, and it lets one system serve users writing in many languages.

Example. A Hindi query "मौसम कैसा है" retrieves an English page titled "Weather forecast".

Related: Sentence embedding, Cross-lingual transfer, Semantic similarity

A contiguous run of n items, usually words or characters, taken from a text. Counting n-grams captures local word order and common phrases, and they serve as features for classifiers and as the building blocks of simple language models.

Example. The word 3-grams of "I like hot chai" are "I like hot" and "like hot chai".

Related: Unigram, Bigram, N-gram language model

A language model that estimates the probability of the next word from counts of the previous n-1 words in a corpus. It is fast and easy to build, but cannot use long-range context and struggles with word sequences it has not seen.

Example. With counts "good morning" = 90 and "good" = 300, a bigram model gives P(morning | good) = 90 / 300 = 0.3.

Related: Markov assumption, Smoothing, Language model

A word or phrase that refers to a specific real thing, like a named person, company, place, date, monetary value or product, as opposed to a general noun. Common category sets include PER, ORG, LOC and MISC.

Example. "Tata Motors" is a named entity of type organisation, while "company" is not.

Related: Named entity recognition, Entity linking, Gazetteer

The task of finding spans of text that name real-world things and labelling their type, such as person, organisation, location, date or amount. It turns free text into structured data and is usually solved as sequence labelling.

Example. In "Infosys hired 200 people in Pune on 3 March", it tags Infosys as ORG, Pune as LOC and 3 March as DATE.

Related: Named entity, Sequence labelling, BIO tagging

The field of computer science that builds programs able to read, interpret, search, classify or generate human language in text or speech form. It covers tasks from splitting sentences into words to answering questions, and it is the foundation on which chatbots, search engines and translation tools are built.

Example. A bank uses NLP to read customer emails and route each one to the loans, cards or complaints team automatically.

Related: Text classification, Tokenisation, Language model

The part of language processing concerned with extracting meaning from text, for example working out what a user wants and which details they mention. In assistants it often means intent classification plus slot extraction.

Example. From "Book a train to Pune tomorrow", it finds the intent book_train and the slots destination=Pune and date=tomorrow.

Related: Text classification, Named entity recognition, Natural language processing

A training shortcut for Word2Vec that, for each true word-context pair, also draws a few random unrelated words as negative examples, and trains the model to tell real pairs from fake ones. It avoids computing a softmax over the entire vocabulary.

Example. For each real pair (hot, chai), 5 random words such as "bridge" or "tax" are used as negatives.

Related: Skip-gram, Contrastive learning

An open-source Python library and teaching toolkit for language processing, with tools for tokenising, stemming, tagging and parsing, and access to many corpora. It is widely used for learning and for prototyping.

Example. nltk.word_tokenize("Chai is ready") returns ["Chai", "is", "ready"].

Related: Tokenisation, Stemming, Corpus

A vector as long as the vocabulary with a single 1 at the word's position and 0 everywhere else. It uniquely identifies a word but is huge, mostly zeros, and treats every pair of words as equally unrelated.

Example. In a 5-word vocabulary [cat, dog, car, bus, tea], "car" is [0, 0, 1, 0, 0].

Related: Word embedding, Sparse vector, Vocabulary

A word that does not appear in the tokeniser's or model's fixed vocabulary. Word-level systems typically replace it with a generic unknown token, losing its meaning; subword and byte-level systems avoid the problem by splitting it into known pieces.

Example. A word-level model trained on formal English sees "vadapav" for the first time and maps it to <unk>.

Related: Vocabulary, Word tokenisation, Subword tokenisation

A special filler token appended to shorter sequences so every sequence in a batch has the same length. An attention mask or similar signal tells the model to ignore these positions so they do not influence results.

Example. Batching sequences of lengths 5 and 8 pads the first with three <pad> tokens to reach length 8.

Related: Special tokens, Token ID, Truncation

Assigning each word its grammatical category, such as noun, verb, adjective or preposition, using the context of the sentence to settle ambiguous words. It supports parsing, lemmatisation and information extraction, and is a classic example of sequence labelling.

Example. In "Time flies", "Time" is tagged noun and "flies" verb; in "Time flies like an arrow" context helps settle the tags.

Related: Sequence labelling, Dependency parsing, Hidden Markov model

A score of how much more often two words occur together than they would if independent: PMI(x, y) = log( P(x, y) / (P(x) P(y)) ). Positive values mean association and negative mean avoidance. It is used for collocations and for weighting co-occurrence counts.

Example. If P(x, y) = 0.002 while P(x) = 0.01 and P(y) = 0.02, PMI = log(0.002 / 0.0002) = log 10, about 2.3 in natural log.

Related: Positive PMI, Collocation, Co-occurrence matrix

A variant of pointwise mutual information that replaces all negative values with zero: PPMI = max(PMI, 0). Negative values are unreliable with limited data, so this version is preferred when building word vectors from co-occurrence counts.

Example. A PMI of -1.2 becomes 0, while a PMI of 2.3 stays 2.3.

Related: Pointwise mutual information, Co-occurrence matrix, Word embedding

Finding and labelling relationships between entities in text, producing facts such as (person, works_at, company). It commonly follows named entity recognition and feeds knowledge graphs, which store such facts as connected records.

Example. From "Sundar Pichai leads Google", it extracts the triple (Sundar Pichai, CEO_of, Google).

Related: Named entity recognition, Semantic role labelling, Entity linking

Identifying, for each predicate in a sentence, the roles played by other phrases, such as who did the action, what was affected, and when or where. It captures meaning that stays the same across different sentence forms.

Example. In "Ravi sold the car to Meera", Ravi is the seller, the car is the thing sold and Meera is the recipient.

Related: Dependency parsing, Relation extraction, Sequence labelling

How close two pieces of text are in meaning, regardless of the exact words used. In practice it is estimated by comparing the embedding vectors of the two texts, often with cosine similarity, and it is the basis of semantic search and duplicate detection.

Example. "cheap flights to Goa" and "low-cost air tickets to Goa" have high semantic similarity though they share few words.

Related: Cosine similarity, Sentence embedding, Semantic textual similarity

A benchmark task in which systems score how similar in meaning two sentences are, and are compared with human ratings, typically using correlation. It is a standard way to evaluate sentence embeddings.

Example. A model gives 0.87 to a sentence pair humans rated 4.5 out of 5, and its scores are correlated with the human ratings across the dataset.

Related: Sentence embedding, Semantic similarity, Cross-encoder

A single dense vector that represents the meaning of a whole sentence or short passage, so texts with similar meaning have nearby vectors. It powers semantic search, clustering, duplicate detection and retrieval systems.

Example. "How do I reset my UPI PIN?" and "Forgot my UPI PIN, what now?" get vectors with cosine similarity around 0.9.

Related: Semantic similarity, Mean pooling, Bi-encoder

Dividing a text into individual sentences. It is harder than splitting at full stops because periods also appear in abbreviations, decimals and initials, so rule-based or learned methods are used to decide which ones truly end a sentence.

Example. In "Dr. Rao paid Rs. 4.5 lakh. He left." only two of the four full stops end a sentence.

Related: Tokenisation, Text normalisation, Corpus

An approach that fine-tunes a BERT-style model in a twin (siamese) setup so that pooled outputs of two sentences can be compared directly with cosine similarity. It makes semantic search over thousands of sentences fast, since each sentence is encoded only once.

Example. Encoding 100,000 FAQ answers once with SBERT lets a query be matched by one encoding and quick vector comparisons.

Related: Siamese network, Bi-encoder, Sentence embedding

A tokeniser library that learns subword units directly from raw text, treating the input as a stream of characters including spaces, so it needs no language-specific pre-splitting. It supports byte-pair encoding and unigram methods, which makes it convenient for languages that do not separate words with spaces.

Example. A single SentencePiece model can be trained on mixed Hindi, Tamil and English text without writing separate word-splitting rules.

Related: Subword tokenisation, Byte-pair encoding, Unigram language model tokeniser

Determining the attitude expressed in text, typically positive, negative or neutral, or a score along that range. It is applied to reviews, social posts and support messages to track how people feel.

Example. "Delivery was late but the food was great" may be labelled mixed or slightly positive.

Related: Text classification, TF-IDF, Sequence labelling

A task where a model assigns one label to every item in an input sequence, usually every token, while taking neighbouring items into account. Named entity recognition, part-of-speech tagging and chunking are typical examples.

Example. For the tokens [Ravi, lives, in, Delhi] the labels might be [PER, O, O, LOC].

Related: Named entity recognition, Part-of-speech tagging, BIO tagging

A neural architecture that passes two inputs through the same network with shared weights and then compares the outputs. It is trained so that similar pairs land close together and dissimilar pairs land far apart.

Example. Two job descriptions are encoded by the same network, and their vectors are compared to decide if they describe the same role.

Related: Sentence-BERT, Triplet loss, Contrastive learning

A Word2Vec architecture that takes one centre word and learns to predict the words around it within a window. It works well for rare words and small corpora, and its learned input weights become the word vectors.

Example. For "we drink hot chai" with centre word "hot", the model tries to predict "drink" and "chai".

Related: Word2Vec, Continuous bag of words, Negative sampling

Extracting the specific values needed to carry out a user's request, such as a city, date or amount, from their message. It is normally treated as sequence labelling with tags for each slot type.

Example. In "Transfer 5000 rupees to Anil", the amount slot is 5000 and the recipient slot is Anil.

Related: Intent classification, Sequence labelling, Named entity recognition

A family of techniques that move a little probability from seen word sequences to unseen ones, so that no valid sequence gets probability zero. Without it, a single unseen n-gram would make an entire sentence impossible under the model.

Example. After smoothing, a bigram never seen in training gets a small probability such as 0.0001 instead of exactly 0.

Related: Laplace smoothing, Backoff, N-gram language model

An open-source Python library for practical NLP, offering pretrained pipelines for tokenisation, tagging, parsing and named entity recognition, plus tools for training custom models. It is designed for fast, production-style text processing.

Example. nlp = spacy.load("en_core_web_sm") loads an English pipeline, and doc.ents lists the entities it found.

Related: Named entity recognition, Tokenisation, Part-of-speech tagging

Reserved vocabulary entries that carry structure rather than ordinary text, such as markers for padding, unknown words, start or end of a sequence, or a separator between two sentences. They are added by the tokeniser and tell the model how to read the input.

Example. A sentence pair may be encoded as [CLS] first sentence [SEP] second sentence [SEP], where [CLS] and [SEP] are special tokens.

Related: Padding token, Token ID, Vocabulary

Detecting misspelt words and suggesting the intended ones, usually by generating candidates a small number of edits away and choosing the one a language model finds most probable in context.

Example. "recieve" is corrected to "receive" because it is one swap away and far more frequent in the corpus.

Related: Fuzzy matching, Language model, N-gram language model

A word embedding that gives each word exactly one vector regardless of the sentence it appears in. It is cheap and easy to use, but cannot separate different senses of a word.

Example. "bank" has the same vector in "river bank" and "bank loan" under Word2Vec or GloVe.

Related: Contextual embedding, Word2Vec, Word sense disambiguation

A training trick that randomly samples different valid subword splits of the same text instead of always using the single best split. Showing the model several segmentations makes it more robust to spelling variation and noisy input.

Example. The word "unbelievable" is sometimes fed as ["un", "believable"] and sometimes as ["unbeliev", "able"] across epochs.

Related: Unigram language model tokeniser, SentencePiece, Tokenisation

A middle path between word and character splitting: frequent words stay whole while rare words are broken into smaller reusable pieces. This keeps the vocabulary manageable, avoids unknown words, and lets the model relate words that share pieces such as prefixes and suffixes.

Example. "tokenisation" could be split as ["token", "isation"], and "tokenising" as ["token", "ising"], sharing the first piece.

Related: Byte-pair encoding, WordPiece, Vocabulary

How often a term appears in one document, either as a raw count or scaled by document length. It measures how strongly a document is about a term, but common words score high everywhere unless weighted by rarity.

Example. The word "cricket" appears 6 times in a 200-word article, giving a term frequency of 6 / 200 = 0.03.

Related: Inverse document frequency, TF-IDF, Document-term matrix

Assigning one or more predefined labels to a piece of text, such as spam or not spam, positive or negative, or the topic of an article. It is one of the most common NLP tasks and works with features such as tf-idf or embeddings.

Example. A model labels customer emails as billing, delivery or refund with a tf-idf plus logistic regression pipeline.

Related: TF-IDF, Sentiment analysis, Sentence embedding

A trained model that converts text of any length into a fixed-size vector so that texts with similar meaning are close together. It is used for search, clustering, recommendation and retrieval, and is typically available through a library or hosted service.

Example. A model turns each of 10,000 support tickets into a 768-number vector, and near vectors group similar complaints.

Related: Sentence embedding, Semantic similarity, Bi-encoder

Converting text into a consistent standard form before further processing, for example lowercasing, removing extra spaces, expanding contractions or converting numbers and dates to a fixed style. It reduces meaningless variation so equal things look equal to the model.

Example. " Won't STOP " is normalised to "will not stop".

Related: Case folding, Unicode normalisation, Tokenisation

A weighting scheme that multiplies how often a term appears in a document by how rare it is across the collection: tf-idf = tf x idf. It highlights words that characterise a document and suppresses words common everywhere, and it is a classic feature for search and text classification.

Example. A word with tf 0.03 and idf 2.3 scores 0.03 x 2.3 = 0.069, while "the" with idf near 0 scores almost nothing.

Related: Term frequency, Inverse document frequency, Document-term matrix

The smallest unit of text a language system treats as one piece after splitting. Depending on the method, a token can be a whole word, part of a word, a single character or a punctuation mark. Models never see raw text; they see sequences of tokens converted to numbers.

Example. The sentence "Unhappiness is rare" might become the tokens ["un", "happiness", "is", "rare"] under a subword tokeniser.

Related: Tokenisation, Subword tokenisation, Token ID

The integer that represents one token, equal to its position in the vocabulary. Models take sequences of token IDs as input and look up a vector for each ID; they never handle the token strings directly.

Example. If the vocabulary lists ["<pad>", "the", "cat"], then "cat" has token ID 2.

Related: Vocabulary, Token, Padding token

The step that splits raw text into tokens and maps each token to an integer identifier from a fixed list. It is the first stage of almost every language pipeline, and the choice of method affects vocabulary size, sequence length, handling of rare words and cost of processing.

Example. "I love biryani!" split on spaces and punctuation gives ["I", "love", "biryani", "!"], which are then mapped to IDs such as [40, 812, 15007, 3].

Related: Token, Vocabulary, Byte-pair encoding

The average number of tokens a tokeniser produces per word of a language. High fertility means the language is split into many small pieces, making text longer, slower and costlier to process than languages the tokeniser was designed around.

Example. If English averages 1.3 tokens per word but Malayalam averages 4.5, Malayalam text costs about three and a half times more tokens for the same word count.

Related: Subword tokenisation, Byte-level BPE, Vocabulary size

A family of unsupervised methods that discover groups of words that tend to appear together and describe each document as a mix of these groups. It is used to explore large text collections without labels.

Example. Running topic modelling on 50,000 news articles may reveal clusters such as elections, cricket and monsoon.

Related: Latent Dirichlet allocation, Latent semantic analysis, Document-term matrix

Writing text of one language using the script of another while keeping the sounds, without translating the meaning. It is widespread in Indian messaging and creates spelling variation that NLP systems must handle.

Example. "namaste" is the Roman-script transliteration of "नमस्ते".

Related: Code-mixing, Text normalisation, Fuzzy matching

An n-gram of three consecutive words or characters. Trigrams capture more context than bigrams but appear much less often, so counts become sparse and many valid trigrams are never seen in training data.

Example. In "the quick brown fox" the trigrams are "the quick brown" and "quick brown fox".

Related: Bigram, N-gram, Smoothing

A loss that uses an anchor, a positive example similar to it and a negative example unlike it, and penalises the model unless the anchor is closer to the positive than to the negative by a margin: L = max(0, d(a, p) - d(a, n) + margin).

Example. With d(a, p) = 0.4, d(a, n) = 0.5 and margin 0.2, loss = max(0, 0.4 - 0.5 + 0.2) = 0.1.

Related: Contrastive learning, Siamese network, Hard negative mining

Cutting a token sequence down to a maximum length so it fits the limit a model can accept. Text beyond the limit is dropped, usually from the end, which can lose important information if the limit is small.

Example. A 700-token review truncated to 512 tokens loses its final 188 tokens, which may contain the verdict.

Related: Padding token, Tokenisation, Token

Rewriting text so that different code-point sequences that look identical become one canonical form. Standard forms named NFC, NFD, NFKC and NFKD exist. Without it, visually equal strings may not match, which breaks search and tokenisation.

Example. The letter "é" can be stored as one code point or as "e" plus a combining accent; normalising to NFC makes both identical.

Related: Unicode, Text normalisation, Tokenisation

An n-gram of length one: a single word or character considered by itself, with no context. Unigram counts give word frequencies, and a unigram model assumes each word is independent of its neighbours.

Example. In "to be or not to be" the unigram "to" appears twice.

Related: N-gram, Bigram, Bag of words

A subword method that starts with a large candidate vocabulary and repeatedly removes the pieces that least reduce the likelihood of the corpus, until the target size is reached. Each word can be split several ways, and the most probable split is chosen, or one is sampled for regularisation.

Example. "nation" might be split as ["nat", "ion"] or ["na", "tion"]; the tokeniser picks the split with the higher probability.

Related: SentencePiece, Subword regularisation, Subword tokenisation

A dynamic-programming method that finds the single most probable sequence of hidden states for an observed sequence, by keeping only the best path into each state at every step. Its cost grows linearly with length and with the square of the number of states.

Example. With 3 tags and 5 words, Viterbi evaluates about 5 x 3 x 3 = 45 transitions instead of 3^5 = 243 full sequences.

Related: Hidden Markov model, Conditional random field, Sequence labelling

The fixed list of distinct tokens a model or tokeniser knows, each paired with a unique integer ID. Text is converted to IDs using this list, and anything outside it must be split further or replaced by an unknown token.

Example. A vocabulary of 32,000 entries maps "the" to ID 5, "chai" to ID 9021, and so on.

Related: Token ID, Vocabulary size, Out-of-vocabulary word

The number of distinct tokens a tokeniser holds. A larger vocabulary shortens sequences but increases the size of the embedding table and gives rare tokens too few training examples; a smaller one lengthens sequences and raises compute cost.

Example. Moving from a 30,000-token to a 100,000-token vocabulary with 768-dimensional embeddings raises the embedding table from about 23 million to 77 million numbers.

Related: Vocabulary, Byte-pair encoding, Embedding dimension

A test of embeddings that checks whether vector arithmetic reflects relationships, such as king - man + woman landing near queen. It is a popular, though imperfect, way to show that embeddings capture regularities.

Example. In some embeddings, vec(Paris) - vec(France) + vec(India) is closest to vec(Delhi).

Related: Word embedding, Word2Vec, Semantic similarity

A dense list of real numbers, typically 50 to 1,000 long, that represents a word so that words with similar meaning have vectors that are close together. Embeddings are learned from text and let models generalise across related words.

Example. In a good embedding, the vector for "king" is closer to "queen" than to "banana".

Related: Word2Vec, GloVe, Static embedding

Choosing the correct meaning of a word that has several senses, using its surrounding context. Static embeddings blend all senses of a word into one vector, so context-aware methods are usually needed to tell the senses apart.

Example. In "He deposited cash at the bank", "bank" means a financial institution, not a river edge.

Related: Static embedding, Contextual embedding, Entity linking

Splitting text into whole words, usually using spaces and punctuation rules. It is simple and easy to read, but the vocabulary grows very large, and any word not seen during training becomes an unknown token, which loses information.

Example. "Don't stop" may be split into ["Do", "n't", "stop"] or ["Don't", "stop"] depending on the tokeniser's rules.

Related: Tokenisation, Out-of-vocabulary word, Subword tokenisation

A family of shallow neural-network methods that learn word vectors by predicting words from their neighbours in a large corpus. Its two architectures are skip-gram and continuous bag of words. It showed that simple training on raw text yields vectors capturing meaning.

Example. Trained on millions of sentences, Word2Vec places "mumbai" and "delhi" near each other in vector space.

Related: Skip-gram, Continuous bag of words, Negative sampling

A subword tokenisation algorithm that builds its vocabulary by merging pieces chosen to maximise the likelihood of the training data. Continuation pieces are marked with a prefix such as ##. It is the tokeniser used by BERT-style models.

Example. "playing" may be split into ["play", "##ing"], where ## shows the piece continues the previous word.

Related: Subword tokenisation, Byte-pair encoding, Vocabulary

An empirical pattern in language: a word's frequency is roughly inversely proportional to its rank in the frequency table, freq ~ 1 / rank. A few words are extremely common while most words are rare, which explains long-tailed vocabularies and sparse counts.

Example. If the most common word appears 60,000 times, the second most common appears about 30,000 times and the tenth about 6,000.

Related: Vocabulary, Corpus, Out-of-vocabulary word