Top-p  (top-p, top_p, top p, nucleus sampling)

The top-p sampling method, also known as nucleus sampling, is a technique used in natural language processing (NLP) for generating text from probabilistic large language models (LLMs). This method allows for dynamically selecting a subset of tokens based on their cumulative probability, ensuring a balance between diversity and coherence in the generated text. Here’s a more technical and mathematical breakdown of how top-p sampling works:

Probability Distribution and Model Output

Given a language model, let V be the vocabulary of possible tokens (words, subwords, or characters). When the model generates a token, it produces a probability distribution P(xi ∣ x1, x2, …, xi−1) over the entire vocabulary V. Here, x1, x2, …, xi−1​ represent the sequence of tokens generated so far.

Mathematically, the model outputs a probability distribution over the next possible tokens:

where pj represents the probability of the j-th token in the vocabulary being the next token. These probabilities are non-negative and sum to 1.

Top-p Sampling (Nucleus Sampling)

The goal of top-p sampling is to select a subset of tokens from the vocabulary such that their cumulative probability is greater than or equal to a specified threshold pthreshold​, which is the value of the top-p parameter.

1. Sorting the Tokens by Probability:

First, sort the tokens in descending order based on their probabilities:

where p(1)​ is the highest probability and p(∣V∣)​ is the lowest.

2. Cumulative Probability Calculation:

Next, calculate the cumulative probability starting from the highest probability token:

The cumulative probability Pcumulative(k) at the k-th token represents the total probability mass of the top k tokens.

3. Selecting the Nucleus:

Determine the smallest k such that the cumulative probability meets or exceeds the top-p threshold:

where k∗ is the size of the nucleus, the smallest set of tokens whose cumulative probability is at least pthreshold.

4. Sampling from the Nucleus:

Finally, the model randomly selects the next token from this nucleus, using the normalized probabilities within the selected set:

This approach ensures that the model considers a flexible number of tokens based on their probability distribution. If the distribution is sharp (few high-probability tokens), the nucleus will be small; if the distribution is flat (many tokens with similar probabilities), the nucleus will be larger, including more tokens in the sampling process.

Advantages and Trade-offs

• Diversity vs. Coherence
  By adjusting the top-p value, users can control the diversity of the generated text. A higher top-p value (closer to 1) includes more tokens in the nucleus, leading to more diverse and potentially creative outputs, but at the risk of generating less coherent text. A lower top-p value (closer to 0) restricts the nucleus to fewer tokens, producing more deterministic and coherent text, but with reduced creativity.

• Dynamic Selection
  Unlike top-k sampling, which fixes the number of tokens considered, top-p sampling dynamically adjusts based on the context, leading to more contextually appropriate text generation.

In essence, top-p sampling offers a flexible and adaptive method for text generation, allowing models to maintain a balance between creativity and coherence depending on the context and user preferences.