Concepts / Independent Sampling

Independent Sampling

D is the distribution; z ∼ D describes sampling from it.

  • Programming

From Distribution to Sample

Probability notation becomes easier when you read it from the distribution outward. Start with D: it names a probability distribution over a set Z. Then read z ∼ D as a statement about the relationship between z and D. It says that z is sampled according to D.

Key point

The symbol z ∼ D does not identify one permanently fixed value of z. It tells you to regard z as a value obtained by sampling from D.

Ddistributionzsampled valuesample according to
What happens when a value z is sampled from the distribution D, and how is that different from z being fixed?

Using a Sampled Variable

Once z is understood as sampled from D, later expressions involving z are evaluated with respect to that sampling convention. For example, a function can receive z as its input, or a condition can be evaluated for the sampled value. The notation keeps track of the distribution that governs the variable.

Definition

The expression E z ∼ D [f(z)] denotes the expected value of f(z) when z is sampled according to D. The subscript identifies the distribution governing the variable inside the brackets.

Worked example

Following the sampling context

Interpret the expression E z ∼ D [f(z)].

  1. 1
    Identify D

    D is the distribution over the underlying set Z.

  2. 2
    Read the subscript

    The notation z ∼ D says that z is sampled according to D.

  3. 3
    Read the brackets

    The function f is evaluated using the sampled variable z.

  4. 4
    Interpret the whole expression

    The expression summarizes the expected value of the real-valued quantity f(z) under that sampling process.

Result

It is an expected value governed by D, not a statement that z is one fixed value.

Averages and Event Probabilities

Expected-value notation and probability notation both refer to sampling from D, but they summarize different things. Expected value summarizes a real-valued function of the sampled variable. Probability notation summarizes whether a Boolean condition is true.

NotationWhat is evaluatedWhat is summarized
E z ∼ D [f(z)]A real-valued function f(z)The expected value of that quantity
P z ∼ D [f(z)]A Boolean condition f(z)The probability that the condition is true

Both expressions use the sampling convention z ∼ D, but they answer different questions.

E z ∼ D [f(z)]real-valued quantityP z ∼ D [f(z)]Boolean condition
What is the difference between averaging a numerical quantity over samples and measuring the probability that an event occurs?

If f maps Z to {true, false}, then P z ∼ D [f(z)] denotes the probability that f(z) is true when z is sampled according to D. In set notation, this is the probability assigned by D to the collection of values z for which f(z) is true.

Building the Product Distribution

A single sampled variable is not the only object that can have a distribution. Suppose m points are sampled from D. The resulting object is an m-component tuple written as (z₁, …, zₘ). Each zᵢ is sampled from D independently of the other points.

Definition

Dm refers to the probability over Zm induced by sampling m points from D independently. D governs individual samples from Z; Dm governs the larger space Zm of m-component sample tuples.

Ddistribution over Zz₁sample from D(z₁, …, zₘ)m-component tupleD^m probability over Z^m z₂sample from Dzₘsample from Dsample independentlysample independentlysample independentlycomponentcomponentcomponentinduces
How do m independently sampled points from D combine to form the product distribution Dm?
Key point

The superscript in Dm signals repeated independent sampling and the larger sample space Zm. It does not describe a new individual sample variable.

Common Reading Errors

  • Common mistake

    Treating z ∼ D as if z were a permanently fixed value.

    What it looks like

    Reading the notation as though it simply named one particular element of Z.

    Why it fails

    The notation describes z as a value obtained by sampling from D.

    Do this instead

    Carry the sampling context into later expressions involving z.

  • Common mistake

    Confusing an expected value with an event probability.

    What it looks like

    Interpreting E z ∼ D [f(z)] as the probability that f(z) is true.

    Why it fails

    Expected value summarizes a real-valued function, while probability notation summarizes when a Boolean condition is true.

    Do this instead

    Ask whether the expression is averaging a numerical quantity or measuring the probability of a true condition.

  • Common mistake

    Reading Dm as an individual sample from D.

    What it looks like

    Treating the superscript m as if it changed the name of one sampled value.

    Why it fails

    Dm refers to the probability over m-component tuples in Zm.

    Do this instead

    Look for the tuple (z₁, …, zₘ) and remember that its components are sampled independently from D.

  • Common mistake

    Assuming the shorthand E[f] introduces a different expected value.

    What it looks like
    Treating E[f] as unrelated to E z ∼ D [f(z)].
    Why it fails

    The shorter notation omits sampling information that is already clear from context.

    Do this instead

    Restore the distribution and sampled variable mentally when interpreting the shorthand.

Practice the Notation

MEDIUM

For each expression, identify what is being described: the distribution of one sample, an expected value, an event probability, or a distribution over tuples. Then explain the role of D.

Hints
  • Read D first as a distribution over Z.
  • Interpret z ∼ D as the sampling relationship.
  • For an expectation, look for a real-valued function.
  • For a probability, look for a Boolean condition.
  • For Dm, identify the m-component tuple and the space Zm.
Worked example

Classifying Four Notations

Classify the roles of D, z ∼ D, E z ∼ D [f(z)], P z ∼ D [f(z)], and Dm.

  1. 1
    D

    This names a probability distribution over the set Z.

  2. 2
    z ∼ D

    This states that z is sampled according to D.

  3. 3
    E z ∼ D [f(z)]

    This is the expected value of a real-valued function of the sampled variable.

  4. 4
    P z ∼ D [f(z)]

    This is the probability that the Boolean condition f(z) is true under the sampling convention.

  5. 5
    Dm

    This is the induced probability over m-component tuples in Zm when the components are sampled independently from D.

Result

The notation changes meaning according to the object being summarized: one distribution, one sampled variable, a numerical average, an event probability, or a tuple distribution.

Key Takeaways

  • D names a probability distribution over an underlying set Z.
  • z ∼ D says that z is sampled according to D rather than treated as one permanently fixed value.
  • E z ∼ D [f(z)] summarizes the expected value of a real-valued function, while P z ∼ D [f(z)] summarizes the probability that a Boolean condition is true.
  • Dm is the induced probability over m-component tuples in Zm formed by sampling each component independently from D.
  • The superscript in Dm describes repeated independent sampling and a larger sample space, not a new individual sample variable.

Related Topics

Related concepts

More concepts starting with I