SMSTC (2007/08)

Probability

                www.smstc.ac.uk

Contents

4 Joint distributions and independence 4–1

4.1 Joint distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1

4.1.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y} . . . . . . . . . . . . . . . 4–2 4.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.3 Joint density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.4 Joint moment generating function . . . . . . . . . . . . . . . . . . . . . . . . . . 4–4 4.5 Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–5 4.6 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6

4.6.1 Naïve conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6 4.6.2 Conditional density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6 4.6.3 Conditional probability mass function . . . . . . . . . . . . . . . . . . . . 4–6

SMSTC (2007/08) Probability

Lecture 4: Joint distributions and independence

a

                  www.smstc.ac.uk

\underline{Contents}

4.1 Joint distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–1 4.1.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y} . . . . . . . . . . . . . . 4–2 4.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.3 Joint density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–3 4.4 Joint moment generating function . . . . . . . . . . . . . . . . . . . . 4–4 4.5 Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–5 4.6 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6 4.6.1 Naïve conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6 4.6.2 Conditional density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6 4.6.3 Conditional probability mass function . . . . . . . . . . . . . . . . . . . 4–6

a

4.1 Joint distributions

Consider a random element (X, Y) : (Ω, ℱ) → (ℝ^{2}, ℬ(ℝ^{2})). This random variable is called two random variables.

If P is a probability of (Ω, ℱ), the joint distribution of (X, Y) is another name for the law P_{X,Y} of the random variable (X, Y).

The joint distribution function of (X, Y) is the function

F_{X,Y} (x, y) := P(X ≤ x, Y ≤ y) = P_{X,Y} ((−∞, x] × (−∞, y]), (x, y) ∈ ℝ^{2}.  (4.1)

The law P_{X} of X is referred to as the first marginal of the law P_{X,Y}. The distribution function F_{X} of X is referred to as the first marginal distribution function of the joint distribution function F_{X,Y} and, of course,

F_{X}(x) = lim_{y→∞} F_{X,Y} (x, y).

Note that we chose ti use ≤ instead of < in (4.1) for no good reason other than a mere arbitrary convention.

4–1

aThese notes contain almost no proofs. For a complete set of notes, see [3]; alternatively, read the introductory book [4], or the more advanced book [1].

SMST C: Probability 4–2

4.1.1 Knowledge of F_{X,Y} implies knowledge of P_{X,Y}

Explaining why knowledge of the function F_{X,Y} implies knowledge of P_{X,Y} (B) for all B ∈ ℬ(ℝ^{2}) is beyond the scope of these lectures. The explanation can be found in [1, 3, 4]. But we give some intuition. Consider a rectangle (with sides parallel to the axes–please think geometrically)

(a_{1}, b_{1}] × (a_{2}, b_{2}] := {(x, y) ∈ ℝ^{2} : a_{1} < x ≤ b_{1}, a_{2} < y ≤ b_{2}}.  (4.2)

We allow a_{1}, a_{2} to take any value, including −∞. Since (−∞, b_{1}]×(−∞, b_{2}] is the disjoint union of four rectangles, using additivity, we obtain

P_{X,Y} ((a_{1}, b_{1}] × (a_{2}, b_{2}]) = F_{X,Y} (b_{1}, b_{2}) − F_{X,Y} (b_{1}, a_{2}) − F_{X,Y} (a_{1}, b_{2}) + F_{X,Y} (b_{1}, b_{2}).

Rectangles of the form (4.2) have the following nice properties: First, intersection of two of them is a rectangle of the same form. Thus, if ℛ denotes the collection of these rectangles we have that ℛ is closed under finite intersection, i.e. it is a π-system. Second, the complement of a rectangle from ℛ is a finite union of disjoint rectangles from ℛ.

EXERCISE 1. Write, explicitly, the complement of (a_{1}, b_{1}] × (a_{2}, b_{2}] as the disjoint union of elements of ℛ. Notice we can do that in at least two different ways. Do so.

Now consider the class

𝒞 := {finite disjoint unions of elements of ℛ }.

It is easy to visualise, geometrically, what kind of elements 𝒞 contains. Then

EXERCISE 2. Show that 𝒞 is a field, i.e. if A ∈ 𝒞 then A^{c} ∈ 𝒞 and if A, B ∈ 𝒞 then A ∪ B ∈ 𝒞.

Hence, if A ∈ 𝒞, we can write A = ∪_{i=1}^{m}B_{i}, where B_{i} are disjoint rectangles from ℛ, and, since for each such rectangle B_{i} we can use F_{X,Y} to compute P_{X,Y} (B), we have that

P_{X,Y} (A) = ∑_{i=1}^{m} P_{X,Y} (B), A ∈ 𝒞 ,

can be computed by using F_{X,Y} only. But there are many sets in ℬ(ℝ^{2}) that do not belong to 𝒞, so we wish to continue our endeavour. If it were true that every set in ℬ(ℝ^{2}) was a limit of sets of 𝒞 then we would be finished, by the continuity property of a probability. However, there are many elements in ℬ(ℝ^{2}) that are not limits of sets in 𝒞.

Example 4.1. The set of all points (x, y) where x, y are rationals cannot be obtained as a limit of elements of 𝒞^{2}. To illuminate this point, consider “straightforward” procedure that places a little rectangle around each such point and then let the little rectangle shrink. Specifically, let ℚ = {q_{1}, q_{2}, …} be an enumeration of the rationals. To each (q_{m}, q_{n}) associate the rectangle

I_{m,n}(ε) := (q_{m} − ε2^{−m}, q_{m} + ε2^{−m}] × (q_{n} − ε2^{−n}, q_{n} + ε2^{−n}],

and let I(ε) := ∪_{m,n}I_{m,n}(ε). Show that ∩_{ε>0,ε∈ℚ}I(ε) is not equal to ℚ × ℚ. Hint: The set I(ε) is uncountable.

We will denote by P_{0} the function P_{X,Y} restricted to 𝒞. Clearly, P_{0} is uniquely specified by F_{X,Y}. If P_{0} can be uniquely extended to a probability on (ℛ^{2}, ℬ(ℝ^{2})) then our claim that F_{X,Y} completely specifies P_{X,Y} will be proved. The proof of this is contained in [3]. The most important step in the proof is to show that P_{0} is countably additive on 𝒞.

EXERCISE 3. Let V, W be independent random variables with F_{V} (t) = F_{W} (t) = 1 − e^{−t}, t > 0. Let X = 2V, Y = V − W. Compute the joint distribution function F_{X,Y} of the random element (X, Y).

4.2 Independence

Recall that X, Y are independent random variables on (Ω, ℱ P) if σ(X), σ(Y) are independent σ-fields. We would like to show that

Proposition 4.1. X, Y are independent on (Ω, ℱ , P) if and only if F_{X,Y} (x, y) := P(X ≤ x, Y ≤ Y), F_{X}(x) := P(X ≤ x), F_{Y} (y) := P(Y ≤ y) are related by

F_{X,Y} (x, y) = F_{X}(x)F_{Y} (y).

This is a consequence of the following:

Lemma 4.1. Let (Ω, ℱ , P) be a probability space. Let ℛ_{1}, ℛ_{2} be two classes of sets, each of which is closed under the intersection operation. Then σ(ℛ_{1}), σ(ℛ_{2}) are independent if and only if

P(B_{1}B_{2}) = P(B_{1})P(B_{2}), B_{1} ∈ ℛ_{1}, B_{2} ∈ ℛ_{2}.

For a proof see [3]. Proof of Proposition 4.1: Let ℛ_{1} = {X^{−1}(a, b] : a < b}, ℛ_{2} = {Y^{−1}(a, b] : a < b}. These are both classes of sets closed under intersection. It is easy to see that the assumption F_{X,Y} (x, y) = F_{X}(x)F_{Y} (y) implies that P(B_{1}B_{2}) = P(B_{1})P(B_{2}), B_{1} ∈ ℛ_{1}, B_{2} ∈ ℛ_{2}. Hence P(A_{1}A_{2}) = P(A_{1})P(A_{2}), A_{1} ∈ σ(ℛ_{1}) = σ(X), A_{2} ∈ σ(ℛ_{2}) = σ(Y).

4.3 Joint density

The concept of joint density requires understanding of the Lebesgue integral of functions of two variables. You will not be asked to understand this, but you may think of it, roughly, as a Riemann integral. For a complete understanding, see [1, 4], and also consult the notes [3]. We do mention, without proof of course, the following:

Theorem 4.1. (i) If h is Riemann integrable on a rectangle R = [a, b] × [c, d] then its Lebesgue integral on R coincides with its Riemann integral on R. (ii) If h is bounded and measurable then it is Riemann integrable on R of the set of discontinuities D of h satisfies P((U_{1}, U_{2}) ∈ D) = 0.

Given two distribution functions F_{1}, F_{2} on ℝ, we can define the product measure F_{1} × F_{2} on (ℝ^{2}, ℬ(ℝ^{2})) by defining it first on rectangles,

(F_{1} × F_{2})(B_{1} × B_{2}) := P(F_{1}^{−1}(U_{1}) ∈ B_{1}, F_{2}^{−1}(U_{2}) ∈ B_{2}),

and then extending it to ℬ(ℝ^{2}) using the procedure explained in [3]. One could define the Lebesgue-Stieltjes integral by

∫_{ℝ^{2}} h d(F_{1} × F_{2}) := Eh(F_{1}^{−1}(U_{1}), F_{2}^{−1}(U_{2})),

where U_{1}, U_{2} are independent random variables, uniform in [0, 1].

Theorem 4.2 (Fubini). If h ≥ 0 or if ∫_{ℝ^{2}} |h| d(F_{1} × F_{2}) < ∞, then

∫_{ℝ^{2}} h d(F_{1} × F_{2}) = ∫_{ℝ} (∫_{ℝ} h(x, y)F_{2}(dy)) F_{1}(dx) = ∫_{ℝ} (∫_{ℝ} h(x, y)F_{1}(dx)) F_{2}(dy).

Proof [sketch]: The statement is obvious when h(x, y) = 1_{B_{1}}(x)1_{B_{2}}(y). Hence it is obvious for linear combinations of such functions. The general statement follows from an approximation procedure (see [1, 4] and the notes [3].) Occasionally, it so happens that there exists a function f such that F_{X,Y} can be written as a Lebesgue integral:

F_{X,Y} (x, y) = ∫_{(−∞,x]×(−∞,y]} f(s, t)dsdt.

In such a case, we say that (X, Y) is absolutely continuous (jointly absolutely continuous, I suppose, if you want to be pedantic) and that f_{X,Y} is a density. Using Fubini’s theorem, we see that

f_{X}(x) = ∫_{(−∞,x]} f(s)ds

is a density for X (i.e. X is absolutely continuous, and so is Y). Another consequence of Fubini’s theorem is:

Lemma 4.2. If X, Y are independent then

E(XY) = (EX)(EY),

whenever the expectations are defined.

And another, useful consequence of Fubini’s theorem is:

Lemma 4.3. If X is a positive random variable then

EX = ∫_{0}^{∞} P(X > x) dx.

A standard criterion for independence between X, Y, for absolutely continuous (X, Y), is:

Lemma 4.4. Suppose that (X, Y) is absolutely continuous. Let f_{X,Y} be a density of (X, Y). Let f_{X}, f_{Y} be densities of X, Y, respectively. Then X, Y are independent if and only if

f(x, y) = f_{X}(x)f_{Y} (y),

for all (x, y) except, possibly, on a set of measure zero.

4.4 Joint moment generating function

When (X, Y) is a random variable in ℝ^{2} we can define its joint moment generating function by

M_{X,Y} (η, θ) := Ee^{ηX+θY} , η, θ ∈ ℝ.

whenever it exists. Let also M_{X}, M_{Y} be the moment generating functions of X, Y. One can prove that:

Lemma 4.5. Suppose that M_{X,Y} exists in a (one sided) neighbourhood of zero. If M_{X,Y} (η, θ) = M_{X}(η)M_{Y} (θ) then X, Y are independent.

EXERCISE 4. Let X, Y be independent random variables with common moment generating function M_{X}(θ) = M_{Y} (θ) = e^{θ^{2}}. Let X′ := X + Y, Y′ := X − Y. Compute the joint moment generating function of (X′, Y′). Show that X′, Y′ are independent.

4.5 Correlations

We now consider random variables X on some probability space (Ω, ℱ , P) with EX^{2} < ∞. The aggregate of all these random variables will be denoted by

L^{2}(Ω, ℱ , P).

If EX^{2} < ∞, EY^{2} < ∞, then E(X + Y)^{2} < ∞ (indeed, (x + y)^{2} ≤ 2x^{2} + 2y^{2}). This means that if X, Y ∈ L^{2}(Ω, ℱ , P) then, for any a, b ∈ ℝ, aX + bY ∈ L^{2}(Ω, ℱ , P). Hence L^{2}(Ω, ℱ , P) is a linear space (a linear subspace of ℝ^{Ω}). The correlation between X and Y is defined by

E(XY).

The covariance between X and Y is defined by

cov(X, Y) = E(X − EX)(Y − EY).

Since, for X_{1}, X_{2}, Y ∈ L^{2}(Ω, ℱ , P), a_{1}, a_{2} ∈ ℝ,

cov(a_{1}X_{1} + a_{2}X_{2}, Y) = a_{1} cov(X_{1}, Y) + a_{2} cov(X_{2}, Y),

the covariance is linear in each of its arguments when the other is kept fixed and it can thus be used to define an inner product:

⟨X, Y ⟩ := cov(X, Y).

We also define the semi-norm

||X|| := \sqrt{cov(X, X)} = \sqrt{var(X, X)},

where the word ‘semi-norm’ means that it has the following properties:

1. ||X|| ≥ 0

2. ||a_{1}X_{1} + a_{2}X_{2}|| = |a_{1}| ||X_{1}|| + |a_{2}| ||X_{2}||.

3. ||X + Y || ≤ ||X|| + ||Y ||.

Another name for ||X|| is ‘standard deviation’. We also note that

If ||X|| = 0 then P(X = 0) = 1.

We \underline{cannot} deduce, from ||X|| = 0 alone that X(ω) = 0 for all ω ∈ Ω, but only that X(ω) = 0 for all ω except those in a set of probability zero. If we could deduce that X(ω) = 0 for all ω ∈ Ω, we would say that || · || is a norm. To get around this problem, we merely identify all random variables in L^{2}(Ω, ℱ , P) which differ on a set of measure zero: That is, we let [X] be the set of all Y such that P(X ≠ Y) = 0, and redefine L^{2}(Ω, ℱ , P) to be the collection of all such [X]. It is not hard to see that this is still a linear space and if we let ||[X]|| := ||X|| (which is well defined), then this is a norm. Being a normed space with an inner product, L^{2}(Ω, ℱ , P) has a structure much like the geometric structure of the usual Euclidean space, for instance, Pythagoras’ theorem holds:

||X + Y ||^{2} = ||X||^{2} + ||Y ||^{2} if ⟨X, Y ⟩ = 0.

For more properties, see [3].

EXERCISE 5. Let U_{1}, U_{2} be independent, both uniform in [0, 1]. Compute the correlation and covariance between X_{1} := U_{1} + e^{U_{2}}, X_{2} = U_{1}^{2} − U_{2}^{3}.

4.6 Conditioning

4.6.1 Naïve conditioning

Recall that, if (Ω, ℱ , P) is a probability space and B ∈ ℱ, P(B) ≠ 0, we define

P(A|B) = P(AB)/P(B), A ∈ ℱ .

This is a new probability, called P conditional on B. We easily see that (Ω, ℱ , P(·|B)) is a new probability space. The problem is that, in many cases, we want to condition with respect to an event B that has probability zero. This can be done. Loosely speaking, what saves us is the fact that if P(B) = 0 then P(AB) = 0 as well. We can use many methods for defining conditioning in a more general sense. There is a geometric approach based on the structure of L^{2}(Ω, ℱ , P), and this can be found in any good book, like [1, 4], and also in the notes [3]. In Chapter 8 you will see another approach. What we will do, instead, is to define conditioning in very special cases, using formulae.

4.6.2 Conditional density

Suppose that (X, Y) is an absolutely continuous random element of ℝ^{2} with some joint density f : ℝ^{2} → ℝ. Then X is an absolutely continuous random variable. Let f_{1} : ℝ → ℝ be a density for X. Similarly, Y is an absolutely continuous random variable. Let f_{2} be a density for Y. We call f_{1}, f_{2} marginal densities of f. Define the conditional density of X given Y by

f_{X|Y} (x|y) := \begin{cases}\frac{f(x, y)}{f_{2}(y)}, & if f_{2}(y) ≠ 0, \\ 0, & otherwise. \end{cases}

Lemma 4.6. For B, C ∈ ℬ, let

g_{B}(y) := ∫_{B} f_{X|Y} (x|y)dx.

Then

P (X ∈ B, Y ∈ C) = E[g_{B}(Y)1_{C}(Y)],

This is hard to prove in general. But you can sort of see it is true by considering

EXERCISE 6. Convince yourselves that the lemma is true if B, C are intervals.

4.6.3 Conditional probability mass function

Let (X, Y) be a random variable on (Ω, ℱ , P) with values in the discrete set (S_{1} × S_{2}, 2^{S_{1}×S_{2}}). As usual, define its joint probability mass function and marginal probability mass functions by

p(x, y) := P((X, Y) = (x, y)), p_{1}(x) := P(X = x), p_{2}(y) := P(Y = y), (x, y) ∈ S_{1} × S_{2}.

Define the conditional probability mass function of X given Y by ^{b}

p_{X|Y} (x|y) := \begin{cases}\frac{p(x, y)}{p_{2}(y)}, & if p_{2}(y) ≠ 0, \\ 0, & otherwise. \end{cases}  (4.3)

bThe notation p_{X|Y} is terrible. We only use it out of some respect to the undergraduate probability courses.

The reason that the notation is terrible is that in the subscript ‘X|Y ’ in p_{X|Y} (x|Y) plays a merely cosmetic rôle, as opposed to the essential rôle played by the last variable Y inside the parenthesis.

Lemma 4.7.

P(X = x|Y = y) = p_{X|Y} (x|y)p_{2}(y).

Proof If p_{2}(y) = 0 then P(X = x, Y = y) = 0. If p_{2}(y) > 0 then multiply both sides of (4.3) by p_{2}(y) to obtain the desired result. □

References

[1] B. Fristedt & L. Gray, A Modern Approach to Probability Theory, Birkhäuser, 1997.

[2] E. Hewitt & K. Stromberg, Real and Abstract Analysis, Springer, 1965.

[3] T. Konstantopoulos, Extended set of lecture notes, with proofs,

   www.ma.hw.ac.uk/∼takis

[4] D. Williams, Probability with Martingales, Cambridge, 1991.