Chapter-1-Probability-Theory.ipynb

Kernel: Python 3

'''
Import here useful libraries
Run this cell first for convenience
'''
import numpy as np
from scipy import stats
import scipy
import warnings
warnings.simplefilter('ignore', DeprecationWarning)

Chapter 1 - Probability Theory

Probabilities

Introduction

• Statistics and probability theory constitute a branch of mathematics for dealing with uncertainty. The probability theory provides a basis for the science of statistical inference from data

• Sample: (of size n) obtained from a mother population assumed to be represented by a probability

• Descriptive statistics: description of the sample

• Inferential statistics: making a decision or an inference from a sample of our problem

Sample Spaces

• Experiment: any process or procedure for which more than one outcome is possible

• Sample Space: set of all the possible experimental outcomes

Probability Values

A set of probability values for an experiment with sample space $S = \{ O_1, O_2, \cdots, O_n \}$ consists of some probabilities that satisfy: $$\begin{equation} 0 \leq p_i \leq 1, \hspace{0.5cm} i= 1,2, \cdots, n \end{equation}$$ and $$\begin{equation} p_1 +p_2 + \cdots +p_n = 1 \end{equation}$$ The probability of outcome $O_i$ occurring is said to be $p_i$ and it is written: $$\begin{equation} P(O_i) = p_i \end{equation}$$ In cases in which the $n$ outcomes are equally likely, then each probability will have a value of $1/n$.

Events

Events and Complements

• Events: subset of the sample space

• The probability of an event $A$, $P(A)$, is obtained by the probabilities of the outcomes contained withing the event $A$

• An event is said to occur if one of the outcomes contained within the event occurs

• Complement of events: event $A'$ is the event consisting of everything in the sample space $S$ that is not contained within $A$ $$\begin{equation} P(A) + P(A ') = 1 \end{equation}$$

• Elementary (or simple) event: event consisting of an individual outcome

Combinations of Events

Intersections of Events

• Intersections of Events: $A \cap B$ consists of the outcomes contained within both events $A$ and $B$

• Probability of the intersection, $P(A \cap B)$, is the probability that both events occur simultaneously

• Properties:

  • $P(A \cap B) +P(A \cap B') = P(A)$

  • Mutually exclusive events: if $A \cap B = \emptyset$

  • $A \cap (B \cap C) = (A \cap B) \cap C$

Union of Events

• Union of Events: $A \cup B$ consists of the outcomes that are contained within at least one of the events $A$ and $B$

• The probability of this event, $P (A \cup B)$ is the probability that at least one of these events $A$ and $B$ occurs

• Properties

  • $P( A \cup B) = P(A \cap B') + P(A' \cap B') + P(A \cap B)$

  • If the events are mutually exclusive, then $P(A \cup B) = P(A) + P(B)$

• Other simple results:

  • $(A \cup B)' = A' \cap B'$

  • $(A \cap B)' = A' \cup B'$

  • $A \cup A' = S$

  • $A \cup (B \cup C) = (A \cup B) \cup C$

Combinations of Three or More Events

• $P(A \cup B \cup C) = P(A) + P(B) + P(C) - P(A \cap B) - P( B \cap C) - P( A \cap C) + P(A \cap B \cap C)$

• Union of Mutually Exclusive Events: given a sequence $A_1, A_2, \cdots , A_n$ of mutually exclusive events: $$\begin{equation} P(A_1 \cup A_2 \cup \cdots \cup A_n) = P(A_1) + \cdots + P(A_n) \end{equation}$$

• Partition of sample space S: given a sequence $A_1, A_2, \cdots , A_n$ of mutually exclusive events such that $A_1 \cup A_2 \cup \cdots \cup A_n = S$ is called a partition of S

Conditional Probability

Definition of Conditional Probability

• Conditional Probability: of an event $A$ conditional on an event $B$ is: $$\begin{equation} P(A \mid B) = \frac{P(A \cap B)}{P(B)} \hspace{0.5cm} \text{for } P(B) >0 \end{equation}$$

Probabilities of Event Intersections

General Multiplication Law

• $P (A \mid B) = \frac{P(A \cap B)}{P(B)} \\ \Longrightarrow P(A \cap B) = P(B)P (A \mid B)$

• $P (A \mid B \cap C) = \frac{P(A \cap B \cap C)}{P(B \cap C)} \\ \Longrightarrow P(A \cap B \cap C) = P(B \cap C)P (A \mid B \cap C)$

• In general, for a sequence of events $A_1, A_2, \cdots,Untitled A_n$: $$\begin{equation} P(A_1, A_2, \cdots, A_n) = P(A_1)P(A_2 \mid A_1)P(A_3 \mid A_1 \cap A_2) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1}) \end{equation}$$

Independent Events

• Two events $A$ and $B$ are independent if:

  1. $P(A \mid B) = P(A)$

  2. $P(B \mid A) = P(B)$

  3. $P(A \cap B) = P(A)P(B)$

• These conditions are also equivalent

• Interpretation: events are independent if the knowledge about one event does not affect the probability of the other event

• Mutual independence: $A_1, A_2, \cdots, A_n$ are mutually independent if for any subset $\{ j_1, j_2, \cdots, j_k \}, k \leq n, \text{of } \{ 1, 2, \cdots, n \}$ : $$\begin{equation} P(A_{j_1}, A_{j_2}, \cdots, A_{j_k} = P(A_{j_1})P(A_{j_2}) \cdots P(A_{j_k}) \end{equation}$$

• Pairwise independence: the events $A_1, A_2, \cdots, A_n$ satisfying: $$\begin{equation} P(A_i \cap A_j) = P(A_i)P(A_j) \text{ for all } i, j \in \{1, 2, \cdots, n \} \text{ with } i \neq j \end{equation}$$

Posterior Probabilities

Law of Total Probability

• Given $\{ A_1, A_2, \cdots, A_n \}$ a partition of sample space $S$, the probability of an event $B$, $P(B)$ can be expressed as: $$\begin{equation} P(B) = \sum_{i=1}^n P(A_i)P(B \mid A_i) \end{equation}$$

Bayes' Theorem

• Given $\{ A_1, A_2, \cdots, A_n \}$ a partition of a sample space, then the posterior probabilities of the event $A_i$ conditional on an event $B$ can be obtained from the probabilities $P(A_i)$ and $P(A_i \mid B)$ using the formula: $$\begin{equation} P(A_i \mid B) = \frac{P(A_i)P(B \mid A_i)}{\sum_{j=1}^n P(A_j)P(B \mid A_j)} \end{equation}$$