$\DeclareMathOperator{\p}{P}$ $\DeclareMathOperator{\P}{P}$ $\DeclareMathOperator{\c}{^C}$ $\DeclareMathOperator{\or}{ or}$ $\DeclareMathOperator{\and}{ and}$ $\DeclareMathOperator{\var}{Var}$ $\DeclareMathOperator{\Var}{Var}$ $\DeclareMathOperator{\Std}{Std}$ $\DeclareMathOperator{\E}{E}$ $\DeclareMathOperator{\std}{Std}$ $\DeclareMathOperator{\Ber}{Bern}$ $\DeclareMathOperator{\Bin}{Bin}$ $\DeclareMathOperator{\Poi}{Poi}$ $\DeclareMathOperator{\Uni}{Uni}$ $\DeclareMathOperator{\Geo}{Geo}$ $\DeclareMathOperator{\NegBin}{NegBin}$ $\DeclareMathOperator{\Beta}{Beta}$ $\DeclareMathOperator{\Exp}{Exp}$ $\DeclareMathOperator{\N}{N}$ $\DeclareMathOperator{\R}{\mathbb{R}}$ $\DeclareMathOperator*{\argmax}{arg\,max}$ $\newcommand{\d}{\, d}$

Core Probability Reference

Definition: Empirical Definition of Probability

The probability of any event $E$ can be defined as:

$$ \p(E) = \lim_{n \rightarrow \infty} \frac {\text{count}(E)} {n} $$

Where $\text{count}(E)$ is the number of times that $E$ occured in $n$ experiments.

Definition: Core Identities

For an event $E$ and a sample space $S$

$0 ≤ \p(E) ≤ 1$ All probabilities are numbers between 0 and 1.
$\p(S) = 1$ All outcomes must be from the Sample Space.
$\P(E) = 1 - \P(E^\c)$ The probability of an event from its complement.

Definition: Probability of Equally Likely Outcomes

If $S$ is a sample space with equally likely outcomes, for an event $E$ that is a subset of the outcomes in $S$: $$ \begin{align} \p(E) &= \frac{\text{number of outcomes in $E$}}{\text{number of outcomes in $S$}} = \frac{|E|}{|S|} \end{align} $$

Definition: Conditional Probability.

The probability of $E$ given that (aka conditioned on) event $F$ already happened: $$ \p(E |F) = \frac{\p(E \and F)}{\p(F)} $$

Definition: Probability of or with Mututally Exclusive Events

If two events $E$ and $F$ are mutually exclusive then the probability of $E$ or $F$ occurring is: $$ \p(E \or F) = \p(E) + \p(F) $$

For $n$ events $E_1, E_2, \dots E_n$ where each event is mutually exclusive of one another (in other words, no outcome is in more than one event). Then: $$ \p(E_1 \or E_2 \or \dots \or E_n) = \p(E_1) + \p(E_2) + \dots + \p(E_n) = \sum_{i=1}^n \p(E_i) $$

Definition: General Probability of or (Inclusion-Exclusion)

For any two events $E$ and $F$: $$ \p(E \or F) = \p(E) + \p(F) − \p(E \and F) $$

For three events, $E$, $F$, and $G$ the formula is: $$ \begin{align} \p(E \or F \or G) =& \text{ }\p(E) + \p(F) + \p(G) \\ & −\p(E \and F) − \p(E \and G)−P(F \and G) \\ & +\p(E \and F \and G) \end{align} $$

For more than three events see the chapter of probability of or.

Definition: Probability of and for Independent Events.

If two events: $E$, $F$ are independent then the probability of $E$ and $F$ occurring is: $$ \p(E \and F) = \p(E) \cdot \p(F) $$

For $n$ events $E_1, E_2, \dots E_n$ that are independent of one another: $$ \p(E_1 \and E_2 \and \dots \and E_n) = \prod_{i=1}^n \p(E_i) $$

Definition: General Probability of and (The Chain Rule)

For any two events $E$ and $F$: $$ \p(E \and F) = \p(E | F) \cdot \p(F) $$

For $n$ events $E_1, E_2, \dots E_n$: $$ \begin{align} \p(E_1 \and E_2 \and \dots \and E_n) = &\p(E_1) \cdot \p(E_2|E_1) \cdot \p(E_3 |E_1 \and E_2) \dots \\ &\p(E_n|E_1 \dots E_{n−1}) \end{align} $$

Definition: The Law of Total Probability
For any two events $E$ and $F$: $$ \begin{align} \p(E) &= \p(E \and F) + \p(E \and F\c)\\ &=\p(E | F) \p(F) + \p(E | F\c) \p(F\c) \end{align} $$

For mutually exclusive events: $B_1, B_2, \dots B_n$ such that every outcome in the sample space falls into one of those events: $$ \begin{align} \p(E) &= \sum_{i=1}^n \p(E \and B_i) && \text{Extension of our observation}\\ &= \sum_{i=1}^n \p(E | B_i) \p(B_i) && \text{Using chain rule on each term} \end{align} $$

Definition: Bayes' Theorem

The most common form of Bayes' Theorem is Bayes' Theorem Classic: $$ \p(B|E) = \frac{\p(E | B) \cdot \p(B)}{\p(E)} $$

Bayes' Theorem combined with the Law of Total Probability: $$ \p(B|E) = \frac{\p(E | B) \cdot \p(B)}{\p(E|B)\cdot \p(B) + \p(E|B\c) \cdot \p(B\c)} $$