Markov Chains and Stochastic Processes
Markov chains represent a fundamental concept in probability theory that models sequences of events where the probability of each event depends only on the state of the previous event.
Core Concepts
A Markov chain is a mathematical system that transitions from one state to another according to certain probabilistic rules. The defining characteristic of a Markov chain is the Markov property: the probability of transitioning to any particular state depends solely on the current state and time elapsed, and not on the sequence of states that preceded it.
This property can be stated mathematically as:
P(Xn+1 = x | X1 = x1, X2 = x2, ..., Xn = xn) = P(Xn+1 = x | Xn = xn)
In simpler terms, the future state depends only on the present state, not on how the system arrived at the present state.
Transition Matrices
The behavior of a Markov chain is typically represented using a transition matrix P, where each element pij represents the probability of transitioning from state i to state j in a single step:
P = [pij] where pij = P(Xn+1 = j | Xn = i)
For a transition matrix with m states, P is an m × m matrix where each row sums to 1 (since the probabilities of transitioning from a given state to all possible states must sum to 1).
State Classifications
States in a Markov chain can be classified in several ways:
- Absorbing state: A state that, once entered, cannot be left (pii = 1)
- Transient state: A state that, once left, has a non-zero probability of never being revisited
- Recurrent state: A state that will be revisited with probability 1