A. A. Markov (1856--1922) came up with the idea for ``Markov'' chains when he wanted to estimate the frequency at which consonants occur in Pushkin's poem Eugene Onegin. See the History section of the Wikipedia article Markov Chain.
The weather problem has this Maple worksheet.
The market share problem has this Maple worksheet.
Illustration of the redundant equation.
The Maple file to find the communicating classes of a Markov chain. The rough explanation of the algorithm used is given here (Kao, pp. 172--174).
Other examples: Irreducible-Cinlar-page-133.mw | Irreducible-Isaacson-Madsen-page-58.mw
Here is a (rather crude) method for determining the period of a Markov chain using Maple: Period.mw
When the MC is irreducible but periodic with period d, then subsequences p[j,j]^(nd) converge to d*pi[j] as we see in this file.
Here's another (crude) method for testing the recurrence properties of a Markov chain where we attempt to establish that Sum((p[i,i])^n,n=1..infinity) = infinity: Recurrence.mw
To calculate the mean first passage times (for a recurrent chain, e.g., our inventory problem) we can use this Maple file: muij.mw
Maple worksheet for the periodic review (s,S) inventory problem. This MC is irreducible, aperiodic and positive recurrent, thus thus ergodic. Here, we just compute the transition probabilities.
Maple worksheet for the same (s,S) problem where we find the limiting probabilities by simply computing the higher powers of the transition matrix.
Here, we use the results discussed in class to find the stationary distribution.
This .pdf file has a summary of the classification of the states.
Even though I did not discuss absorbing Markov chains, this is a rather important topic which the students should be familiar with. Pleasee see this .pdf file of my handwritten notes.