LeatherSagiKnowledgebase

Discrete-Time Markov Jump Linear Systems

8 min read

Problem

Engineering and economic systems are often well described by a family of linear models — one per “operation mode” — with abrupt jumps between modes governed by random events (component failures, regime changes, environmental disturbances, boom/recession transitions). When the jumps are modelled as a finite-state Markov chain {θ(k)} and each mode i carries its own linear dynamics x(k+1) = A_i x(k) + B_i u(k) + G_i w(k), the resulting object is a discrete-time Markov Jump Linear System (MJLS). Although MJLS look like “many Kalman filters glued together,” the joint process (x(k), θ(k)) has subtleties that defy linear intuition: a system whose every individual mode is unstable can be globally mean square stable, and vice versa. The book develops a unified analytical theory of stability, optimal control, filtering, and H∞-control for discrete-time MJLS, hosted on a Banach-space framework over the operator space H^n of N-tuples of matrices.

Key idea

Replace the non-Markov state x(k) with the augmented state (x(k), θ(k)) and work with the indicator-weighted second-moment operators Q_i(k) = E[x(k)x(k)* 1_{θ(k)=i}]. On the Hilbert space H^n the second-moment recursion becomes a single linear operator equation Q(k+1) = T(Q(k)) where T_j(Q) = Σ_i p_{ij} Γ_i Q_i Γ_i*. From this one operator the whole theory drops out:

  • Stability (Chapter 3): MJLS is mean square stable ⇔ rσ(T) < 1 ⇔ rσ(A1) < 1 ⇔ a coupled Lyapunov equation has a positive-definite solution. All four conditions are equivalent and finitely checkable.
  • Optimal control (Chapter 4): The finite-horizon LQ problem is solved by a set of N coupled Riccati difference equations indexed by the operation mode; the infinite-horizon LQ problem is solved by the mean-square-stabilizing solution of the corresponding coupled algebraic Riccati equations (CARE).
  • Filtering (Chapter 5, primary entry point): With the mode known, the optimal Markov jump filter is again a set of N coupled Riccati difference equations — not a sample-path Kalman filter. This is essential because the sample-path Kalman filter would require N + N² + … + N^T = O(N^T) precomputed gains, while the mode-restricted filter only needs N·T. With the mode unknown, the optimal nonlinear filter has memory and computation that grow exponentially in time, so the book derives the optimal linear MMSE estimator (LMMSE), an Nn-dimensional Riccati recursion plus an extra term V(Q(k)) that vanishes when N=1.
  • CARE existence (Appendix A): The mean-square stabilizing solution to the control / filtering CARE exists and is unique under MS stabilizability and detectability; control and filtering CARE are dual under the redefinition s_{ij} ↔ s_{ji}.

Method

The text fixes a stochastic basis (Ω, F, {F_k}, P), an N-state Markov chain {θ(k)} with transition matrix P = [p_{ij}], and a family of dynamics G_i = (A_i, B_i, G_i, L_i, H_i, …). Chapter 2 sets up the operator space H^n and the four canonical operators E, L, T, V, J (eqs. 3.6–3.10) — with T = L* and V = J* — and proves r_σ is invariant across them. Chapter 3 uses these operators to characterize MSS via the augmented matrix A_1 = A_2 = (P’ ⊗ I_n²) diag(A_i ⊗ A_i). Chapter 4 develops dynamic programming through the value function W(i, x, k) = x* X_i(k) x + α(k), giving the coupled Riccati difference equation (4.14) with optimal feedback F_i(k) = -R_i(k)^{-1} B_i* E_i(X(k+1)) A_i. Chapter 5 specializes the same machinery to filtering: the finite-horizon filtering coupled Riccati difference equation (5.13) for Y_i(k), the infinite-horizon filtering CARE (5.31), the LMMSE recursion (5.44–5.49) for the Nn-dimensional augmented state z(k) = (x(k)1_{θ(k)=1}, …, x(k)1_{θ(k)=N}), and Lemma 5.11’s direct Riccati form (5.51) with the extra operator V(Q(k), k) capturing the cost of not knowing the mode. Section 5.5 gives an LMI alternative that handles parametric uncertainty. Appendix A proves maximal-and-stabilizing-solution coincidence and convergence Y(k) → Y for the CARE under MS stabilizability + detectability.

Results

  • Equivalent forms of MSS for the homogeneous MJLS (Theorem 3.9): rσ(T) < 1 ⇔ rσ(A1) < 1 ⇔ existence of positive-definite solution to coupled Lyapunov equation V − T(V) = S, V > 0. The pathological Examples 3.17 and 3.18 show stability of every individual mode is neither necessary nor sufficient for MSS.
  • Finite-horizon LQ optimum (Theorem 4.2): u(k) = F_{θ(k)}(k) x(k) with F_i(k) given by the coupled Riccati DE (4.14)–(4.15); minimal cost is Σ_i v_i x_0* X_i(0) x_0 + Σ_t δ(t).
  • Finite-horizon mode-known filter (Theorems 5.3–5.5): The estimator x̂_e(k+1) = A_{θ(k)} x̂_e(k) + B_{θ(k)} u(k) − M_{θ(k)}(y(k) − L_{θ(k)} x̂_e(k)) with gain M_i(k) given by (5.14) and Y_i(k) by the filtering Riccati DE (5.13) is optimal among Markov-jump linear filters; the optimal cost is Σ_k Σ_i tr(Y_i(k)).
  • Infinite-horizon mode-known filter (Theorem 5.8): The optimal filter is the H_2-norm-minimizing time-invariant Markov jump filter with gains M_i = −A_i Y_i L_i* (H_i H_i* π_i + L_i Y_i L_i*)^{-1} where Y is the MS-stabilizing solution of the filtering CARE (5.31).
  • Mode-unknown filter (Theorem 5.9, Lemma 5.11, Theorem 5.12): The LMMSE is an Nn-dimensional Riccati recursion (5.51) with the additional operator V(Q(k), k) (5.50) accounting for the unobserved mode; under MSS + ergodicity the error covariance Z̃(k+1|k) → P, the unique p.s.d. solution of the Nn-dimensional algebraic Riccati equation (5.60). The stationary filter rσ(A + T(P)L) < 1 is automatically stable.
  • CARE existence (Appendix A.5–A.23): MS stabilizability of (A, B, p) plus MS detectability of (p, C, A) is sufficient for existence and uniqueness of the mean-square-stabilizing solution to the control CARE; convergence X(k) → X from any X(0) ∈ H^{n+}; control–filtering duality (Proposition A.3) gives the filtering CARE result for free.
  • The textbook’s filter and stability framework is the canonical reference for Hamilton-style mode-aware filters, IMM filters, and Rao-Blackwellized particle filters in finance, econometrics, fault-tolerant control, and target tracking.

Limitations

  • The mode-known assumption is restrictive in practice. Sections 5.4 and 5.5 treat the mode-unknown case but only at the linear MMSE level — the optimal nonlinear estimator is admitted to be intractable (memory exponential in T).
  • Theory is finite-state Markov chain (N < ∞). Costa cites [67] for the countable-N case where the MSS / SS equivalence breaks down.
  • The LMMSE filter dimension grows as Nn — for large mode counts this becomes expensive even though it is “linear in T.”
  • The H_∞ chapter (Ch. 7) and LMI sections (5.5) lean heavily on convex optimization solvers; the LMI route does not give the same closed-form insight as the CARE route.

Open questions

  • How tight are the LMMSE error bounds against the true (intractable) conditional mean? The book gives the Nn-dim Riccati recursion but does not characterize its gap to the optimal nonlinear filter except in special cases.
  • For the partial-information control problem (Chapter 6) the separation principle is established, but the conditions on the filter side are stronger than the dual CARE existence; sharper conditions are an open area.
  • Convergence-rate results for the coupled Riccati recursion are mostly asymptotic; explicit rate bounds (analogous to the Kalman filter’s observability-Gramian rate) are not the focus of Appendix A.

My take

This is the canonical reference for everything an asset-pricing or econometrics-oriented Markov-switching state-space user actually needs from the control-theory side. The H^n / T-operator framing is the right abstraction: once you accept that the second moment lives in H^n and that one operator T governs MSS, control, and filtering, the same coupled-Riccati template solves all three problems and the duality between control CARE and filtering CARE comes practically for free.

For the CRE asset-pricing project specifically, Chapter 5 is the most load-bearing: it formalizes why the Rao-Blackwellized particle filter is the right architecture. The mode-known Markov jump filter (Theorem 5.5) is exactly the inner Kalman recursion that runs conditional on a regime path; the exponential-blowup observation in Section 5.4.1 is exactly the reason particles must carry regime histories rather than just continuous states; and the Nn-dimensional LMMSE (Theorem 5.9) is the principled mode-unknown alternative when one cannot afford particles at all. The control-CARE / filtering-CARE duality in Appendix A is also the reason the project’s pricing-side coupled Riccati and the filter-side coupled Riccati can both be analysed with the same existence theorem.

Two cautions for reuse: (1) the book consistently assumes mode-known or fully linear filters; the project’s RBPF is genuinely mode-unknown and nonlinear in the regime path, so Chapter 5’s optimality results give bounds on the RBPF rather than its exact analysis. (2) The CARE existence proof in Appendix A is a big abstract machine — for numerical work the iterative scheme X(k+1) = X(X(k), ς(k)) (eq. A.3) is what one actually runs.

Concepts:

Concept aliases / variants captured under the three concepts above: Markov jump filter (mode-known) and LMMSE filter (mode-unknown) are recorded as variants of coupled-algebraic-riccati-equation and markov-jump-linear-system; the H^n operator space, separation principle for MJLS, and exponential filter complexity in T are recorded as aliases / formal-notation entries of those same three concepts. (Honors the importance-5 ≤ 3-new-concept hard limit from /ingest.)

Topics (linked, not created in INIT MODE):

People:

Claims:

Graph View

Backlinks