LeatherSagiKnowledgebase

Constrained Estimation Branch: m_g Pin + MC Lookup + BOBYQA in 54-D

3 min read

Objective

Establish a working constrained MLE pipeline for the full 54-parameter CRE model. Three innovations combined: (1) the IS-stationarity constraint pinning m_g (55 → 54 free params), (2) MC cap-rate lookup tables replacing the BC Riccati recursion, (3) BOBYQA derivative-free trust-region optimizer supporting per-eval MC rebuild.

Setup

  • m_g constraint: pins m_g = phi*(r_bar - pi_bar), removing a single direction with gradient 2.52e11 that defeats all gradient-based optimizers. Net: 70,350x conditioning improvement.
  • MC lookup: per-eval rebuild at H=150, R=200, 5.2 s/eval. Fixed-lookup drift is catastrophic (+5269 nats).
  • Optimizers tested: NelderMead, BOBYQA, L-BFGS (with FD gradients / AD / EM-style rebuild).
  • TSM baseline comparison: {Hamilton, RBPF} x {BC, MC} four-combo experiment at 29-parameter TSM-only level.

Procedure

  • TSM baseline: 4-combo sweep confirms BC-constrained within 3.7 nats of unconstrained; MC pricing changes the likelihood surface (17.4 nats worse at the BC-optimal theta).
  • Constraint validation: 5-test suite; T1 FAIL (expected — constraint shifts optimum), T2 PASS (MC lookup runs), T3 PASS (regime probabilities preserved), T5 PASS (70,350x conditioning).
  • NelderMead: -4255 in ~1000 iterations, classic 54-D stagnation.
  • BOBYQA: warm-start from NM best, 4000 evals in 4.1 hrs → NLL -5031.62 (1542-nat improvement from warm start).
  • L-BFGS: all 4 variants fail (fixed lookup drift, per-eval MC too slow at 36.9-158 s/eval, EM-style rebuild diverges).
  • RBPF verification (N=3000, H=500, R=500, 10 seeds): -4950.56 +/- 14.97 at BOBYQA best.

Results

  • BOBYQA best NLL: -5031.62 (Hamilton H=150) / -4989.33 (H=500 re-eval). RBPF verification: -4950.56 +/- 14.97.
  • m_g constraint: 70,350x gradient conditioning improvement; 575.86-nat LL shift (expected — optimum moves in the constrained manifold).
  • NelderMead: stagnates at -4255 in 54-D (simplex collapses without finding the basin).
  • L-BFGS: all variants FAIL — fixed-lookup drift +5269 nats; per-eval FD 36.9 s/eval + steps into infeasible region; AD 158 s/eval prohibitive.
  • BOBYQA dominates for noisy MC objectives in 54-D: steady convergence trace, no stagnation plateaus.
  • Full model vs TSM-only: NLL -5032 vs -3798 (1234-nat improvement from adding cap rates and income).

Analysis

This branch establishes the production estimation pipeline: m_g pin + per-eval MC rebuild inside BOBYQA. The three innovations are individually necessary and jointly sufficient: without the m_g constraint, gradient-based methods are hopeless; without per-eval rebuild, fixed lookups drift catastrophically; without BOBYQA, neither NelderMead nor L-BFGS navigates 54-D reliably. The -5031.62 NLL on real data is the baseline that the subsequent simulated-data arc improves upon.

Claim updates

  • forward-solution-unique-fundamental-ree-with-nbc: tested_by, strength strong. The full constrained estimation pipeline operates inside the forward-solution/NBC-feasible region, and the 70,350x conditioning improvement from the m_g constraint confirms that the IS-stationarity manifold is the correct estimation subspace.
  • bond-dividend-strip-prices-regime-switching: tested_by, strength moderate. Per-eval MC rebuild replaces BC Riccati; the 5269-nat fixed-lookup drift demonstrates that Riccati-based lookups are not reusable across theta.

Follow-up

Graph View

Backlinks