Forecasting Real Estate Prices

Problem

How predictable are residential and commercial real estate prices, and what predictors and econometric specifications actually work? Real estate is the largest US asset class (~$25Tatendof2011,ofwhich$16T residential — larger than the US stock market), and recent boom/bust dynamics have shown its capacity to drive macroeconomic outcomes. Yet predicting it is hard for two reasons: (i) the underlying asset is extremely heterogeneous, illiquid, and traded with very large transaction costs, and (ii) reliable price data exists only at low frequency over short samples — precluding most of the standard out-of-sample forecasting toolbox. Before any predictability question can even be asked, the chapter must address the more basic problem of how to construct a reliable, “constant-quality” price index from sparse, heterogeneous transaction data.

Key idea

The chapter is a methodological survey organized around three layers:

1. Index construction is part of the model: median-price, repeat-sales, hedonic, hybrid, and stock-market-based REIT indices each impose distinct statistical artifacts on the resulting series (e.g. Case-Shiller has serial correlation ~0.94 in growth rates; Census median has -0.52). The choice of index materially shapes any predictability finding, so the literature cannot be read without first understanding how the data were constructed.
2. Three families of predictive regressions: lagged returns (weak-form efficiency tests), valuation ratios (rent-price, price-income — motivated by Campbell-Shiller present-value identities), and economic predictors (income, population, construction costs, mortgage rates, monetary policy, leverage). Each is reviewed in turn against six aggregate US indices (Census, Case-Shiller, FHFA/OFHEO for residential; NCREIF, TBI, REIT for commercial), 1991:Q2-2010:Q4.
3. REITs as a special case: because REITs trade daily with low costs and long sample (1980-2010), they enable the full small-sample-bias-corrected GMM and out-of-sample machinery (Stambaugh correction, Lettau-Van Nieuwerburgh cross-horizon restrictions, Welch-Goyal OOS R^2, Campbell-Thompson sign/positivity constraints) that the rest of the literature cannot run.

Method

Index construction (Section 2): Repeat-sales models price aggregate log-index changes from a regression of the form y_i = β X_i + u_i where X_i is a dummy vector for transaction periods. OLS is GLS-inefficient because Var(u_i) ∝ T_i. The Case-Shiller (1987) Weighted Repeat Sales (WRS) correction adds a constant noise term 2σ_n² and uses a 3-step WLS-then-GLS procedure. Hedonic models regress log price on a vector of property attributes plus time dummies; the time-dummy coefficients identify the quality-adjusted index. Repeat-sales is shown to be a special case of hedonic where shadow prices are assumed time-invariant. Hybrid models (Case-Quigley 1991, Hill-Sirmans-Knight 1997) jointly estimate both via GMM. The chapter also covers Kalman-filter-based latent-price approaches (Brown et al. 1997; Giaccotto-Clapp 1992) and spatial econometric models for geographic dependence.

Predictive regressions (Section 3): Long-horizon r_{t+1:t+T} = α(T) + β(T) r_{t-T+1:t} + ε for serial correlation; r_{t+1:t+T} = β_r(T) hp_t + τ for cap-rate predictability (where hp_t = log(rent/price) is the log cap rate); and analogous equations with macro/local conditioning variables. Regime-switching specifications (Section 3.1) use Hamilton (1989) AR(1) models r_t = c_{s_t} + φ_{s_t} r_{t-1} + ε_t, ε_t ~ N(0, σ²_{s_t}) with 2-3 regimes capturing booms/busts. Crawford-Fratantoni (2003) compare regime-switching to ARIMA/GARCH on state-level OFHEO data; regime models win in-sample but lose out-of-sample due to overfitting.

REIT predictive system (Section 4): The standard system r_{t+1} = μ_r + β_r x_t + τ^r, Δd_{t+1} = μ_d + β_d x_t + τ^d, x_{t+1} = μ_x + φ x_t + τ^{dp} is estimated with the Stambaugh (1999) small-sample bias correction and the Lettau-Van Nieuwerburgh (2008) cross-horizon restriction β_r(T) = β_r (1-φ^T)/(1-φ). The Campbell-Shiller present-value identity β_r - β_d = 1 - ρφ provides additional GMM cross-equation restrictions. Out-of-sample evaluation uses the Welch-Goyal OOS R² with Campbell-Thompson sign and non-negativity constraints, on a 1980:01-1995:12 estimation window with monthly recursive expansion through 2010:12.

Results

Index summary statistics (Table 2): Repeat-sales indices (Case-Shiller, FHFA) have growth-rate AR(1) of 0.5-0.94, far higher than median indices (-0.52 for Census). REITs have AR(1) similar to small-cap equity (~0.1-0.2). The chapter emphasizes that much of the repeat-sales serial correlation is construction artifact, not market inefficiency.

Stylized facts (Section 3, Section 6):

• Repeat-sales and hedonic price changes are strongly positively serially correlated at monthly/quarterly frequencies; median indices show negative serial correlation; REITs behave like small-cap stocks.
• Transaction costs (~6% of property value) are typically too large for observed serial correlation to translate into trading profits in non-REIT markets.
• Valuation ratios (rent-price, income-price) have weak in-sample predictive power for returns, mostly attributable to time-varying expected returns rather than exploitable inefficiencies.
• Local economic variables (income growth, population, construction costs, zoning) have sizeable in-sample R² (Case-Shiller 1990 reach 33-62% R² for 4-quarter excess returns) but their out-of-sample performance is largely unexplored.
• For REITs: weak in-sample return predictability, much stronger rent/dividend-growth predictability (R² up to 40-50% for industrial and office REITs over 1980-2010). Out-of-sample REIT return forecasts underperform the unconditional mean for almost all predictors except the lagged stock market return; sign/positivity constraints provide some improvement. The dividend-price ratio is a much stronger predictor of REIT cash flows than of REIT returns.
• Leverage (Section 5.1) is positively related to future returns and amplifies the response of house prices to economic shocks, especially in high-LTV cities (Lamont-Stein 1999). Credit supply (Mian-Sufi 2009; Favilukis-Kohn-Ludvigson-Van Nieuwerburgh 2012) is “by far the most powerful” forecaster of contemporaneous house price growth in 2000-2010, with R² > 0.40 in 1-4 quarter ahead regressions.
• Monetary policy (Section 5.2): Tight monetary policy reduces house prices and GDP (Iacoviello 2005); the effect is stronger and more rapid in housing than in the rest of the economy; the regional response is heterogeneous (Fratantoni-Schuh 2003 HAVAR — peak responses vary by >1pp, mean lags by >1 year). Del Negro-Otrok (2007) attribute the recent boom primarily to local rather than national factors and find a small role for monetary policy.

Cross-sectional dispersion (Section 3.3): MSA-level FHFA regressions show average R² of ~49% (matching the national 48%), but range from 25% (St. Louis) to 79% (Los Angeles). The first principal component of the 25-MSA covariance explains ~70% of variance; this rises sharply during 2008-2010, consistent with a crisis-driven common factor.

Limitations

• Data limitations are first-order: most studies use samples too short for meaningful out-of-sample evaluation (the only deep-history exception is Eichholtz 1997’s bi-annual Amsterdam index 1628-1973). Most cited work is in-sample, with cross-sectional pooling substituting for time-series length.
• No structural model: predictive regressions are reduced-form. They cannot identify the economic source of forecastability (demand vs. supply vs. credit conditions vs. monetary policy) without additional restrictions.
• Index revisions: repeat-sales index estimates change as new transactions arrive (1-2pp per year on annualized series), which renders out-of-sample tests hard to evaluate cleanly.
• Selection biases: repeat-sales selection (homes that transact twice are not representative); hedonic omitted-variable bias; appraisal smoothing in NCREIF.
• The chapter does NOT propose a new structural macro-finance model — it is explicitly a survey, not a contribution to the term-structure or asset-pricing literature on real estate.

Open questions

• How to construct a quality-adjusted index that does not introduce artificial serial correlation. The current state-of-the-art repeat-sales filtering may be appropriate for tracking the market state but is unsuitable for forecasting because the construction-induced AR(1) confounds true predictability.
• Are the predictive results robust out-of-sample? The chapter explicitly flags this as the open question for non-REIT real estate.
• Are there asymmetric responses of prices to positive vs. negative economic shocks (especially through the leverage channel)?
• Can a structural model with credit constraints, heterogeneous agents, and monetary policy generate the empirically-observed amount of return and rent-growth predictability?
• What is the right way to combine the high-frequency, market-priced REIT signal with the low-frequency, slow-moving direct-real-estate series? The chapter notes that REITs and direct CRE returns are strongly correlated (MacKinnon-Al Zaman 2009 find REITs are nearly redundant once direct CRE is in the portfolio) but the literature has no agreed framework.

My take

For the CRE asset pricing project this chapter is the standard reference on the empirical landscape we are trying to model. Three implications for the SimMdlPrices model and the global-opt pipeline:

1. Asset heterogeneity is real, not noise: the chapter documents strong cross-property-type differences in predictability — apartments vs. industrial vs. office vs. retail all behave differently both in return and in cash-flow forecastability. This justifies the project’s 3-asset (apartments / industrial / office) decomposition and warns against pooling them.
2. Cap rates are the right state variable for direct CRE: the chapter identifies the log rent-price ratio (a.k.a. log cap rate) as the theoretically-grounded predictor via the Campbell-Shiller present-value identity, and notes its statistical significance is weak in-sample but improves with structural restrictions. The project’s no-arbitrage NK asset-pricing block is exactly the structural model the chapter calls for: it imposes that cap rates load on macro state variables (output gap, inflation, short rate) via Riccati exponential-quadratic loadings.
3. Regime-switching is well-motivated but data-hungry: Crawford-Fratantoni (2003) found regime-switching wins in-sample but loses out-of-sample to ARIMA. This is consistent with the project’s choice of 4 compound regimes from two 2-state chains (small parameter count) rather than free-form high-state regime models, and with the difficulty of identifying regime structure from a 119-quarter panel. The chapter does NOT cover the monetary-policy-regime literature that the project draws from (Bansal-Zhou 2002, Dai-Singleton-Yang 2007 — both in our documents library).
4. The chapter is silent on the structural-NK direction: it does not review the macro-finance/RE asset pricing literature (Iacoviello-Neri, Piazzesi-Schneider-Tuzel) in any depth. So this paper is the empirical reference, not the theoretical reference, for our project.

Related

real-estate-price-indices — central concept introduced here; the chapter catalogs median, repeat-sales (Bailey-Muth-Nourse, Case-Shiller WRS, FHFA), hedonic, hybrid, and stock-market-based (REIT) indices.

supports: real-estate-returns-show-short-run — short-run momentum and long-run reversal in real estate returns, the chapter’s strongest cross-study stylized fact.

eric-ghysels alberto-plazzi walter-torous rossen-valkanov

expected-returns-expected-growth-rents-commercial — Plazzi-Torous-Valkanov (2010), the key underlying paper on CRE cap rate predictability that this Handbook chapter surveys extensively. Applies the dynamic Campbell-Shiller model to NCREIF transaction data across 53 MSAs.

other-commercial-real-estate-boom-bust — Duca-Ling (2015), extends the present-value cap rate framework with credit-augmented VECM and regulatory capital channels.

cre-cap-rate-decomposition — the analytical concept underlying the Campbell-Shiller / Gordon Growth Model decomposition applied to CRE cap rates.

Forward-link backlog (concepts to be created by future ingests when more papers in this area land — kept here so we don’t have to re-read the chapter):

• “real estate price forecasting” (predictive regressions, valuation ratios, economic predictors)
• “regime-switching real estate models” (Hamilton 1989-style applied to housing booms/busts; Crawford-Fratantoni 2003 is the canonical comparator)
• “residential vs commercial real estate distinction” (different trading dynamics, REIT vs NCREIF/TBI vs Case-Shiller; Geltner-Miller 2006)
• “Campbell-Shiller present-value identity” (foundational; chapter applies it to log cap rates rather than log dividend-price ratios)
• “Stambaugh 1999 small-sample bias correction” (used in the REIT regression block)