Real Estate Price Indices

Definition

A real estate price index is an estimated time series of the aggregate price level of a class of real estate assets (residential or commercial, national or local) constructed from individual transaction or appraisal data on heterogeneous, infrequently-traded properties. Because no single property trades every period and no two properties are identical, the estimator must simultaneously address (i) the missing-data structure of sparse repeat sales and (ii) the quality heterogeneity of the underlying asset. Different choices about how to do this give rise to four main families of indices: median-price, repeat-sales, hedonic, and stock-market-based (REIT) indices, plus hybrids that combine the first three.

Intuition

Suppose we want a single number for “the value of a typical US single-family home” each quarter. We cannot just average all transactions, because the mix of which homes are sold changes over time and with the business cycle — high-quality homes tend to come on the market in expansions. The four families differ in how they remove this composition bias:

Median-price: take the median transaction price each period. Easy to compute but does not adjust for quality, so the index conflates composition shifts with true price changes.
Repeat-sales: only use homes that sold at least twice; the same home acts as its own control for unobserved quality. Throws away most of the data and can be biased by selection (homes that turn over frequently are not representative).
Hedonic: regress log price on a vector of quality attributes (lot size, square footage, bedrooms, location dummies); the time fixed effects are the index. Uses all transactions but is sensitive to omitted attributes and to functional-form choices.
Hybrid: estimate repeat-sales and hedonic models jointly under cross-equation restrictions to combine information.
Stock-market-based / REIT: read the index off the trading prices of publicly-traded REIT securities. Daily, low transaction costs, but REITs are levered small-cap equities and may not represent the underlying property market.

The choice is not innocent: the resulting time series have very different serial-correlation, volatility, and revision properties, and any predictability finding can be an artifact of construction rather than a real economic regularity.

Formal notation

Repeat-sales (Bailey-Muth-Nourse 1963): Let $p_{i, t} = p_{m, t} + e_{i, t}$ where $p_{m, t}$ is the unobserved log market index and $e_{i, t}$ is a property-specific drift with iid increments $ε_{i, t}$ of variance $σ_{ε}^{2}$ . For a property with two sales at $t_{i}$ and $t_{i} + T_{i}$ :

p_{i, t_i + T_i} - p_{i, t_i} = (p_{m, t_i + T_i} - p_{m, t_i}) + Σ_{τ} ε_{i,τ}

This motivates the cross-sectional regression $y_{i} = β X_{i} + u_{i}$ where $X_{i}$ is a $T \times 1$ dummy vector with +1 at the second sale, -1 at the first, 0 elsewhere. The estimated $\betâ$ is the index. OLS is inefficient because $Va r (u_{i}) = T_{i} σ_{ε}^{2}$ , so the BLUE is GLS with weights 1 / sqrt(T_i).

Weighted Repeat Sales (Case-Shiller 1987): Adds a market-wide noise term $n_{i, t}$ with variance $σ_{n}^{2}$ capturing imperfections: $p_{i, t} = p_{m, t} + e_{i, t} + n_{i, t}$ . Then $Va r (u_{i}) = 2 σ_{n}^{2} + T_{i} σ_{ε}^{2}$ . The 3-step estimator is OLS → WLS regression of squared residuals on time interval (estimating $σ_{n}^{2}$ and $σ_{ε}^{2}$ ) → final GLS with the estimated weights.

Hedonic semi-log (Rosen 1974 framework): $p_{i, t} = β Z_{i} + δD + ε_{i, t}$ where $Z_{i}$ is a $C \times 1$ vector of property attributes (with constant) and $D$ is a $T - 1 \times 1$ vector of time dummies. The index is $\deltâ_t$ for each period; $\betâ$ is the vector of shadow prices of attributes.

Repeat-sales as a constrained hedonic (Meese-Wallace 1997): if shadow prices are constant across the two transactions of property i, differencing the two hedonic equations yields exactly the repeat-sales regression. So repeat-sales imposes the testable restrictions that (i) homes that sell twice are representative, and (ii) attribute shadow prices do not change between sales.

Variants

Bailey-Muth-Nourse (1963) — original OLS/GLS repeat-sales.
Case-Shiller (1987) Weighted Repeat Sales (WRS) — adds the market noise term, used by S&P/Case-Shiller Home Price Index.
FHFA / OFHEO HPI — variant of WRS that adds a quadratic term in the second-stage residual regression; uses Fannie/Freddie conforming mortgages so excludes jumbos and subprime.
CoStar CCRSI / Moody’s-REAL CPPI — repeat-sales applied to commercial properties.
NCREIF NPI — appraisal-based commercial index, value-weighted by property market value, smoothed (responds with a lag to true market).
MIT TBI (Fisher-Geltner-Pollakowski 2007) — two-stage hedonic that uses NCREIF appraisals as a hedonic variable to construct transaction-frequency commercial index without the appraisal lag.
Goetzmann (1992) Bayesian repeat-sales — shrinkage / Bayesian alternatives to GLS.
Goetzmann-Spiegel (1995) — adds a “non-temporal” intercept to capture sale-time renovations.
Hill-Sirmans-Knight (1997) — joint MLE of hedonic depreciation and AR(1) repeat-sales errors; reported large efficiency gains.
Spatial-econometric variants (Pace et al. 1998, 2000; Caplin et al. 2008) — add geographic correlation structure.
Kalman-filter latent-price (Brown-Song-McGillivray 1997; Giaccotto-Clapp 1992) — treats the true index as a latent state estimated from noisy transaction prices.

Comparison

Family	Data needed	Key assumption	Statistical property
Median	Just transaction prices	None on quality	Volatile, biased by composition
Repeat-sales	Two-or-more sales of same property	Quality constant between sales; properties that sell twice are representative	High serial correlation by construction
Hedonic	Attribute data on each transaction	Functional form correctly specified; no omitted attributes	Sensitive to specification
Hybrid	Both	Joint identification	More efficient when both data sources available
REIT	Market trading prices	REIT cash flows reflect underlying real estate	Equity-like dynamics, daily, but levered & small-cap

The Case-Shiller (repeat-sales) index has growth-rate $A R (1) \approx$ 0.94; the Census median has $A R (1) \approx$ -0.52; REITs have AR(1) similar to small-cap equity. These differences are NOT primarily about market behavior — they are construction artifacts.

When to use

Index construction is part of the model, not a preprocessing step. Always check whether the serial correlation in the index is consistent with the index’s known construction artifacts before claiming it as evidence of market inefficiency.
For forecasting, repeat-sales and hedonic indices are problematic because their construction-induced serial correlation crowds out the signal you are trying to forecast. The chapter recommends using lower frequencies (annual rather than quarterly) and being very careful with inference.
For measuring the current state of the market, repeat-sales and TBI are appropriate.
For higher-frequency dynamics and out-of-sample testing, REIT indices are the only game in town despite the leverage and small-cap caveats.

Known limitations

Spurious autocorrelation: repeat-sales price changes can be serially correlated even if the true price follows a random walk (Webb 1981, Goetzmann 1992 — autocorrelation can approach -0.5 asymptotically depending on sale timing).
Index revisions: as new sales are added, historical repeat-sales estimates change. Revisions of 1-2pp on annualized series are typical (Abraham-Schauman 1991), making real out-of-sample evaluation hard because the original “vintage” is no longer available.
Selection bias: homes that sell twice are not representative (Clapp-Giaccotto-Tirtiroglu 1991, Gatzlaff-Haurin 1998, Korteweg-Sorensen 2011). Renovations between sales are usually filtered out by ad-hoc rules (e.g., excluding sales within 6 months).
Geometric vs arithmetic average: hedonic and repeat-sales both estimate the geometric average return, which is below the log of the arithmetic average return by Jensen’s inequality. The bias does not vanish with sample size.
Appraisal smoothing: NCREIF-style appraisal indices respond with a lag and are much smoother than true transaction prices (Fisher 2005), introducing autocorrelation that has nothing to do with market efficiency.
Thin markets: when very few properties sell in a period, the regression matrix is near-singular; standard practice is to coarsen the time grid, which induces additional autocorrelation in the high-frequency returns.

Open problems

How to construct an index that is simultaneously quality-adjusted, spatially representative, free of construction-induced autocorrelation, and revisable in a forecast-friendly way. No current method achieves all four.
How to combine high-frequency REIT signals with low-frequency direct CRE indices in a single statistical framework that respects both the leverage gap and the appraisal lag.
Whether spatial-econometric and machine-learning approaches (e.g. large-database hedonic with random-forest functional forms) actually improve out-of-sample performance over WRS once selection bias is accounted for.

Key papers

ghysels-forecasting-real-estate-prices — Handbook chapter containing the canonical taxonomy and comparison of all four families (Sections 2.1.1-2.1.6).
cap-rate-please-explain — NCREIF (2011) practitioner note explaining the six NCREIF cap rate definitions and the 40 bp systematic differences between appraisal vs. transaction and equal- vs. value-weighted series.
expected-returns-expected-growth-rents-commercial — Plazzi-Torous-Valkanov (2010) use NCREIF transaction-based data across 53 MSAs, exploiting the advantages of transaction over appraisal data for return predictability analysis.

My understanding

For the CRE asset pricing project, the relevant index choice is the NCREIF-derived TBI hedonic transaction index for direct CRE (the project’s capData), supplemented by the four asset-type panels (apartments, industrial, office; the chapter notes retail is the fourth NCREIF type but our model uses three). The construction-induced autocorrelation in TBI is one reason the project’s RBPF needs the $cr i s i s_{r} an g e = 57 : 70$ , $cr i s i s_{s} c a l e = 10.0$ adjustment to widen cap-rate observation noise during the 2008-2009 window — the appraisal-smoothing-driven serial correlation in NCREIF/TBI is least informative exactly when the market is moving fastest. The Ghysels et al. chapter does not directly cover the state-space / Kalman approach the project uses, but it cites the same methodological ancestry (Engle-Lilien-Watson 1985; Brown-Song-McGillivray 1997; Giaccotto-Clapp 1992).

LeatherSagiKnowledgebase

Explorer