
Keywords: Measurement error models, news and noise
Abstract:
This paper proposes a simple generalization of the classical measurement error model and studies its implications for ordinary least squares (OLS) regression. The usual model starts with the true variable of interest and adds noise, which we call the classical measurement error (CME) term; see Fuller (1987) or virtually any econometrics textbook. The generalization discussed here incorporates a different kind of measurement error that subtracts signal from the true variable; this new error term is called the Lack of Signal Error, or LoSE for short. This additional term adds some much-needed flexibility to the classical measurement error model: it allows the mismeasured variable to have either more or less variance than the true variable of interest, in contrast to the classical model which imposes that the mismeasured variable have greater variance. This restriction does not hold in some important applications in macroeconomics and elsewhere.
The implications of LoSE for OLS regression are opposite the usual intuition about measurement error, which is applicable to CME only. The CME intuition says that measurement error in the dependent variable
of a regression poses no real problems for standard estimation and inference. Parameter estimates are unbiased and consistent, while hypotheses are more difficult to reject because CME increases the variance of regression residuals and parameter estimates; inference under these
circumstances has a cautious slant. CME in the explanatory variables
causes the real problems for OLS regression, namely attenuation bias and inconsistency. However with LoSE these results
are reversed. For the baseline case considered here, LoSE in the explanatory variables
produces no bias or inconsistency, and similar to CME in
, LoSE in
boosts the variance of regresson residuals and hence standard errors. And, LoSE in the dependent
variable
introduces an attenuation-type bias and inconsistency into the regression under some circumstances (in particular, when the explanatory variables contain some signal missing
from the dependent variable). In addition, LoSE in
shrinks the variance of the regression residuals, thus shrinking parameter standard errors compared to what they would be without this type
of mismeasurement. Standard errors are potentially misleadingly small, complicating inferences about the relation between true
and
. Under these conditions, a more cautious approach to inference than has been taken in the past may be warranted.
Mismeasurement in many types of data used for empirical work in economics and other disciplines may be better described by the generalized measurement error model with LoSE than by the pure classical measurement error model. This paper focuses on US macroeconomic quantity data such as gross domestic product (GDP) and gross domestic income (GDI), which attempt to measure the same underlying concept using different source data; see Fixler and Nalewaik (2007) and Nalewaik (2007a). These data pass through numerous revisions, and the more poorly-measured initial estimates have less variance than the revised estimates, providing a concrete example of measurement error that cannot be CME; see also Mankiw and Shapiro (1986). Section 2 of the paper makes this point while motivating the generalization of the CME model outlined here.
Section 3 of the paper discusses the nature of the source data used to compute US macroeconomic quantity data, and points out some reasons why LoSE may be present in the estimates even after they have passed through all their revisions. The fact that GDP growth should equal GDI growth, but does not in the fully revised quarterly or annual frequency data, proves that some mismeasurement remains in either GDP or GDI growth. The evidence in Fixler and Nalewaik (2007) and Nalewaik (2007a,b) supports the notion that this mismeasurement is largely LoSE, with more LoSE in GDP growth than in GDI growth. Those arguments are recounted briefly in this section, and some simple calculations comparing GDP and GDI growth show that the LoSE in GDP growth is likely substantial: after 1984, at least 30% of the variance of the true growth rate of the economy appears to be missing.
Some of the implications of substantial LoSE in GDP growth are fairly obvious. Realizations of GDP growth are simply less informative about the true growth rate of the economy than many macroeconomists currently believe, given the common but incorrect presumption that fully-revised GDP growth is measured with little error. In a macro forecasting context, true forecast errors are larger, on average, than forecast errors computed using data mismeasured with LoSE. The implications for regression estimation are more complicated, and working out those implications is the main contribution of the paper. This is done in section 4, which illustrates the effect of LoSE in the dependent and explanatory variables under different sets of assumptions. For concreteness, examples of popular regressions from macroeconomics that may conform to each set of assumptions are provided.
In a wide variety of econometric specifications employed in the macroeconomics and finance, variables like GDP growth, investment growth and consumption growth are regressed on asset prices - interest rates, stock price changes, exchange rate changes, etc. These regressions are of particular interest because asset prices potentially capture some signal missing from the mismeasured quantities, implying attenuation-type biases in the coefficients. Section 4 tests for these biases, regressing different output growth measures contaminated with more or less LoSE on a fixed set of stock or bond prices. When the dependent variable is contaminated with more LoSE, the regression coefficients are smaller, and the differences across regressions are often statistically significant. For example, the coefficients increase when we switch the dependent variable from the early GDP growth estimates based on limited source data to later GDP growth estimates based on more-comprehensive data. Tellingly, the coefficients increase again when we switch the dependent variable from GDP growth to GDI growth. The hypothesis that measurement error in the dependent variable does not bias OLS regression coefficients, a core piece of conventional wisdom in the profession, is rejected by the data, just as the paper predicts if the measurement error is LoSE. Section 5 concludes the paper.
Let
be the true value of the variable of interest, while
is an
vector of possibly stochastic variables used to construct
. The mismeasured estimate of the variable of interest is
. In many cases a government statistical agency or some other organization computes
based on information from surveys, administrative records, and other data sources (source data for short); then
will be variables drawn from the source data, possibly including non-linear functions of the original source data.
Under the classical measurement error model,
Under the generalized model of mismeasurement considered here, the mismeasured estimate
is as in Fixler and Nalewaik (2007):
| (1) |
Define the deviation of the variable of interest from its conditional expectation as:
| (2) |
| (3) |
Since the CME is independent of
, it is naturally independent of the LoSE as well. However the LoSE is clearly correlated with
, with
in fact. Taking variances of (3):
| (4) |
The restrictions imposed by the CME model on variances and covariances have obvious drawbacks, as it is easy to think of hypothetical and actual counterexamples. As a hypothetical counterexample, assume that
has positive variance, while the estimate
is just a constant for all
. The estimate
is clearly mismeasured, but the classical measurement error model cannot handle this case. The growth rates of US GDP and GDI provide an actual
counterexample. These estimates pass through numerous revisions that plausibly reduce measurement error, since they incorporate more comprehensive and higher-quality source data. For example, suitable source data is simply unavailable for many components of the "advance" current quarterly GDP
estimate released about a month after the quarter ends. Source data for some of those components is incorporated into the revised "final" current quarterly estimate released about two months later, and higher-quality data are incorporated at subsequent annual and benchmark revisions, likely
bringing the estimate closer to its true value.1 Then an early estimate of GDP growth or GDI growth can be modelled as a later revised estimate (the analog to
) plus a measurement error term, which disappears with revision. Table 1 shows that the initial estimates have less variance than the revised estimates, violating the variance
restrictions of the CME model.2 The generalized model here implies that the bulk of the measurement error is LoSE, as noted by Mankiw and Shapiro (1986).
While the generalized model here is less restrictive than the CME model, some restrictions do remain. Writing:
| (5) |
Before concluding this section, it is worth emphasizing that
need not be an exhaustive information set - i.e. it need not contain all available relevant pieces of information about
. Resource and other constraints certainly preclude this from being the case, and the sections below considering the implications of LoSE allow for this possibility.
Each estimated growth rate of a macro quantity such as gross domestic product (GDP) is an attempt at measuring the change in the value of all relevant economic transactions, in the entire economy, from one fixed time period to the next. For an entity as large as the U.S. economy, this is a daunting, almost mind-boggling task, as the number of transactors and transactions is typically enormous, with little or no information recorded about many of them at high frequencies. Attempts to measure changes in these macro quantities are much more ambitious than attempts to measure similar changes for a single person, household, or even company. Simply due to their broad, universal nature, estimates of macro quantities are likely to miss more information - i.e. be contaminated with more LoSE - than are estimates of micro quantities (although some micro data sources may be contaminated with LoSE as well).6
Of course, the nature of the available source data determines the information content of macro quantity growth rates of interest, and frequency is important in this regard in the case of data from the U.S. National Income and Product Accounts (NIPA).7 The most comprehensive data on GDP and other major NIPA aggregates are only available at the quinquennial frequency (every five years), at the time of the major economic censuses. Even then, resource constraints make true census counts impossible. Many transactions in the underground economy remain unobserved and must be estimated, and some "above-ground" transactions are simply missed by any census.8 At the annual frequency, the GDP source data are typically samples drawn from the census universe. These samples can be quite large, capturing a sizeable fraction of the relevant value of transactions, but they are typically skewed towards measuring the transactions of larger businesses. As such, they may miss variation arising from the transactions of small companies and from businesses starting, shutting down, and operating in the underground economy. The lack of representation of these segments of the economy may add or substract variance to the official estimates, depending on the relative variance of the non-measured segments and their covariance with the measured segments, but this mismeasurement has the potential to add some LoSE to the data. At the annual frequency, and also at the quarterly frequency to some extent, government tax and administrative records are used as an additional source of information about the value of transactions, especially on the income (GDI) side of the accounts. These data can be informative, but underreporting makes them less than fully comprehensive.
At the quarterly and monthly frequency,9 reliance on samples is more pronounced, and the samples are less comprehensive. Smaller samples introduce larger sampling errors, which have traditionally been thought of as introducing CME into the estimates. The samples are typically random, after all, so part of the difference between the population and sample moments is likely random variation uncorrelated with the variation in the population moments. However smaller samples may introduce some LoSE as well, since smaller samples are simply less informative than larger samples: when different segments of the economy behave quite differently, small samples which are not fully representative may miss variation arising from some segments.10In addition, usable data on the value of transactions at a frequency higher than annual is simply unavailable for a substantial share of some NIPA aggregates such as GDP; many of the services categories of personal consumption expenditures (PCE) lack usable source data, for example.11 Quarterly and monthly growth rates are typically interpolated from annual totals, or estimated as "trend extrapolations." The lack of hard information for these categories must introduce some LoSE into the quarterly and monthly estimates.
Some users of quarterly or annual US NIPA data take the view that the variance of CME is negligible after the data have passed through its sequence of revisions, particularly for the more highly aggregated NIPA quantities like GDP growth where any remaining CME variance in its subcomponents may
be diminished by averaging. Absent knowledge of the possible existence of LoSE, this view would imply that the variance of overall mismeasurement is close to zero, since
in the pure CME model.
However there is ample evidence that the variance of overall mismeasurement in the most aggregated NIPA aggregate, GDP, is not close to zero, especially since the mid-1980s. GDI is an alternative estimate of the same quantity, so examining the relation between GDP and GDI provides some direct evidence on mismeasurement. Table 2 shows that, prior to 1984, the variance of each estimate is close to the covariance between the two, for both annual and annualized quarterly growth rates. The two estimates diverge very little, providing little direct evidence of mimeasurement. However after 1984, when the variance of both estimates drops dramatically (see McConnell and Perez-Quiros (2000)), the correlation between the estimates also falls, as the covariance falls relative to the variances on average. This is especially true for the quarterly growth rates, where the correlation falls from 0.93 to 0.68.12Interestingly, the variance of GDI growth also increases relative to the variance of GDP growth. Under the generalized CME model of section 2, this relatively large GDI variance may stem from some combination of two possible sources: (1) a relatively large amount of CME in GDI growth, boosting its variance, and (2) a relatively large amount of LoSE in GDP growth, damping its variance. The evidence favors the latter as the more important source of mismeasurement.
First, consider the results in Nalewaik (2007a), who estimates a two-state bivariate Markov switching model where mean GDP and GDI growth switch with the state; the low-growth states identified by the model encompass recessions as defined by the NBER. The conditional variance of GDI in that model, conditional on the estimated state of the world, is actually slightly lower than the conditional variance of GDP, despite the fact that the unconditional variance of GDI growth is higher. The higher unconditional variance stems from GDI growing faster than GDP in high-growth periods and slower than GDP in slow-growth periods in and around recessions. In other words, GDI growth appears to contain more signal about the state of the world than GDP growth: the larger spread between its high- and low-growth means implies greater informativeness about the state. Greater signal in GDI growth implies some LoSE in GDP growth, relatively more than in GDI growth.
Second, table 1 shows that the variance of GDI growth becomes relatively large only after the data pass through annual and benchmark revisions; in the earlier current quarterly estimates, the variance of GDP growth actually exceeds the variance of GDI growth. Since the revisions plausibly bring estimated GDI growth closer to the truth, they must either reduce LoSE, which would increase its variance, or reduce CME, which would decrease its variance. The increase in variance from the revisions, then, is likely a decrease in LoSE, suggesting the relatively large variance of revised GDI stems from relatively less LoSE. The annual and benchmark revisions appear to add more signal to GDI growth than GDP growth, increasing the variance of GDI growth relative to GDP growth.13 Fixler and Nalewaik (2007) discuss the revisions evidence in more detail, testing the hypothesis that the idiosyncratic variation of GDI growth is purely CME and rejecting at conventional significance levels. This again implies some LoSE in GDP growth.
To get a sense of the magnitude of the potential variance missing from GDP growth due to LoSE, assume that the CME variance in each estimate is negligible, so the differences between GDP and GDI growth stem entirely from differential LoSE:
Going forward, if these post-1984 variances and covariances are the norm, the implications of a potentially non-trivial amount of LoSE in macro data such as GDP growth should be taken seriously. For estimation and inference, the post-1984 portion of many samples will become increasingly large and important. The next section explores the implications of LoSE for estimation and inference using the most ubiquitous tool in econometrics: OLS regression.
Consider ordinary least squares estimation of the relation between a mismeasured variable
and a
set of mismeasured explanatory variables
, using a
sample of length
. When stacking together the observations, time subscripts are dropped for convenience:
![]() |
| (6) | |||
| (7) |
| (8) | |||
| (9) |
The inconsistency of
can be corrected by instrumenting with a
set of instruments
, with
, if the instruments meet the following set of assumptions:
| (10) |
Given the traditional focus on mismeasurement in
on regression estimation, we begin with this subsection making the following assumption, in addition to assumption 1:
The OLS regression estimator in this case is:
Of course, the CME in
produces the usual attenuation bias. By way of review, and for comparison with later results:
| (11) | |||
| (12) |
To focus more tightly on the implications of LoSE, the remainder of this subsection considers the case of no CME in
:
![]() |
|||
Since nominal asset prices are measured with virtually no error, they provide a candidate
variable meeting the assumptions of this section.16 Given the evidence from the prior section indicating that macroeconomic quantities are measured with LoSE, a specification such as the human capital CAPM, essentially a regression of stock
prices on labor income growth, may be well represented by the regression model developed here.
In addition to assumption 1, this subsection makes the following assumptions:
Since
, we have:
in this case. The LoSE impacts only
, so
, and
. The OLS regression estimates
as:
The standard errors around the point estimates are more interesting. For the variance of the point estimates,
, and:
LoSE in the dependent variable has an opposite effect, decreasing the variance of the regression residuals and parameter estimates. Measurement error of this type actually increases the power of hypothesis tests. Is more power a good thing? I would argue not. If a hypothesis is rejected at a prescribed significance level, an immediate concern is that the rejection was due to LoSE-induced power. If the data were free of mismeasurement, the variance of the parameter estimates would be larger and the rejection may not have occured.
Note that the parameter variances here are correct for the parameters governing the relation between mismeasured
and
, and are not biased down in the same sense as they would be if, for example, we ignored positive autocorrelation in the residuals of the regression. However in most applications the parameters of interest are those governing the relation between true
and
, and the econometrician uses the mismeasured data to make inferences
about that true relation out of necessity, because the mismeasured data are all that is available. Under the assumptions of this subsection, the parameters estimated using the mismeasured data are unbiased and consistent for the parameters governing the true relation, but the parameter variances
are smaller than they would be without the LoSE in
. Standard errors computed using data mismeasured in this way then give a misleading sense of precision about the relation between true
and
, and the power of hypothesis tests is misleadingly large.
An example clarifies some of these issues. Imagine a situation where the econometrician has access to true
,
, and a list of other variables
that are orthogonal to
but related to
. Hypotheses about the relation between
and
are of interest. Testing hypotheses using parameter estimates and standard errors from a regression of
on
is the natural and correct way to procede. But consider the following two step procedure: (1) regress
on
and some subset of
, and compute predicted values which we call
, and then (2) regress
on
, testing hypotheses about the relation between
and
using standard errors from this second regression. The parameter estimates in the second regression are the same as in a regression of
on
, but the standard errors are smaller and tests have greater power because the
first stage generates LoSE in
. In fact, by making
arbitrarily small, the
standard errors can be made arbitrarily small and any hypothesis may be rejected; setting
to the null set drives the standard errors to zero. Artificially generating LoSE in
in this fashion is clearly ridiculous, and makes hypothesis tests meaningless.
In reality, if the data meet the conditions of this section, the econometrician starts out in stage (2) of this two step procedure, sadly without access to
. Something analogous to
step (1) has already taken place with the creation of the data
, by a government statistical agency or some other entity. The main issue is not materially different if it is a government
statistical agency rather than the econometrician who has generated the LoSE in
.
If the variance of LoSE
were known, the most prudent course of action would be to increase the estimated variance of
by
. This essentially brings us back the pure CME case, where econometricians are comfortable using standard procedures for inference.19Unfortunately,
is typically unknowable; the results in sections 3.2 and 4.3.1 for measured GDP growth provide only an approximate lower bound on
. The bottom line is that inferences are fundamentally less definitive when the dependent variable is contaminated with LoSE.
In a forecasting context, it should be noted that the LoSE also shrinks the variance of out-of-sample forecast errors. The actual variance of the out-of-sample forecast error for the true variable of interest,
, with
estimated using mismeasured
, is
. However the LoSE reduces
this variance by
, and if the increase in variance from CME
does not offset this increase, the forecast errors computed using mismeasured
give a misleading sense of precision: the deviations of
from the forecasts are larger than those mismeasured forecast errors suggest. For example, table 1 shows that for GDP growth forecasts, forecast errors computed using the
"advance" GDP growth estimates will be smaller, on average, than the true forecast errors.
Regressions of mismeasured macroeconomic quantities like GDP growth on time trends, dummies, and other deterministic variables certainly meet the conditions outlined in this subsection. These types of regressions are run most frequently in a forecasting context, where the primary goal is to
forecast
, not to estimate the deep structural parameters of an economic model relating ![]()