Abstract/Summary

Optimal estimation (OE) is a core method in quantitative Earth observation. The optimality of OE depends on the errors in the prior, measurements and forward model being zero mean and having well-known error covariance. Often these assumptions are not met. We show how to use matches of satellite observations to in situ reference measurements to estimate parameters for use in OE that bring the retrieval framework closer to the theoretical optimality. This is done by retrieving bias correction and error covariance parameters. Bias correction parameters for some components of the retrieved state and for the satellite radiances are anchored by the in situ reference measurements, and are obtained by a modification of Kalman filtering. Error covariance matrices for the prior state and for the observation-simulation difference are iteratively obtained by applying equations for diagnosing internal retrieval consistency. The theory is applied to the case of OE of sea surface temperature from a sensor on a geostationary platform. Relative to an initial OE implementation, all measures of retrieval performance are improved when the optimised OE is tested on independent data: mean difference from validation data is reduced from -0.08 K to -0.01 K, and the standard deviation from 0.47 to 0.45 K; retrieval sensitivity to sea surface temperature increases from 71% to 76%; and a 20% underestimation of retrieval uncertainty is corrected. Perhaps more significant than the quantitative improvements are the coherent new insights into the forward model simulations and prior assumptions that are also obtained. These include estimates of prior bias in the absence of in situ information, an important consideration when in situ information is not globally distributed. Biases and lack of information about error covariances arise in remote sensing very often. While illustrated here for a particular case, the principles and methods we present for constraining that lack of knowledge systematically using ground truth will be widely applicable in remote sensing.