In evaluating population estimates, it is necessary to define measures of their quality. Population estimates can be characterized by four aspects of quality: bias, scale, outliers and extreme errors. This paper axiomatically examines each of these quality aspects and describes and criticizes statistics commonly used to measure them. An important distinction is drawn between extreme errors and outliers: the former are simply the largest errors, while the latter, normally large, indicate possible problems with their data generation processes. The entire framework is based on the assumption of nonidentically distributed errors, the implications of which are discussed.