Population and Dwelling Count Highlight Tables, 2011 Census

Data quality

The 2011 Census was a large and complex undertaking. While considerable effort was taken to ensure high standards throughout all collection and processing operations, the resulting estimates are inevitably subject to a certain degree of error. Users of census data should be aware that such error exists, and should have some appreciation of its main components, so that they can assess the usefulness of census data for their purposes and the risks involved in basing conclusions or decisions on these data.

Errors can arise at virtually every stage of the census process, from the preparation of collection materials through data processing, including the listing of dwellings and the collection of data. Some errors occur at random, and when the individual responses are aggregated for a sufficiently large group, such errors tend to cancel out. For errors of this nature, the larger the group, the more accurate the corresponding estimate. It is for this reason that users are advised to be cautious when using small area estimates. There are some errors, however, which might occur more systematically, and which result in 'biased' estimates. Because the bias from such errors is persistent no matter how large the group for which responses are aggregated, and because bias is particularly difficult to measure, systematic errors are a more serious problem for most data users than the random errors referred to previously.

For census data in general, the principal types of error are as follows:

1. coverage errors, which occur when dwellings or individuals are missed, incorrectly enumerated or counted more than once;
2. non response errors, which result when responses cannot be obtained from a certain number of households and/or individuals, because of extended absence or some other reason or when responses cannot be obtained from a certain number of questions in a complete questionnaire;
3. response errors, which occur when the respondent, or sometimes the census representative, misunderstands a census question, and records an incorrect response or simply uses the wrong response box;
4. processing errors, which can occur at various steps, especially during data capture, when responses are transferred from the census questionnaire in an electronic format, by an optical character recognition system or by key entry operators; during coding, when 'write in' responses are transformed into numerical codes; during imputation, when a 'valid', but not necessarily correct, response is inserted into a record by the computer to replace missing or 'invalid' data ('valid' and 'invalid' referring to whether or not the response is consistent with other information on the record).

The above types of error each have both random and systematic components. These components may be significant.

Coverage errors

Coverage errors affect the accuracy of the census counts, that is, the sizes of the various census universes: population, families, households and dwellings. While steps have been taken to correct certain identifiable errors, the final counts are still subject to some degree of error because persons or dwellings have been missed, incorrectly enumerated in the census or counted more than once.

Missed dwellings or persons result in undercoverage. Dwellings can be missed because of the misunderstanding of collection unit boundaries, or because either they do not look like dwellings or they appear uninhabitable. Persons can be missed when their dwelling is missed or is classified as vacant, or because the respondent misinterprets the instructions on whom to include in the questionnaire. Some individuals may be missed because they have no usual residence and did not spend census night in a dwelling.

Dwellings or persons incorrectly enumerated or double-counted result in overcoverage. Overcoverage of dwellings can occur when structures unfit for habitation are listed as dwellings (incorrectly enumerated), when there is a certain ambiguity regarding the collection unit boundaries or when units (for example, rooms) are listed separately instead of being treated as part of one dwelling (double-counted). Persons can be counted more than once because their dwelling is double counted or because the guidelines on whom to include on the questionnaire have been misunderstood. Occasionally, someone who is not in the census population universe, such as a foreign resident or a fictitious person, may, incorrectly, be enumerated in the census. On average, overcoverage is less likely to occur than undercoverage and, as a result, counts of dwellings and persons are likely to be slightly underestimated.

For the 2011 Census, three studies are used to measure coverage error. In the Dwelling Classification Study, dwellings listed as vacant are revisited to verify that they were vacant on Census Day, and dwellings whose households were listed as non-respondent are revisited to determine the number of usual residents and their characteristics. Adjustments are made to the final census counts to account for households and persons missed because their dwelling was incorrectly classified as vacant and for dwellings classified as non-respondent. Despite these adjustments, the final counts still may be subject to some undercoverage. Undercoverage tends to be higher for certain segments of the population, such as young adults (especially young adult males) and recent immigrants. The Reverse Record Check Study is used to measure the residual undercoverage for Canada, and each province and territory. The Census Overcoverage Study is designed to investigate overcoverage errors caused by persons who are enumerated more than once. The results of the Reverse Record Check and the Census Overcoverage Study, when taken together, furnish an estimate of net undercoverage.

Other error sources

While coverage errors affect the number of units in the various census universes, other errors affect the characteristics of those units.

Sometimes it is not possible to obtain a complete response from a household, even though the dwelling was identified as occupied. The household members may have been away throughout the census collection period or, in rare instances, the householder may have refused to complete the form. More frequently, the questionnaire is returned by mail or submitted through the Internet, but no response is provided to certain questions. Effort is devoted to ensure as complete a questionnaire as possible. Analysis is performed to detect significant cases of partial non-response and follow-up interviews are attempted to get the missing information. Despite this, at the end of collection, a small number of responses are still missing, i.e., non response errors. Although missing responses are eliminated during processing by replacing each one of them by the corresponding response for a 'similar' record, there remain some potential imputation errors. This is particularly serious if the non respondents differ in some respects from the respondents; this procedure will then introduce a non response bias.

Even when a response is obtained, it may not be entirely accurate. The respondent may have misinterpreted the question or may have guessed the answer, especially when answering on behalf of another, possibly absent, household member. The respondent may also have entered the answer in the wrong place on the questionnaire. Such errors are referred to as response errors. While response errors usually arise from inaccurate information provided by respondents, they can also result from mistakes by the census representative who completed certain parts of the questionnaire, for example, during follow-up to obtain a missing response.

Some of the census questions require a written response. During processing, these 'write in' entries are given a numeric code. Coding errors can occur when the written response is ambiguous, incomplete, and difficult to read or when the code list is extensive (e.g., major field of study, place of work). A formal quality control operation is used to detect, rectify and reduce coding errors. Within each work unit, a sample of responses is independently coded a second time. The resolution of discrepancies between the first and second codings determines whether recoding of the work unit is necessary. Census coding is now entirely automated, resulting in a reduction of coding errors.

The questionnaire page images are scanned and the information on these images is captured into a computer file. To monitor and to ensure that the number of data capture errors is within tolerable limits, a sample of responses is re-sampled. Analysis of the two captures is done. Unsatisfactory work is identified and corrected by a last capture.

The data are edited where they undergo a series of computer checks to identify missing or inconsistent responses. These are replaced during the imputation stage of processing where either a response consistent with the other respondents' data is inferred or a response from a similar donor is substituted. Imputation ensures a complete database where the data correspond to the census counts and facilitate multivariate analyses. Although errors may have been introduced during imputation, the methods used have been rigorously tested to minimize systematic errors.

Various studies are being carried out to evaluate the quality of the responses obtained in the 2011 Census. For each question, non-response rates and edit failure rates have been calculated. These can be useful in identifying the potential for non response errors and other types of errors. Also, tabulations from the 2011 Census have been or will be compared with corresponding estimates from previous censuses, from sample surveys (such as the Labour Force Survey) and from various administrative records (such as birth registrations and municipal assessment records). Such comparisons can indicate potential quality problems or at least discrepancies between the sources.

In addition to these aggregate level comparisons, there are some micro-match studies, in which census responses are compared with another source of information at the individual record level. For certain 'stable' characteristics (such as age, sex and mother tongue), the responses obtained in the 2011 Census, for a sample of individuals, are being compared with those for the same individuals in the 2006 Census.

For further information on the quality of census data, contact the Social Survey Methods Division at Statistics Canada, Ottawa, Ontario, Canada K1A 0T6, or by calling 613-951-4783.