Some of the major challenges facing the official statistical system are as follows. First, key government databases on population, household consumption, employment and national income have been in disarray for more than a decade due to delayed release and politicisation.
Second, the official statistical system — originally designed for a planned economy — has not adapted to the changing realities of post-liberalisation and the digital era. The inability of official statistics to adequately capture changing patterns of migration, employment and consumption is a case in point.
Third, the official statistical system has to compete with private survey agencies such as the Centre for Monitoring Indian Economy, NGOs such as Pratham and philanthropic organisations such as Tata Trusts, all of which have acquired the capacity to conduct their own large-scale surveys. Big data sources held by private entities are further undermining the government’s monopoly over data generation by making high volumes of data available at higher frequency.
Notwithstanding the emergence of non-governmental sources of data and newer types of data, conventional government statistics such as the population census will continue to occupy a key position in policymaking and public life for the foreseeable future. India’s Census Act was enacted in 1948, two years before the constitution was adopted. Several constitutional provisions link the census to power and resource sharing at different levels of government, as well as to various affirmative action policies.
Recent controversies have shown that the use of census data in policymaking remains politically sensitive and the quality of data has a significant bearing on decision-making. The choice of reference year for the delimitation of constituencies in the union territory of Jammu and Kashmir and a few north-eastern states has proven contentious. This is because of an uneven change in the relative population shares of various ethnic communities that is partly explained by poor quality census data.
The mandate of the Fifteenth Finance Commission — set up by the union government to give recommendations for devolution of taxes among other matters — to change the reference year and weight of population figures used in federal redistribution has also been questioned. It is likely to reduce the shares of more urbanised and economically vibrant southern states due to their lower fertility rates and poor accounting of the migrant population.
The most recent protests surrounding the hugely controversial Citizenship Amendment Act and the decision to link the National Register of Citizens with the decennial census have demonstrated that large sections of society continue to see a fair and conventional census as a guarantor of their constitutional rights.
Despite its continued relevance, the official statistical system requires substantive reforms. Recent changes to the system mostly involve mechanical solutions to fix data quality through the introduction of newer data collection tools such as Computer Assisted Personal Interviewing. But technological and legalistic solutions will only go so far in solving existing problems.
India’s ‘data deficit’ is embedded in a mutually constitutive relationship with its fraying democracy and deficient development in several respects. First, growing political interference has been a major problem facing official statistics on religion, caste, employment, consumption and national income.
Second, most government bodies entrusted with collecting data are struggling with a shortage of trained manpower to handle the volume of conventional data.
Third, biometric databases built by the government are deficient in underdeveloped areas where conventional databases have also been incomplete or flawed.
Fourth, adding punitive provisions to the laws governing the collection of data has in the past been followed by some of the most egregious instances of politically-motivated data manipulation.
A multi-pronged strategy is needed to holistically address India’s ‘data deficit’.
The government should start by helping to strengthen the independence of the judiciary, press freedom, the Right to Information Act and the autonomy of government statistical agencies. All of these are essential to ensuring the timely release and critical examination of data.
Statistical agencies should sensitise non-governmental stakeholders to secure their cooperation in the field to reduce non-response rate. This would also serve as a check on interference driven by political and economic considerations.
Statistical agencies should also trim unwieldy questionnaires to better utilise scarce resources and improve data quality. Nearly half of the questions in the census’ ‘House listing and Housing Schedule’ and a quarter in its ‘Household Schedule’ seek information on household amenities and assets, and employment, respectively. These questions can be dropped without impairing the constitutional and policy obligations of the census.
Existing sources of data should be used intensively. The continued neglect of state counterparts of the national sample surveys is a case in point. Governments at different levels should better utilise administrative statistics instead of adding to the growing pool of sample surveys of questionable quality. Moreover, digitisation of administrative statistics will improve access to data and allow for greater scrutiny and use of that information.
Lastly, the government should shed its self-image as the only source of large-scale data. It must engage non-government players as sources of ideas and solutions, particularly in the case of big data, because its manpower and technological constraints cannot be overcome overnight.
Vikas Kumar is Assistant Professor at Azim Premji University, Bengaluru, and is co-author of Numbers in India’s Periphery: The Political Economy of Government Statistics (Cambridge University Press, 2020).