The first PISA results for India: The end of the beginning

by Lant Pritchett.

    
Now this is not the end. It is not even the beginning

of the end. But it is, perhaps, the end of the beginning.

Winston Churchill, November 1942

The PISA 2009+ results are the end of the beginning. For the last decade there has been a debate. Some argued the levels of learning inside Indian elementary schools (primary and upper primary) are a national scandal and a threat to the future of India's society, polity, and economy. Others appeared to believe that the main, if not only, problem with Indian schools was that not enough children attend them and that with more money and more of the same, all would be well. The last five years saw a relentless accumulation of evidence about the crisis of learning. The establishment has tried to deny, deflect, and dismiss the evidence on learning. Eventually the Government of India agreed to participate in the PISA (Programme for International Student Assessment) - but only for two states, Tamil Nadu and Himachal Pradesh - and both sides agreed PISA was the litmus test. The PISA 2009+ results, which are both official and are beyond gain-saying are unspeakably bad. They confirm the worst of what anyone has been saying about the levels of learning in India elementary education.

• In reading of the 74 regions participating in PISA 2009 or 2009+ these two states beat out only Kyrgyzstan.
  
• In mathematics of the 74 regions participating the two states finished again, second and third to last, again beating only Kyrgyzstan.
  
• In science the results were even worse, Himachal Pradesh came in dead last, behind Kyrgyzstan, while Tamil Nadu inched ahead to finish 72^nd of 74.
  

But just coming in last (if we can dismiss as a relevant comparator for India a tiny Central Asian state) does not convey the enormity of how bad these results were, as not only was India last, it was far, far, behind its aspirations, both at the bottom and at the top levels of performance.

PISA expresses the levels of performance in two ways, an overall index number and the fraction of students achieving various "levels" of achievement. The PISA index numbers for each subject are scaled so that the typical OECD student is at 500 and the standard deviation across OECD students is 100. The testing of thousands of students allows the results to present not only the average but also the worst (5^th percentile) and best (95^th percentile) students do in each country/region. PISA also classifies student performance into "levels" that represent different degrees of mastery of the material.

Table 1 compares India's performance to three groups of countries. The economic superstars have successfully completed the transition from poor to rich economies in just two generations - Singapore, Hong Kong, Korea (China's only results are just for the city of Shanghai, which are the highest scores of any region tested, but this is too a typical to really be comparable) and India aspires to their sustained success economically. The current super powers are represented by the USA and the OECD average reflects India's aspirations as a superpower. The rising powers are represented by the BRIC countries of Russia and Brazil which reflect the rise of the emerging markets.

Compared to the economic superstars India is almost unfathomably far behind. The TN/HP average 15 year old is over 200 points behind. If a typical grade gain is 40 points a year Indian eighth graders are at the level of Korea third graders in their mathematics mastery. In fact the average TN/HP child is 40 to 50 points behind the worst students in the economic superstars. Equally worrisome is that the best performers in TN/HP - the top 5 percent who India will need in science and technology to complete globally - were almost 100 points behind the average child in Singapore and 83 points behind the average Korean - and a staggering 250 points behind the best in the best.

As the current superpowers are behind the East Asian economic superstars in learning performance the distance to India is not quite as far, but still the average TN/HP child is right at the level of the worst OECD or American students (only 1.5 or 7.5 points ahead). Indians often deride America's schools but the average child placed in an American school would be among the weakest students. Indians might have believed, with President Obama, that American schools were under threat from India but the best TN/HP students are 24 points behind the average American 15 year old.

Even among other "developing" nations that make up the BRICs India lags - from Russia by almost as much as the USA and only for Brazil, which like the rest of Latin America is infamous for lagging education performance does India even come close - and then not even that close.

To put these results in perspective, in the USA there has been huge
and continuous concern that has caused seismic shifts in the discourse
about education driven, in part, by the fact that the USA is lagging
the economic superstars like Korea. But the average US 15 year old is
59 points behind Koreans. TN/HP students are 41.5 points behind
Brazil, and twice as far behind Russia (123.5 points) as the US is
Korea, and almost four times further behind Singapore (217.5 vs 59)
that the US is behind Korea. Yet so far this disastrous performance
has yet to occasion a ripple in the education establishment.

Table 1: Comparing Indian (Tamil Nadu and Himachal Pradesh) students mastery of mathematics to economic superstars, current superpowers, and rising superpowers
Country/Region	5th	mean	95th	HP+TN average to comparator average	HP+TN average to comparator 5th percentile	HP+TN best (95th) to comparator's average	HP+TN best (95th) to comparator's 95th
				Points TN/HP is behind (-)/ahead(+)
Economic Superstars
Singapore	383	562	725	-217.5	-38.5	-99	-262
Hong Kong	390	555	703	-210.5	-45.5	-92	-240
Korea	397	546	689	-201.5	-52.5	-83	-226
Current Superpower
OECD avg.	343	496	643	-151.5	1.5	-33	-180
USA	337	487	637	-142.5	7.5	-24	-174
Rising Superpowers
Russia	329	468	609	-123.5	15.5	-5	-146
Brazil	261	386	531	-41.5	83.5	77	-68
Indian States
Tamil Nadu	241	351	468
Himachal Pradesh	223	338	458
Average of TN and HP	232	344.5	463
Source: PISA 2009 Plus Results, Table B.3.1 for first three columns and author's calculations.

I have emphasised Mathematics because many believed math was an
Indian strong suit. The results for reading and science are similarly
bad. Table 2 shows science results in a different format, which shows
the proportion of children in various categories of performance. There
are three points:

1. "Below level 1" doesn't even have a description as it implies
   that so little proficiency is demonstrated it is impossible to
   distinguish from not knowing anything at all. In the USA, even with
   its socio-economic and racial inequalities and language inequalities
   and its failing inner city schools, only 4.2 percent are in this
   category. In HP 57.9 percent of 15 year olds in school cannot be
   distinguished from not having learned any science at all and in
   TN 43.6 percent all in this category - ten times as many as the
   USA.
   
2. PISA considers "level 2" as the minimum level that
   provides the science competencies that will enable them to
   participate actively in life situations related to science and
   technology. Since more than 80 percent of students in both HP and TN
   are level 1 or below this most students in these states have reached
   age 15 ill-equipped for the century they will face.
   
3. While a
   thin elite that competes for the few highly selective technical
   institutes are globally competitive, this is a tiny fraction of the
   population. The estimate of the fraction of TN or HP students at level
   6 in science proficiency was zero. Their estimate of the
   fraction at level 5: also zero. Of course this does not mean there are
   not such students in these states, of course there are, just that from
   the samples available in the study the best estimate was so small as
   to be indistinguishable from zero.
   

colspan=5> Table 2: Comparison of science proficiency in Tamil Nadu
and Himachal Pradesh to India's aspirations

Country/Region	Below level 1	Level 1 1	Level 5 5	Level 6 6
Singapore	2.8	8.7	15.3	4.6
Hong Kong	1.4	5.2	14.2	2
Korea	1.1	5.2	10.5	1.1
OECD avg.	5	13	7.4	1.2
USA	4.2	13.9	7.9	1.3
Russia	5.5	16.5	3.9	0.4
Brazil	19.7	34.5	0.6	0
Tamil Nadu	43.6	40.9	0a	0a
Himachal Pradesh	57.9	30.9	0a	0a
Source: PISA 2009 Plus Results. Description of levels Table 3.2, percentages Table B.3.4. 1) At Level 1, students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow explicitly from given evidence. 5) At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis. 6) At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they demonstrate willingness to use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations. a) In Table B.3.4 these are reported as blank but the estimated percentages in below 1 to level 4 sum to exactly 100 percent. Obviously this not imply that there are exactly zero students in all of these two states meeting these levels but that with the sample sizes assess students of 1616 in HP and 3210 in TN there was insufficient information to create a non-zero estimate.

These results on PISA 2009+, while tragic for what they imply for Indian youth and perhaps shocking to newcomers to this subject, come as no surprise to those who have been working on basic education in India:

• Das and Zajonc (2008) used results from Orissa and Rajasthan to create indices on mathematics performance similar to those of TIMSS (Trends in Mathematics and Science Study) and found these states near the bottom of the global rankings.
  
  
• Educational Initiatives carried out an 18 state study using sophisticated testing instruments and found levels of performance on TIMSS comparable items that were stunningly lower. For instance on the open ended question "Write a fraction larger than 2/7" less than 30 percent of Indian students in standard 8 could answer correctly compared to more than 70 percent internationally.
  
  
• The APRest study led by Karthik Muralidharan and Venkatesh Sundararaman in rural AP asked the same questions of students in grades 2 to 5 and found very slow rates of learning progress.
  
  
• The results year after year from the ASER [2010 2009] study supported by Pratham find that significant fractions of students in Standard 8 cannot master even Standard 2 curricular basics. In rural areas nationwide a third of children in grade 8 could not do a simple division problem and almost 20 percent could not read a level 2 text. The 2011 results, due out in a few weeks will show continued stagnation or even retrogress in learning.
  
  
• Numerous studies by MIT's JPAL, World Bank, NCAER/University of Maryland and other researchers found levels of performance that were shockingly low compared to curricular expectations.
  

These PISA 2009+ results are the end of the beginning. The debate is over. No one can still deny there is a deep crisis in the ability of the existing education system to produce child learning. India's education system is undermining India's legitimate aspirations to be at the global forefront as a prosperous economy, as a global great power, as an emulated polity, and as a fair and just society. As the beginning ends, the question now is: what is to be done?