Opinion - News Analysis   

How CSDS fine-tunes polling exercise

K. Narayanan

The dust has settled, and so too, I hope, the doubts. Doubts over the exit polls jointly sponsored and published by The Hindu and CNN-IBN. The accuracy rate of the exit polls conducted by the Centre for the Study of Developing Societies (CSDS), Delhi, led by Professor Yogendra Yadav, varied from 80 per cent in Kerala to 99 per cent in Tamil Nadu.

Unlike in 1999 and 2004, the conduct and publication of the polls was not seriously challenged this time in court. In those years, the Election Commission of India moved the Supreme Court to bar such surveys of voter preferences but failed to have its way. The basic legal contention on the side of the news media was that opinion and exit polls were not part of the electoral process, but were exercises that fell outside the scope of electoral laws and the powers of the Election Commission. Any curb on the publication of such polls would be a violation of freedom of speech and expression guaranteed by Article 19 of the Constitution.

The merits and demerits, and accuracy, of opinion and exit polls have been debated upon at length. I propose only to deal with doubts and reservations expressed by readers over the latest polls in communications to me.

A reader who says he has 41 years of experience in conducting similar surveys in industry questioned the methodology adopted and whether Bureau of Indian Standards (BIS) guidelines on random sampling were followed. Another reader suggested publication of exit poll results of previous elections to judge whether the paper had been able to predict accurately. While pollsters could be well versed in arithmetic, they were not alive to the chemistry and logic of the voters, was another comment, which added that the large voter turnout especially of women would disprove the findings.

One reader cited my own words in an earlier column on not publishing unverifiable reports that did not have evidentiary support. For him, a scientifically conducted survey with statistical records to back the findings belonged to that category.

Some of the captions for the statistical interpretations were commented upon. It was contended that to describe the DMK's lead among women as narrow, when it was 6 per cent, and to say the richer the voter, the higher the support for the DMK alliance, when it had a 7 percentage point lead even among the "poor", was misleading. The community and religion-wise categorisation of the respondents as was done made the sampling size small, one reader felt. The whole exercise was "neither authentic, nor scientific, and only to scare the public," said a dismissive voice.

There have always been questions about polls — can a randomly selected small percentage of the population, especially in a society as diverse as ours, represent the general view? Some critics have pointed to "non-sampling" errors such as the wording and order of questions, the process of interviewing, the motivation of the pollsters, and so on.

Many of these issues were anticipated and resolved by the CSDS team. As it pointed out in The Hindu on May 12, the aim was accuracy, transparency and accountability and all details were shared with the readers. The results are visible.

The best way to answer doubting readers, I thought, was to let Professor Yadav respond. I posed the questions to him when he was in the thick of election analysis. After "recovering from the election exhaustion," he was gracious enough to provide detailed answers. Welcoming the public scrutiny and attention the polling exercise had received ("opinion polls have a tendency to degenerate in the absence of monitoring and criticism,") he said his team would be happy to answer further queries at election@csdsdelhi.

Dealing with the entire range of polls that the CSDS team conducted, including the pre-election and post-poll surveys and the exit polls, Professor Yadav made the following points:

BIS Guidelines: To the best of my knowledge, the BIS has no specific guidelines for opinion polling in general or election-related exit or post poll surveys in particular. Its guidelines pertain to general principles of randomisation, which need to be applied to specific situations. We have devised our own methods for implementing the principle of random selection of the sample. We select our sample in three stages. First we select a few representative constituencies (numbers always reported in each report) usually by the circular random sampling (Probability Proportionate to Size) method. We verify this selection to ensure that the past electoral record of this set matched the record of the entire State. In the second stage, we take the list of polling stations of the selected constituencies and again use circular random sampling to select about four to six polling stations (numbers always reported). Finally, we obtain the latest electoral rolls of these selected polling stations and do circular random sampling to select the specified number of respondents (e.g. if the rolls have 1,200 names and we want to select 30 names, we pick every 40th name on the rolls beginning with a random number. If the random number is, say, 27, we select the elector number 27, 67, 107, 147, 187 and so on). We hand the list of these names to our investigators and ask them to interview these respondents. What distinguished our method from that of all other polling agencies (and may account for the moderately better results that we get) is that we do not leave the selection of the location or the respondent to the investigators and thus reduce much of the sample selection bias that polls usually suffer from.

Methodology: There was a methodology box with every report. We are the only poll to give methodological details like the date, sample size, number of locations and constituencies sampled and the team that conducted the survey. (This should be mandatory and unfortunately, many polls have stopped giving even this minimum information.) We have also started the practice of sharing with our readers the social composition of our sampled respondents in terms of some select and significant attributes (rural/urban, men/women, major social groups and minorities, etc.) so that the reader can judge the representativeness of our sample. There was one lapse though. In our exit poll story on Tamil Nadu, we carried all the relevant details in the story itself but did not have a separate box on the methodology. We should have explained the exact procedure of exit polls and how it differed from the post-poll we used in other States.

It is not possible for exit polls to take into account the gender composition of the turnout, for this data is not available at the time of making the forecast. All we can do is to ensure a fair representation of women voters in our sample. Among the respondents we interviewed for the exit poll, 44.5 per cent were women. Among the actual voters, women constituted 49.1 per cent. This discrepancy did affect our vote estimate, for the AIADMK performed better among women.

Table captions: All the table captions referred to make a relative point. The point of these detailed break-ups was not just to repeat that the winner was leading everywhere, but to see if the lead had a pattern across different social groups. We were simply reporting that the DMK enjoyed a higher lead among some sections compared with others.

Disaggregation into caste and regions does pose a problem of small sample size. We have followed the standard rule of not reporting any category that has fewer than 35 cases in it. In the case of Tamil Nadu, we were dealing with a very large sample size and therefore did not have any problem of this kind.

Professor Yadav concluded with a request to readers and a request to politicians on how to relate to the poll forecasts.

To the reader: Survey estimates are after all estimates, at best scientific estimates. It can be a good overall guide, but these are not magic numbers. There is no method that can guarantee a precise forecast of the number of seats in our first-past-the-post electoral system. We were lucky to get our forecast so accurate in three States this time. Please do not expect this to happen every time.

To the politicians: Please do question and criticise polls by all means. But please do not attribute motives. Also, please do not offer unintelligent and uninformed criticism like "how can a few thousand interviews tell you about several crore people?"

readerseditor@thehindu.co.in

Printer friendly page  
Send this article to Friends by E-Mail

Stories in this Section

End of the road for Gyanendra
Planet football
Prospecting for `green' diesel
Meeting the challenge of Mandal II
Three weddings and a funeral
How CSDS fine-tunes polling exercise
"Smoking can blind you"
Right policy
Unacceptable
Temple priests
Over-reaction

Copyright © 2006, The Hindu. Republication or redissemination of the contents of this screen are expressly prohibited without the written consent of The Hindu