This article started life as a post by JS on Talking Philosophy, The Philosophers’ Magazine’s new blog. The post criticised the shortcomings of an opinion poll commissioned by the British Humanist Association. David Pollock and Jemima Hooper of the BHA later commented on the post, and Casper Melville posted a comment on the New Humanist blog, to which Julian Baggini replied. JS and OB then wrote separate comments on the BHA reaction – a reaction which gave them an odd feeling that they would have to write Why Truth Matters all over again, or at least give the BHA a tutorial in its subject matter.

The Opinion Poll

The recent opinion poll commissioned by The British Humanist Association (BHA) almost defies belief in its absurdity. Indeed, so bad is it, that it is difficult to know where to start a critique.

But consider the following claim:

Humanist outlook on life is calculated as those choosing the following three statements:

• Scientific and other evidence provides the best way to understand the universe
• Human nature by itself gives us an understanding of what is right and wrong
• What is right and wrong depends on the effects on people and the consequences for society and the world

This is all over the place.

First, there are many religious people who would accept that "Scientific and other evidence provides the best way to understand the universe" – they would simply dispute what this tells us, if anything, about origins, etc.

Second, "Human nature by itself gives us an understanding of what is right and wrong": This proposition just about makes sense (though it seems to make redundant a lot of philosophising about ethics), but (a) it is quite possible to be a humanist and think this is nonsense; and (b) it is quite possible to be religious, and think that something like this might be true (if, for example, God has given us such an ability).

Third, "What is right and wrong depends on the effects on people and the consequences for society and the world". This suggests that it is necessary to be some kind of consequentialist to be a humanist (nb. it might appear here that we are confusing IF and IFF, but actually if you look at how the question is posed and the analysis constructed, you’ll find that this is not so). Which is absurd.

This is bad enough. But now have a look at how the questions were posed.

I am going to read out some pairs of statements to you. I’d like you to tell me on balance which one in each pair most closely matches your view.

And then, for example:

Scientific and other evidence provides the best way to understand the universe

Or:

Religious beliefs are needed for a complete understanding of the universe

It takes very little training in sociology – indeed perhaps none – to realise that this is fatally flawed as a methodology. The questions are pitched as a forced choice, but they are not in any sense mutually exclusive (certainly not in terms of how they will be taken). In the UK, at least, you’re going to catch a lot of religious people in the first category. It’s no good then claiming – ah, these people think science is the best way to understand the universe, they’re really humanist. That’s just an argument by redefinition. A humanist becomes a person who thinks science is the best way to understand the universe.

Also, look at the instructions more carefully:

 Where respondents were unsure, interviewers were allowed to select "Neither" or "Don’t know", but these options were not presented to respondents and they were encouraged to choose a statement from each set if they could.

That’s ridiculous. You can’t encourage people to answer forced choice questions, and then claim that their answers genuinely represent their opinions. You’ll end up with what are called "doorstep opinions" (see, for example: Schuman, H. and Presser, S., “The open and closed question”, American Sociological Review, (1979), 44 692-712).

This whole polling exercise was frankly a disaster, and the BHA should be embarrassed about it all.

*****************************

A Response to Critics

David Pollock of the British Humanist Association (BHA) complains about an ‘unforgiving onslaught of abuse against an unassuming opinion poll‘. He has since been joined by a Jemma Hooper (also of the BHA), who thinks ‘professionals should applaud quantitative data’ (more of which later); and Caspar Melville at New Humanist magazine, who seems to be blaming Julian Baggini for all the whole affair.

Pollock claims that :

even without any pretensions to being serious sociological research (on a budget of £5,000?), the poll is surely indicative.

This is just wrong. Here is why.

Pollock argues that:

religious people who reject both the option of saying that ‘religious beliefs are needed for a complete understanding of the universe’ and the option of saying that ‘People need religious teachings in order to understand what is right and wrong’ have at most a pretty attenuated sort of religion…

Variously:

1. The poll does not give people the option to reject the view that each statement represents. This is not how it was set up (which obviously is part of the problem). This is what the instructions say:

I’d like you to tell me on balance which one in each pair most closely matches your view. You might find that the statements overlap a little, however please tell me which one you feel most closely matches your view. (If you had to choose just one of the statements which one best matches your view?).

And also:

Where respondents were unsure, interviewers were allowed to select "Neither" or "Don’t know", but these options were not presented to respondents and they were encouraged to choose a statement from each set if they could.

Nobody was rejecting anything substantive when they responded to this poll (and Pollock should retract his statement to the contrary).

People were presented with a forced choice, and then they were encouraged to answer one way or the other even though it was conceded that there might be overlap in the statements (too right there is). It is staggering that anybody would think that this works as a methodology. One cannot help but suspect that the BHA didn’t bother to run its poll design past a social scientist.

2. Notwithstanding the methodological disaster, the opinion poll is conceptually up the creek as well. Christians in the UK are not particularly anti-science. It is entirely plausible, therefore, that people whose religious belief is not attenuated – not that it is obvious that an attenuated religious belief equates to humanism anyway – will choose the statement ‘Scientific and other evidence provides the best way to understand the universe’ rather than ‘Religious beliefs are needed for a complete understanding of the universe’ when they are forced to choose (there’s that irritating methodological point again). Also it is possible that this is what religious people would say even if they weren’t forced to choose. Perhaps, for example, they think that science explains what happens in the universe; religion what happens outside of it.

The statement about religious teachings, and right and wrong is also a conceptual mess. It is possible to be an atheist, and yet to think that people need religious teachings in order to understand what is right and wrong. Sociologists in the functionalist tradition, for example, think something like this (not quite like this, but not far off it). Also, religious people might not think that religious teachings are necessary for an understanding of right and wrong. It is easy to imagine that some religious people will think that God has granted human beings an innate ability to understand the difference between the two.

So even if the poll wasn’t a methodological disaster, it’s a conceptual disaster.

It doesn’t show what is being claimed for it.

3. Even if there were no methodological or conceptual problems, it still doesn’t suggest the conclusion that the BHA was so desperate to find. It is possible to elicit opinions on all kinds of things by means of questionnaires. It does not follow that people have these opinions. If one went out onto the street to ask people about their eschatological views, then one would find eschatological views, even if people had never given eschatology a thought before. The fact that people say that they think science is the best way to understand the universe doesn’t mean that they actually think it is the best way to understand the universe. They might have no idea about what constitutes science. They might think that homeopathy is science, for example. Perhaps they are amongst the one-third of British people who think that the sun goes around the earth.

How about Jemma Hooper’s point that professionals should applaud quantitative data? Well, whether people should applaud quantitative data depends (partly) on whether the data is any good. This poll’s data is hopeless. Therefore, it should not be applauded. One wonders whether Hopper thinks that we should applaud the quantitative data that predicted a win for Thomas Dewey in the 1948 US Presidential Election? It’s a famous polling error. Truman, having won, appeared on the news holding a copy of the Chicago Tribune, which had printed "Dewey Beats Truman" on its front page on the basis of polling data.

Moreover, even if quantitative data is good, not everybody thinks it is useful. This is why some people are committed to using qualitative research methods. This is very basic sociology. The kind of thing which people learn about at GCSE level. (To say so is not gratuitous rudeness, it is the kind of thing people learn about in GCSE Sociology.)

The most disappointing thing about this whole affair is that the BHA, an organisation presumably committed to reason, proper enquiry, etc., has been so cavalier in the way that it approaches social research. They ought to be ashamed of themselves. They should stop defending the indefensible (when you’re in a hole, stop digging). And they should apologise to their members for wasting £5000.00 on a worthless poll.

***************************

The ‘We’re a campaigning organisation’ defence

It might be thought that people like rationalists and humanists and similar are supposed to value reason and truth and accuracy and getting things right.

However, some of the response to the criticisms of the BHA opinion poll make one wonder. Here’s an example:

So, less of this ivory tower disdain, please, for the honest labours of those who are trying to defend the secular principle in the face of sustained attack by the most religious government for over 100 years…In the real world of politics you cannot always be academically nice – your opponents will make mincemeat of you if you try…On rationality and truth – come down out of your ivory tower! The BHA is a campaigning organisation, not a university department.

So the response to criticism is to say that such concerns are ivory tower disdain, being academically nice, the result of high-altitude occupation of that ivory tower, confusion between campaigning and a university department? In other words, criticism of a flawed poll is pedantic and (as it were) elitist, and campaigning organizations needn’t and even shouldn’t worry about rationality and truth? But if rationality and truth aren’t the issue – then what is? Why are they humanists at all? Are they just allergic to communion wafers or something?

This comment is if anything even odder.

As a Marketing professional, I notice something distasteful about the not so subtle prejudice against marketing in the casual dismissing of a professional study. Yes, I’m aware that the profession has a mixed reputation but Philosophers and Sociologists, are in no position to throw stones either. On a professional level, I would expect you to rally to the support of fellow professionals, undertaking quantitative research to support the defence of the secular freedoms which we have enjoyed to-date.

Five uses of the word ‘profession’ or ‘professional’ in four lines, and the whole concept deployed as some sort of loyalty imperative. This is very strange. Why are professionals supposed to rally to the support of fellow professionals? Is that how the world is carved up? Do all professionals supporteach other? And what exactly is a ‘professional’ anyway? And why is it seen as some sort of hurrah-word?

We have talked here about the 1948 Dewey-Truman polling error. This was part of the background of the childhood of one of the authors of this piece (OB). The mistake haunted George Gallup’s polling organisation. They bent every nerve to figure out how they’d got it wrong, they revamped everything, and they sweated bullets over subsequent elections. OB hung out there once on the evening of a presidential election – it was like being at NASA during a mission: hours of huge tension, followed by shrieks of euphoria. But what they did not do was shrug and pout and say it was no big deal. They didn’t bother murmuring about academic niceties or ivory towers, they just turned everything upside down to correct the mistake.

Julian Baggini makes much the same point on the New Humanist blog.

Is it really the case that none of my fellow humanists can see and admit that this poll was frankly flaky and there is a real issue here of how much a movement committed to rationality can be prepared to say, "let’s not worry too much about the niceties of truth – let’s just get campaigning."

Well exactly.

Jeremy Stangroom and Ophelia Benson are the authors of Why Truth Matters (Continuum 2006).