3. MEASUREMENT AND METHOD

3.1 The ISSS/A and the 1994 Survey
3.2 Topics Covered in the ISSS/A 1994 Survey
3.3 Issues of Wording
3.4 Do Well-Formed Attitudes Exist?
3.5 Effects of Knowledge on Measurement
3.6 Measurement of Background and Demographic Variables
3.7 Attitude Scales
3.8 Methods
3.9 The Author (e-mail: Kelley@international-survey.org)
3.10 Summary: Measurement & Method

3.1 The ISSS/A and the 1994 Survey.

The International Social Science Survey / Australia is a nation-wide survey conducted by researchers at the Australian National University and the University of Melbourne. Begun in 1984, it is Australia's leading academic survey and is devoted entirely to academic research in the social sciences, is non-profit, and is not connected with any business or political party. The survey's core sponsor is the Research School of Social Sciences at ANU.

The ISSS is a founding member of the International Social Survey Programme, sociology's leading cross-cultural survey project which conducts annual surveys in 29 nations throughout the world. The ISSS group recently founded the International Survey of Economic Attitudes, which is now conducted bi-annually in five nations.

The ISSS group plans to establish a new bi-annual International Bioethics Survey, hopefully with parallel surveys in Australia, the United States, and possibly other nations.

The ISSS is based on large, representative national samples of all states and territories, drawn from the electoral roll. Non-citizens are not surveyed but other evidence shows that they differ little from citizens, save only in length of residence in Australia (Evans, 1988).

The detailed and comprehensive ISSS survey takes about two hours to complete. It is conducted by mail. The first mailing includes a cover letter from the Australian National University and a postage-paid reply envelope, followed by a further letter about two weeks later. Anyone who did not respond within a month or so is then pursued by up to three more mailings over a six month period. Survey fieldwork and management was by Datacol, a respected private firm.

Comparison with the census shows the samples collected in this way to be representative of the Australian population in age, sex, education, occupation, and other characteristics (Bean, 1991). Numerous academic papers based on ISSS data and written by the ISSS group have been published in the world's two top sociology journals (Kelley and Evans, 1993 and 1995; Evans, Kelley, and Kolosi, 1992; Evans and Kelley, 1991).

Jonathan Kelley Ph.D. is Director and principal investigator of the ISSS; M.D.R. Evans Ph.D. and Krzysztof Zagorski Ph.D. are co-principal investigators.

This report is based on 1378 respondents from the 1994/95 survey conducted in late 1994 and the early months of 1995.

3.1.1 Developmental Survey

The conceptualization of the ISSS survey was based on a developmental survey designed to explore a wide range of issues relating to genetic engineering. The purposes of the developmental survey are two: First to ascertain the broad outlines of public opinion on the matter. Second, to provide systematic evidence on which to base decisions about which topics warrant fuller investigation in the main survey.

The data for the developmental survey are from 318 randomly chosen ACT residents. They were interviewed by telephone in early May, 1994. Data collection was by Datacol . With a sample of this size and the usual uncertainties of a developmental survey, percentages should be accurate to within 5 to 10 percent. Past experience suggests that ACT residents differ little from the rest of Australia, particularly on topics for which educational differences are modest. Genetic engineering is such a topic.

The developmental questionnaire provides broad coverage of genetic engineering topics, with over 75 individual questions. This is roughly twice the length of the final questionnaire.

The developmental data was analyzed extensively. Scales were conceptualized in advance (at the design stage); the items in each potential scale were factor analyzed; and concepts rethought if required. Multiple item scales were constructed on the basis of this work. The main analyses envisioned for the final data were then run on using the preliminary data and preliminary scales, mainly using ordinary least squares regression. This gave a fair idea of the broad outlines of public opinion about genetic engineering, in particular showing what concepts could be reliably measured and which helped to explain the Australian public's views on genetic engineering.

3.2 Topics Covered in the ISSS/A 1994 Survey

The 1994/95 International Social Science Survey covers a wide range of topics. It includes more than 1,000 questions and takes over two hours to complete. Topics covered in 1994 include
  1. The International Social Survey Programme's module on the environment;
  2. Questions about acceptance of a scientific world-view;
  3. The genetic engineering module; and
  4. An extensive inventory of background and demographic questions, and modules on many other topics ranging from attitudes on sex to attitudes on politics to educational and occupational careers.
Most of the questionnaire items are copyright by Jonathan Kelley, M.D.R. Evans, Clive Bean and Krzysztof Zagorski and may not be reproduced without prior written permission, save for brief extracts for the purpose of fair comment.

The genetic engineering questions are one module in the ISSS, taking up about two and a half pages. Results from all the items in the module will be presented below. The complete text of the genetic engineering module is in an appendix.

3.2.1 Sequencing of modules

We located the module in the questionnaire following the five page "Attitudes towards the Environment" module of the International Social Survey Programme (ISSP) and the ISSS/A's "Scientific world-view" and "Technological Understanding" questions.

The advantage of this placement was thematic continuity. A possible disadvantage is that the ISSP module asks people to rate the degree of danger from a long list of environmental hazards -- it is possible that this focus on hazard may slightly bias answers to the subsequent genetic engineering module against genetic engineering by making environmental dangers more salient. However, past ISSS experience suggests that the bias, if any, is probably small.

3.2.2 Attitude Formation

This is a baseline survey, conducted at a time when the commercial application of "genetically modified organisms" (GMOs) in Australia is in its infancy. The attitudes I describe here may or may not persist into the future: public attitudes towards some technologies have shifted in a clearly positive direction (for example, the moral qualms that some people felt during the pioneering stages of organ transplant procedures have vanished), while support has plummeted for others (for example, the growth of opposition to nuclear power in many countries).

Because this is so new, and hypothetical, it is possible that the public might not be able to form attitudes on the topic. Our developmental pre-test was designed, in part to assess this possibility, but it instead suggested clear opinions and pronounced social differences in them. This might seem surprising, given how little people know about science in general, let alone about genetic engineering, but it is worth remembering that people often must think hypothetically and on the basis of very limited knowledge in their lives and in forming political opinions (for example in developing opinions about tariff policies -- an important and long-standing political issue -- or developing opinions about the likely consequences of various forms of government being mooted in the debate on an Australian republic).

We can tell whether the opinions offered are disorganized and random (indicating that there really is no underlying opinion) or whether they are real attitudes by assessing their correlations with one another and their correlations with "criterion variables" reflecting expected social differences using factor analysis and related maximum likelihood LISREL techniques. The results suggest that the Australian public in fact has well-formed attitudes in this domain.

For details, see (Kelley, 1995c).

3.3 Issues of Wording

Genetic engineering is not an easy concept and there are, as yet, no universally accepted question wording. We think our questions, developed after careful pretesting, are quite satisfactory but measurement is a difficult and potentially controversial issue. Details on question wording are in a later chapter. In this section, we consider some measurement issues that might be of interest to specialists.

Our introductory 'tomato' question is hypothetical, but hypothetical questions are common and pose few special problems -- citizens are quite accustomed to deciding about policies that do not exist and may never exist (for example, the GST). Our 'tomato' scenario actually paralleled events in the USA (Schibeci et al. 1994: 25-26): tomatoes were in fact carefully vetted by an appropriate government regulatory committee, met with little or no scientific opposition, and were labeled (although labeling was not legally required). Our genetic engineering questions followed a 15 minute International Social Survey Programme module which discussed an assortment of environmental risks, and so would, if anything, sensitize respondents to risk, not lull them into acquiescence. Moreover the tomato is the least popular genetically engineered product on our list. We might instead have introduced the subject with the leukemia cure, blood pressure treatment, or genetically engineered cotton. These are overwhelmingly popular and so sensible alternative wordings of our questions would surely also discover that.

3.4 Do Well-Formed Attitudes Exist?

Do ordinary Australians have coherent views on genetic engineering? To be sure, there are no hard and fast rules about the best way to measure difficult concepts but the standard procedure, and the one that usually works best, is to ask a number of specific, concrete questions and then average the answers. For example, to discover what voters think about government regulation of business, best practice is to ask a number of specific questions about regulation in particular industries (railways, steel manufacturing, cars, farms, etc.) and then construct a combined 'government regulation' scale from the answers (Kelley 1988; Headey, Kelley and Wearing, 1993). This is the strategy we followed for genetic engineering, choosing projects from among those already well into development in Australia and overseas (Australian Science and Technology Council, 1993).

This approach allows us to use standard multivariate statistical procedures to discover whether the public really does have coherent attitudes toward genetic engineering or whether the issue is so novel and complex that ordinary people as yet have no clear views. The evidence comes from the correlations among answers: if people have no clear views, their answers to different genetic engineering questions will be uncorrelated (and measurement reliability will be zero). But if they have well-defined views on genetic engineering, as they do on many political and economic issues, correlations will be positive, typically in the range of .20 to .60, and factor analysis will find a single factor. Table 1 gives the evidence.

Table 1: Genetic engineering questions: Correlations and factor analysis show that attitudes are well-formed. Australia, 1994.
Question (1) (2) (3) (4) (5) (6) (7) (8) (9) Factor loading 
1 Cure cancer 1.00 .72 
2 Blood pressure .79 1.00 .80 
3 Cotton .61 .67 1.00 .72 
4 Cooking oil .45 .51 .49 1.00 .73 
5 Control animals .35 .43 .46 .48 1.00 .66 
6 Control insects .40 .46 .50 .47 .72 1.00 .68 
7 Lean pork .39 .44 .44 .68 .53 .50 1.00 .74 
8 Tomatoes .31 .34 .32 .46 .31 .32 .53 1.00 .55 
9 Benefit vs. risk .36 .39 .35 .43 .34 .33 .45 .52 1.00 .56
 

Source: International Social Science Survey / Australia, N=1378. The questions are #4c, #4e to #4k, and #7a on pages 62 and 63 of the questionnaire.

These results clearly show that the Australian public has well-formed attitudes about genetic engineering. The correlations among questions average a substantial .46 and the factor analysis shows a single, clear factor. For comparison, correlations average .31 among items measuring attitudes toward government regulation, .42 among price control items, and .56 among trade union questions. Thus attitudes to genetic engineering are well within the normal range for Australian political and social attitudes (Evans and Kelley 1995; Kelley 1988:60-70; Kelley, Evans and Headey 1993).

3.5 Effects of Knowledge on Measurement

The public's (allegedly) low level of knowledge of genetic engineering worries many researchers, who wonder whether ill-informed citizens as yet have any well-formed views about genetic engineering at all. But in answer to one of our questions, 68% of Australians say they have "heard much about genetic engineering" and in answer to another, 63% claim to have "a basic understanding" of it (Kelley 1995). So in fact there is a fair level of comprehension. Moreover, in a democracy voters routinely make decisions about policies about which they have no detailed academic understanding (Bean and Kelley 1995).

Importantly, even those who are less knowledgeable about genetic engineering nonetheless have reasonably coherent attitudes about it (table 2). Correlations among their answers (.39) are well within the normal range, although lower than correlations for more knowledgeable respondents (.50).

Table 2: Knowledge of genetic engineering. Australia, 1994.
Less knowledgeable  More knowledgeable 
Correlation among attitude questions (mean)  .39  .50 
Support for genetic engineering: 
Cure cancer (% favour)  93%  94% 
Blood pressure (% favour)  92%  93% 
Cotton (% favour)  92%  93% 
Cooking oil (% favour)  84%  82% 
Control animals (% favour)  75%  74% 
Control insects (% favour)  76%  72% 
Lean pork (% favour)  75%  73% 
Tomatoes (% favour)  57%  70% 
Personally use products: 
Wear cotton (% yes)  68%  84% 
Eat tomatoes (% yes)  51%  68% 
Eat lean pork (% yes)  47%  63% 
Use cooking oil (% yes)  59%  73% 
Benefits outweigh risks (% yes)  55%  70% 
(Number of cases)  (589)  (737)
 

Source: International Social Science Survey / Australia. The questions are #4c, #4e to #4k, and #7a on pages 62 and 63 of the questionnaire. The knowledge scale, based on items #7b and #7c, is described in the next chapter; it is dichotomized into high vs. low at the mean.

Moreover, knowledge of genetic engineering does not lead to opposition. Quite to the contrary (table 2), there is little systematic difference in support for most of the genetically modified products on our list. Knowledgeable respondents are actually keener on genetically modified tomatoes, more likely to say they would themselves use genetically modified cotton, pork and cooking oil, and more likely to believe that the benefits of genetic engineering will outweigh the risks,

These results suggest that Australian public will become more supportive of genetic engineering in the future as levels of knowledge increase.

3.6 Measurement of Background and Demographic Variables

Background and demographic variables come from the extensive battery of measures available elsewhere in the ISSS/A survey. They are measured conventionally:

Gender: Male = 1, female = 0

Age: Years

Education: Years of school and tertiary training

Status: An approximation of Kelley's world-wide occupational status score, ranging from 0 to 100.

Politics: Sympathy for the Liberal-National Coalition: Average score for Liberal Party and National Party, on a conventional Michigan feeling thermometer. Range: 0 (very cold or unfavorable feeling). through 50 (no feeling either way), to 100 (very warm or favorable feeling).

Politics: Sympathy for Labor: Average score for the Labor Party on a conventional Michigan feeling thermometer. Range: 0 (very cold or unfavorable feeling). through 50 (no feeling either way), to 100 (very warm or favorable feeling).

Politics: Sympathy for Environmentalists. Average score for environmentalists on a conventional Michigan feeling thermometer. Range: 0 (very cold or unfavorable feeling). through 50 (no feeling either way), to 100 (very warm or favorable feeling).

Christian belief: Average score on 5 items measuring belief in God, the devil, heaven, hell and life after death. Range: 0 (unbeliever) to 100 (devout).

Catholic = 1, all others = 0.

Fearfulness: A 4 item scale measuring fear of spiders, illness, etc. Range: 0 (not fearful) to 100 (highly fearful).

3.7 Attitude Scales

Attitude and value measures (described in detail below) are almost all based on multiple item scales. The use of multiple item scales is a vast improvement over the more usual reliance on single questions. By including many measures of the same concept, it becomes possible to:
  1. Test the assumption that they measure what you assume they do using factor analysis and other sophisticated statistical procedures;
  2. Refine the measurement by excluding items that show statistical or conceptual weaknesses;
  3. Reduce arbitrariness by relying on the average of several items rather than putting all your eggs in one basket by choosing a single question which may be unrepresentative in ways you could not anticipate; and
  4. Reduce random measurement error that inevitably arises from the statistical imprecision of any single item.
The cumulative impact of these advantages is so great that the use of single item measures of attitudes, values, or perceptions can rarely be justified in serious work.

3.8 Methods

3.8.1 Factor and Regression Analyses

Items in the developmental questionnaire were extensively factor analyzed (mainly principal axis factor analysis rotated to simple structure by the varimax criterion). Items in the final questionnaire were again factor analyzed and the scale construction based on those analyses and also on their correlations with demographic and other variables following the standard logic for multiple item indicators. Scale reliability was generally high.

Effects are estimated by ordinary least squares regression. No correction was made for attenuation due to random measurement error but this is likely to be small for the variables of main interest here, as our attitude measures are mostly quite reliable multiple item scales and demographic and background variables typically have little measurement error.

In the text, I report standardized partial regression coefficients (betas). These range from -1 (for a perfect negative relationship) to +1 (for a perfect positive relationship) but such extreme values are most unusual.. To give a sense of scale, note that a straightforward genetically inherited trait (like height, for example) produces a beta of .50, so that corresponds to a very strong effect indeed. In most circumstances -- and assuming that a reasonably large range of other relevant variables are included in the equation, as they are in this report, the following would be a reasonable practical guideline:

Unless otherwise noted, all betas reported are significantly different from zero at p<.01 or better.

3.8.2 Scoring: Points out of 100

For convenience in comparing questions with different answer categories, all are converted to range from a low of zero (for the lowest answer category, for example "definitely no") to a high of 100 (for the highest answer category, for example "definitely yes"). Intermediate answers are given scores at equal intervals in-between. For example answers to many questions are in standard yes/no categories:
Definitely yes  [[scored 100 points]] 
Yes [[scored 75 points]] 
Hard to say, mixed feelings [[scored 50 points]] 
No [[scored 25 points]] 
Definitely no [[scored 0 points]] 
Means are then easy to compare. This scoring is purely cosmetic, and leads to the same conclusions as conventional scoring would (for example, scoring "definitely no" as 1 point, "no" as 2, and so on with "definitely yes" getting 5).

3.9 The Author

Jonathan Kelley is Senior Fellow in the Institute of Advanced Studies, Australian National University and Director of the National Social Science Survey. He is a graduate of Cambridge University (BA) and the University of California (Ph.D.). He is currently studying bioethics, inequality (with M. D. R. Evans), social mobility, and attitudes toward the economy (with Krzysztof Zagorski).

Many of these analyses are based on cross-national data from the International Social Survey Project, which he co-founded in 1984 . It is now conducted annually in 29 nations. Other analyses are from his new bi-annual International Survey of Economic Attitudes, founded a few years ago (with Evans and Zagorski) and now conducted in five nations. To date, he has been principal investigator for 16 large national surveys in three nations.

He has published widely in academic journals in Australia and overseas, including Australia (Australian and New Zealand Journal of Sociology); Britain (Sociology; British Journal of Sociology); Europe (International Social Science Journal; Social Indicators Research); and the USA (American Journal of Sociology; American Sociological Review; American Political Science Review; American Journal of Political Science; Public Opinion Quarterly). His publications include 4 major recent articles in the world's best sociology journals: Kelley and Evans, 1993, 1995; Evans and Kelley 1991; Evans, Kelley and Kolosi 1992). Indeed, over the five years to 1993 (the period evaluated for the ARC/ANU review of the Institute of Advanced Studies), he stands fourth in the world in amount published in the world's two leading sociology journals. (Complete CV).

3.10 Summary: Measurement & Method

This is a report of data collected for the Department of Industry, Science and Technology by the International Social Science Survey / Australia. The ISSS, Australia's leading academic survey, is conducted by researchers at the Australian National University and the University of Melbourne. The results are based 1378 respondents from a large, representative national sample of all states and territories, drawn from the electoral roll. The survey was conducted in late 1994 and the early months of 1995. The conceptualization was based on a earlier developmental survey designed to explore a wide range of issues relating to genetic engineering

The author is Senior Fellow in the Institute of Advanced Studies, Australian National University and Director of the International Social Science Survey. He has published widely in academic journals in Australia and overseas, including numerous publications in the world's best sociology and political science journals. 

To top of page
To next chaper
Table of Contents