8/16/2019 Chiappetta (1991)

1/13

JOURNAL

OF

RESEARCH IN SCIENCE TEACHING VOL. 28, NO. 8, PP. 713-725 (1991)

A Method to Quantify Major Themes of Scientific Literacy

in Science Textbooks

Eugene

L .

Chiappetta

Department

of

Curriculum and Instruction, University

of

Houston,

Houston, Texas 77204

David A. Fillman

Galena Park High School) School District, Galena Park, T exas 77545

Godrej

H.

Sethna

Houston Museum of Natural Science, Houston, Texas 77030

Abstract

Science textbooks are frequently used to convey a great deal of the information that students

receive in science courses. They influence how science teachers organize the curriculum and

how students perceive the scientific enterprise. An overreliance

on

these teaching aids often

results in an overemphasis

on

terminology and vocabulary, and presents a false impression of

the nature of science. s a result of their importance, a method was developed to assess the

curricular emphasis in science textbooks. The procedure is explained in a 25-page manual to

train researchers to determine the relative emphasis that has been given to (a) science as a body

of knowledge, (b) science as a way of investigating, (c) science as a way of thinking, and (d)

the interaction among science, technology, and society. Textbooks in the areas of life science,

earth science, physical science, biology, and chemistry were used in the analyses. Interrater

agreements of at least

80

and kappas of at least 0 . 7 3 were achieved in the content analyses

amon g two experienced researchers and one science teacher who were given the training manual

to learn the assessment procedure.

Science textbooks have long been an object of interest and concern among science

educators. These teaching aids are widely used in science courses (Exline, 1984; Harms

&

Yager, 1981); thus they convey

a

great deal of the scientific information that students

receive. Most importantly, these instructional materials influence how students and

their teachers perceive the scientific enterprise. Unfortunately, m any science teachers

rely heavily on the assigned text, which probably gives students a false impression of

the nature of science (Yager, 1984). Many of the com mercially available texts stress

facts and present science as a complete body of information that was derived

in

an

errorless manner. Science textbooks place too much emphasis on terminology and

991

by

the National Association for Research in Science Teaching

Published

by

John Wiley & Sons,

Inc.

CCC

0022-4308/91/080713-13 04.00

8/16/2019 Chiappetta (1991)

2/13

714 CHIAPPETTA, FILLMAN, A N D SETHNA

vocabulary (Yager, 1983), which results

in

students memorizing large amounts of

information and giving it back on tests.

Obviously, science textbooks play a very important role in science teaching;

consequently this teaching aid should be as useful as possible. Science textbooks must

convey a valid conception of the scientific enterprise. In the process of making sc ience

as relevant as possible, these teaching aids must relate science to the everyday lives

of

students without compromising the integrity of the field of study. Sc ience textbooks

can be interesting to students and at the same time illustrate how science, technology,

and society are interrelated.

Since science textbooks play such an important role in science teaching, researchers

must determine the extent to which these teaching aids present an appropriate delivery

system for science course instruction at the middle and secondary school levels. This

type of inquiry necessitates a valid and reliable method in order to provide accurate

information regarding the messages that science textbooks convey to students, many

of whom are being “turned

off”

to science.

Purpose

The purpose of this study was to develop a valid and reliable method to quantitatively

analyze the content

of

science textbooks, especially those used in middle and senior

high school science courses. The approach employed four aspects of scientific literacy

to determine curriculum balance

in

textbooks. The specific research question was: Can

a quantitative content analysis procedure be developed that will result in interrater

agreement of at least 80 and a kappa of at least 0.70 , to determine the emphases in

written materials for science courses?

Review of Literature

A limited number of content analysis studies have been conducted in the field of

science education, w hereas in the field of com munication this procedure is a comm only

used research method. The studies that have been conducted to analyze the content

of

science textbooks have reported high measures of reliability in their procedures. However,

many of these investigations used statistical tests that do not take into account agreement

by chance among raters. The authors often report percent agreement among the raters,

in

spite of the “waming against percent agreement as a reliability yardstick” (Krippendorff,

1980; p. 135), while other authors do not report interrater agreement. In addition, the

authors of some of these studies are not clear on how the validity of their procedures

was established.

Levin and Lindbeck (1979) analyzed five secondary schoo l biology textbooks for

coverage of 11 controversial issues and biosocial problems. Tw o science educators

rated these textbooks for quantitative and qualitative coverage of the 11 issues. The

Pearson product moment correlations of the ratings for the quantitative coverage ranged

from 0.71 to 1

O

and for the qualitative coverage ranged from 0.8 7 to 1

O

Prosser (1983) analyzed the conceptual difficulty (either concrete or formal) of

two chapters taken from a college physics textbook. He concluded that much of the

subject matter required formal-operational thinking. Prosser reported that there w as

an intraclass correlational agreement among three raters of 0.91.

8/16/2019 Chiappetta (1991)

3/13

METHOD

To

QUANTIFY MAJOR THEMES

715

Skoog (1979) studied the inclusion of evolution in 93 biology textbooks published

between 1900 and 1977. He identified 44 aspects of evolution to look for in these

texts and performed a type of word count to determine how much written material

was devoted to evolution. It is not clear from Skoog’s report how he validated the

various aspects of evolution that were used in the analyses or how he determined the

reliability of his method.

Gannaway (1980) examined two secondary school chemistry textbooks to determine

their content, objectives, and pedagogical approach. The coders in this study analyzed

paragraphs, pictures, etc. Gannaway established the validity for the coding list by

presenting a rationale for these ideas. Reliability for this procedure was determined

by using the test-retest method and was reported to range from 86 to 93 over a

six-week period of time. Krippendorff (1980), an authority in the field of content

anaysis, refers to this type of reliability as “stability” and suggests that it is the weakest

type of reliability to establish.

Ro sen thal( l984 ) investigated the extent to which 22 high school biology textbooks

included socia l issues. She asked 25 experts either to classify 87 social issues into one

of

14

categories that she developed or to create new categories. Rosenthal reported

an 84 agreem ent among these individuals in establishing the validity of the categories.

Subsequently, four raters were asked to classify 100 paragraphs from the selected

textbooks using the established categories of social issues. Rosentha l reported an 86

agreement between raters and her coding, thus establishing the reliability of this

procedure.

A large-scale study was carried out by the Science Council of Canada (Orpw ood

&

Souque, 1984) to examine the contents and aims of science textbooks used in

Canada. The them es were selected from those contained in the Ministry of Education

guidelines, which are related to science content, acquisition of scientific skills, and

the relationship between science and society. The Council’s analyses included

6

textbooks used

in

the elementary, middle, and senior high schools. Unfortunately,

Orpwood and Souque (1984) did not report the procedure used to establish the reliability

or validity of the assessment process.

The present study grew out of the investigation of Garcia (1985), who analyzed

earth science textbooks for their presentation of various aspects of scientific literacy.

Garcia selected scientific literacy as the major theme of her content analysis because

of its broad conceptual framework for the outcomes in science education. She examined

the work of many science educational researchers and organizations in order to form

broad and discrete categories of scientific literacy. Among the works on scientific

literacy which were analyzed were those written by Pella, O’H eam , and Gale (1966);

Showalter (1974); Harm s and Yager (1981); NSTA (1982); Roberts (1983); Fensham

(1983); Orpwood and Alam (1984); and Collette and Chiappetta (1986).

From these works, Garcia (1985) identified many descriptors, each of which was

placed on a card. The cards were given to two science educators to categorize using

a modified Q-sort procedure described by Rakow (1985). This procedure produced

four categories

of

scientific literacy: (a) The basic knowledge of science, (b) the

investigative nature of science, (c) the thinking processes

of

science, and (d) the

interaction of science, technology, and society. Many descriptors were provided for

each of these categories. In the few cases where ambiguity or disagreement occurred

among the science educators, the descriptors were reworded

so

that agreement was

achieved with regard to the categories and their descriptors.

8/16/2019 Chiappetta (1991)

4/13

716

CHIAPPETTA, FILLMAN, AND SETHNA

Procedure

The first problem to resolve in the present study was to insure that a valid method

be used to analyze science textbooks written for life science, earth science, physical

science, chemistry, and biology. The three authors found that Garcia’s descriptors,

which were used to analyze earth science textbooks, needed to be modified

so

that

the written material that appears in a variety of science textbooks could be properly

categorized. This phase involved the identification of all the important ideas that appear

in a variety

of

science textbooks in order to insure the content validity of the procedure.

The authors had to find descriptors which had a high rate of recognition for the four

major themes. This required many iterations of analyzing a large variety of science

textbooks, resulting in the construction of a 25-page training manual (Chiappetta,

Fillman, & Sethna, 1991). The four major themes (categories) of scientific literacy

and their descriptors, as they appear in the procedures manual, are as follows:

Categor ies fo r Analyzing Science Textbooks

1

The knowledge of science.

Check this category

if

the intent of the text is to

present, discuss, or ask the student to recall information, facts, concepts, principles,

laws, theories, etc. It reflects the transmission of scientific know ledge where the student

receives information. This category typifies most textbooks and presents information

to be learned by the reader. Textbook material in this category:

(a) Presents facts, concepts, principles and laws.

(b) Presedts hypotheses, theories, and models.

(c) Asks students to recall knowledge or information.

2 .

The investigative nature of science.

Check this category if the intent of the

text is

to stimulate thinking

and

doing

by asking the student to “find out.” It reflects

the active aspect of inquiry and learning, which involves the student in the methods

and processes of science such as observing, measuring, classifying , inferring, recording

data, making calculations, experimenting, etc. This type

of

instruction can include

paper and pencil as well as hands-on activities. Textbook material in this category:

(a) Requires students to answer a question through the use of materials.

(b) Requires students to answer a question through the use of charts, tables, etc .

(c) Requires students to make a calculation.

(d) Requires students to reason out an answer.

e )

Engages students in a thought experiment or activity.

However,

if

a question simply asks for recall of information or is immediately answered

in the text, check Category

1.

3 Science as a way of thinking.

Check this category if the intent of the text is

to illustrate how science

in

general or a certain scientist in particular, went about

“finding out.” This aspect of the nature

of

science represents thinking, reasoning , and

rejlection, where the student is told about how the scientific enterprise operates. Textbook

material in this category:

8/16/2019 Chiappetta (1991)

5/13

METHOD o QUANTIFY MAJOR THEMES

I17

(a) Describes how a scientist experimented.

(b) Shows the historical development of an idea.

(c) Emphasizes the empirical nature and objectivity of science.

(d) Illustrates the use of assumptions.

(e) Shows how science proceeds by inductive and deductive reasoning.

(f) Gives cause and effect relationships.

(8) Discusses evidence and proof.

(h) Presents the scientific method and problem solving.

4 .

Interaction of science, technology, and society.

Check this category if the

intent of the text is to illustrate the

effects

or

impacts of

science on society. This aspect

of scientific literacy pertains to the application

of

science and how technology helps

or hinders humankind. In addition, it involves social issues and careers. N evertheless,

the student receives this information and generally does not have to find out. Textbook

material in this category:

(a) Describes the usefulness of science and technology to society,

(b) Points out the negative effects of science and technology on society,

(c) Discusses social issues related to science or technology, and

(d) Mentions careers and jobs in scientific and technological fields.

In addition to the above the manual contains:

(1)

A

presentation of scientific literacy and its role in the analysis of a science

textbook.

2 )

A

description of the four categories

of

scientific literacy and their descriptors

(subcategories).

(3)

A

list of text elements (units of analysis) that appear

on

the pages

of

science

textbooks that should be used for analyzing content themes. The units of

analysis include: complete paragraphs, questions, figures, tables with captions,

marginal comm ents, and com plete steps in a laboratory or hands-on activity.

4)

A list

of

pages that should not be analyzed in a science textbook, such as a

page with fewer than two analyzable units, a page that contains only review

questions and vocabulary words, and goal and objective statements.

(5)

Directions

on

how to identify and number the units of analysis on each page.

(6) A data sheet upon which the units of analysis identified on each textbook page

can be classified into the four aspects of scientific literacy.

(7) Sev en practice sets to aid in developing the skill of categorizing units of analysis

on a given page of a textbook. Each set consists

of

three or four paragraphs

from a different science textbook published over the past 20 years, and which

was written for science courses taught in Grades 7-12. The user is instructed

to analyze each paragraph and categorize it into one of the four aspects or

themes of scientific literacy and its appropriate subcategory. Then the user

checks the answers and explanations

of

these ratings

on

the next page.

(8) A review that requests the user to construct many short paragraphs, each of

which illustrates a different aspect of scientific literacy.

Categorizing units of analysis that present “the knowledge

of

science” (Category

1)

are usually an easy matter. Most

of

the paragraphs, figures, pictures with captions,

and m arginal comments that appear in science textbooks tell about phenomena which

8/16/2019 Chiappetta (1991)

6/13

718

CHIAPPETTA, FILLMAN, AND SETHNA

are easy to recognize. Similarly, it is easy to categorize units of analysis that involve

the reader in carrying out a manulative or

a

mental task (“the investigative nature of

science,” Category

2).

A more difficult categoriza tion requires distinguishing between

“science as a way

of

thinking” (Category

3)

and “the knowledge of science” (Category

I .

For example:

Roentigen and Thompson found, independently, that the ionization of air produced

by x-ray discharges electrified bodies. The rate of discharge was shown to depend

on the intensity of the x-rays. This property was therefore used as a quantitative

means

of

measuring the intensity of an x-ray beam. As a result, carefu l quantitative

measurement of the properties an d effects of x-rays could be made. [Harvard Project

Physics.

(1968).

An

introduction to

physics:

Models

of

the atom (Vol. 5 , p. 56.)

New York: Holt, Rinehart and Winston.

The paragraph above indicates how the work of two scientists was used to further

scientific knowledge. The paragraph also provides information about the properties of

x-rays. In addition, the paragraph indicates how empirical data was used to study a

phenomena. These three ideas taken together place the unit of analysis into Category

3 ,

because it illustrates how scientists use empirical data

to

advance science and how

scientists go about their work. This unit of analysis should indicate the difficulty

encountered by raters, because the paragraph not only contains information about the

work of scientists, but also presents information about x-rays. W hen one presents the

work of a scientist,

it

invariably is accompanied by a discussion of scientific facts,

concepts, and principles. Units of ana lysis that contain more than one theme are difficult

to rate accurately and consistently, which is the reason 25 different units of analysis

were selected from a variety of science textbooks and placed in the procedures manual.

In

the development of a reliable procedure, one m ust also consider sample size.

How many textbook pages should be selected from a given text in order to insure that

a representative sample of all the major categories of scientific literacy have been

identified and that obscure categories have been included in the frequency

in

which

they exist in a given text? One must select the smallest sample size that does not omit

these important aspects of science education. For example, in some science textbooks

the authors write one page at the end

of

each chapter that describes career opportunities

as they relate to the topic under study.

As

career opportunities relate to an important

aspect of developing scientific literacy (“the interaction of science, technology, and

society”), these occurrences must not be overlooked in the sampling.

Garcia (1985) took several 5 random samples from one earth science textbook

and found that this relatively small proportion of total textbook pages produced the

same frequency distribution of the four aspects of scientific literacy. Similarily, one

of the authors of the present study took two random, 5 samples from a high school

biology textbook and found that these samples had roughly the same proportion of the

four aspects of scientific literacy in them: 78.0 versus 82.0 (Category l), 11.3

versus 11.2 (Category 2), 2.6 versus

2.9

(Category 3), and 8.1 versus 5.0

(Category 4 .

Most science textbooks are quite lengthy. Therefore, when one analyzes a 5

sample of the total pages of a textbook, the procedure results in many categorizations.

For exam ple, there was an average number of

731.4

pages

in

the five biology textbooks

adopted by the State of Texas for 1987-88. The average number of pages in a 5

8/16/2019 Chiappetta (1991)

7/13

METHOD

QUANTIFY

MAJOR THEMES

719

sample of these textbooks is 36.6. The average number of units of analysis is 298.0,

and the average number of units of analysis per page is 8.1.

In the early phase of this work, an analysis was done

on

five physical science

textbooks which were recommended for adoption in senior high schools by the Texas

Education Agency. Interrater agreements of 78 , 78 , 79 , 82 , 84 , and their

respective kappas of 0.71, 0.71, 0.72, 0.76, 0.79 (Table 1) were obtained for the five

textbooks (Chiappetta, Sethna, & Fillman; 1987). These results show that the percent

agreements had almost reached the 80 level, and the kappas had reached 0.70. The

kappa statistic (Cohen, 1960; Fleiss, Cohen,& Everett, 1969; Fleiss, 1971; and Tinsley

& Weiss, 1975) is an appropriate statistic to compute interrater agreement when: (a)

two judges are working independently; (b) the units of analysis are independent; and

(c) the categories are independent, mutually exclusive, and contain nominal data.

Cohen’s kappa takes guessing into account. The kappa statistic has a range of

.OO

1

.OO

with

0

representing chance agreement among raters. Rubinstein and

Brown (1984) state that kappas greater than 0.75 indicate excellent agreement among

coders and that kappas between 0.40 and 0.75 indicate fair to good agreement.

Following the analysis of the physical science textbooks, the authors modified the

procedure and selected five different types of science textbooks to examine: life science,

earth science, physical science, biology, and chemistry. Each textbook was randomly

selected from the five science textbooks which were in use during the 1980s and which

had been recommended for that science discipline by the Texas Education Agency.

The textbooks selected are listed below:

Barr, B. B ., &Leyden, M .B . (1986). Life science. Menlo Park, CA: Addison-

Wesley.

Brown, E M . , & Kemper, G. H . (1979).

Earth science.

Morristown, NJ: Silver

Burdett.

Heimler, C .H ., &Price, J . (1981). Focus onphysical science. Columbus,

OH:

Charles

E.

Merrill.

Otto, J.H.,

&

Towle, A . (1985). Modern biology. New York: Holt, Rinehart

and Winston.

Wilbraham, A .C ., Staley, D.D., Simpson, C .J .,& Matta, M .S . (1987). Chem-

istry.

Menlo Park, CA: Addison-Wesley.

Table 1

Intercoder Agreement for the Analysis

of

Five Physical Science

Textbooks between Tw o Raters

Textbook Percent agreement Kappa

Energy: A Physical Science

Holt Physical Science

Spacesh ip Earth-P hysical Science

Focus

on

Physical Science

Physical Science

(Harcourt Brace)

78 0.71

(Holt, Rinehart and Winston) 79 0 . 7 2

(Houghton Mifflin)

84 0 .79

(Charles Memll)

78 0 .74

(Prentice-Hall)

8 2 0 . 7 6

8/16/2019 Chiappetta (1991)

8/13

720 CHIAPPETTA, FILLMAN, AND SETHNA

Three individuals analyzed a 5 random sample of textbook pages taken from

each of the science textbooks. Two of the raters, who had extensive experience with

this method, were the authors of this report. The third rater was a science teacher who

had over

10

years of high school teaching experience in physical science and chem istry,

but who had

no

previous knowledge of this method. The science teacher was asked

to study the procedura l manual and to do all of the exerc ises in

it

before she was given

the

five

science textbook samples

to

analyze. Beyond a few comments regarding the

purpose of this activity, very little discussion took place between the authors and the

science teacher regarding the m ethodology.

Results

Th e percentage of interrater agreements and their kappas exceeded those that were

established by the authors as acceptable indicators of a reliable procedure . The in terrater

agreements between two

of

the researchers (A/C in Table

2

anged from

83%

to 93 .

The interrater agreements between one of the researchers and the science teacher (A/B

in Table 2) ranged from

83%

to

94 .

The interrater agreements between the other

researcher and the science teacher (B/C

in

Table

2)

ranged between

80

and

97 .

All of these ranges reached or exceeded the level of acceptable percent agreement

(80 )

that was set before this investigation was undertaken.

The kappas between the two researchers (A/C

in

Table 2) ranged from 0.77 to

0.91.

The kappas between one of the researchers and the science teacher (A/B in Table

2

anged from

0.77

to 0.9 2. The kappas between the other researcher and the science

teacher (B/C in Table 2) ranged from

0.73

to 0.9 6. This kappa range exceeded the

0.70 level set at the beginning of the study.

Table

3

presents the occurrence of the four them es or aspects of scientific literacy

in the five science textbooks. The overall mean percentages indicate that “science as

a body of knowledge” is the predominant theme among these texts (mean

=

65 .7 .

Table 2

Interrater Agreement for the Analysis of Five Science Textbooks among Three Raters

A/B

A/C

BIC

Mean

Textbook agree Kappa agree Kappa agree Kappa agree Kappa

Life Science

Earth Science

Focus on

(Addison-Wesley) 93.9 0.92 88.9 0.85 89.5 0.86 90.8 0.88

(Silver

Burdett)

90.1 0.87 92.2 0.90 92.3 0.90 91.5 0.89

Physical Science

(Menill) 89.8 0.86 92.9 0.91 89.7 0.86 90.8 0.88

Modern Biology

(Holt) 94.3 0.92 92.7 0.90 96.9 0.96 94.6 0.93

Chemistry

(Addison-Wesley) 82.8 0.77 82.8 0.77 80.0 0.73 81.9 0.76

Mean 90.2 0.87 89.9 0.87 89.7 0.86 89.9 0.87

Nore.

A =

first researcher,

B =

the science teacher,

C

= second researcher.

8/16/2019 Chiappetta (1991)

9/13

METHOD

QUANTIFY MAJOR

THEMES

121

The second most emphasized theme is “science as a way of investigating” (mean

=

24.2). The “interaction of science, technology, and society” appears to be receiving

some coverage (mean

=

9.0). “Science as a way of thinking,” however, seems to be

neglected

in

most of the science textbooks (mean

=

1.1)

analyzed in this study.

Discussion

The ultimate goal of this line of research is to determine how students, teachers,

and those who select science textbooks perceive these written materials. W hat makes

a particular science textbook interesting to students or desirable for adoption by teachers?

What impression do science textbooks give students regarding the nature of science?

Do science teachers prefer a text w ith information that can be easily assessed on paper-

and-pencil tests or do science teachers need an outline of what to teach and select

textbooks that best fulfill this need? One step

in

the process of determining the worth

of a text is to characterize

it

objectively,

so

that its attributes and the perceptions of

others about it can be discussed.

A procedure to quantify the major themes in science textbooks is necessary

in

order to analyze the content of these materials. The method must address all of the

major themes that authors include in these teaching aids. Work done by science

educational researchers

on

scientific literacy was useful in the identification

of

major

themes, which

in

turn formed the categories of analysis. Certainly, other conceptual

schem es would produce a different set of categories, which would characterize science

textbooks differently. For exam ple, the four-goal cluster generated from Project Synthesis

(Harms & Yager, 198 1)-Personal Needs, Societal Issues, Academ ic Preparation,

and Career EducationalIAwareness-would produce a different set of them es by which

science textbooks could be studied. No doubt researchers should “experiment” with

other classificatory schemes to analyze the content of science textbooks.

While we believe that thematic units are a valid approach to this type of project,

they do present some problems. Krippendorff (1 980) points out:

Thematic units require a deep understanding of the source language with all of its

shades and nuances of meaning and content. While

it

is often easy for ordinary

readers to recognize themes,

i t

is generally difficult to identify them reliably.

Although the purpose of the research is important in judging which kind of units

are most meaningful, for many content analyses thematic units are probably the

most preferable. But because of the long chains

of

cognitive operations involved

in the identification of thematic units , even carefully trained observers can be easily

led astray. Thematic units are therefore often avoided in content analysis

or

at best

used to circumscribe the fuzzy universe from which a sample or propositional units

are drawn. (p.

63)

The realities of Krippendorff

’s

statements were realized in this investigation.

Although it may have been m ore reliable to use science words

to

characterize science

textbooks, this would not have provided as meaningful a description of science textbooks

as using themes of scientific literacy. The authors improved on the reliability of the

procedures by carefully defining the descriptors for the four categories in order to

facilitate the identification of the themes.

The descriptors used by Garcia (1985) in her study of earth science textbooks

were modified

so

that raters could accurately place the units of analysis into a given

8/16/2019 Chiappetta (1991)

10/13

122 CHIAPPETTA, FILLMAN, A N D SETHNA

category when examining any science textbook. Assigning units of analysis to Categories

1 and 4 did not cause difficulty.

A

considerable amount of written material in science

textbooks emphasizes “basic knowledge of science,” which is Category 1. This category

was coded with relative ease when the reader was presented with inform ation or asked

to recall it. For example, facts, concepts, principles, laws, and theories, which are

placed in Category 1 , “the knowledge of science,” are encoun tered with high frequency

in science textbooks. Category 4 , “the interaction of science, technology, and society,”

is also relatively easy to code consistently, partly because this category occurs with

little frequency. In add ition, it is relatively easy

to

identify units of analysis that stress

the positive or negative effects of science and technology, discuss a social issue, or

describe careers related

to

science and technology.

The refinement of descriptors for Categories

2

and

3

required considerable work.

Category

2

was defined

so

that instructions appearing on textbook pages, which engage

the reader in mental

or

manipulative activities, were coded as Category

2,

“the investigative

nature of sc ience.” If the reader was asked to use a chart or a table to answer a question,

this unit of analysis was placed in Category

2.

Similarly, if the reader was asked to

make a calculation, refer to a table to produce an answer or even participate in a

“thought experiment,” the unit of analysis was placed in Category 2 .

Category 3 was defined

so

that it would be coded when a unit of analysis illustrates

how a person in general, or a scientist in particular, makes discoveries. A general

definition along with specific descriptors were construc ted for this category that stress

how scientists engage in experimentation, gather empirical data, use assumptions,

show cause and effect, are disposed toward self-exam ination, etc. Th is helped to reduce

the problem of distinguishing between Categories 1 and 3 .

In addition to m odifying the descriptors that G arcia (1985) recommended for this

procedure, the selection and definition of units of analysis were modified. For exam ple,

some of the textbooks that were analyzed contained goal and objective statements.

These elements were found to be confusing and reduced the consistency of the coding.

Consequently, these elements were identified as units of analysis that were not to be

coded.

The percentages of agreement found am ong the researchers and the science teacher

were above the levels set at the beginning of this inquiry. The authors hoped to obtain

interrater agreements of at least

80

and kappas of at least .70 between pairs of raters.

The overall range of percent agreements was between

80

o 97 , while the kappas

ranged from 0.7 3 to

0.96.

The fact that the science teacher was able

to

categorize the

units of analysis in a manner that resulted in high agreem ent with the researchers, who

had much more experience with this procedure, suggests that the procedure has reached

a high level of reliability.

This procedure shou ld be repeated by other researchers to verify the reliability of

the method, even though the results suggest that the procedure may be reliable and

can be used to determine the content messages in science textbooks, especially those

messages that

pertain

to the broad curriculum goals of scientific literacy. The importance

of replicating investigations cannot be overstated, since different results are often ob-

tained (Turner, 1988).

The researchers in this study noted that when science textbook authors attempt to

weave two or more themes into a textbook paragraph, this may or may not enhance

the quality of the presentation . In any event, this style of presentation lowered interrater

agreement regarding the m eaning of the m essage about science being conveyed to the

8/16/2019 Chiappetta (1991)

11/13

METHOD

To

QUANTIFY MAJOR THEMES

723

reader. The authors found that interrater agreements were lower in a few of the most

recently published science textbooks, because authors include4 several themes in a

given paragraph, making it difficult to code consistently. Note that one of the researchers

coded 8 of the chem istry text in Category 3 “science as a way

of

thinking” (Table

3),

while another researcher coded

0

in this category. When interrater agreement

drops to the

80

level (Table 2), or lower, the percentage

of

coverage reported can

be misleading. The emphasis on the interactiun

of

science, technology, and society

averages approximately 9 , which suggests that some publishers are attempting to

make science textbooks more relevant for students. If one were to analyze some of

the most recent editions of high school chemistry textbooks, he/she might ascertain

that a significant percentage

of

a few

of

these texts are devoted to science, technology,

and society (STS), a theme that is attracting more attention in science education

(Chiappetta, Sethna,

&

Fillman, 1989).

Table 3

Percentage o Themes o Scientific Literacy Found among Five Science

Textbooks

Textbook

Categories

Rater I I

111

IV

Life Science

(Addison-Wesley)

Earth Science

(Silver Burdett)

Focus

on

Physical Science

(Merrill)

Modern Biology

(Holt)

Chemistry

(Addison-Wesley)

Overall mean

A

B

C

Mean

A

B

C

Mean

A

B

C

Mean

A

B

C

Mean

A

B

C

Mean

46.4

49.7

49.9

47.7

49.4

53.9

53.1

52.1

60.0

61.4

62.1

61.6

92.8

93.8

95.4

94.0

66.9

71.3

81.3

73.2

65.7

42.0

34.3

41.9

39.4

35.4

34.2

37.0

35.5

28.3

29.9

32.5

30.2

1.5

0.5

3.6

1.9

14.0

14.0

14.2

14.1

24.2

0 0

0.0

0.0

0 0

1 . 3

0 0

0 0

0.4

0 0

0 0

0.0

0.0

2.6

2.6

0 0

1.7

8.1

1.5

0 0

3.2

1 . 1

11.6

16.0

11.2

12.9

13.9

11.8

9.9

11.9

11.0

8.7

4.8

8.2

3.1

3 .1

1 o

2.4

11.0

13.2

4.5

9.6

9.0

Raters: A

=

researcher one , B

=

the science teacher, C

=

researcher tw o.

Categories:

I .

knowledge of science, 11. investigative nature

of

science, 111. science as a way

of thinking,

IV.

interaction

of

science, technology, and society.

8/16/2019 Chiappetta (1991)

12/13

724

CHIAPPETTA, FILLMAN, A ND SETHNA

When researchers analyze phenom ena in the behavioral and social sciences, they

will experience difficulty developing methods of acceptable validity and reliability.

Human activity is complicated, and when researchers improve on the reliability of a

procedure, they often compromise on its validity. In the present study, the researchers

realized the importance of refining a procedure to place units of analysis into only one

category, because without this type of agreement, the method would be confusing

(Holsti,

1969;

Krippendorff,

1980).

In the analysis of most typical science textbooks,

this was not a significant problem. With some textbooks, however, where authors

place several themes in one paragraph and the raters must place units of analysis in

one category, the task of quantifying aspects

of

scientific literacy becomes quite

difficult. Nevertheless, this procedure has shown to

be

reliable with the science textbooks

currently on the market. However, the authors of this research are looking for textual

materials which utilize novel approaches to convey science to secondary school students.

This type of material could be characterized in order to determine its impact on student

interest and achievement.

There is a need for science education researchers to thoroughly study the contents

of science textbooks, given the central role they play in the cumculu m. M any different

paradigms should be used to analyze these m aterials. The four-goal clusters

of

Project

Synthesis with its emphasis on student needs might provide one good model, as would

the literacy goals of science, mathematics, and technology for Project

2061.

The

outcom es of these analyses can be used to determine the relationships between textbook

charateristics and student interest, and a greater insight into why science teachers adopt

certain textbooks. This line of research m ight be more meaningful than the readability

and comprehension studies that have been conducted in the past on science textbooks.

References

Chiappetta,

E.L.,

Fillman, D.A .,

&

Sethna, G.H.

(1991).Procedures fo r conducting

content analysis o fscien ce textbooks . Houston, TX : University of Houston, Department

of Curriculum

&

Instruction.

Chiappetta, E.L., Sethna, G.H.,

&

Fillman, D .A.

(1987).

Curriculum balance

in

science textbooks. The Texas Science Teach er,

16(2), 9- 12.

Chiappetta, E.L., Sethna, G.H., & Fillman, D.A.

(1989,

March). Examination

of hi gh school chemistry textbooks . Paper presented at the annual meeting of National

Association for Research

in

Science Teaching, San Francisco.

Cohen, J.A.

(1960). A

coefficient of agreement for nominal scales. Educational

and Psychological Measurement,

20, 27-46.

Collette, A.T. & Chiappetta, E.L.

(1986).

Science instruction in the middle and

secondary schools.

Columbus, OH: Charles Merrill.

Exline, J.D.

(1984).

National survey: Science textbook adoption process. The

Science Teacher,

51(1), 92-93.

Fensham, P.J.

(1983).

A research base for new objectives of science teaching.

Science Education, 67, 3 12.

Fleiss J.

L . (1971 .

Measuring nominal scale agreement among many raters.

Psychological Bulletin,

'76, 378-382.

Fleiss, J.L ., Cohen, J. ,

&

Everitt,

B . S .

(1969).

Large sample standard errors of

kappa and weighted kappa. Psychological Bulletin,

72, 323-327.

Gannaway, S.P.

(1980).

Development of a high school chemistry textbook evaluation

8/16/2019 Chiappetta (1991)

13/13

METHOD

TO QUANTIFY

MAJOR THEMES

125

instrument using survey and content techniques. Dissertation Abstracts International,

41,

101

1A. (University Microfilms No. 8019878)

Garcia, T.D.

(1985).

An analysis of earth science textbooks fo r presentation of

aspects

of

scientijc literacy.

Unpublished dissertation, U niversity of Houston.

Harms, N.C., & Yager, R.E. (1981). What research says to the science teacher

(Vol.

3 ,

No.

471-1 14776).

Washington, DC: National Science Teachers Association.

Holsti, O.R.

(1969).

Content analysis fo r the social sciences and humanities.

Reading, MA: Addison Wesley.

Krippendorff, K. (1980). Content analysis: An introduction to its methodology.

Beverly Hills, CA: Sage Publications.

Levin, F.,

&

Lindbeck, J.S.

(1979).

Analysis

of

selected biology textbooks for

the treatment of controversial issues and biosocial problems. Journal

of

Research in

Science Teaching, 16, 199-204.

National Science Teachers Association

(

1982).

Science, technology, society-

Science education fo r the

1980s:

An NSTA position statement. Washington,

DC

uthor.

Orpwood, G.W.F.,

&

Alam,

I .

(1984). Background Study

52,

Science education

in Canadian schools. Ottawa, Ontario, Canada: Science Council of Canada.

Orpwood, G.W.F.,

&

Soque, J.P.

(1984).

Summary of background study

52 ,

science education in Canad ian schools. Ottawa, O ntario, Canada: Science Council

of

Canada.

Pella, M.O., O’Heam, G.T.,

&

Gale, C.W.

(1966).

Scientific literacy-Its referents.

The Science Teacher,

33(5), 44 .

Prosser, M.

(1983).

Relationship between the cognitive ability of a group

of

tertiary

physics students and the cognitive requirements

o f

their textbook. Science Education,

Rakow, S.J.

(1985).

Excellence in middle/junior high science teaching: The teacher’s

perspective. School Science and Mathematics, 85(8),

63

1-632.

Roberts, D.A. (1983). Scie ntijc literacy towards balance in setting goals fo r

schools science programs

(Cat. No.

SS21-5/1983-2E).

Ontario,

Canada: The Publication

Office, Science Council of Canada.

Rosenthal, D.B. (1984). Social issues in high school biology textbooks: 1963-

1983. Journal of Research in Science Teaching, 21, 819-831.

Rubinstein, R.A.,

&

Brown, R.T.

(1984).

An evaluation of the validity

of

the

diagnostic category

of

attention deficit disorder. American Journal of Orthopsychiatry,

Showalter, V.M.

(1974).

What is united science education? (Part

5):

Program

Skoog,

G.

(1979). The topic of evolution in secondary biology textbooks: 1900-

Tinsley,

H.E., &

Weiss,

D.J. (1975).

Research methodology.

Journal of Counseling

Turner, A.

(1988).

Getting it right.

New Scientist,

117(1595), 70-71 .

Yager, R.E. (1983). The importance of terminology in teaching K-12 science.

Yager, R.E.

(1984).

The major crisis in science education. School Science and

67(1), 75-83.

54, 398-414.

objectives and scientific literacy. Prism,

II(2), 3,4.

1977. Science Education, 63, 621-640.

Psychology, 22,

358-376.

Journal of Research in Science Teaching, 20, 577-588.

Mathematics,

84,

189-198.

Manuscript accepted November

18, 1990.