surveys of enacted curriculum for english language learner instruction, standards, and assessments:...

Surveys of Enacted Curriculum for English Language Learner Instruction, Standards,

and Assessments:Evaluating Language Demands

and Language Complexity

Edynn Sato, WestEdJohn Smithson, WCER

Jaime Usma, WCERPeter Worth, WestEd

Enhanced Assessment Grant: SEC-ELL Coding

Washington, DCMay 5-7, 2008

2

Training Goals

• Introduce ELL components of study– Language demands– Language complexity

• Training and calibration in SEC-ELL coding protocol

3

Agenda

• Introducing the linguistic analysis aspect of the study

• Components of linguistic analysis

• Coding procedures

• Coding practice

• Questions

4

Study Goals

• Improve methods of aligning instruction to standards and assessments for English language learners.

• Analyze relationship of alignment to student achievement.

5

What is the nature of the language that students need to meaningfully engage with and achieve academic content?

6

Content and Academic Language

What is the relationship?

Content Academic Language

Content Academic Language

Academic LanguageContent

7

Interacting Dimensions of Content and Language

• Content/Topics

• Cognitive Demands

• Language Demands—Academic English Language Functions

• Language Complexity

8

Given the expected student outcome (content, learning), how do we expect students to use (receive, produce) language to acquire/demonstrate understanding of the content?

9

Example

Content• Mean, median,

mode• Operations• Multiply decimals

Language• Description• Inquiring

The average weight of 50 prize-winning tomatoes is 2.36 pounds. What is the combined weight, in pounds, of these 50 tomatoes? (Multiple-choice math item)

10

Example

Content• Main idea(s), key concepts

Language• Inquiring• Description• Summarization

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

11

ExampleContent• Geometric

concepts

Language• Identify• Classify• Compare/Contrast• Define• Analyze• Generalize• Evaluate• Variation in:

– Length– Amount of detail– Level of abstraction– Vocabulary– Sentence structure– Discourse style

12

Components to be analyzed

• Focus• Mode

– Mode category

• Language demand• Language complexity

– Density– Construction– Overall complexity

13

Example

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

14

Focus

• Is the language in this standard/item academic or social/functional?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

15

Mode

• Is the language required of the student in this standard/item receptive or productive? Or both?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

16

Mode Category

• Specifically, does the standard/item require the student to listen or read? Speak or write?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

17

Language Demand

• What function(s) does academic language in each mode serve in this standard/item?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

18

Language Demands

2008

19

Language Complexity—Density

• Is the density of the minimum amount of academic language required to achieve this standard/item low or high?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

20

Language Complexity—Density

Low HighLength ranges from a word to paragraphs    No/little variation in words and/or phrases in sentences/paragraphs; consistent use of language    Repetition of key words/phrases/sentences reinforces information    Language is used to present critical/central details    No/little abstraction; language reflects more literal/concrete information; illustrative language is used; language is used to define/explain abstract information     Graphics and/or relevant text features reinforce critical information/details

Length ranges from a word to paragraphs   Some variation in words and/or phrases in sentences/paragraphs   Repetition of key words/phrases/sentences introduces new or extends information   Language is used to present critical/central details, but non-essential detail also is presented   Some abstraction; language may or may not be used to define/explain abstract information; illustrative language may or may not be used; technical words/phrases are used   Graphics and/or relevant text features may or may not reinforce critical information/details

21

Language Complexity—Construction

• Is the construction of the minimum amount of academic language required to achieve this standard/item simple or complex?

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

22

Language Complexity—Construction

Simple ComplexMostly common/familiar words/phrases; no/few uncommon words/phrases, compound words, gerunds, figurative language, and/or idioms   Language is organized/structured   Mostly simple sentence construction   No/little passive voice   Little variation in tense   Mostly one idea/detail per sentence   Mostly familiar construction

(e.g., ’s for possessive; s and es for plural)   Mostly familiar text features

(e.g., bulleted lists, bold face)

Some common/familiar words/phrases; some uncommon words/phrases, compound words, gerunds, figurative language, and/or idioms   Language may or may not be organized/structured    Varied sentence construction, including complex sentence construction   Some passive voice   Variation in tense   Multiple ideas/details per sentence   Some less familiar/irregular construction   Some less familiar text features

(e.g., pronunciation keys, text boxes)

23

Overall complexity

• What is the overall complexity of the academic language in the standard/item? (1-4)

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

24

Coding ProceduresRater: Document: Date: Page # of

Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High)

Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex)

Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest;

X= Complexity cannot be determined)

Content Codes

Language Codes

Strand ID

TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1

2

3

4

5

6

25

Coding Sample

Which sentence best sums up the selection?

A. Two Presidents gave Jesse Owens important awards for his achievements.

B. Jesse Owens set a new Olympic record in the long jump.

C. Talent and hard work helped Jesse Owens become a great runner.

Content Codes

Language Codes

Strand ID

TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 Item 7 A RD D L S 2

2 E

3 H

4

26

Decision Rules

Guidelines that help ensure standardization and reliability of the coding process.

Examples: Grades 11-12 Written and Oral Language Conventions

standard 1.2 is understood to address only producing one's own writing, not correcting other authors' writing.

In grades 11-12 Reading standard 2.1, the phrase “public documents” was interpreted as including editorials, commentaries and opinion pieces on topics of public concern.

27

Coding Standards

• Work in groups of 3 or 4

• Code standard for each component

• Not a consensus model

• Ensure intra-rater consistency

28

Coding a Standard

ELL Standard I (Grades 3–6) Standard: Listening—Comprehend and interpret the

spoken language of others.

Objective: Listen to others to clarify, question, and extend.

Expectation: Mark or write a response that shows comprehension of evidence or conclusions in a selection.

29

Expectation: Mark or write a response that shows comprehension of evidence or conclusions in a selection.

Content Codes

Language Codes

Strand ID

TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 I.2.2.I A LI E L S 1

2 G

3 WR G L S 1

4 I

5

30

Algebra (Grade 4)

4.1 Generate algebraic rules and use all four operations to describe patterns, including nonnumeric growing or repeating patterns.

31

Algebra 4.1 Generate algebraic rules and use all

four operations to describe patterns, including nonnumeric growing or repeating patterns.

Content Codes

Language Codes

Strand ID TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 MA.4.A.4.1 A RD P L S 1

2 WR E L S 1

3 P

32

Reading (Grades Nine and Ten)

2.1 Analyze the structure and format of functional workplace documents, including the graphics and headers, and explain how authors use the features to achieve their purposes.

33

Reading (Grades Nine and Ten) 2.1 Analyze the structure and format of functional

workplace documents, including the graphics and headers, and explain how authors use the features to achieve their purposes.

Content Codes

Language Codes

Strand ID TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 Reading 2.1

A RD A L S 1 Reading may be more or less complex based on state's understanding of which functional workplace documents are studied.

2 B

3 WR A L S 2

4 B

5 G

6 J

7 K

34

ELL Standard I (Grades 3–6)Standard : Listening – comprehend and interpret the

spoken language of others.

Objective: Listen to others to clarify, question, and extend.

Expectation: Create a picture or mark a correct response that shows comprehension of evidence or conclusions in a selection.

35

Expectation: Create a picture or mark a correct response that shows comprehension of evidence or conclusions in a selection.

Content Codes

Language Codes

Strand ID

TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 I.2.2.E A LI A L S 1 Language demand for writing unclear– “create a picture” interpreted as early writing.

2 E

3 WR Z L S X

4

36

Algebra (Grade 4)

2.4 Compare and order decimals and estimate fraction and decimal amounts in real-world problems.

37

Algebra 2.4 Compare and order decimals and

estimate fraction and decimal amounts in real-world problems.

Content Codes

Language Codes

Strand ID TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 MA.4.A.2.4 A RD B L S 1 Compare and order assumed to be in print. If oral, Mode Category could also be LI and SP.

2 C

3 P

4 WR

A L S 1

P

38

Geometry (High School)

2.0 Students write geometric proofs, including proofs by contradiction.

39

Geometry

2.0 Students write geometric proofs, including proofs by contradiction.

Content Codes

Language Codes

Strand ID TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 Geo 2.0 A RD A H S 2 High Density because of abstraction and technical terms.

2 P

3 WR B H S 2

4 G

40

Reading (Grades Nine and Ten)

3.9 Explain how voice, persona, and the choice of a narrator affect characterization and the tone, plot, and credibility of a text.

41

Reading (Grades Nine and Ten)

3.9 Explain how voice, persona, and the choice of a narrator

affect characterization and the tone, plot, and credibility of a text. Content Codes

Language Codes

Strand ID TPC1

CGD1

Focus

(A/S)

Mode Language Demand

(A-P, Z)

Density

(L/H)

Construction

(S/C)

Overall Complexity

(1-4, X)

Comments

Rec. Pro.

1 Reading 3.9

A WR G L S 1 Explanation is based on reading and understanding a grade-level text, which is an ELA skill. However, the specific Language Demands can not be determined without additional information on the type and content of text.

2 SP I

3 J

4 L

42

Reminders

• Code up to six language demands for each mode

• Consider complexity of the minimum language required to achieve the standard

• Use decision rules to record interpretations

43

Questions

44

If you have questions or would like additional information, please contact

Edynn Sato, Ph.D. esato@wested.org