AHELO’s Civil Engineering StrandReflections, outputs and lessons from the NPM

perspective

Hesham Gomma, ADEC 　　　　 Satoko Fukahori, NIERMary Catharine Lennon, HEQCO Daniel Edwards, ACER

Facilitator: Ashley Ater Kranov, Washington State University

AHELO – What can we Learn from OECD’s Feasibility Study?The Westin Georgetown Hotel, Washington DC

July 11-12, 2013

Overview of the Session• Introduction (Ashley Ater Kranov )

–Why Civil engineering.–International quality assurance, and the importance of direct measures.

• The AHELO experience–Within institutions (Hesham Gomma)

•Catalytic effects: The Abu Dhabi experience working with institutions and faculty.–Within a jurisdiction (Mary Catharine Lennon)

•What information can a jurisdictional report provide, and why is it valuable? The Canadian experience.

–Across jurisdictions (Satoko Fukahori)•What can be gained by sharing analytic methods, data, results, etc. across nations? Examples from a three country analysis : Australia, Japan, and Canada.

• Lessons and the Future (Daniel Edwards and Mary Catharine Lennon)–FS data limitations, and future assessment design.

•Discussion (facilitated by Ashley Ater Kranov)

2

Measuring if students can think like an Engineerhttp://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf (pp.252-264)

The Hoover Dam is a 221-metre high concrete arch-gravity dam in the Black Canyon of the Colorado River in the United States of America. It was built to provide irrigation water, to control floods and to provide water for a hydroelectric power station at the base of the dam.

Figure 1. Hoover DamFigures 2-5. (Images of construction sites, and plans)

1. Explain why this is a good dam site for hydroelectric power generation. You should discuss at least two aspects.a) Dam height/high potential energy, b) high flow rate of the river, c) lake capacity, d) minimal social impact, e) characteristics of the rock, f) narrow gorge.

2. Explain the two main design features that contribute to the structural strength and stability of the Hoover dam.3. The maximum electrical power generated by the turbines at the Hoover Dam is 2.08*109.. What is the

approximate amount of water that flows through the turbines at this power output, if the power station operates at 90% efficiency?

4. Imagine that a new dam is being planned today in a different location. Briefly explain two environmental effects of the dam (which could also be upstream or downstream) that an engineer would need to consider in an environmental impact statement.

3

Measuring the Basic Engineering Science Competency http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf (pp.265-266)

A load P is applied to a Warren truss as shown below.If the self-weight of the members is ignored, which of the following statements is correct?

A. Compressive force exists in both the upper-chord member (p-q) and the lower-chord member (r-s).

B. Tensile force exists in both the upper-chord member (p-q) and the lower-chord member (r-s).

C. Compressive force exists in the upper-chord member (p-q), while tensile force is applied to the lower-chord member (r-s).

D. Tensile force exists in the upper-chord member (p-q), while compressive force is applied to the lower-chord member (r-s).

4

International Civil Engineering Strand

5

Guiding Questions

• To whom of this of interest and why?

• Did the results meet these needs and interests?

• What purposes can professional field assessments serve in the future?

6

Within institutions Catalytic effects: The Abu Dhabi experience working with institutions and faculty

Hesham Gomma

7

Institutions expectations from AHELO• Leadership

– Information on the investment efficiency– Performance of their graduates versus local/international peers– Baseline data on their institutions– International visibility and recognition

• Faculty– International benchmark– Information about their graduate capacity – Data to review their programs

• Students– Know the capacity of their institutions– Participate in international assessment– Support their institutions in international exercise– Visibility of their institutions

Engineering faculty perspective on AHELO• Assessment content

– Partial irrelevance to Abu Dhabi Civil engineering programs– Captured only some topics– There is a need for Academic Reference Standards at the international level

• Scoring rubric– Faculty did not participate in the development process (top-down)– Not always reflecting the student capacity

• Data output– Contextual data reflects a realistic picture of the institutions context– The institutions data did not cover learning outcomes and cannot be

utilized in reviewing individual institution program– The jurisdiction data is satisfactory for cross institutions analysis and

collective program review

Outcomes at the institutional level• Institutions have not actively proceeded with data analysis• Responding to ADEC’s invitation, one institution out of the

three institutions showed interest To work on reviewing its curriculum and learning

outcomes To work with employers and other institutions on

mapping and aligning learning outcomes with the employers needs

Talks are in place with the other institutions

Outcomes of AHELO at the Emirate level

• Promoting the assessment culture at the HE level in Abu Dhabi

• Developing Academic Reference Standards at the Emirate level

• ADEC is on its way to develop two major assessment exercises at the Emirate level for the IT and Education fields

Within a jurisdictionWhat information can a jurisdictional report provide, and why is it valuable?The Canadian Experience

Mary Catharine Lennon

12

Broad outcomes

Analyses – Secondary outcome (Feasibility)• Existing analyses

– OECD international reports on validity and reliability – Institutional reports – Jurisdictional reports

---------------------• Additional analysis:

– Jurisdictional reports (internationally leveled data) with international comparisons– Comparative report of Japan, Australia and Canada (recalibrated data) – Combined data report examining contextual strand of Japan, Australia and Canada

(recalibrated data)

Collaboration/experience – KEY outcome from Feasibility Study• Countries working together• Institutions working together• Experts working together• Dialogue about learning outcomes expanding

13

Institutional reporting and analysis

• Institutions provided a report on student and faculty context information, student scores and the international average

• Institutional reports did not have an impact• Shared though committees

• No institutional analysis conducted• Institutions were invited to data analysis

discussion/training. • More interested in jurisdiction/international

comparative report

14

Ontario’s Jurisdictional Report

• Intended to provide Ontario (Canadian) stakeholders: – An overview of the entire AHELO project– A summary of the Ontario experience in

administration and implementation– A de-identified cautious analysis of the data– A comparison to Australia– A discussion of the potential value of this type of

assessment

15

Working with the data

• Challenges of the jurisdictional report– Unable to provide detailed information on student

competencies– Sample sizes too small to present detailed information by

institution– Cautious not to stretch the data or read too much into it

• Benefits from the jurisdictional report– Collaboration with Australia extremely helpful– Gaining better picture of characteristics of Civil Engineering

departments and students– Providing programs with comparative information

16

Across jurisdictionsWhat can be gained by sharing analytic methods, data, results, etc. across nations? Examples from a Three Country Analysis: Australia, Japan and Canada (Ontario)..

Satoko Fukahori

17

Sharing Data

• Memorandum of Understanding– Signed by ACER, NIER, and HEQCO.– Intention of working together in good faith.– Requirement for written consent prior to dissemination

of disclosed data and findings from data analysis. • Not intended for ranking purposes– Intended to highlight differences in systems.

• Strengths and weaknesses, trends.– Intended to produce information that will benefit

institutions in their effort to improve their education.

18

Working with data across jurisdictions

• Benefits With caveats on data limitations: unrepresentative data.

– Providing comparative information on systemic differences, which should be interpreted within policy contexts, and potentially used to draw policy implications.

– Incorporating cross-jurisdictional information within jurisdictional reports (as in the Canadian case) will allow HEIs to potentially globally benchmark themselves, and understand their strengths and weaknesses.

• Challenges– Data limitations of the feasibility study did not allow producing the

kind of information that HEIs requested for. • Analysis on competence clusters - to be discussed in the next section.

19

Further work underway

• Australia, Canada and Japan collaborating on academic papers exploring system differences.

• One example underway is analysis focusing on Contextual Variables and the extent to which they explain scores when system differences are controlled for.

20

LESSONS AND THE FUTUREDaniel Edwards, Mary Catharine Lennon

21

Key lessons for Civil EngineeringGOOD

– Introduced a much wider audience to the concept of learning outcomes in higher education.

– Gained international consensus on detailed content of assessments.

– Initiated new relationships nationally and internationally.

Not so GOOD– Output data-wise not as fruitful…next slides explore

this

22

Data limitations (Remember: Feasibility Study only!)

1. Not a nationally representative sample of institutions/programs/students. Cannot draw inferences about countries (or many institutions).

2. Limited test time, covering multiple items – focus on a wide range of content, concepts, logistics of translation and delivery. Less focus on producing meaningful data (esp. at small group level).

3. Only one ‘total’ score derived. Not able to analyse by competence clusters.

4. Tasks more challenging than expert group anticipated, so less variation in outcomes = less nuanced data

23

Future: Assessment design

• Identify desired outputs; i.e. competence level detail.

• Identify reporting levels; i.e. student, institution, jurisdiction, international.

• Structure assessment to suit desired outputs.

24

Engineering Processes

Analysis Design Practice

Basic/Engineering Sciences Generic Skills

Branch Specific Engineering Non-engineering

General

Example: Eng Competence Clusters

In an ideal case:1) Demonstrate the strengths and weaknesses of programs in developing different types of competences.2) Analyze the relationships between competence clusters.

i.e. Does competence in the Basic/Engineering Sciences and in Engineering Generic Skills underlie competence in the Engineering Processes?

25

BUT BE REALISTIC!• Recognise that assessing to multiple levels of

detail requires many items and very long assessments!

26

Future: Outputs

• Identify desired outputs and reporting levels.• Establish agreed data sharing protocols

(between governments, institutions or regions – whichever is relevant).

• Ensure assessment and framework will deliver desired outputs.

27

These seem simple – but AHELO Feasibility Study makes this seem so…

We have the horse…now for the cart!

28

Thank you