Implementation of Large-scale Education Assessments

Presents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments.

In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD's Programme for International Student Assessment (PISA), the IEA's Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale.

This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better use of the results. The book will also give readers an understanding of the resources needed to undertake and improve the design of educational assessments in their own countries and regions.

Implementation of Large-Scale Education Assessments:

• Brings together the editors’ extensive experience in creating, designing, implementing, analysing and reporting results on a wide range of assessments.
• Emphasizes methods for implementing international studies of student achievement and obtaining highquality data from cognitive tests and contextual questionnaires.
• Discusses the methods of sampling, weighting, and variance estimation that are commonly encountered in international large-scale assessments.
• Provides direction and stimulus for improving global educational assessment and student learning
• Is written by experts in the field, with an international perspective.

Survey researchers, market researchers and practitioners engaged in comparative projects will all benefit from the unparalleled breadth of knowledge and experience in large-scale educational assessments gathered in this one volume.

Edited by
Petra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia

Keith F. Rust, Westat and the University of Maryland at College Park, USA

Raymond J. Adams, Australian Council for Educational Research (ACER), Australia

Notes on Contributors xv

Foreword xvii

Acknowledgements xx

Abbreviations xxi

1 Implementation of Large]Scale Education Assessments 1
Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams

1.1 Introduction 1

1.2 International, Regional and National Assessment Programmes in Education 3

1.3 Purposes of LSAs in Education 4

1.4 Key Areas for the Implementation of LSAs in Education 10

1.5 Summary and Outlook 16

Appendix 1.A 18

References 22

2 Test Design and Objectives 26
Dara Ramalingam

2.1 Introduction 26

2.2 PISA 27

2.3 TIMSS 34

2.4 PIRLS and Pre]PIRLS 41

2.5 ASER 45

2.6 SACMEQ 52

2.7 Conclusion 56

References 58

3 Test Development 63
Juliette Mendelovits

3.1 Introduction 63

3.2 Developing an Assessment Framework: A Collaborative and Iterative Process 65

3.3 Generating and Collecting Test Material 68

3.4 Refinement of Test Material 72

3.5 Beyond Professional Test Development: External Qualitative Review of Test Material 81

3.6 Introducing Innovation 86

3.7 Conclusion 90

References 90

4 Design, Development and Implementation of Contextual Questionnaires in Large]Scale Assessments 92
Petra Lietz

4.1 Introduction 92

4.2 The Role of Questionnaires in LSAs 93

4.3 Steps in Questionnaire Design and Implementation 95

4.4 Questions and Response Options in LSAs 115

4.5 Alternative Item Formats 119

4.6 Computer]Based/Online Questionnaire Instruments 128

4.7 Conclusion and Future Perspectives 131

Acknowledgements 132

References 132

5 Sample Design, Weighting, and Calculation of Sampling Variance 137
Keith F. Rust, Sheila Krawchuk and Christian Monseur

5.1 Introduction 137

5.2 Target Population 138

5.3 Sample Design 144

5.4 Weighting 148

5.5 Sampling Adjudication Standards 153

5.6 Estimation of Sampling Variance 156

References 166

6 Translation and Cultural Appropriateness of Survey Material in Large]Scale Assessments 168
Steve Dept, Andrea Ferrari and Béatrice Halleux

6.1 Introduction 168

6.2 Overview of Translation/Adaptation and Verification Approaches Used in Current Multilingual Comparative Surveys 169

6.3 Step]by]Step Breakdown of a Sophisticated Localisation Design 174

6.4 Measuring the Benefits of a Good Localisation Design 184

6.5 Checklist of Requirements for a Robust Localisation Design 190

References 191

7 Quality Assurance 193
John C. Cresswell

7.1 Introduction 193

7.2 The Development and Agreement of Standardised Implementation Procedures 194

7.3 The Production of Manuals which Reflect Agreed Procedures 196

7.4 The Recruitment and Training of Personnel in Administration and Organisation: Especially the Test Administrator and the School Coordinator 197

7.5 The Quality Monitoring Processes: Recruiting and Training Quality Monitors to Visit National Centres and Schools 198

7.6 Other Quality Monitoring Procedures 201

7.7 Conclusion 204

Reference 204

8 Processing Responses to Open]Ended Survey Questions 205
Ross Turner

8.1 Introduction 205

8.2 The Fundamental Objective 207

8.3 Contextual Factors: Survey Respondents and Items 207

8.4 Administration of the Coding Process 214

8.5 Quality Assurance and Control: Ensuring Consistent and Reliable Coding 221

8.6 Conclusion 229

References 229

9 Computer]Based Delivery of Cognitive Assessment and Questionnaires 231
Maurice Walker

9.1 Introduction 231

9.2 Why Implement Computer]Based Assessments? 232

9.3 Implementation of International Comparative Computer]Based Assessments 238

9.4 Assessment Architecture 244

9.5 Item Design Issues 247

9.6 State]of]the]Art and Emerging Technologies 250

9.7 Summary and Conclusion 250

References 251

10 Data Management Procedures 253
Falk Brese and Mark Cockle

10.1 Introduction 253

10.2 Historical Review: From Data Entry and Data Cleaning to Integration into the Entire Study Process 254

10.3 The Life Cycle of a LSA Study 255

10.4 Standards for Data Management 256

10.5 The Data Management Process 258

10.6 Outlook 272

References 274

11 Test Implementation in the Field: The Case of PASEC 276
Oswald Koussihouèdé, Antoine Marivin and Vanessa Sy

11.1 Introduction 276

11.2 Test Implementation 278

11.3 Data Entry 294

11.4 Data Cleaning 295

11.5 Data Analysis 295

11.6 Governance and Financial Management of the Assessments 295

Acknowledgments 296

References 297

12 Test Implementation in the Field: The Experience of Chile in International Large]Scale Assessments 298
Ema Lagos Campos

12.1 Introduction 298

12.2 International Studies in Chile 302

Annex A 321

References 321

13 Why Large]Scale Assessments Use Scaling and Item Response Theory 323
Alla Berezner and Raymond J. Adams

13.1 Introduction 323

13.2 Item Response Theory 325

13.3 Test Development and Construct Validation 329

13.4 Rotated Test Booklets 345

13.5 Comparability of Scales Across Settings and Over Time 347

13.6 Construction of Performance Indicators 349

13.7 Conclusion 354

References 354

14 Describing Learning Growth 357
Ross Turner and Raymond J. Adams

14.1 Background 357

14.2 Terminology: The Elements of a Learning Metric 358

14.3 Example of a Learning Metric 360

14.4 Issues for Consideration 360

14.5 PISA Described Proficiency Scales 365

14.6 Defining and Interpreting Proficiency Levels 374

14.7 Use of Learning Metrics 379

Acknowledgement 380

References 381

15 Scaling of Questionnaire Data in International Large]Scale Assessments 384
Wolfram Schulz

15.1 Introduction 384

15.2 Methodologies for Construct Validation and Scaling 386

15.3 Classical Item Analysis 387

15.4 Exploratory Factor Analysis 388

15.5 Confirmatory Factor Analysis 389

15.6 IRT Scaling 392

15.7 Described IRT Questionnaire Scales 396

15.8 Deriving Composite Measures of Socio]economic Status 399

15.9 Conclusion and Future Perspectives 404

References 405

16 Database Production for Large]Scale Educational Assessments 411
Eveline Gebhardt and Alla Berezner

16.1 Introduction 411

16.2 Data Collection 412

16.3 Cleaning, Recoding and Scaling 416

16.4 Database Construction 418

16.5 Assistance 421

References 423

17 Dissemination and Reporting 424
John C. Cresswell

17.1 Introduction 424

17.2 Frameworks 425

17.3 Sample Items 426

17.4 Questionnaires 427

17.5 Video 427

17.6 Regional and International Reports 428

17.7 National Reports 428

17.8 Thematic Reports 429

17.9 Summary Reports 429

17.10 Analytical Services and Support 430

17.11 Policy Papers 430

17.12 Web]Based Interactive Display 431

17.13 Capacity]Building Workshops 432

17.14 Manuals 432

17.15 Technical Reports 432

17.16 Conclusion 433

References 433

Index 436

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.