Implementation of Large-Scale Education Assessments (eBook)
469 Seiten
John Wiley & Sons (Verlag)
978-1-118-76247-9 (ISBN)
Presents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments.
In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD's Programme for International Student Assessment (PISA), the IEA's Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale.
This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better use of the results. The book will also give readers an understanding of the resources needed to undertake and improve the design of educational assessments in their own countries and regions.
Implementation of Large-Scale Education Assessments:
- Brings together the editors' extensive experience in creating, designing, implementing, analysing and reporting results on a wide range of assessments.
- Emphasizes methods for implementing international studies of student achievement and obtaining highquality data from cognitive tests and contextual questionnaires.
- Discusses the methods of sampling, weighting, and variance estimation that are commonly encountered in international large-scale assessments.
- Provides direction and stimulus for improving global educational assessment and student learning
- Is written by experts in the field, with an international perspective.
Survey researchers, market researchers and practitioners engaged in comparative projects will all benefit from the unparalleled breadth of knowledge and experience in large-scale educational assessments gathered in this one volume.
Edited by
Petra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia
Keith F. Rust, Westat and the University of Maryland at College Park, USA
Raymond J. Adams, Australian Council for Educational Research (ACER), Australia
Presents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments. In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD's Programme for International Student Assessment (PISA), the IEA's Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale. This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better use of the results. The book will also give readers an understanding of the resources needed to undertake and improve the design of educational assessments in their own countries and regions. Implementation of Large-Scale Education Assessments: Brings together the editors extensive experience in creating, designing, implementing, analysing and reporting results on a wide range of assessments. Emphasizes methods for implementing international studies of student achievement and obtaining highquality data from cognitive tests and contextual questionnaires. Discusses the methods of sampling, weighting, and variance estimation that are commonly encountered in international large-scale assessments. Provides direction and stimulus for improving global educational assessment and student learning Is written by experts in the field, with an international perspective. Survey researchers, market researchers and practitioners engaged in comparative projects will all benefit from the unparalleled breadth of knowledge and experience in large-scale educational assessments gathered in this one volume.
Edited by Petra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia Keith F. Rust, Westat and the University of Maryland at College Park, USA Raymond J. Adams, Australian Council for Educational Research (ACER), Australia
Title Page 5
Copyright Page 6
Contents 7
Notes on Contributors 17
Foreword 19
Acknowledgements 22
Abbreviations 23
Chapter 1 Implementation of Large-Scale Education Assessments 27
1.1 Introduction 27
1.2 International, Regional and National Assessment Programmes in Education 29
1.3 Purposes of LSAs in Education 30
1.3.1 Trend as a Specific Purpose of LSAs in Education 34
1.4 Key Areas for the Implementation of LSAs in Education 36
1.5 Summary and Outlook 42
Appendix 1.A 44
References 48
Chapter 2 Test Design and Objectives 52
2.1 Introduction 52
2.2 PISA 53
2.2.1 Purpose and Guiding Principles 53
2.2.2 Target Population 53
2.2.3 Sampling Approach 54
2.2.4 Assessment Content 55
2.2.5 Test Design 55
2.2.6 Link Items 56
2.3 TIMSS 60
2.3.1 Purpose and Guiding Principles 60
2.3.2 Target Population 60
2.3.3 Sampling Approach 62
2.3.4 Assessment Content 62
2.3.5 Test Design 64
2.4 PIRLS and Pre-PIRLS 67
2.4.1 Assessment Content 67
2.4.2 Test Design 68
2.5 ASER 71
2.5.1 Purpose and Guiding Principles 71
2.5.2 Target Population 72
2.5.3 Sampling Approach 73
2.5.4 Assessment Content 74
2.5.5 Test Design 75
2.6 SACMEQ 78
2.6.1 Purpose and Guiding Principles 78
2.6.2 Target Population 79
2.6.3 Sampling Approach 79
2.6.4 Assessment Content 80
2.6.5 Test Design 81
2.7 Conclusion 82
References 84
Chapter 3 Test Development 89
3.1 Introduction 89
3.2 Developing an Assessment Framework: A Collaborative and Iterative Process 91
3.2.1 What is an Assessment Framework? 92
3.2.2 Who Should Develop the Framework? 93
3.2.3 Framework Development as an Iterative Process 93
3.3 Generating and Collecting Test Material 94
3.3.1 How Should Assessment Material Be Generated? 95
3.3.2 Who Should Contribute the Material? 95
3.3.3 Processing Contributions of Assessment Material 97
3.4 Refinement of Test Material 98
3.4.1 Panelling of Test Material by Test Developers 99
3.4.2 Panelling Stimulus 99
3.4.3 Panelling Items 100
3.4.4 Cognitive Interviews and Pilot Studies 101
3.4.5 Preparations for Trial Testing 103
3.4.6 Analysis of Trial Test Data 104
3.5 Beyond Professional Test Development: External Qualitative Review of Test Material 107
3.5.1 Jurisdictional Representatives 107
3.5.2 Domain Experts 109
3.5.3 The Commissioning Body 110
3.5.4 Dealing with Diverse Views 110
3.6 Introducing Innovation 112
3.6.1 Case Study 1: The Introduction of Digital Reading in PISA 2009 113
3.6.2 Case Study 2: The Introduction of New Levels of Described Proficiency to PISA in 2009 and 2012 115
3.7 Conclusion 116
References 116
Chapter 4 Design, Development and Implementation of Contextual Questionnaires in Large-Scale Assessments 118
4.1 Introduction 118
4.2 The Role of Questionnaires in LSAs 119
4.3 Steps in Questionnaire Design and Implementation 121
4.3.1 Management of Questionnaire Development Process and Input from Relevant Stakeholders 121
4.3.2 Clarification of Aims and Content Priorities 122
4.3.3 Development of Questionnaires 133
4.3.4 Permissions (Copyright/IP) Requests 135
4.3.5 Cognitive Interviews with Respondents from the Target Population 135
4.3.6 Cultural/Linguistic Adaptations to Questionnaires 137
4.3.7 Ethics Application to Approach Schools and Students 138
4.3.8 Field Trial Questionnaire Administration 138
4.3.9 Analyses of Field Trial Data to Finalise the Questionnaire 139
4.3.10 Selection of Material for the Final MS Questionnaire 140
4.3.11 MS Questionnaire Administration 140
4.3.12 Preparation of Questionnaire Data for Public Release 140
4.4 Questions and Response Options in LSAs 141
4.5 Alternative Item Formats 145
4.6 Computer-Based/Online Questionnaire Instruments 154
4.6.1 Field Trial of Computer?Based Questionnaires 155
4.6.2 Beta Testing 157
4.7 Conclusion and Future Perspectives 157
Acknowledgements 158
References 158
Chapter 5 Sample Design, Weighting, and Calculation of Sampling Variance 163
5.1 Introduction 163
5.2 Target Population 164
5.2.1 Target Population and Data Collection Levels 164
5.2.2 Target Populations of Major Surveys in Education 165
5.2.3 Exclusion 169
5.3 Sample Design 170
5.3.1 Multistage Sample Design 170
5.3.2 Unequal Probabilities of Selection 171
5.3.3 Stratification and School Sample Size 172
5.3.4 School Nonresponse and Replacement Schools 173
5.4 Weighting 174
5.4.1 Reasons for Weighting 174
5.4.2 Components of the Final Student Weight 175
5.4.3 The School Base Weight 176
5.4.4 The School Base Weight Trimming Factor 177
5.4.5 The Within-School Base Weight 177
5.4.6 The School Nonresponse Adjustment 178
5.4.7 The Student Nonresponse Adjustment 178
5.4.8 Trimming the Student Weights 179
5.5 Sampling Adjudication Standards 179
5.5.1 Departures from Standards Arising from Implementation 181
5.6 Estimation of Sampling Variance 182
5.6.1 Introduction 182
5.6.2 Methods of Variance Estimation for Complex Samples 183
5.6.3 Replicated Variance Estimation Procedures for LSA Surveys 184
5.6.4 Computer Software for Variance Estimation 191
5.6.5 Concluding Remarks 191
References 192
Chapter 6 Translation and Cultural Appropriateness of Survey Material in Large-Scale Assessments 194
6.1 Introduction 194
6.2 Overview of Translation/Adaptation and Verification Approaches Used in Current Multilingual Comparative Surveys 195
6.2.1 The Seven Guiding Principles 196
6.2.2 Components from Current Localisation Designs 198
6.3 Step-by-Step Breakdown of a Sophisticated Localisation Design 200
6.3.1 Developing the Source Version(s) 200
6.3.2 Translation/Adaptation 208
6.3.3 Linguistic Quality Control: Verification and Final Check 208
6.4 Measuring the Benefits of a Good Localisation Design 210
6.4.1 A Work in Progress: Proxy Indicators of Translation/Adaptation Quality 212
6.4.2 The Focused MS Localisation Design 213
6.5 Checklist of Requirements for a Robust Localisation Design 216
References 217
Chapter 7 Quality Assurance 219
7.1 Introduction 219
7.2 The Development and Agreement of Standardised Implementation Procedures 220
7.3 The Production of Manuals which Reflect Agreed Procedures 222
7.4 The Recruitment and Training of Personnel in Administration and Organisation: Especially the Test Administrator and the School Coordinator 223
7.5 The Quality Monitoring Processes: Recruiting and Training Quality Monitors to Visit National Centres and Schools 224
7.5.1 National Quality Monitors 224
7.5.2 School-Level Quality Monitors 225
7.6 Other Quality Monitoring Procedures 227
7.6.1 Test Administration Session Reports 227
7.6.2 Assessment Review Procedures 228
7.6.3 Checking Print Quality (Optical Check) 228
7.6.4 Post-final Optical Check 228
7.6.5 Data Adjudication Processes 228
7.7 Conclusion 230
Reference 230
Chapter 8 Processing Responses to Open-Ended Survey Questions 231
8.1 Introduction 231
8.2 The Fundamental Objective 233
8.3 Contextual Factors: Survey Respondents and Items 233
8.4 Administration of the Coding Process 240
8.4.1 Design and Management of a Coding Process 241
8.4.2 Handling Survey Materials 244
8.4.3 Management of Data 244
8.4.4 Recruitment and Training of Coding Personnel 245
8.5 Quality Assurance and Control: Ensuring Consistent and Reliable Coding 247
8.5.1 Achieving and Monitoring Between-Coder Consistency 249
8.5.2 Monitoring Consistency across Different Coding Operations 251
8.6 Conclusion 255
References 255
Chapter 9 Computer-Based Delivery of Cognitive Assessment and Questionnaires 257
9.1 Introduction 257
9.2 Why Implement CBAs? 258
9.2.1 Assessment Framework Coverage 259
9.2.2 Student Motivation 259
9.2.3 Control of Workflow 260
9.2.4 Resource Efficiency 263
9.3 Implementation of International Comparative CBAs 264
9.3.1 Internet Delivery 264
9.3.2 Portable Application 267
9.3.3 Live System 268
9.4 Assessment Architecture 270
9.4.1 Test-Taker Registration 270
9.4.2 Navigation Architecture 271
9.4.3 Assessment Interface 271
9.4.4 Aspect Ratio 273
9.4.5 Accessibility Issues 273
9.5 Item Design Issues 273
9.5.1 Look and Feel 274
9.5.2 Digital Literacy 274
9.5.3 Translation 275
9.6 State-of-the-Art and Emerging Technologies 276
9.7 Summary and Conclusion 276
References 277
Chapter 10 Data Management Procedures 279
10.1 Introduction 279
10.2 Historical Review: From Data Entry and Data Cleaning to Integration into the Entire Study Process 280
10.3 The Life Cycle of a LSA Study 281
10.4 Standards for Data Management 282
10.5 The Data Management Process 284
10.5.1 Collection of Sampling Frame Information and Sampling Frames 286
10.5.2 School Sample Selection 287
10.5.3 Software or Web-Based Solutions for Student Listing and Tracking 288
10.5.4 Software or Web-Based Solutions for Within-School Listing and Sampling Procedures 289
10.5.5 Adaptation and Documentation of Deviations from International Instruments 291
10.5.6 The Translation Verification Process 292
10.5.7 Data Collection from Respondents 293
10.6 Outlook 298
References 300
Chapter 11 Test Implementation in the Field: The Case of PASEC 302
11.1 Introduction 302
11.2 Test Implementation 304
11.2.1 Human Resources 304
11.2.2 Sample Size and Sampling 304
11.2.3 PASEC’s Instruments 305
11.2.4 Cultural Adaptation and Linguistic Transposition of the Instruments 315
11.2.5 Preparation of Administrative Documents 315
11.2.6 Document Printing and Supplies Purchase 315
11.2.7 Recruitment of Test Administrators 316
11.2.8 Training, Preparation and Implementation 316
11.2.9 Test Administration 318
11.2.10 Supervision of the Field Work 320
11.2.11 Data Collection Report 320
11.3 Data Entry 320
11.4 Data Cleaning 321
11.5 Data Analysis 321
11.6 Governance and Financial Management of the Assessments 321
Acknowledgments 322
References 323
Chapter 12 Test Implementation in the Field: The Experience of Chile in International Large-Scale Assessments 324
12.1 Introduction 324
12.2 International Studies in Chile 328
12.2.1 Human Resources Required in the National Centre 328
12.2.2 Country Input into Instruments and Tests Development 330
12.2.3 Sampling 331
12.2.4 Preparation of Test Materials 333
12.2.5 Preparation and Adaptation of Administrative Documents (Manuals) 335
12.2.6 Preparation of Field Work 336
12.2.7 Actual Field Work 338
12.2.8 Coding Paper and Computer-Based Test 341
12.2.9 Data Entry Process 344
12.2.10 Report Writing 344
12.2.11 Dissemination 346
12.2.12 Final Words 346
Annex A 347
References 347
Chapter 13 Why Large-Scale Assessments Use Scaling and Item Response Theory 349
13.1 Introduction 349
13.2 Item Response Theory 351
13.2.1 Logits and Scales 353
13.2.2 Choosing an IRT Model 354
13.3 Test Development and Construct Validation 355
13.4 Rotated Test Booklets 371
13.5 Comparability of Scales Across Settings and Over Time 373
13.6 Construction of Performance Indicators 375
13.7 Conclusion 380
References 380
Chapter 14 Describing Learning Growth 383
14.1 Background 383
14.2 Terminology: The Elements of a Learning Metric 384
14.3 Example of a Learning Metric 386
14.4 Issues for Consideration 386
14.4.1 Number of Descriptions or Number of Levels 386
14.4.2 Mapping Domain Content onto the Scale 388
14.4.3 Alternative Approaches to Mapping Content to the Metric 389
14.5 PISA Described Proficiency Scales 391
14.5.1 Stage 1: Identifying Scales and Possible Subscales 392
14.5.2 Stage 2: Assigning Items to Subscales 395
14.5.3 Stage 3: Skills Audit 396
14.5.4 Stage 4: Analysing Preliminary Trial Data 397
14.5.5 Stage 5: Describing the Dimension 400
14.5.6 Stage 6: Revising and Refining with Final Survey Data 400
14.6 Defining and Interpreting Proficiency Levels 400
14.7 Use of Learning Metrics 405
Acknowledgement 406
References 407
Chapter 15 Scaling of Questionnaire Data in International Large-Scale Assessments 410
15.1 Introduction 410
15.2 Methodologies for Construct Validation and Scaling 412
15.3 Classical Item Analysis 413
15.4 Exploratory Factor Analysis 414
15.5 Confirmatory Factor Analysis 415
15.6 IRT Scaling 418
15.7 Described IRT Questionnaire Scales 422
15.8 Deriving Composite Measures of Socio-economic Status 425
15.9 Conclusion and Future Perspectives 430
References 431
Chapter 16 Database Production for Large-Scale Educational Assessments 437
16.1 Introduction 437
16.2 Data Collection 438
16.3 Cleaning, Recoding and Scaling 442
16.4 Database Construction 444
16.5 Assistance 447
References 449
Chapter 17 Dissemination and Reporting 450
17.1 Introduction 450
17.2 Frameworks 451
17.2.1 Assessment Frameworks 451
17.2.2 Questionnaire Frameworks 452
17.3 Sample Items 452
17.4 Questionnaires 453
17.5 Video 453
17.6 Regional and International Reports 454
17.7 National Reports 454
17.8 Thematic Reports 455
17.9 Summary Reports 455
17.10 Analytical Services and Support 456
17.11 Policy Papers 456
17.12 Web-Based Interactive Display 457
17.13 Capacity-Building Workshops 458
17.14 Manuals 458
17.15 Technical Reports 458
17.16 Conclusion 459
References 459
Index 462
EULA 469
| Erscheint lt. Verlag | 7.3.2017 |
|---|---|
| Reihe/Serie | Wiley Series in Survey Methodology | Wiley Series in Survey Methodology |
| Sprache | englisch |
| Themenwelt | Mathematik / Informatik ► Mathematik ► Angewandte Mathematik |
| Mathematik / Informatik ► Mathematik ► Statistik | |
| Mathematik / Informatik ► Mathematik ► Wahrscheinlichkeit / Kombinatorik | |
| Naturwissenschaften | |
| Sozialwissenschaften ► Pädagogik | |
| Technik | |
| Schlagworte | Angewandte Wahrscheinlichkeitsrechnung u. Statistik • Applied Probability & Statistics • Educational Researcher • ILSA • Implementation of large-scale Assessments • Implementation of Large-Scale Education Assessments • John Cresswell • Keith F. Rust • Methoden der Daten- u. Stichprobenerhebung • Petra Lietz • Raymond D. Adams • sampling, weighting, and variance estimation • Statistics • Statistics for Social Sciences • Statistik • Statistik in den Sozialwissenschaften • Survey Research Methods & Sampling |
| ISBN-10 | 1-118-76247-9 / 1118762479 |
| ISBN-13 | 978-1-118-76247-9 / 9781118762479 |
| Informationen gemäß Produktsicherheitsverordnung (GPSR) | |
| Haben Sie eine Frage zum Produkt? |
PDF (Adobe DRM)Kopierschutz: Adobe-DRM
Adobe-DRM ist ein Kopierschutz, der das eBook vor Mißbrauch schützen soll. Dabei wird das eBook bereits beim Download auf Ihre persönliche Adobe-ID autorisiert. Lesen können Sie das eBook dann nur auf den Geräten, welche ebenfalls auf Ihre Adobe-ID registriert sind.
Details zum Adobe-DRM
Dateiformat: PDF (Portable Document Format)
Mit einem festen Seitenlayout eignet sich die PDF besonders für Fachbücher mit Spalten, Tabellen und Abbildungen. Eine PDF kann auf fast allen Geräten angezeigt werden, ist aber für kleine Displays (Smartphone, eReader) nur eingeschränkt geeignet.
Systemvoraussetzungen:
PC/Mac: Mit einem PC oder Mac können Sie dieses eBook lesen. Sie benötigen eine
eReader: Dieses eBook kann mit (fast) allen eBook-Readern gelesen werden. Mit dem amazon-Kindle ist es aber nicht kompatibel.
Smartphone/Tablet: Egal ob Apple oder Android, dieses eBook können Sie lesen. Sie benötigen eine
Geräteliste und zusätzliche Hinweise
Buying eBooks from abroad
For tax law reasons we can sell eBooks just within Germany and Switzerland. Regrettably we cannot fulfill eBook-orders from other countries.
aus dem Bereich
