Implementation of Large-Scale Education Assessments (eBook)

Petra Lietz, John C. Cresswell, Keith F. Rust, Raymond J. Adams (Herausgeber)

eBook Download: EPUB

2017
John Wiley & Sons (Verlag)
978-1-118-76249-3 (ISBN)

Infos

Presents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments.

In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD's Programme for International Student Assessment (PISA), the IEA's Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale.

This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better use of the results. The book will also give readers an understanding of the resources needed to undertake and improve the design of educational assessments in their own countries and regions.

Implementation of Large-Scale Education Assessments:

• Brings together the editors' extensive experience in creating, designing, implementing, analysing and reporting results on a wide range of assessments.
• Emphasizes methods for implementing international studies of student achievement and obtaining highquality data from cognitive tests and contextual questionnaires.
• Discusses the methods of sampling, weighting, and variance estimation that are commonly encountered in international large-scale assessments.
• Provides direction and stimulus for improving global educational assessment and student learning
• Is written by experts in the field, with an international perspective.

Survey researchers, market researchers and practitioners engaged in comparative projects will all benefit from the unparalleled breadth of knowledge and experience in large-scale educational assessments gathered in this one volume.

Edited by
Petra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia

Keith F. Rust, Westat and the University of Maryland at College Park, USA

Raymond J. Adams, Australian Council for Educational Research (ACER), Australia

Inhalt

1. Implementation of Large-Scale Education Assessments

2. Test Design and Objectives

3. Test development

4. Design, development and implementation of contextual questionnaires in large-scale assessments

5. Sample Design, Weighting, and Calculation of Sampling Variance

6. Translation and cultural appropriateness of survey material in large scale assessments

7. Quality assurance

8. Processing Responses to Open-Ended Survey Questions

9. Computer-based Delivery of Cognitive Assessment and Questionnaires

10. Data management procedures

11. Test Implementation in the Field: The Case of PASEC

12. Test Implementation in the Field: The Experience of Chile in International Large Scale Assessments

13. Why Large Scale Assessments Use Scaling and Item Response Theory Scale Assessments Use Scaling and Item Response Theory

14. Describing Learning Growth

15. Scaling of Questionnaire Data in International Large-Scale Assessments

16. Database Production for Large-Scale Educational Assessments

17. Dissemination and Reporting

Textauszug

1
Implementation of Large‐Scale Education Assessments

Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams

1.1 Introduction

The 60 years that followed a study of mathematics in 12 countries conducted by the International Association for the Evaluation of Educational Achievement (IEA) in 1964 have seen a proliferation of large‐scale assessments (LSAs) in education. In a recent systematic review of the impact of LSAs on education policy (Best et al., 2013), it was estimated that LSAs in education are now being undertaken in about 70% of the countries in the world.

The Programme for International Student Assessment (PISA) conducted by the Organisation for Economic Co‐operation and Development (OECD) was implemented in 75 countries in 2015 with around 510 000 participating students and their schools. Similarly, the Trends in International Mathematics and Science Study (TIMSS), conducted by the IEA, collected information from schools and students in 59 countries in 2015.

This book is about the implementation of LSAs in schools which can be considered to involve 13 key areas. These start with the explication of policy goals and issues, assessment frameworks, test and questionnaire designs, item development, translation and linguistic control as well as sampling. They also cover field operations, technical standards, data collection, coding and management as well as quality assurance measures. Finally, test and questionnaire data have to be scaled and analysed while a database is produced and accompanied by dissemination and the reporting of results. While much of the book has been written from a central coordinating and management perspective, two chapters illustrate the actual implementation of LSAs which highlight the requirements regarding project teams and infrastructure required for participation in such assessments. Figure 1.2 in the concluding section of this chapter provides details regarding where each of these 13 key areas is covered in the chapters of this book.

Participation in these studies, on a continuing basis, is now widespread, as is indicated in . Furthermore, their results have become integral to the general public discussion of educational progress and international comparisons in a wide range of countries with the impact of LSAs on education policy being demonstrated (e.g. Baker & LeTendre, 2005; Best et al., 2013; Breakspear, 2012; Gilmore, 2005). Therefore, it seems timely to bring together in one place the collective knowledge of those who routinely conduct these studies, with the aim of informing users of the results as to how such studies are conducted and providing a handbook for future practitioners of current and prospective studies.

While the emphasis throughout the book is on the practical implementation of LSAs, it is grounded in theories of psychometrics, statistics, quality improvement and survey communication. The chapters of this book seek to cover in one place almost every aspect of the design, implementation and analysis of LSAs, (see Figure 1.2), with perhaps greater emphasis on the aspects of implementation than can be found elsewhere. This emphasis is intended to complement other recent texts with related content but which have a greater focus on the analysis of data from LSAs (e.g. Rutkowski, von Davier & Rutkowski, 2013).

This introductory chapter first provides some context in terms of the development of international, regional and national assessments and the policy context in which they occur. Then, the purposes for countries to undertake such assessments, particularly with a view to evidence‐based policymaking in education, are discussed. This is followed by a description of the content of the book. The chapter finishes with considerations as to where LSAs might be headed and what is likely to shape their development.

1.2 International, Regional and National Assessment Programmes in Education

The IEA first started a programme of large‐scale evaluation studies in education with a pilot study to explore the feasibility of such an endeavour in 1959–1961 (Foshay et al., 1962). After the feasibility study had shown that international comparative studies in education were indeed possible, the first content areas to be tested were mathematics with the First International Mathematics Study conducted by 12 countries in 1962–1967 (Husén, 1967; Postlethwaite, 1967) and the content areas of the six subject surveys, namely, civic education, English as a foreign language, French as a foreign language, literature education, reading, comprehension and science, conducted in 18 countries in 1970–1971. Since then, as can be seen in Appendix 1.A, participation in international studies of education has grown considerably with 59 and 75 countries and economies, respectively, participating in the latest administrations of the TIMSS by the IEA in 2015 and the PISA by the OECD in 2015.

In addition to international studies conducted by the IEA since the late 1950s and by the OECD since 2000, commencing in the mid 1990s, three assessment programmes with a regional focus have been designed and implemented. First, the Conference of Education Ministers of Countries Using French as the Language of Communication (Conférence des ministres de l’Education des États et gouvernements de la Francophonie – CONFEMEN) conducts the Programme d’Analyse des Systèmes Educatifs de la CONFEMEN (PASEC). Since its first data collection in 1991, assessments have been undertaken in over 20 francophone countries not only in Africa but other parts of the world (e.g. Cambodia, Laos and Vietnam). Second, the Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ), with the support of the UNESCO International Institute for Educational Planning (IIEP) in Paris, has undertaken four data collections since 1995, with the latest assessment in 2012–2014 (SACMEQ IV) involving 15 countries in Southeast Africa. Third, the Latin‐American Laboratory for Assessment of the Quality in Education (LLECE is the Spanish acronym), with the assistance of UNESCO’s Regional Bureau for Education in Latin America and the Caribbean (UREALC), has undertaken three rounds of data collection since 1997, with 15 countries participating in the Third Regional Comparative and Explanatory Study (TERCE) in 2013. First steps towards an assessment in the Asia‐Pacific region are currently being undertaken through the Southeast Asian Primary Learning Metrics (SEA‐PLM) initiative.

In terms of LSAs of student learning, a distinction is made here between LSAs that are intended to be representative of an entire education system, which may measure and monitor learning outcomes for various subgroups (e.g. by gender or socio‐economic background), and large‐scale examinations that are usually national in scope and which report or certify individual student’s achievement (Kellaghan, Greaney & Murray, 2009). Certifying examinations may be used by education systems to attest achievement at the end of primary or secondary education, for example, or education systems may use examinations to select students and allocate placements for further or specialised study, such as university entrance or scholarship examinations. The focus of this book is on the implementation of LSAs of student learning that are representative of education systems, particularly international assessments that compare education systems and student learning across participating countries.

Parallel to the growth in international assessments, the number of countries around the world administering national assessments in any year has also increased – from 28 in 1995 to 57 in 2006 (Benavot & Tanner, 2007). For economically developing countries in the period from 1959 to 2009, Kamens and Benavot (2011) reported the highest number of national assessments in one year as 37 in 1999. Also in the 1990s, most of the countries in Central and South America introduced national assessments (e.g. Argentina, Bolivia, Brazil, Colombia, Dominican Republic, Ecuador, El Salvador, Guatemala, Paraguay, Peru, Uruguay and Venezuela) through the Partnership for Educational Revitalization in the Americas (PREAL) (Ferrer, 2006) although some introduced them earlier (e.g. Chile in 1982 and Costa Rica in 1986).

International, regional and national assessment programmes can all be considered as LSAs in education. While this book focuses mainly on international assessment programmes conducted in primary and secondary education, it also contains examples and illustrations from regional and national assessments where appropriate.

1.3 Purposes of LSAs in Education

Data from LSAs provide information regarding the extent to which students of a particular age or grade in an education system are learning what is expected in terms of certain content and skills. In addition, they assess differences in achievement levels by subgroups such as gender or region and factors that are correlated with different levels of achievement. Thus, a general purpose of participation in LSAs is to obtain information on a system’s educational outcomes and – if questionnaires are administered to obtain background information from students, teachers, parents and/or schools – the associated factors, which, in turn, can assist policymakers and other stakeholders in the education system in making policy and resourcing decisions for improvement (Anderson, Chiu & Yore, 2010; Benavot & Tanner, 2007; Braun, Kanjee & Bettinger, 2006;...

Produkt-Details

• Erscheint lt. Verlag: 7.3.2017
• Reihe/Serie: Wiley Series in Survey Methodology
• Reihe/Serie: Wiley Series in Survey Methodology
• Reihe/Serie: Wiley Series in Survey Methodology
• Sprache: englisch
• Themenwelt: Mathematik / Informatik ► Mathematik ► Angewandte Mathematik
• Themenwelt: Mathematik / Informatik ► Mathematik ► Statistik
• Themenwelt: Mathematik / Informatik ► Mathematik ► Wahrscheinlichkeit / Kombinatorik
• Themenwelt: Naturwissenschaften
• Themenwelt: Sozialwissenschaften ► Pädagogik
• Themenwelt: Technik
• Schlagworte: Angewandte Wahrscheinlichkeitsrechnung u. Statistik • Applied Probability & Statistics • Educational Researcher • ILSA • Implementation of large-scale Assessments • Implementation of Large-Scale Education Assessments • John Cresswell • Keith F. Rust • Methoden der Daten- u. Stichprobenerhebung • Petra Lietz • Raymond D. Adams • sampling, weighting, and variance estimation • Statistics • Statistics for Social Sciences • Statistik • Statistik in den Sozialwissenschaften • Survey Research Methods & Sampling
• ISBN-10: 1-118-76249-5 / 1118762495
• ISBN-13: 978-1-118-76249-3 / 9781118762493