Analysis of Large-Scale Assessment Data
Autumn 2021 (see semester page for autumn 2021 for detailed schedule)
Dr. Isa Steinmann
This workshop aims to communicate both knowledge and hands-on analytical skills in the field of international large-scale assessment (ILSA) data. ILSAs like PIRLS (Progress in International Reading Literacy Study), TIMSS (Trends in International Mathematics and Science Study), or PISA (Programme for International Student Assessment) provide unique opportunities to investigate both substantive and methodological research questions. ILSAs are conducted in recurring cycles and in numerous countries. They assess data on different levels, often including education systems (e.g., curriculum information, quantity of schooling), schools (e.g., school resources, composition), teachers (e.g., qualification for teaching, teaching practices), students (e.g., achievement in standardized tests, learning motivations), or familial backgrounds (e.g., socio-economic status, home support for learning). The aim of the international design of the tests and questionnaires is to make educational inputs, frameworks, and outputs comparable across countries and time. By implication, these studies face numerous methodological challenges and provide a plethora of interesting research opportunities.
This workshop addresses the following ILSA-related topics:
- Central aims
- Sampling
- Target populations
- Sampling procedures and representativeness of the data
- Data collection
- Instrument development
- Data collection procedures
- Quality assurance measures
- ILSA data
- Achievement tests
- Questionnaire data
- Comparison of educational systems
- Central limitations and challenges
The workshop encompasses lecture and interaction formats, group work, as well as practical exercises in the R environment. The required literature has to be read before the respective sessions. Further optional literature is recommended.
After a successful completion of the course, the students
- have a broad knowledge about ILSAs’ backgrounds, methods, and scope,
- can critically read and interpret results of ILSA reports and ILSA-based studies,
- can handle methodological peculiarities of ILSA data in own analyses, and
- can develop own ILSA-based research questions and approaches.
The workshop covers 13 sessions. After the course, students are required to hand in an outline of a written assignment for which individual feedback will be provided. These outlines have to be approved before the students submit the final versions of their assignment papers.
Participation in the course requires the formal criteria described on the course page as well as good knowledge about basic statistical methods. The participants are expected to bring laptops with administrator rights and the newest version of R installed.
Click here to read about the requirements for assignments in this workshop
Machine Learning in Education
Autumn 2021 (see semester page for autumn 2021 for detailed schedule)
Dr. Denise Reis Costa
Objective: This workshop will introduce participants to the field of machine learning (ML) for the analysis of educational data. It will favor a hands-on approach to grow an intuitive understanding of ML algorithms and the associated theory. Two types of ML methods will be discussed: supervised and unsupervised learning from data. Whereas supervised methods mainly focus on prediction, unsupervised methods aim to build descriptive models.
Participants will develop and structure codes in R to apply these techniques in the context of fields such as educational data mining and learning analytics. Proper scientific practice and ethical conduct will be emphasized throughout the workshop.
Learning outcomes: After completing, participants will:
- Have an understanding of supervised and unsupervised learning algorithms as applied to educational data
- Be familiar with basic principles underlying ML as well as with the interpretation of typical ML output and results
- Have experience in using machine learning tools in R
Structure: The workshop includes lectures, group work, as well as practical exercises in the R environment.
Information about the assignment will be given in Canvas.
The Uses of Process Data in Educational Assessment
Dr Bryan Maddox, Professor of Educational Assessment at the University of East Anglia, England.
Spring 2022 (early January). The schedule will be published on the semester pages for spring 2022 in the end of November 2021.
This workshop will introduce participants to the collection and use of multimodal process data in educational assessment to improve assessment design and validation, user experience, quality control, and to deepen the analysis of performance data. We will look at how small data is used to deepen the analysis of big data in large-scale assessments, how it informs next generation assessment designs, and digital learning analytics. The workshop will introduce the participants to the key sources of multimodal process data, including log data, the use of video, eye tracking, cognitive interviews, and retrospective think aloud. We will investigate their strengths and weaknesses, and how they inform large-scale educational assessment. The workshop will involve 4 days of teaching and tutorials delivered over two weeks. It will involve participatory activities, reading of research literature, and critical discussion and critical reflection on theory and methods. By the end of the workshop, the participants will have gained sufficient knowledge and skills to enable you to collect and use process data with confidence, and to understand its significance in contemporary educational measurement.
Information about the assignment will be given in Canvas.
Methods for Causal Inference in Educational Research
Spring 2022 (mid-January until end of February). The schedule will be published on the semester pages for spring 2022 in the end of November 2021.
Dr. Isa Steinmann
This workshop aims to communicate theoretical knowledge about causal and non-causal inference, illustrate methods for causal inference from experimental and observational data, and exemplify hands-on applications of the methods in R.
Causal research questions aim to isolate effects of a treatment on those who got the treatment, independent of other effects and differences between the treated and untreated. The most straightforward way to answer causal research questions is to conduct randomized trials like in pharmaceutical studies, for instance. In many fields including education, randomized trials are however very difficult to conduct, due to practical, ethical, or financial reasons, among others. Therefore, educational research often has to rely on observational data, i.e. information that stems from pure observations of educational processes and outcomes without an interference of the researchers. Under specific circumstances, it is however possible to isolate causal effects anyway. This course will cover some of these methods both from a theoretical and practical perspective. Real example studies from the field of education will help to understand the assumptions and prerequisites behind these methods and to evaluate their scope critically.
The workshop will cover the following topics:
- The potential outcome framework
- Randomized trials
- Regression models
- Instrumental variable approaches
- Propensity score matching
- Regression discontinuity designs
- Differences-in-differences approaches
The workshop encompasses lecture and interaction formats, group work, as well as practical exercises in the R environment. After a successful completion of the course, the students
- have a deep understanding of the potential outcome framework,
- have a broad knowledge about the assumptions, prerequisites, and procedures of randomized trials, regression models, propensity score matching, regression discontinuity designs, differences-in-differences approaches, and instrumental variable approaches,
- can critically read and interpret results of causal inference studies,
- can apply methodological peculiarities of the causal inference methods in own analyses, and
- can develop own causal inference research questions and approaches.
Further information about the assignments, the schedule, as well as participation prerequisites and preparations follow in due time.