Digital student monitoring systems (DSMS) usage results, in comparison to other interventions aimed at small groups of pupils, in a positive and relatively large impact on learning outcomes.
More and more Dutch schools use digital student management systems (Inspectorate of Education, 2013). Also in other countries, and continents schools increasingly make use of such systems, e.g., as a result of the increase in assessment tests (Heritage & Yeagley, 2005). Digital student monitoring systems (DSMSs) can be defined as systems through which teachers receive feedback based on test results about the results of the education provided.
The information received by teachers on the basis of assessment tests can be considered as feedback to teachers about the results of his or her teaching (Visscher & Coe, 2002). Teachers can adjust their instructions based on this feedback so that the teaching is tailored to the specific learning needs of students. When this is the case, the use of a DSMS can result in higher learning outcomes. Both governments and schools therefore heavily invest in DSMSs (Ministry of Education, Culture and Science, 2007). It is therefore important to investigate whether the use of DSMSs actually leads to higher learning results.
In this research, the results of a meta-analysis are described in which this connection has been investigated. For this meta-analysis qualitatively strong experimental studies on the effects of DSMSs are selected. The following questions have been answered with the analysis:
- What is the effect of the use of digital student management systems by teachers on student achievements?
- What factors hinder or facilitate the intended effect of the use of digital student management systems on student performance?
In the meta-analysis 40 effects are included derived from fifteen different studies. The studies have been found in one of the six databases. A systematic search for relevant studies has been conducted on the basis of predefined keyword lists. Also 126 international contacts were approached and asked if they knew of other relevant studies, or performed relevant studies themselves. In the databases 38 studies were found, and 32 studies were found by accessing the contacts. The studies were read and evaluated by two researchers who eventually determined that fifteen studies met all predefined content and methodological criteria.
The effects of the selected studies were calculated using the Cohen's d and Hedges's g formulas.
Each effect had a weight attached that determined how influencing the corresponding effects were in the average effect size. These weights were assigned based on the variance within the effects. Use has been made of a random effect model for the determination of the average of the effect size.
The analyzes were performed with the program Comprehensive Meta-analyzes.
In the studies in which the aim of the intervention was to increase the learning outcomes of small groups of students (for example, the intervention was directed at a specific number of students within a class) a significant effect size of 0.4 was found (see Table 5). DSMS usage results, in comparison to other interventions aimed at small groups of pupils, in a positive and relatively large impact on learning outcomes. In the studies where the intervention was aimed at increasing the learning outcomes of all students within an entire school or school board a significant effect size 0.06 has been found (see Table 5). DSMS usage, in comparison to other interventions aimed at schools, results in a slightly lower than average effect on learning outcomes.
The following factors seem to enhance the intended effect of a DSMS on outcomes:
- high feedback frequency,
- systems that apart from feedback also provide advice on instruction and processing methods that match the feedback received,
- a supportive intervention which takes place at least monthly.
The on average high effect of a DSMS in studies with intervention aimed at small groups was not found in the studies of interventions aimed at schools or school boards. It is therefore worth further investigating how the successful approach of the use of DSMS for small groups of students can be translated into approaches at school or school board level resulting in comparable high effects.