Stability and sensitivity of learning analytics based prediction models

Type: Evidence | Proposition: B: Teaching | Polarity: | Sector: | Country:

Evidence of the month: July 2015

This paper focuses on the issues of stability and sensitivity of learning-analytics-based prediction models. It investigates whether prediction models remain the same, when the instructional context is repeated with a new cohort of students, and whether prediction models change when relevant aspects of the instructional context are adapted.

The research applies Buckingham Shum and Deakin Crick’s theoretical framework of dispositional learning analytics: an infrastructure that combines learning dispositions data with data extracted from computer-assisted, formative assessments and learning management systems.

The paper compares two cohorts of a large introductory quantitative methods module, with 1,005 students in the 2013-2014 cohort, and 1,006 students in the 2014-2015 cohort. Both modules were based on principles of blended learning – combining face-to-face problem-based learning sessions with e-tutorials – and had a similar instructional design, except for an intervention into the design of quizzes administered in the module. Focusing on their predictive power, the paper provides evidence of the stability and the sensitivity of regression-type prediction models.

The paper reports that the value of dispositional data is strongly dependent on the time at which richer (assessment) data become available, and on the need for timely signalling of under performance. If timely feedback is required, the combination of data extracted from e-tutorials (both in practice and test modes) and learning disposition data was found to be the best mix to support learning analytics applications.

Feedback related to learning dispositions (for example, by flagging suboptimal learning strategies, or inappropriate learning regulation) is generally open to interventions to improve the learning process. The same is true of feedback related to suboptimal use of e-tutorials; it is both predictive and open for intervention.

This paper won Best Paper award at the 7th International conference on Computer Supported Education

Citation: Tempelaar, D. T.; Rienties, B. and Giesbers, B. (2015). Stability and sensitivity of Learning Analytics based prediction models. In: Proceedings of 7th International conference on Computer Supported Education (Helfert, Markus ; Restivo, Maria Teresa; Zvacek, Susan and Uho, James eds.), 23-25 May 2015, Lisbon, Portugal, CSEDU, pp. 156�166. | Url: