In this paper we discuss the results of a study of students' academic performance in first year general education courses. Using data from 566 students who received intensive academic advising as part of their enrollment in the institution's pre-major/general education program, we investigate individual student, organizational, and disciplinary factors that might predict a students' potential classification in an Early Warning System as well as factors that predict improvement and decline in their academic performance. Disciplinary course type (based on Biglan's  typology) was significantly related to a student's likelihood to enter below average performance classifications. Students were the most likely to enter a classification in fields like the natural science, mathematics, and engineering in comparison to humanities courses. We attribute these disparities in academic performance to disciplinary norms around teaching and assessment. In particular, the timing of assessments played a major role in students' ability to exit a classification. Implications for the design of Early Warning analytics systems as well as academic course planning in higher education are offered.
In recent years, Massive Open Online Courses (MOOCs) have become a phenomenon offering the possibility to teach thousands of participants simultaneously. In the same time the platforms used to deliver these courses are still in their fledgling stages. While course content and didactics of those massive courses are the primary key factors for the success of courses, still an smart platform may increase or decrease the learners experience and his learning outcome. This paper at hand proposes the usage of an A/B testing framework that is able to be used within an microservice architecture to validate hypotheses about how learners use the platform and to enable data-driven decisions about new features and settings. To evaluate this framework three new features (Onboarding Tour, Reminder Mails and a Pinboard Digest) have been identified based on a user survey. They have been implemented and introduced on two large MOOC platforms and their influence on the learners behavior have been measured. Finally this paper proposes a data driven decision workflow for the introduction of new features and settings on e-learning platforms.
We study how the use of online learning systems stimulates cognitive activities, by conducting an experiment with the use of eye tracking technology to monitor eye fixations of 60 final year students engaging in online interactive tutorials at the start of their Final Year Project module. Our findings show that the students' learning behaviours fall into three different types of eye fixation patterns, and the data corresponding to the different types of learners relates to the performance of the students in other related academic modules. We conclude that this method of studying eye fixation patterns can identify different types of learners with respect to their cognitive capability and academic potential, and also allow educators to understand how their instructional design and online learning environment can stimulate higher-order cognitive activities.
Statway is one of the Community College Pathways initiatives designed to promote students' success in their developmental math sequence and reduce the time required to earn college credit. A recent causal analysis confirmed that Statway dramatically increased students' success rates in half the time across two different cohorts. These impressive results also obtain results across the gender and race/ethnicity groups. However, there is still room for improvement. Students who did not succeed overall in Statway often did not complete the first of the two-course sequence. Therefore, the objective of this study is to formulate a series of indicators from self-report and online learning system data, alerting instructors to students' progress during the first weeks of the first course in the Statway sequence.
Student mistakes are often not random but, rather, reflect thoughtful yet incorrect strategies. In order for educational technologies to make full use of students' performance data to estimate the knowledge of a student, it is important to model not only the conceptions but also the misconceptions that a student's particular pattern of successes and errors may indicate. The student models that drive the 'outer loop' of Intelligent Tutoring Systems typically only track positive skills and conceptions, not misconceptions. Here, we present a method of representing misconceptions in the kinds of Knowledge Component models, or Q-Matrices, that are used by student models to estimate latent knowledge. We show, in a case study on a fraction arithmetic dataset, that incorporating a misconception into the Knowledge Component model dramatically improves model fit. We also derive qualitative insights from comparing predicted learning curves across models that incorporate varying misconception-related parameters. Finally, we show that the inclusion of a misconception in a Knowledge Component model can yield individual student estimates of misconception strength. These estimates are significantly correlated with out-of-tutor individual measures of student errors indicative of the misconception.
With the aim of better scaffolding discussion to improve learning in a MOOC context, this work investigates what kinds of discussion behaviors contribute to learning. We explored whether engaging in higher-order thinking behaviors results in more learning than paying general or focused attention to course materials. In order to evaluate whether to attribute the effect to engagement in the associated behaviors versus persistent characteristics of the students, we adopted two approaches. First, we used propensity score matching to pair students who exhibit a similar level of involvement in other course activities. Second, we explored individual variation in engagement in higher-order thinking behaviors across weeks. The results of both analyses support the attribution of the effect to the behavioral interpretation. A further analysis using LDA applied to course materials suggests that more social oriented topics triggered richer discussion than more biopsychology oriented topics.
The 'doer effect' is an association between the number of online interactive practice activities students' do and their learning outcomes that is not only statistically reliable but has much higher positive effects than other learning resources, such as watching videos or reading text. Such an association suggests a causal interpretation--more doing yields better learning--which requires randomized experimentation to most rigorously confirm. But such experiments are expensive, and any single experiment in a particular course context does not provide rigorous evidence that the causal link will generalize to other course content. We suggest that analytics of increasingly available online learning data sets can complement experimental efforts by facilitating more widespread evaluation of the generalizability of claims about what learning methods produce better student learning outcomes. We illustrate with analytics that narrow in on a causal interpretation of the doer effect by showing that doing within a course unit predicts learning of that unit content more than doing in units before or after. We also provide generalizability evidence across four different courses involving over 12,500 students that the learning effect of doing is about six times greater than that of reading.
This paper describes the development and evaluation of an affect-aware intelligent support component that is part of a learning environment known as iTalk2Learn. The intelligent support component is able to tailor feedback according to a student's affective state, which is deduced both from speech and interaction. The affect prediction is used to determine which type of feedback is provided and how that feedback is presented (interruptive or non-interruptive). The system includes two Bayesian networks that were trained with data gathered in a series of ecologically-valid Wizard-of-Oz studies, where the effect of the type of feedback and the presentation of feedback on students' affective states was investigated. This paper reports results from an experiment that compared a version that provided affect-aware feedback (affect condition) with one that provided feedback based on performance only (non-affect condition). Results show that students who were in the affect condition were less off-task, a result that was statistically significant. Additionally, the results indicate that students in the affect condition were less bored. The results also show that students in both conditions made learning gains that were statistically significant, while students in the affect condition had higher learning gains than those in the non-affect condition, although this result was not statistically significant in this study's sample. Taken all together, the results point to the potential and positive impact of affect-aware intelligent support.
Many pedagogical models in the field of learning analytics are implicit and do not overtly direct learner behavior. While this allows flexibility of use, this could also result in misaligned practice, and there are calls for more explicit pedagogical models in learning analytics. This paper presents an explicit pedagogical model, the Team and Self Diagnostic Learning (TSDL) framework, in the context of collaborative inquiry tasks. Key informing theories include experiential learning, collaborative learning, and the learning analytics process model. The framework was trialed through a teamwork competency awareness program for 14 year old students. A total of 272 students participated in the program. This paper foregrounds students' and teachers' evaluative accounts of the program. Findings reveal positive perceptions of the stages of the TSDL framework, despite identified challenges, which points to its potential usefulness for teaching and learning. The TSDL framework aims to provide theoretical clarity of the learning process, and foster alignment between learning analytics and the learning design. The current work provides trial outcomes of a teamwork competency awareness program that used dispositional analytics, and further efforts are underway to develop the discourse layer of the analytic engine. Future work will also be dedicated to application and refinement of the framework for other contexts and participants, both learners and teachers alike.
The adequate emotional state of students has proved to be essential for favoring learning. This paper explores the possibility of obtaining students' feedback about the emotions they feel in class in order to discover potential emotion patterns that might indicate learning fails. This paper presents a visual dashboard that allows students to track their emotions and follow up on their evolution during the course. We have compiled the principal classroom related emotions and developed a two-phase inquiry process to: verify the possibility to measure students' emotions in classroom; discover how emotions can be displayed to promote self-reflection; and confirm the impact of emotions on learning performance. Our results suggest that students' emotions in class are related to evaluation marks. This shows that early information about students' emotions can be useful for teachers and students to improve classroom results and learning outcomes.