Tag Archives: ev-qualitative

Evidence is primarily from rigorous qualitative research methods.

Type: Evidence | Proposition: A: Learning | Polarity: | Sector: | Country:

There is no proven direct effect between the availability of this form of learning analytics and learning outcomes. Leeuwen, A. van,  Janssen, J., Erkens, G. (2014) Effecten van learning analytics bij computer ondersteund samenwerkend leren, Kennisnet, 4W Issue 4-2014

4W Issue 4-2014

When students work together in a computer-assisted learning environment, it is difficult to get a good picture of the progress of the groups. Learning analytics, where data on the learning activities of students are displayed graphically, helps teachers to better guide the groups and make students aware of the cooperation process. The expectation is that insight into the course of the cooperation process contributes not only to better cooperation, but also results in better educational results.

It is of all times: in a group assignment one of the group members is taking the task not serious and as a result a part of the assignment stagnates and / or more work ends up on the shoulders of others. As a teacher you hear this often afterwards, when the assignment has already been submitted. In group assignments in a computer-assisted learning environment these kind of obstacles already should have come to light in an earlier stage as the teacher can watch the cooperation process, thanks to the information gathered by the computer system on activities of students. In practice, however, the amount of information collected is so large that teachers cannot see the trees from the forest. By visualizing relevant data, the information is much more manageable: students see from each other how much everyone contributes and the teacher understands the process. This graphical representation of student activities is a form of learning analytics.

Effect of learning analytics on guidance by the teacher

Teachers can get feed back from the system on the activities of the students in the form of visualizations as well. Can they can better assist students with this information (arrow 3 -> 4 in Figure 1)? Our research on the Participation Tool (Van Leeuwen et al.) shows that teachers, thanks to information of the individual contributions of group members, can better detect imbalances in the groups and act on them. Then they seem to send relatively more messages to groups where participation is problematic then to groups in which this is not the case. In addition, their assessment of the participation is more specific because they can accurately identify which group member does not participate proportionally.

In a study completed at the time, teachers were given information about which task-relevant key words students use in their discussions (Draft Trail Tool) and to which extent they made progress on the task. The result was that teachers in general were sending more messages, and again often to groups with task problems, for instance, when there was discussions about things other than the subject of the task.

The presence of learning analytics not only seems to assist teachers in keeping an eye on the problems, but also to activate them to provide help to groups with problems.

Effect of learning analytics on learning outcomes

In summary, we can conclude that learning analytics have beneficial effects on the course of collaboration. Understanding participation ensures that the attitude of students changes and becomes more active, as well as the guidance by the teachers. Reasoning back, it was expected that this would also lead to better learning outcomes (arrow 5 in Figure 1). Is that the case? No, this expectation is to date not supported by research: there is no proven direct effect between the availability of this form of learning analytics and learning outcomes. All students have a better understanding of the participation of their group members, and although teachers can guide the groups better on the basis of learning analytics, it does not provide better group outcomes, nor a better individual performance. Therewith the added value of learning analytics is demonstrable for the cooperation process, but not for the learning outcomes.

Citation: Leeuwen, A. van, Janssen, J., Erkens, G. (2014) Effecten van learning analytics bij computer ondersteund samenwerkend leren, Kennisnet, 4W Issue 4-2014 | Url: http://4w.kennisnet.nl/artikelen/2014/12/10/effecten-van-learning-analytics-bij-computer-onder/

Type: Evidence | Proposition: A: Learning | Polarity: | Sector: | Country:

The right mix of forms of control (in adaptive learning systems E.B.) can provide a positive contribution to the learning process of learners.

In this research report (in Dutch), the influence of digital learning resources with adaptive features and capabilities on student learning is assessed through a qualitative, exploratory research design.

In light of the introduction of Tailored education in August 2014 and the high expectations of the role that ICT could play, this report tries to provide insights into the extent to which certain adaptive characteristics and capabilities of digital programs can have a positive effect on student learning.

The study was conducted at 37 students divided between middle and upper primary education and class 2 HAVO / VWO of secondary education.

In the study 12 girls and 25 boys participated. Regarding the digital learning resources, the choice has been made for four digital learning resources / programs and learning tasks in the field of Language / Dutch.

The general question of the research is as follows:

What impact do adaptive characteristics and capabilities of digital learning resources have on student learning?

From the findings regarding the theoretical framework, the following research questions were formulated.

Regarding the theoretical framework following conclusions are drawn:

  • Little is known about the effects of adaptive digital learning resources
  • The effects seem to be especially in acquiring knowledge of highly structured knowledge domains and less in the acquisition of abstract and complex competencies
  • A form of shared control seems to be preferred over full learner control or complete control program
  • Teachers attach importance to adaptive learning systems in which students work independently and in which feedback and reflection are seen as key instructional goals
  • Especially elaborated feedback seems to have been effective.

View the complete summary (in Dutch): http://www.kennisnet.nl/onderzoek/alle-onderzoeken/adaptiviteit-van-digitale-leermiddelen/

Citation: Reints, A., Roll, G. and Wilkens, H. (2014) Adaptiviteit van Digitale Leermiddelen, Expertisecentrum Leermiddelenontwikkeling Universiteit van Utrecht) | Url: http://www.kennisnet.nl/uploads/tx_kncontentelements/2014_08_21_Definitieve_versie.pdf

Type: Evidence | Proposition: A: Learning | Polarity: | Sector: | Country:

Abstract: In this paper, we explain the design research process that we used to develop the learning analytics for the a fractions intervention program called HALF. In particular, we highlight a set of qualitative interviews that we conducted with individual students after a short study in which students in three classes at the same school learned to use the virtual manipulatives embedded in the technology-enhanced learning environment (TELE) to compare pairs of proper fractions and order groups of 3 proper fractions. During the intervention, the analytics we collected for each of these 7 students indicated that they failed to master the content taught during the intervention, but they provided little insight into why and how the students were struggling. In contrast, the qualitative interviews provided us with considerable information that helped us diagnose and address misconceptions and incomplete understandings. These insights led to design changes for the lessons students who use the most up-to-date version of HALF experience via the TELE, which in turn improved the learning analytics for our system.

Overview from the Hub: This paper by Mendiburo et al. discusses the value that qualitative research (in this case student interviews) can offer to improving the interpretation of data analytics and the effective use of these analytics in an intervention programme teaching fractions. Drawing from 7 student interviews undertaken as part of their study, the highlight that learning analytics data gives a misleading impression about what, and how much, has been learnt. They explain, ‘the quantitative data that we collected about these students …yielded mostly inconclusive or misleading results while the qualitative data we collected provide particularly informative insights that led to design changes in future versions of the system.’ The case studies provided reveal how fragile the relationship can be between a learning analytic (data) and the correct interpretation of that data. In their report, the authors conclude, “once we better understood these students' process, we were able to redesign the interaction students had with the TELE in ways that provided more useful quantitative data. Because the meaning of interaction traces is defined by the designs of the interactions, refinement of analytics and user interaction with a TELE go hand in hand.”


Citation: Mendiburo, M., Sulcer, B. and Hasselbring, T. (2014) Interaction Design for Improved Analytics. The 4th International Conference on Learning Analytics and Knowledge. March 24-28, 2014. Indianapolis, USA. http://lak14indy.wordpress.com/lak-2014-conference-program-schedule/