Tag Archives: evaluation

Type: Evidence | Proposition: A: Learning | Polarity: | Sector:

This paper is a preliminary study on how learning analytics can support tracking and measurement of informal learing, which represents a large part learning at the workplace.

The paper is in tight relationship with a previous evidence identified by this Hub (please , see http://evidence.laceproject.eu/?evidence=new-framework-informal-learning-workplace) about the development of Social Semantic Server (SSS). In this paper it is highlighted how SSS can generate semantically-enriched Actor-Artifact Networks (AANs) to describe the relationships among actors and artifacts in different learning contexts.

To simply  determine how an AAN can be generated in SSS, a preliminary study was developed using SSS and two of its applications, Bookmarker (a Chrome extension that allows users to submit bookmarks and tags to the SSS while browsing) and Attacher (a WordPress plugin that integrates a blog editor to the SSS and displays a tagcloud that includes tags registered in the SSS).  In this study, ten students of a training course for future teachers were asked to bookmark web resources they considered relevant (using Bookmarker) and to write "a blog post about their reflections of the subject using WordPress and Attacher to browse the bookmarks published in the social semantic infrastructure".

Results show that three AANs can be easily identified, helping the teacher to analyze the learning of the students and to understand their behavior. This study describes a first example on how the SSS can integrate the data from different applications and coherently combine it to support learning analytics at the workplace.

Citation: A. Ruiz-Calleja , S. Dennerlein, V. Tomberg, K. Pata, T. Ley, D. Theiler, E. Lex. Supporting Learning Analytics for Informal Workplace Learning with a Social Semantic Infrastructure. (2015)10th European Conference on Technology Enhanced Learning, EC-TEL 2015, Toledo, Spain, September 15-18, 2015, Proceedings, pp 634-637 | Url: http://link.springer.com/chapter/10.1007%2F978-3-319-24258-3_76

Type: Evidence | Proposition: B: Teaching | Polarity: | Sector: | Country:

The paper describes a procedure evaluation/e-training tool (named PeT) for the oil and gas industry, related to the tracking of knowledge and confidence of trainees in emergency operating procedures.

The issue that this kind of tools try to solve is the fact that one of the main responsible for incidents in the oil & gas industry is the lacking of knowledge, both during standard and emergency situations. This lack of training can have serious consequences over the safety of workforce, such as losses of life and serious injuries. But incidents in oil & gas platforms can also lead to severe issues on the productivity of the plant, such as operations downtime, heavy costs and loss of reputation (e.g. the oil spill in the Gulf of Mexico on 2010).

PeT is a training and testing environment for both standard (SOP) and emergency operating procedures (EOP), implementing multiple-choice knowledge tests for two emergency procedures. The main objectives of PeT are:

  • Verify the knowledge of the standard procedures, with the aim to avoid incidents.
  • Ensure that workforce takes appropriate decisions in an emergency setting.
  • Create a competence portfolio for each operator while taking into account his/her past experience and expertise.
  • Provide remedial instructions and feedback to address gaps in the knowledge and competences of operators in the execution of both critical and non-critical tasks.


Concerning Learning Analytics, PeT tracks three kinds of data:

  • Session data, related to the time of completion of the overall knowledge verification session, ID of the employee and code of the operating procedure undertaken.
  • Question data, related to the time an user has spent in each single step of the procedure.
  • Choice data, related to the correctness of answers.


Moreover, the study has introduced an overall confidence metric (for the measurement of the ability of an operator to execute a task, both standard and emergency operating ones, correctly in the proper timeline with an optimal mindset) and a concept of criticality of each step.

The PeT tool has been tested and improved in two separate sessions in an oil & gas company in Canada in 2014. The experiment was conducted with several operators with different backgrounds and different levels of expertise.

Results show that, on the analysed group of workers, PeT is capable to efficiently assess the knowledge and behaviour of workforce in the oil and natural gas industry, in both standard and emergency procedures, ensuring traceable training for operators and leading to greater safety for workforce together with less risk of productivity losses for the plant.

Type: Evidence | Proposition: B: Teaching | Polarity: | Sector: | Country:

This study provides strong indications that there are discrete online behaviour patterns that are representative of students’ overall engagement and final assessment grade. The visualisation of online
student engagement/effort affords instructors with early opportunities for providing additional student learning assistance and intervention – when and where it is required. The capacity to establish early indicators of ‘at-risk’ students provides timely opportunities for instructors to re-direct or add resources to facilitate progression towards patterns of behaviour that represent what has been termed a 'low-risk category' (e.g. participation in group discussions, regular access of discipline content and review of assessment criteria).

The paper identifies opportunities for instructors to make these changes, but that would be a next step - so it does not provide evidence that learning analytics improve learning support and teaching, only that they offer potential to achieve this.

Citation: Dawson, Shane, McWIlliam, Erica, & Tan, Jen Pei-Ling. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Paper presented at the ascilite 2008, Melbourne,