Performance Tracing

Performance Tracing Report Assistant

The Performance Tracing Report Assistant (or PeTRA) is a real-time reporting tool for the Information Trails assessment framework (Loh, 2007, 2012). The purpose of PeTRA is to visualize players gameplay data (i.e., what their actions and behaviors are within the virtual environments) for communication. After tracing user-generated data in situ the virtual environment using Information Trails, we subjected the data for machine/statistical learning (exploratory study).

Since we already know the order of learning is important, we also need to know the order learners go about their learning. Information Trails can be used to trace the navigational path of learners within virtual environments to provide us with the data needed for analysis. Thus far, there are very little metrics or methods suitable for this kind of data analysis because no one has thought of using Virtual Environment for performance assessment. We need new metrics and method for performance assessment in training with virtual environments. 

Loh and Sheng (2013, 2015) have examine the problem and devised several new metrics to address this problem. The most basic one, Similarity Index measures a learners’ performance against that of an establish expert based on the navigational path chosen. The Maximum Similarity Index (MSI), on the other hand, measure the performance indices of novices against a group of experts, particularly when the expertise should not be averaged. (After all, averaged expertise is no expert at all.) The issue is of paramount importance because in situation such as medical diagnosis, surgeons may have opposing opinions in how to treat a condition. Military decision-making is another area where averaging may not make sense, or will result in meaningless actions. A reporting tool can then be used to visualize the learners path graphically for communications with stakeholders (e.g., trainers, learners, and administrators) who are likely to have little statistical training.

Partial Screenshot of the Performance Tracing Report Assistant (PeTRA), Copyright 2010-2016

Partial Screenshot of the Performance Tracing Report Assistant (PeTRA), Copyright 2010-2016

In the image shown above, the yellow lines on the game map indicate movements and the dots depict game events. Since actions are a product of decision, the players’ actions in the game also infer his/her decisions. The learning events are traced and translated into behavioral data, which can then be used for ad hoc (real time, formative) or post hoc (after-action, summative, debriefing) reports for peer- or self-evaluation.

Pedagogic Agents

In the future, it may be possible to create an Indexable Expert Learning pedagogic agent to teach or coach learners the way of the experts.

divider

History of PeTRA

Timeline PeTRA Development
11/2015 PeTRA is now fully powered by R and RStudio server.
10/2015 Unity3D Maze – first public game with Information Trails. Open for data collection.
08/2015 PeTRA now accept data from Unity3D games.
01/2015 The Lost World – First Unity3D game with Information Trails.
09/2014 PeTRA being upgraded for Unity3D-driven assessment.
11/2010 – 05/2014 On-going game-based learning assessment research projects
08/2010 Information Trails tracer was renamed into “Performance Tracer Reporting Assistant” (PeTRA).
05/2010 Information Trails (version 2) was completed (developed using Adobe Flex).
10/2009 – 07/2010 Setup of the Virtual Environment Lab (V-LAB), funded through DURIP by the ARO.
08/2009 Porting game modding platform to Neverwinter Nights 2
01/2009 Porting of Information Trails from a JAVA application (version 1) to Adobe Flex (version 2).
07/2008 We applied and received a provisional patent status for the Information Trails.
05/2008 The prototype of Information Trails (version 0.9) was completed, but we ran into problem with JAVA.
12/2007 We created our first serious game, Saving Adryanee, which won a Finalist Entry at the 2nd Serious Games Showcase & Challenge (sponsored by I/ITSEC).
07/2007 Dr. Loh received a $26,000 Seed Grant to create the first prototype of the assessment tool.
10/2006 The original concept of Information Trails was conceptualized and presented at the annual conference of AECT by Dr. Christian S. Loh.

divider

Reference:

  • Loh, C. S., & Sheng, Y. (2014). Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior. 39: 322-330. DOI: 10.1016/j.chb.2014.07.022
  • Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-)similarity between expert and novice behaviors as Serious Games Analytics. Education and Information Technologies. 20(1): 5-19. DOI: 10.1007/s10639-013-9263-y
  • Loh, C. S. (2012). Information Trails: In-process assessment for game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds). Assessment in game-based learning: Foundations, innovations, and perspectives. (pp.123-144) New York, NY: Springer. DOI: 10.1007/978-1-4614-3546-4_8
  • Loh, C. S. (Jan 2007). Designing Online Games Assessment as ‘Information Trails.’ In D. Gibson, C. Aldrich & M. Prensky (Eds.), Games and Simulation in Online Learning: Research and Development Frameworks. (pp. 323-348). Hershey, PA: Idea Group, Inc.