Performance Tracing Report Assistant
Since we already know the order of learning is important, we also need to know the order learners go about their learning. Information Trails can be used to trace the navigational path of learners within virtual environments to provide us with the data needed for analysis. Thus far, there are very little metrics or methods suitable for this kind of data analysis because no one has thought of using Virtual Environment for performance assessment. We need new metrics and method for performance assessment in training with virtual environments.
Loh and Sheng (2013, 2015) have examine the problem and devised several new metrics to address this problem. The most basic one, Similarity Index measures a learners’ performance against that of an establish expert based on the navigational path chosen. The Maximum Similarity Index (MSI), on the other hand, measure the performance indices of novices against a group of experts, particularly when the expertise should not be averaged. (After all, averaged expertise is no expert at all.) The issue is of paramount importance because in situation such as medical diagnosis, surgeons may have opposing opinions in how to treat a condition. Military decision-making is another area where averaging may not make sense, or will result in meaningless actions. A reporting tool can then be used to visualize the learners path graphically for communications with stakeholders (e.g., trainers, learners, and administrators) who are likely to have little statistical training.
In the image shown above, the yellow lines on the game map indicate movements and the dots depict game events. Since actions are a product of decision, the players’ actions in the game also infer his/her decisions. The learning events are traced and translated into behavioral data, which can then be used for ad hoc (real time, formative) or post hoc (after-action, summative, debriefing) reports for peer- or self-evaluation.
In the future, it may be possible to create an Indexable Expert Learning pedagogic agent to teach or coach learners the way of the experts.
History of PeTRA
|11/2015||PeTRA is now fully powered by R and RStudio server.|
|10/2015||Unity3D Maze – first public game with Information Trails. Open for data collection.|
|08/2015||PeTRA now accept data from Unity3D games.|
|01/2015||The Lost World – First Unity3D game with Information Trails.|
|09/2014||PeTRA being upgraded for Unity3D-driven assessment.|
|11/2010 – 05/2014||On-going game-based learning assessment research projects|
|08/2010||Information Trails tracer was renamed into “Performance Tracer Reporting Assistant” (PeTRA).|
|05/2010||Information Trails (version 2) was completed (developed using Adobe Flex).|
|10/2009 – 07/2010||Setup of the Virtual Environment Lab (V-LAB), funded through DURIP by the ARO.|
|08/2009||Porting game modding platform to Neverwinter Nights 2|
|01/2009||Porting of Information Trails from a JAVA application (version 1) to Adobe Flex (version 2).|
|07/2008||We applied and received a provisional patent status for the Information Trails.|
|05/2008||The prototype of Information Trails (version 0.9) was completed, but we ran into problem with JAVA.|
|12/2007||We created our first serious game, Saving Adryanee, which won a Finalist Entry at the 2nd Serious Games Showcase & Challenge (sponsored by I/ITSEC).|
|07/2007||Dr. Loh received a $26,000 Seed Grant to create the first prototype of the assessment tool.|
|10/2006||The original concept of Information Trails was conceptualized and presented at the annual conference of AECT by Dr. Christian S. Loh.|
- Loh, C. S., & Sheng, Y. (2014). Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior. 39: 322-330. DOI: 10.1016/j.chb.2014.07.022
- Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-)similarity between expert and novice behaviors as Serious Games Analytics. Education and Information Technologies. 20(1): 5-19. DOI: 10.1007/s10639-013-9263-y
- Loh, C. S. (2012). Information Trails: In-process assessment for game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds). Assessment in game-based learning: Foundations, innovations, and perspectives. (pp.123-144) New York, NY: Springer. DOI: 10.1007/978-1-4614-3546-4_8
- Loh, C. S. (Jan 2007). Designing Online Games Assessment as ‘Information Trails.’ In D. Gibson, C. Aldrich & M. Prensky (Eds.), Games and Simulation in Online Learning: Research and Development Frameworks. (pp. 323-348). Hershey, PA: Idea Group, Inc.