Ruf, A., Niederhauser, M., Jäger, J., Zahn C., & Opwis, K. (2021). Introducing a new approach for investigating learning behavior. In C. Hmelo-Silver, B. de Wever, & J. Oshima (Eds.), 14th International Conference on Computer-Supported Collaborative Learning – CSCL 2021 (pp. 251-252). International Society of the Learning Sciences, 2021.
Ruf., A., Jäger, J., Niederhauser, M., Zahn, C. & Opwis, K. (2021). Logible: Detecting, Analyzing, and Visualizing Behavior Sequences for Investigating Learning Behavior. In Wichmann, A. Hoppe, H. U. & Rummel, N. (Eds.). (2021). General Proceedings of the 1st Annual Meeting of the International Society of the Learning Sciences 2021 (pp. 13-16). Bochum, Germany: International Society of the Learning Sciences.
Ruf, A., Leisner, D., Zahn C., & Opwis, K. (2021). Impact of learners’ video interactions on learning success and cognitive load. In C. Hmelo-Silver, B. de Wever, & J. Oshima (Eds.), 14th International Conference on Computer-Supported Collaborative Learning – CSCL 2021 (pp. 3-10). International Society of the Learning Sciences, 2021.
Educational videos are established around the globe with a continuously increasing popularity. The potential of learners’ video interaction for understanding learning behavior has been recognized in previous research, however, by primarily focusing on frequencies of single interactions (e.g., clicks such as play or pause). Yet, we assume that investigating meaningful behavior sequences can lead to in-depth understanding of learning behavior. The goal of this contribution is to detect and visualize behavior sequences using a newly developed method and application (logible, making log files legible) to get deeper insights into the learning behavior of learners interacting with a video-based environment.
In previous work we enhanced the FrameTrail Video Annotation Software so it can record user interactions inside both the video player and the annotation components. So during the learning process, FrameTrail automatically logged the following actions:
VideoPlay
VideoPause
VideoJumpBackward
VideoJumpForward
AnnotationAdd
AnnotationChangeTime
AnnotationChangeText
To build our method and logible, we used data of a learning session from an experimental study (see references above). Upon completion of a learning session, a chronologically ordered list of actions ("action-strings") including timestamps and meta information was exported from FrameTrail in the form of 92 Excel Sheets (one sheet per session).
About logible
logible is a web-based application that is able to process the previously recorded Excel Sheets as input data, in order to detect and visualize predefined sequences of actions in different settings and learning scenarios (Individual vs. Collaborative leaerning, Annotating vs. Hyperlinking, see references above).
The sequence detection is based on a set of manually defined rules which are used to identify a given behavior sequence within a string of actions.
Rule definitions (example)
In the next step, we additionally implemented the factors priority (higher levels of interactivity were preferred) and length of sequence (longer sequences were preferred) in logible. These factors were necessary to select the most meaningful sequence for further analyses.
Overview of behavior patterns and behavior sequences with prioritization
For this purpose, all automatically detected behavior sequences in logible were divided into blocks of non-overlapping sequences. Within each block, we calculate an overall score for the average priority, the number of actions and the number of behaviour sequences. For sequences with similar priority and length scores, we introduced heterogeneity as a third factor, to prefer candidates with a higher number of different actions. Then, normalized scores for each of the factors were calculated and an overall rating for the given list of behavior sequences was assigned, which also took into account the respective weight of each factor, which was iteratively defined during the development process (priority: 2.2, length: 1, heterogeneity: 0.4). Based on this rating, we choose the top-rated behavior sequence per block.
After several iterations and continuous improvement logible is able to detect all 17 predefined behavior sequences and visualize them color-coded above the action-string (see bottom line) of each learner:
logible: sequence visualization (example)
According to the rule-based prioritization the most valuable behavior sequence is selected (marked with a bold black frame). This visualized representation provides a first impression of students' learning behavior.
logible not only detects, analyzes, and visualizes sequences but allows for comparisons of behavioral data between different experimental conditions without the use of other software. The amount of conducted actions can be compared between experimental conditions (individual vs. collaborative learners and using hyperlinks vs. annotations). Such comparisons can also be conducted with the amount of behavior sequences (including types of sequences) from the uploaded data sets. We also implemented a graph visualization (using so-called Markov-chains; a method to analyze the probability of one action or sequence following another) to provide insights about how behavior sequences are dependent on each other.
Amount of sequence occurrence of example data sets (left) and Markov-chain visualization representing transition probabilities (right)
logible is highly sensitive and customizable. Thus, it allows to analyze interaction data from all kinds of sequentially logged data (from web-based video environments, online learning platforms etc.).
Please get in touch via joscha.jaeger [at] filmicweb.org or alessia.ruf [at] fhnw.ch for more information or if you're interested to collaborate.
For information about using logible in experimental research get in touch with alessia.ruf [at] fhnw.ch
