Temporal commonsense knowledge is often necessary to understand events expressed in natural language. This knowledge involves several temporal aspects of events, such as duration, frequency, and temporal order. This knowledge is still challenging for computers to understand.
Dr. Kanashiro and her colleagues built a deep neural network model to attempt to solve the problem. Their model got the 1st place on the official leaderboard for the temporal commonsense comprehension task.
https://leaderboard.allenai.org/mctaco/submissions/public; Screenshot taken on February 27th, 2020
Temporal commonsense knowledge is often necessary to understand events expressed in natural language. This knowledge involves several temporal aspects of events, such as duration, frequency, and temporal order. Most humans would know that, for example, an event such as “going on a vacation" takes longer and occurs less often than an event such as “going for a walk". However, it is still challenging for computers to understand and reason about temporal commonsense.
Recently, contextual pre-trained language models have advanced the state-of-the-art on various commonsense NLP tasks (Kocijan et al., 2019; He et al., 2019). In our work, we investigate the temporal commonsense ability of the contextual pre-trained language model RoBERTa (Liu et al., 2019). RoBERTa builds on BERT's language masking strategy (Devlin et al., 2019), and includes modifications such as hyperparameter tuning and training set size, and has been shown to generalize better to downstream tasks compared to BERT.
Inspired by the work of Jiang et al. (2019), we used an adversarial training approach to alleviate the overfitting and knowledge forgetting issues typically caused when fine-tuning large scale language models.
We evaluated our model on the MC-TACO dataset (Zhou et al. 2019). This dataset has 13k question-answer pairs that require temporal commonsense comprehension. On the official leaderboard for this task, we ranked first in terms of F1 and Exact Match scores. We confirm previous findings that a notable amount of commonsense knowledge can be acquired via pre-training contextual language models. However, our model is still far behind human performance, motivating further research in this area.