What is Targeted-WTWT?

We find that the models can shortcut stance detection task on the publicly available Twitter stance detection datasets without looking at the target sentence. Targeted-WTWT, a derivative of the WTWT dataset, is a new large-scale stance detection dataset on which target-oblivious models fails to perform well. Thiss serves as a better dataset to train and evaluate models.

For more details about Targeted-WTWT, please refer to our NAACL 2021 paper:

Kaushal et al. 2021

Our code for finding on the other existing stance detection datasets can be found in the following link.

Code

Targeted-WTWT Dataset

Targeted-WTWT is distributed under a CC BY-SA 4.0 License. The link to the dataset is provided below.

Dataset (12.8MB)

We experiments with several baselines on the new dataset. Following is the link to the baselines on this dataset.

Baselines

To add your model to the leaderboard, please submit your request to the first author. We follow the same release format as the WTWT dataset. Please obtain the Tweets for the given tweet-id using the Twitter API. Feel free to use your choice of target sentences for each merger.

Evaluation

As explained in the Sec 3.1 of the paper, we propose a similar cross-target evaluation setting for Targeted-WTWT as WTWT. We consider F₁ is Macro averaged across the classes.
-> For the in domain (health) mergers, train on three health merger (total six targets including negated target for each merger) and test for F₁ on the fourth health merger. Repeat this with having each of the four merger being a test set at least once. The two final scores on in-domain are obtained by averaging and weighted averaging the four F₁ scores.
-> For the out-of-domain evaluation, train on the eight targets corresponding to the 4 health mergers and test on the two targets for entertainment merger.

Citation

If you use Targeted-WTWT dataset in your research, please cite our paper with the following BibTeX entry

          @inproceedings{kaushal2020stance,
            title={tWT–WT: A Dataset to Assert the Role of Target Entities for Detecting Stance},
            author={Kaushal, Ayush and Saha, Avirup and Ganguly, Niloy} 
            booktitle={Proceedings of the 2021 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021)},
            year={2021}
          }