AdvGLUE

The Adversarial GLUE Benchmark

The Stanford Sentiment Treebank (SST-2)

Statistics

Examples

Typo (Word-level)


BENIGN
  • Sentence: The primitive force of this film seems to
    bubble
    up from the vast collective memory of the combatants.
  • Label:
    Positive


ADVERSARIAL
  • Sentence: The primitive force of this film seems to
    bybble
    up from the vast collective memory of the combatants.
  • Model prediction:
    Negative

Context-aware (Word-level)


BENIGN
  • Sentence: In execution, this clever idea is far
    less
    funny than the original, killers from space.
  • Label:
    Negative


ADVERSARIAL
  • Sentence: In execution, this clever idea is far
    smaller
    funny than the original, killers from space.
  • Model prediction:
    Positive

CheckList (Human-crafted)


BENIGN
  • Sentence: I think this movie is perfect, but I used to think it was annoying.
  • Label:
    Positive
  • Model prediction:
    Negative