AdvGLUE

The Adversarial GLUE Benchmark

MultiNLI (MNLI) mismatched

Statistics

Examples

Typo (Word-level)


BENIGN
  • Premise: uh-huh how about any
    matching
    programs?
  • Hypothesis: What about matching programs?
  • Label:
    Entailment


ADVERSARIAL
  • Premise: uh-huh how about any
    mathcing
    programs?
  • Hypothesis: What about matching programs?
  • Model prediction:
    Contradiction

Distraction (Sentence-level)


BENIGN
  • Premise: You and your friends are not welcome here, said Severn.
  • Hypothesis: Severn said the people were not welcome there.
  • Label:
    Entailment


ADVERSARIAL
  • Premise: You and your friends are not welcome here, said Severn.
  • Hypothesis: Severn said the people were not welcome there
    and true is true
    .
  • Model prediction:
    Contradiction

ANLI (Human-crafted)


BENIGN
  • Premise: Kamila Filipcikova (born 1991) is a female Slovakian fashion model. She has modeled in fashion shows for designers such as Marc Jacobs, Chanel, Givenchy, Dolce \& Gabbana, and Sonia Rykiel. And appeared on the cover of Vogue Italia two times in a row.
  • Hypothesis: Filipcikova lives in Italy.
  • Label:
    Neutral
  • Model prediction:
    Contradiction