AdvGLUE

The Adversarial GLUE Benchmark

Quora Question Pairs (QQP)

Statistics

Examples

Embedding (Word-level)


BENIGN
  • Question 1: I am getting fat on my lower body and on the
    chest
    , is there any way I can get fit without looking skinny fat?
  • Question 2: Why I am getting skinny instead of losing body fat?
  • Label:
    Not Equivalent


ADVERSARIAL
  • Question 1: I am getting fat on my lower body and on the
    torso
    , is there any way I can get fit without looking skinny fat?
  • Question 2: Why I am getting skinny instead of losing body fat?
  • Model prediction:
    Equivalent

Syntactic (Sentence-level)


BENIGN
  • Question 1:
    Can I learn MMA at the age of 26?
  • Question 2: Can I learn MMA at the age of 24?
  • Label:
    Not Equivalent


ADVERSARIAL
  • Question 1:
    You can learn MMA at 24?
  • Question 2: Can I learn MMA at the age of 24?
  • Model prediction:
    Equivalent

CheckList (Human-crafted)


BENIGN
  • Question 1: Is Alfred Kennedy an analyst?
  • Question 2: Is Alfred Kennedy becoming an analyst?
  • Label:
    Not Equivalent
  • Model prediction:
    Equivalent