The paper introduces PrimaCons, a four-axis benchmark for whether a Jev model gives the same decision when the same question is asked in different formats, plus a calibrator meant to fix the mismatches.
Benchmark: Tested across three Jev sizes, PrimaCons finds consistency violations on all four axes, and the authors say these follow systematic patterns.
Failure signal: Cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91.
Fix: The Cross-Primitive Residual Calibrator attaches a lightweight residual network to a frozen backbone. Under leave-one-domain-out, it reduces cross-primitive inconsistency by 58.57% and improves accuracy by 2.97%.
Same question, different format, and the paper wants the answers to match.
Sign in to comment.
TL;DR of the paper, generated automatically.
The paper introduces PrimaCons, a four-axis benchmark for whether a Jev model gives the same decision when the same question is asked in different formats, plus a calibrator meant to fix the mismatches.
Same question, different format, and the paper wants the answers to match.