An Embedding-space Aware Calibration Loss Function

Abstract: Confidence calibration (the alignment of a model’s predicted probabilities with its true accuracy) is crucial for the reliability of deep classifiers in real-world scenarios. We present NeSCal, a novel train-time regularizer that improves calibration by enforcing predictive consistency between neighbours in representation space, penalizing the Jensen-Shannon divergence between the predictive distributions of points adjacent in a kNN graph. We develop theory connecting this local smoothness to both manifold learning and calibration error, showing an upper bound on the propagation of calibration error between neighbours, and an exact lower bound for when enforcing smoothness necessarily harms calibration, defining the conditions under which NeSCal should be applied. We show preliminary empirical results indicating that NeSCal’s performance is competitive with existing methods, and superior to an unregularized baseline.