Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation

Abstract

Identity cues are irrelevant to a fixed programming specification, but raw counterfactual differences can arise from unequal samples and prompt wording. We audit 30.73 million executed Python generations from seven checkpoints on HumanEval+ and MBPP+, supplemented by an exploratory 550B slice, under model-assigned gender, country, and occupation personas. Within-task randomization and false-discovery-rate control identify occupation as the most consistent structural axis, CodeBLEU dispersion exceeds its exchangeability null in 10/14 model–benchmark cells, remains significant in 8/12 full-coverage cells, and exceeds the country ratio in every paired cell, although the median excess is only 0.141 points. In six high-pass-rate cells, occupation dispersion replicates across token similarity, length, comments, reference similarity, and complexity, while pass-rate dispersion is significant in none. Country leads raw dispersion in 12/14 cells but has a median calibrated ratio of 1.00. Gender-associated variation cannot be separated from persona wording in this design. Thus, the evidence supports small, reproducible occupation-conditioned changes in code form, not stable disadvantage to named identities or demonstrated downstream harm. We use this finding to motivate a broader commentary on the potential impacts of bias in code generation, including the possibility that models may condition their outputs on identity information available from prior conversational context.

@inproceedings{Eren2026IdentityOP,
  title={Identity or Prompt Noise? A Calibrated Invariance Audit of LLM Code Generation},
  author={Maksim Ekin Eren and Ryan C. Barron and Eric Michalak and Charles Nicholas and Manish Bhattarai},
  year={2026},
  url={https://api.semanticscholar.org/CorpusID:292205973}
}

Maksim E. Eren

Scientist

Maksim E. Eren is a Scientist at Los Alamos National Laboratory, specializing in machine learning and artificial intelligence for large-scale data science applications.