Counterfactual Social Bias Evaluation in Persian LLM-Based Information Systems

Large language models are increasingly used in Persian information systems, yet culturally grounded bias evaluation remains limited. We introduce a natural-scenario counterfactual benchmark constructed through human-directed, ChatGPT-assisted drafting and independently validated by three native Persian speakers. From an initial 100 scenarios, 524 rows, and 424 pairs, validation retained 84 scenarios, 424 rows, and 340 pairs; socioeconomic coverage was reduced to one pair. Five open-weight models were evaluated with deterministic restricted next-token A--E scoring, normal/reverse option orders, scenario-cluster bootstrap intervals, and scenario-level sign-flip tests. Baseline mean absolute gaps were small (0.031--0.043), but Dorna-Llama3-8B and Llama-3.1-8B showed severe option-order sensitivity. Fairness prompting increased gaps for every model and significantly for Qwen2-7B, Qwen2.5-14B, and Qwen3-8B. Qwen2.5-14B and Qwen3-8B achieved the highest BBQ-Fa accuracy (0.817), while Qwen2.5-14B led ISEAR-Fa (0.623). Train-only calibration did not improve held-out gaps. The benchmark, validation documentation, prompts, code, and results are provided in an accompanying anonymous GitHub repository.