AI 译本 · Equalang
新上架PDF · 2026年11月翻译
加载中…
加载中…
arXiv · 2026
Kierston 与 KierstonBench：宾夕法尼亚州阿伦敦市模拟，以及语言模型在犯罪调查和监控任务中的评估
Andrew Semenov, Svyatoslav Dorofeev
Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.
大型语言模型可用于收集和分析刑事调查与大规模监控中的记录。目前，我们还无法衡量它们在这类工作中的能力和准确性。现实部署缺乏评估模型结论所需的真实答案，也没有基准能在模拟环境中评估这些能力。我们推出了城市级模拟引擎 Kierston，以及基准 KierstonBench，用已知的模拟事件来测试这些能力。Kierston 对宾夕法尼亚州阿伦敦市约 12.5 万名居民进行建模，生成他们的日常活动，以及监控和调查系统收集的电信、金融、行政和车牌记录。这些记录源自一条共同的行为时间线，并分散在代表不同服务提供商的数据源中。模型通过受限的记录查询界面开展调查，在无法访问底层模拟数据的情况下关联不同来源的观察结果。该基准包含 32 类任务，涵盖记录关联、日常活动重建、事件调查，以及对欺骗性说法的评估。在对 16 个语言模型进行评估时，大多数模型无法有效调查记录，还提交了记录无法支持的身份判定。少数前沿模型完成了大部分任务，并在证据不足时选择不作判断。有时，这些模型仍会落入常见的欺骗和假设陷阱，而且没有任何一个模型能够取代调查人员。
PDF · 2026年11月翻译