AI 译本 · Equalang
新上架PDF · 2026年11月翻译
加载中…
加载中…
NeurIPS · 2023
DecodingTrust：对 GPT 模型可信度的全面评估
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie et al.
Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives -- including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/ ; our dataset can be previewed at https://huggingface.co/datasets/AI-Secure/DecodingTrust ; a concise version of this work is at https://openreview.net/pdf?id=kaHpo8OZw2 .
生成式预训练 Transformer（GPT）模型的能力取得了令人瞩目的进展，吸引了从业者和公众的广泛关注。然而，关于 GPT 模型可信度的研究仍然有限，与此同时，从业者已提出将能力强大的 GPT 模型应用于医疗、金融等敏感领域——在这些领域，出错可能造成重大损失。为此，本研究提出了一套针对大型语言模型的全面可信度评估，重点考察 GPT-4 和 GPT-3.5，并从多个角度进行分析，包括毒性、刻板印象偏见、对抗鲁棒性、分布外鲁棒性、对抗示例下的鲁棒性、隐私、机器伦理和公平性。根据评估结果，我们发现了此前尚未公开报道的可信度风险漏洞。例如，我们发现，GPT 模型很容易受到误导，生成带有毒性和偏见的内容，并泄露训练数据及对话历史中的私人信息。我们还发现，尽管在标准基准测试中，GPT-4 通常比 GPT-3.5 更可信，但面对用于越狱的系统提示或用户提示时，GPT-4 更容易受到攻击，这可能是因为 GPT-4 会更准确地遵循（误导性）指令。本研究展示了对 GPT 模型可信度的全面评估，并揭示了其中的可信度差距。我们的基准测试可在 https://decodingtrust.github.io/ 公开获取；数据集可在 https://huggingface.co/datasets/AI-Secure/DecodingTrust 预览；本研究的精简版见 https://openreview.net/pdf?id=kaHpo8OZw2。
PDF · 2026年11月翻译