{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/trustllm-trustworthiness-in-large-language","title":"TrustLLM: Trustworthiness in Large Language Models","arxiv_id":"2401.05561","date":"2024-01-10","proceeding":null,"authors":["Yue Huang","Lichao Sun","Haoran Wang","Siyuan Wu","Qihui Zhang","Yuan Li","Chujie Gao","Yixin Huang","Wenhan Lyu","Yixuan Zhang","Xiner Li","Zhengliang Liu","Yixin Liu","Yijue Wang","Zhikun Zhang","Bertie Vidgen","Bhavya Kailkhura","Caiming Xiong","Chaowei Xiao","Chunyuan Li","Eric Xing","Furong Huang","Hao liu","Heng Ji","Hongyi Wang","huan zhang","Huaxiu Yao","Manolis Kellis","Marinka Zitnik","Meng Jiang","Mohit Bansal","James Zou","Jian Pei","Jian Liu","Jianfeng Gao","Jiawei Han","Jieyu Zhao","Jiliang Tang","Jindong Wang","Joaquin Vanschoren","John Mitchell","Kai Shu","Kaidi Xu","Kai-Wei Chang","Lifang He","Lifu Huang","Michael Backes","Neil Zhenqiang Gong","Philip S. Yu","Pin-Yu Chen","Quanquan Gu","ran Xu","Rex Ying","Shuiwang Ji","Suman Jana","Tianlong Chen","Tianming Liu","Tianyi Zhou","William Wang","Xiang Li","Xiangliang Zhang","Xiao Wang","Xing Xie","Xun Chen","Xuyu Wang","Yan Liu","Yanfang Ye","Yinzhi Cao","Yong Chen","Yue Zhao"],"abstract":"Large language models (LLMs), exemplified by ChatGPT, have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs emerges as an important topic. This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. Our findings firstly show that in general trustworthiness and utility (i.e., functional effectiveness) are positively related. Secondly, our observations reveal that proprietary LLMs generally outperform most open-source counterparts in terms of trustworthiness, raising concerns about the potential risks of widely accessible open-source LLMs. However, a few open-source LLMs come very close to proprietary ones. Thirdly, it is important to note that some LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent that they compromise their utility by mistakenly treating benign prompts as harmful and consequently not responding. Finally, we emphasize the importance of ensuring transparency not only in the models themselves but also in the technologies that underpin trustworthiness. Knowing the specific trustworthy technologies that have been employed is crucial for analyzing their effectiveness.","url_abs":"https://arxiv.org/abs/2401.05561v6","url_pdf":"https://arxiv.org/pdf/2401.05561v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trustllm-trustworthiness-in-large-language","repo_url":"https://github.com/HowieHwong/TrustLLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"trustllm-trustworthiness-in-large-language","repo_url":"https://github.com/liano3/RAG-fairness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"ethics","task_name":"Ethics"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.05561","atlas_url":"https://app.syntology.ai/?focus=2401.05561","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2401.05561"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2401.05561\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2401.05561"},"call":"get_concepts_for_paper(arxiv_id=\"2401.05561\")"}],"by_repo":[{"repo":"HowieHwong/TrustLLM","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2401.17882"],"harvested_for_calls":[{"arxiv_id":"2401.17882","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2401.17882"},"call":"get_harvested_code_for_paper(arxiv_id=\"2401.17882\")"}]},{"repo":"liano3/RAG-fairness","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0dc763bc5264e2b9d1f5b3eb010dee0f5daf65ff17c74cec00a44b719973955e","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}