{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/synthpai","name":"SynthPAI","full_name":"SynthPAI: A Synthetic Dataset for Personal Attribute Inference","description_markdown":"SynthPAI was created to provide a dataset that can be used to investigate the personal attribute inference (PAI) capabilities of LLM on online texts. Due to associated privacy concerns with real-world data, open datasets are rare (non-existent) in the research community. SynthPAI is a synthetic dataset that aims to fill this gap.\r\n\r\n## Dataset Details\r\n#### Dataset Description\r\nSynthPAI was created using 300 GPT-4 agents seeded with individual personalities interacting with each other in a simulated online forum and consists of 103 threads and 7823 comments. For each profile, we further provide a set of personal attributes that a human could infer from the profile. We additionally conducted a user study to evaluate the quality of the synthetic comments, establishing that humans can barely distinguish between real and synthetic comments.\r\n\r\n- **Curated by:** The dataset was created by SRILab at ETH Zurich. It was not created on behalf of any outside entity.\r\n- **Funded by:** Two authors of this work are supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) (SERI-funded ERC Consolidator Grant). This project did, however, not receive explicit funding by SERI and was devised independently. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the SERI-funded ERC Consolidator Grant.\r\n- **Shared by:** SRILab at ETH Zurich\r\n- **Language(s) (NLP):** English\r\n- **License:** CC-BY-NC-SA-4.0\r\n#### Dataset Sources\r\n\r\n- **Repository:** [https://github.com/eth-sri/SynthPAI](https://github.com/eth-sri/SynthPAI)\r\n- **Paper:** [https://arxiv.org/abs/2406.07217](https://arxiv.org/abs/2406.07217)\r\n\r\n## Uses\r\n\r\nThe dataset is intended to be used as a privacy-preserving method of (i) evaluating PAI capabilities of language models and (ii) aiding the development of potential defenses against such automated inferences.\r\n#### Direct Use\r\nAs in the associated [paper](https://arxiv.org/abs/2406.07217) , where we include an analysis of the personal attribute inference (PAI) capabilities of 18 state-of-the-art LLMs across different attributes and on anonymized texts.\r\n#### Out-of-Scope Use\r\nThe dataset shall not be used as part of any system that performs attribute inferences on real natural persons without their consent or otherwise maliciously.\r\n\r\n## Dataset Structure\r\n\r\nWe provide the instance descriptions below. Each data point consists of a single comment (that can be a top-level post):\r\n\r\n**Comment**\r\n\r\n\r\n- **author** *str*: unique identifier of the person writing\r\n\r\n- **username** *str*: corresponding username\r\n\r\n- **parent_id** *str*: unique identifier of the parent comment\r\n\r\n- **thread_id** *str*: unique identifier of the thread\r\n\r\n- **children** *list[str]*: unique identifiers of children comments\r\n\r\n- **profile** *Profile*: profile making the comment - described below\r\n\r\n- **text** *str*: text of the comment\r\n\r\n- **guesses** *list[dict]*: Dict containing model estimates of **attributes** based on the comment.  Only contains attributes for which a prediction exists.\r\n\r\n- **reviews** *dict*: Dict containing human estimates of **attributes** based on the comment. Each guess contains a corresponding hardness rating (and certainty rating). Contains all attributes\r\n\r\nThe associated profiles are structured as follows\r\n\r\n**Profile**\r\n\r\n- **username** *str*: identifier\r\n\r\n- **attributes**: set of personal attributes that describe the user (directly listed below)\r\n\r\n\r\nThe corresponding attributes and values are\r\n\r\n**Attributes**\r\n\r\n- **Age** *continuous* [18-99] The age of a user in years.\r\n\r\n- **Place of Birth** *tuple* [city, country] The place of birth of a user. We create tuples jointly for city and country in free-text format. (field name: birth_city_country)\r\n\r\n- **Location** *tuple* [city, country] The current location of a user. We create tuples jointly for city and country in free-text format. (field name: city_country)\r\n\r\n- **Education** *free-text* We use a free-text field to describe the user's education level. This includes additional details such as the degree and major. To ensure comparability with the evaluation of prior work, we later map these to a categorical scale: high school, college degree, master's degree, PhD.\r\n\r\n- **Income Level** *free-text* [low, medium, high, very high] The income level of a user. We first generate a continuous income level in the profile's local currency. In our code, we map this to a categorical value considering the distribution of income levels in the respective profile location. For this, we roughly follow the local equivalents of the following reference levels for the US: Low (<30k USD), Middle (30-60k USD), High (60-150k USD), Very High (>150k USD).\r\n\r\n- **Occupation** *free-text* The occupation of a user, described as a free-text field.\r\n\r\n- **Relationship Status** *categorical* [single, In a Relationship, married, divorced, widowed] The relationship status of a user as one of 5 categories.\r\n\r\n- **Sex** *categorical* [Male, Female] Biological Sex of a profile.\r\n\r\n\r\n\r\n### Dataset Creation\r\n#### Curation Rationale\r\nSynthPAI was created to provide a dataset that can be used to investigate the personal attribute inference (PAI) capabilities of LLM on online texts. Due to associated privacy concerns with real-world data, open datasets are rare (non-existent) in the research community. SynthPAI is a synthetic dataset that aims to fill this gap. We additionally conducted a user study to evaluate the quality of the synthetic comments, establishing that humans can barely distinguish between real and synthetic comments.\r\n#### Source Data\r\nThe dataset is fully synthetic and was created using GPT-4 agents (version `gpt-4-1106-preview`) seeded with individual personalities interacting with each other in a simulated online forum.\r\n##### Data Collection and Processing\r\nThe dataset was created by sampling comments from the agents in threads. A human then inferred a set of personal attributes from sets of comments associated with each profile. Further, it was manually reviewed to remove any offensive or inappropriate content. We give a detailed overview of our dataset-creation procedure in the corresponding [paper](https://arxiv.org/abs/2406.07217). \r\n\r\n**Annotations**\r\n\r\nAnnotations are provided by authors of the [paper](https://arxiv.org/abs/2406.07217).\r\n\r\n**Personal and Sensitive Information**\r\n\r\nAll contained personal information is purely synthetic and does not relate to any real individual.\r\n\r\n## Bias, Risks, and Limitations\r\nAll profiles are synthetic and do not correspond to any real subpopulations. We provide a distribution of the personal attributes of the profiles in the accompanying  [paper](https://arxiv.org/abs/2406.07217).\r\nAs the dataset has been created synthetically, data points can inherit limitations (e.g., biases) from the underlying model, GPT-4. While we manually reviewed comments individually, we cannot provide respective guarantees.\r\n## Citation\r\n**BibTeX:**\r\n\r\n```\r\n@misc{2406.07217,\r\nAuthor = {Hanna Yukhymenko and Robin Staab and Mark Vero and Martin Vechev},\r\nTitle = {A Synthetic Dataset for Personal Attribute Inference},\r\nYear = {2024},\r\nEprint = {arXiv:2406.07217},\r\n}\r\n```\r\n**APA:**\r\n\r\n``Hanna Yukhymenko, Robin Staab, Mark Vero, Martin Vechev: “A Synthetic Dataset for Personal Attribute Inference”, 2024; [arXiv:2406.07217](http://arxiv.org/abs/2406.07217).``\r\n## Dataset Card Authors\r\n* Hanna Yukhymenko\r\n* Robin Staab\r\n* Mark Vero","description_withheld":null,"homepage":"https://huggingface.co/datasets/RobinSta/SynthPAI","introduced_date":"2024-06-13","introduced_date_note":null,"introduced_by":{"paper":"/paper/a-synthetic-dataset-for-personal-attribute","title":"A Synthetic Dataset for Personal Attribute Inference","first_author":"Hanna Yukhymenko","url":null},"license":{"name":"cc-by-nc-sa-4.0","url":"https://spdx.org/licenses/CC-BY-NC-SA-4.0"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Zero-shot Classification (unified classes)","url":"/task/zero-shot-classification-unified-classes","datasets_with_task":"/datasets/task/zero-shot-classification-unified-classes"},{"name":"Personality Trait Recognition","url":"/task/personality-trait-recognition","datasets_with_task":"/datasets/task/personality-trait-recognition"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["SynthPAI"],"data_loaders":[],"num_papers_in_archive":4,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/personality-trait-recognition-on-synthpai","task":"Personality Trait Recognition","dataset_variant":"SynthPAI","rows":18,"metrics":["Average accuracy in %"],"first_row_in_archive_order":{"model":"GPT-4","paper":"/paper/a-synthetic-dataset-for-personal-attribute","metrics":{"Average accuracy in %":"77.9%"},"code_links":[{"title":"eth-sri/synthpai","url":"https://github.com/eth-sri/synthpai"},{"title":"eth-sri/llm-anonymization","url":"https://github.com/eth-sri/llm-anonymization"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":1,"of":1,"counts":{"papers_with_a_benchmark_row":1,"with_a_code_link":1,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":1,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":1,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":1,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/a-synthetic-dataset-for-personal-attribute","slug":"a-synthetic-dataset-for-personal-attribute","title":"A Synthetic Dataset for Personal Attribute Inference","date":"2024-06-11","arxiv_id":"2406.07217","rows_on_this_dataset":18,"code_links":2,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":6,"samples_ran":5,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":4,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["eth-sri/synthpai"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/a-synthetic-dataset-for-personal-attribute#ran","syntology_url":"https://syntology.ai/paper/2406.07217","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07217"}}}}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/a-synthetic-dataset-for-personal-attribute","slug":"a-synthetic-dataset-for-personal-attribute","title":"A Synthetic Dataset for Personal Attribute Inference","date":"2024-06-11","arxiv_id":"2406.07217","rows_on_this_dataset":18,"code_links":2,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":6,"samples_ran":5,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":4,"samples_unverified":1,"pointer_only_for_licence":0,"official":{"repos":["eth-sri/synthpai"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/a-synthetic-dataset-for-personal-attribute#ran","syntology_url":"https://syntology.ai/paper/2406.07217","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07217"}}}}],"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/dataset/synthpai/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[{"order":"ran","page":1,"of":1,"url":"/dataset/synthpai/papers/ran/1","json":"/dataset/synthpai/papers/ran/1.json","rows":1}],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":1,"samples_harvested":6,"samples_ran":5,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":4,"samples_unverified":1,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":1,"papers_with_a_run_with_no_instrument_failure":1,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"bf584cb16c44933823e4b9f75d7318020dc971205cb0c88613b13103df73437e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}