{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/rare-disease-annotation-from-mimic-iii","name":"Rare Diseases Mentions in MIMIC-III","full_name":"Rare disease mention annotations from a sample of MIMIC-III clinical notes","description_markdown":"## Data annotation\r\n\r\nThe 1,073 full rare disease mention annotations (from 312 MIMIC-III **discharge summaries**) are in [`full_set_RD_ann_MIMIC_III_disch.csv`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/full_set_RD_ann_MIMIC_III_disch.csv).\r\n\r\nThe data split:\r\n* the first 400 rows are used for validation, [`validation_set_RD_ann_MIMIC_III_disch.csv`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/validation_set_RD_ann_MIMIC_III_disch.csv), and\r\n* the last 673 rows are used for testing, [`test_set_RD_ann_MIMIC_III_disch.csv`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/test_set_RD_ann_MIMIC_III_disch.csv).\r\n\r\nThe 198 rare disease mention annotations (from 145 MIMIC-III **radiology reports**) are in [`test_set_RD_ann_MIMIC_III_rad.csv`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/test_set_RD_ann_MIMIC_III_rad.csv). To note that radiology reports were only used for testing and not for validation.\r\n\r\n**To note**: a row can only be consider a true phenotype of the patient only when the value of the column **gold mention-to-ORDO label** is 1.\r\n\r\n## Data sampling and annotation procedure\r\n* (i) Randomly sampled 500 discharge summaries (and 1000 radiology reports) from MIMIC-III\r\n\r\n* (ii) 312 of the 500 discharge summaries (and 145 of the 1000 radiology reports) have at least one positive UMLS mention linked to ORDO, as identified by SemEHR; there are altogether 1073 (and 198 in radiology reports) UMLS/ORDO mentions.\r\n\r\n* (iii) 3 medical informatics researchers (staff or PhD students) annotated the 1,073 mentions (and 2 medical informatics researchers annotated the 198 mentions in radiology reports), regarding whether they are the correct patient phenotypes matched to UMLS and ORDO. Contradictions in the annotations were then resolved by another research staff having biomedical background.\r\n\r\n## Data dictionary\r\n\r\n| Column   Name                                | Description                                                                                                                                                                                                   |\r\n|----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\r\n| ROW_ID                                       | Identifier unique to each row, see [`https://mimic.physionet.org/mimictables/noteevents/`](https://mimic.physionet.org/mimictables/noteevents/)                                                                                                                                                     |\r\n| SUBJECT_ID                                | Identifier unique to a patient, see [`https://mimic.physionet.org/mimictables/noteevents/`](https://mimic.physionet.org/mimictables/noteevents/)                                                                                                                                                                                                              |\r\n| HADM_ID                                      | Identifier unique to a patient hospital stay, see [`https://mimic.physionet.org/mimictables/noteevents/`](https://mimic.physionet.org/mimictables/noteevents/)                                                                                                                                                                                                              |\r\n| document structure name                    | The document structure name of the mention. The document structure name is identified by   SemEHR  (only for discharge summaries).                                                                                                          |\r\n| document structure offset in full document | The start and ending offsets of the document structure texts (or template) in the whole discharge summary. The document structure is parsed by SemEHR with regular expressions  (only for discharge summaries).                            |\r\n| mention                                      | The mention identified by SemEHR.                                                                                                                                                                          |\r\n| mention offset in document structure       | The start and ending offsets of the mention in the document structure (only for discharge summaries).                                                                                                                                      |\r\n| mention offset in full document            | The start and ending offsets of the mention in the whole discharge summary. They can be calculated by `document structure offset in full document` and `mention offset in document structure`.                                                                                     |\r\n| UMLS with desc                               | The UMLS identified by SemEHR, corresponding to the mention.                                                                                                                                                |\r\n| ORDO with desc                               | The ORDO matched to the UMLS, using the linkage in the ORDO ontology, see [`https://www.ebi.ac.uk/ols/ontologies/ordo/terms?iri=http%3A%2F%2Fwww.orpha.net%2FORDO%2FOrphanet_3325`](https://www.ebi.ac.uk/ols/ontologies/ordo/terms?iri=http%3A%2F%2Fwww.orpha.net%2FORDO%2FOrphanet_3325) as an example.          |\r\n| gold mention-to-UMLS label                 | Whether the mention-UMLS pair indicate a correct phenotype of the patient (i.e. a positive mention that correctly matches to the UMLS concept), 1 if correct, 0 if not.                                 |\r\n| gold UMLS-to-ORDO label                    | Whether the matching is correct from the UMLS concept to the ORDO concept, 1 if correct, 0 if not.                                                                                                          |\r\n| gold mention-to-ORDO label                 | Whether the mention-ORDO triple indicates a correct phenotype of the patient, 1 if correct, 0 if not. This column is 1 if both the mention-to-UMLS label and the UMLS-to-ORDO label are 1, otherwise 0. |\r\n\r\n**Note:**\r\n* These manual annotations are by no means to be perfect. There are hypothetical mentions which were difficult for the annotators to make a decision (see some notes in the raw annotations). Also, they are based on the output of [`SemEHR`](https://github.com/CogStack/CogStack-SemEHR), which does not have 100% recall, so the annotations may not cover all rare diseases mentions from the sampled discharge summaries.\r\n* In row 323 from the full set or the validation set, the mention `nph` is not in the document structure (due to error in mention extraction), thus the `gold mention-to-UMLS label` is `-1`.\r\n\r\n## Raw annotations (with model predictions)\r\nThe two excel workbooks, \r\n\r\n* [`for validation - SemEHR ori (MIMIC-III-DS, free text removed, with predictions).xlsx`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/raw%20annotations%20(with%20model%20predictions)/for%20validation%20-%20SemEHR%20ori%20(MIMIC-III-DS%2C%20free%20text%20removed%2C%20with%20predictions).xlsx) (annotations starting from column `CX` and also in the third sheet, `distinct umls-ordo`), and \r\n\r\n* [`for validation - 1000 docs - ori - MIMIC-III-rad (free text removed, with predictions).xlsx`](https://github.com/acadTags/Rare-disease-identification/blob/main/data%20annotation/raw%20annotations%20(with%20model%20predictions)/for%20validation%20-%201000%20docs%20-%20ori%20-%20MIMIC-III-rad%20(free%20text%20removed%2C%20with%20predictions).xlsx) (annotations starting from column `Z`), \r\n\r\nshow the raw annotations, including each annotator's results and notes, and the predictions of all baselines approaches/tools. The predictions were not available to the annotators when the annotations were made. Free texts of clinical notes have been removed before the publication of the data.","description_withheld":null,"homepage":"https://github.com/acadTags/Rare-disease-identification/tree/main/data%20annotation","introduced_date":"2021-05-05","introduced_date_note":null,"introduced_by":{"paper":"/paper/rare-disease-identification-from-clinical","title":"Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision","first_author":"Hang Dong","url":null},"license":{"name":"MIT Liscence","url":"https://github.com/acadTags/Rare-disease-identification/blob/main/LICENSE"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Named Entity Recognition (NER)","url":"/task/named-entity-recognition-ner","datasets_with_task":"/datasets/task/named-entity-recognition-ner"},{"name":"Entity Linking","url":"/task/entity-linking","datasets_with_task":"/datasets/task/entity-linking"},{"name":"Low Resource Named Entity Recognition","url":"/task/low-resource-named-entity-recognition","datasets_with_task":"/datasets/task/low-resource-named-entity-recognition"},{"name":"Clinical Note Phenotyping","url":"/task/clinical-note-phenotyping","datasets_with_task":"/datasets/task/clinical-note-phenotyping"},{"name":"Computational Phenotyping","url":"/task/computational-phenotyping","datasets_with_task":"/datasets/task/computational-phenotyping"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Rare Diseases Mentions in MIMIC-III"],"data_loaders":[],"num_papers_in_archive":2,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/entity-linking-on-rare-diseases-mentions-in-1","task":"Entity Linking","dataset_variant":"Rare Diseases Mentions in MIMIC-III","rows":2,"metrics":["F1"],"first_row_in_archive_order":{"model":"SemEHR+WS (rules+BlueBERT) with tuning number of training data","paper":"/paper/ontology-based-and-weakly-supervised-rare","metrics":{"F1":"0.711"},"code_links":[{"title":"acadTags/Rare-disease-identification","url":"https://github.com/acadTags/Rare-disease-identification"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":2,"of":2,"counts":{"papers_with_a_benchmark_row":2,"with_a_code_link":2,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":2,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":1,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":1,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/ontology-based-and-weakly-supervised-rare","slug":"ontology-based-and-weakly-supervised-rare","title":"Ontology-Driven and Weakly Supervised Rare Disease Identification from Clinical Notes","date":"2022-05-11","arxiv_id":"2205.05656","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/rare-disease-identification-from-clinical","slug":"rare-disease-identification-from-clinical","title":"Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision","date":"2021-05-05","arxiv_id":"2105.01995","rows_on_this_dataset":1,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":10,"samples_ran":4,"samples_constructed":0,"samples_ran_checked":4,"samples_ran_instrument_failed":0,"samples_unverified":6,"pointer_only_for_licence":0,"official":{"repos":["acadTags/Rare-disease-identification"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":6,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/rare-disease-identification-from-clinical#ran","syntology_url":"https://syntology.ai/paper/2105.01995","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.01995"}}}}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/ontology-based-and-weakly-supervised-rare","slug":"ontology-based-and-weakly-supervised-rare","title":"Ontology-Driven and Weakly Supervised Rare Disease Identification from Clinical Notes","date":"2022-05-11","arxiv_id":"2205.05656","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/rare-disease-identification-from-clinical","slug":"rare-disease-identification-from-clinical","title":"Rare Disease Identification from Clinical Notes with Ontologies and Weak Supervision","date":"2021-05-05","arxiv_id":"2105.01995","rows_on_this_dataset":1,"code_links":1,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":10,"samples_ran":4,"samples_constructed":0,"samples_ran_checked":4,"samples_ran_instrument_failed":0,"samples_unverified":6,"pointer_only_for_licence":0,"official":{"repos":["acadTags/Rare-disease-identification"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":6,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/rare-disease-identification-from-clinical#ran","syntology_url":"https://syntology.ai/paper/2105.01995","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.01995"}}}}],"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/dataset/rare-disease-annotation-from-mimic-iii/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[{"order":"ran","page":1,"of":1,"url":"/dataset/rare-disease-annotation-from-mimic-iii/papers/ran/1","json":"/dataset/rare-disease-annotation-from-mimic-iii/papers/ran/1.json","rows":1}],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":1,"samples_harvested":10,"samples_ran":4,"samples_constructed":0,"samples_ran_checked":4,"samples_ran_instrument_failed":0,"samples_unverified":6,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":1,"papers_with_a_run_with_no_instrument_failure":1,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"33cfc33f07ffc378ac2ae9e8b0ab990bc06ef9d2757e6afa32a3096cfb12c78c","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}