{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/replication-study-development-and-validation","title":"Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs","arxiv_id":"1803.04337","date":"2018-03-12","proceeding":null,"authors":["Mike Voets","Kajsa Møllersen","Lars Ailo Bongo"],"abstract":"Replication studies are essential for validation of new methods, and are\ncrucial to maintain the high standards of scientific publications, and to use\nthe results in practice. We have attempted to replicate the main method in\n'Development and validation of a deep learning algorithm for detection of\ndiabetic retinopathy in retinal fundus photographs' published in JAMA 2016;\n316(22). We re-implemented the method since the source code is not available,\nand we used publicly available data sets. The original study used non-public\nfundus images from EyePACS and three hospitals in India for training. We used a\ndifferent EyePACS data set from Kaggle. The original study used the benchmark\ndata set Messidor-2 to evaluate the algorithm's performance. We used the same\ndata set. In the original study, ophthalmologists re-graded all images for\ndiabetic retinopathy, macular edema, and image gradability. There was one\ndiabetic retinopathy grade per image for our data sets, and we assessed image\ngradability ourselves. Hyper-parameter settings were not described in the\noriginal study. But some of these were later published. We were not able to\nreplicate the original study. Our algorithm's area under the receiver operating\ncurve (AUC) of 0.94 on the Kaggle EyePACS test set and 0.80 on Messidor-2 did\nnot come close to the reported AUC of 0.99 in the original study. This may be\ncaused by the use of a single grade per image, different data, or different not\ndescribed hyper-parameter settings. This study shows the challenges of\nreplicating deep learning, and the need for more replication studies to\nvalidate deep learning methods, especially for medical image analysis.\n  Our source code and instructions are available at:\nhttps://github.com/mikevoets/jama16-retina-replication","url_abs":"http://arxiv.org/abs/1803.04337v3","url_pdf":"http://arxiv.org/pdf/1803.04337v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"replication-study-development-and-validation","repo_url":"https://github.com/mikevoets/jama16-retina-replication","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"diabetic-retinopathy-detection","task_name":"Diabetic Retinopathy Detection"},{"task_slug":"medical-image-analysis","task_name":"Medical Image Analysis"},{"task_slug":"medical-image-segmentation","task_name":"Medical Image Segmentation"},{"task_slug":"mitosis-detection","task_name":"Mitosis Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"mikevoets/jama16-retina-replication","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"9bd3bea490a9272bca0b88b075952245f63eaf53a14ed521358656101a5d4b9d","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}