{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/ss-cxr-multitask-representation-learning","title":"SPCXR: Self-supervised Pretraining using Chest X-rays Towards a Domain Specific Foundation Model","arxiv_id":"2211.12944","date":"2022-11-23","proceeding":null,"authors":["Syed Muhammad Anwar","Abhijeet Parida","Sara Atito","Muhammad Awais","Gustavo Nino","Josef Kitler","Marius George Linguraru"],"abstract":"Chest X-rays (CXRs) are a widely used imaging modality for the diagnosis and prognosis of lung disease. The image analysis tasks vary. Examples include pathology detection and lung segmentation. There is a large body of work where machine learning algorithms are developed for specific tasks. A significant recent example is Coronavirus disease (covid-19) detection using CXR data. However, the traditional diagnostic tool design methods based on supervised learning are burdened by the need to provide training data annotation, which should be of good quality for better clinical outcomes. Here, we propose an alternative solution, a new self-supervised paradigm, where a general representation from CXRs is learned using a group-masked self-supervised framework. The pre-trained model is then fine-tuned for domain-specific tasks such as covid-19, pneumonia detection, and general health screening. We show that the same pre-training can be used for the lung segmentation task. Our proposed paradigm shows robust performance in multiple downstream tasks which demonstrates the success of the pre-training. Moreover, the performance of the pre-trained models on data with significant drift during test time proves the learning of a better generic representation. The methods are further validated by covid-19 detection in a unique small-scale pediatric data set. The performance gain in accuracy (~25%) is significant when compared to a supervised transformer-based method. This adds credence to the strength and reliability of our proposed framework and pre-training strategy.","url_abs":"https://arxiv.org/abs/2211.12944v2","url_pdf":"https://arxiv.org/pdf/2211.12944v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"covid-19-detection","task_name":"COVID-19 Diagnosis"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"pneumonia-detection","task_name":"Pneumonia Detection"},{"task_slug":"prognosis","task_name":"Prognosis"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"deit","method_name":"DeiT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"test","method_name":"Test"},{"method_slug":"u-net","method_name":"U-Net"},{"method_slug":"unetr","method_name":"UNETR"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/covid-19-diagnosis-on","task":"COVID-19 Diagnosis","dataset":"","model":"Thoracic Radiologist","rank_in_archive_order":2,"of":2,"metrics":{"Per-Class Accuracy":"64.37"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"SS-CXR","rank_in_archive_order":1,"of":7,"metrics":{"Per-Class Accuracy":"98.25"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"DenseNet-169","rank_in_archive_order":2,"of":7,"metrics":{"Per-Class Accuracy":"98.15"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"EfficientNet-B2","rank_in_archive_order":3,"of":7,"metrics":{"Per-Class Accuracy":"97.6"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"Inception Resnet V2","rank_in_archive_order":4,"of":7,"metrics":{"Per-Class Accuracy":"97.55"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"Inception ResNet","rank_in_archive_order":5,"of":7,"metrics":{"Per-Class Accuracy":"97.5"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"DenseNet-121","rank_in_archive_order":6,"of":7,"metrics":{"Per-Class Accuracy":"96.5"},"uses_additional_data":false},{"leaderboard":"/sota/covid-19-diagnosis-on-covidx-cxr-3","task":"COVID-19 Diagnosis","dataset":"COVIDx CXR-3","model":"ViT-S","rank_in_archive_order":7,"of":7,"metrics":{"Per-Class Accuracy":"89.25"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-montgomery-county-x","task":"Semantic Segmentation","dataset":"Montgomery County X-ray Set","model":"UNETR + SS-CXR","rank_in_archive_order":1,"of":3,"metrics":{"F1-score":"0.9561"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-montgomery-county-x","task":"Semantic Segmentation","dataset":"Montgomery County X-ray Set","model":"UNETR+ SS-IN","rank_in_archive_order":2,"of":3,"metrics":{"F1-score":"0.9453"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-montgomery-county-x","task":"Semantic Segmentation","dataset":"Montgomery County X-ray Set","model":"UNETR","rank_in_archive_order":3,"of":3,"metrics":{"F1-score":"0.9227"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4275802cb215441c8ecc800d8e0f0eecb478dcbe21f3b5d4bdc08d08d6425f4b","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}