{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/on-the-effectiveness-of-compact-biomedical","title":"On the Effectiveness of Compact Biomedical Transformers","arxiv_id":"2209.03182","date":"2022-09-07","proceeding":null,"authors":["Omid Rohanian","Mohammadmahdi Nouriborji","Samaneh Kouchaki","David A. Clifton"],"abstract":"Language models pre-trained on biomedical corpora, such as BioBERT, have recently shown promising results on downstream biomedical tasks. Many existing pre-trained models, on the other hand, are resource-intensive and computationally heavy owing to factors such as embedding size, hidden dimension, and number of layers. The natural language processing (NLP) community has developed numerous strategies to compress these models utilising techniques such as pruning, quantisation, and knowledge distillation, resulting in models that are considerably faster, smaller, and subsequently easier to use in practice. By the same token, in this paper we introduce six lightweight models, namely, BioDistilBERT, BioTinyBERT, BioMobileBERT, DistilBioBERT, TinyBioBERT, and CompactBioBERT which are obtained either by knowledge distillation from a biomedical teacher or continual learning on the Pubmed dataset via the Masked Language Modelling (MLM) objective. We evaluate all of our models on three biomedical tasks and compare them with BioBERT-v1.1 to create efficient lightweight models that perform on par with their larger counterparts. All the models will be publicly available on our Huggingface profile at https://huggingface.co/nlpie and the codes used to run the experiments will be available at https://github.com/nlpie-research/Compact-Biomedical-Transformers.","url_abs":"https://arxiv.org/abs/2209.03182v1","url_pdf":"https://arxiv.org/pdf/2209.03182v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-effectiveness-of-compact-biomedical","repo_url":"https://github.com/nlpie-research/compact-biomedical-transformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/named-entity-recognition-on-bc2gm","task":"Named Entity Recognition (NER)","dataset":"BC2GM","model":"BioDistilBERT","rank_in_archive_order":2,"of":13,"metrics":{"F1":"86.97"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc2gm","task":"Named Entity Recognition (NER)","dataset":"BC2GM","model":"CompactBioBERT","rank_in_archive_order":3,"of":13,"metrics":{"F1":"86.71"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc2gm","task":"Named Entity Recognition (NER)","dataset":"BC2GM","model":"DistilBioBERT","rank_in_archive_order":4,"of":13,"metrics":{"F1":"86.6"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc2gm","task":"Named Entity Recognition (NER)","dataset":"BC2GM","model":"BioMobileBERT","rank_in_archive_order":7,"of":13,"metrics":{"F1":"85.26"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-chemical","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-chemical","model":"DistilBioBERT","rank_in_archive_order":4,"of":13,"metrics":{"F1":"94.53"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-chemical","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-chemical","model":"BioDistilBERT","rank_in_archive_order":5,"of":13,"metrics":{"F1":"94.48"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-chemical","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-chemical","model":"CompactBioBERT","rank_in_archive_order":6,"of":13,"metrics":{"F1":"94.31"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-chemical","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-chemical","model":"BioMobileBERT","rank_in_archive_order":7,"of":13,"metrics":{"F1":"94.23"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-disease","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-disease","model":"BioDistilBERT","rank_in_archive_order":7,"of":10,"metrics":{"F1":"85.61"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-disease","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-disease","model":"DistilBioBERT","rank_in_archive_order":8,"of":10,"metrics":{"F1":"85.42"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-disease","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-disease","model":"CompactBioBERT","rank_in_archive_order":9,"of":10,"metrics":{"F1":"85.38"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-on-bc5cdr-disease","task":"Named Entity Recognition (NER)","dataset":"BC5CDR-disease","model":"BioMobileBERT","rank_in_archive_order":10,"of":10,"metrics":{"F1":"84.62"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-jnlpba","task":"Named Entity Recognition (NER)","dataset":"JNLPBA","model":"BioMobileBERT","rank_in_archive_order":5,"of":17,"metrics":{"F1":"80.13"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-jnlpba","task":"Named Entity Recognition (NER)","dataset":"JNLPBA","model":"DistilBioBERT","rank_in_archive_order":7,"of":17,"metrics":{"F1":"79.97"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-jnlpba","task":"Named Entity Recognition (NER)","dataset":"JNLPBA","model":"CompactBioBERT","rank_in_archive_order":8,"of":17,"metrics":{"F1":"79.88"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-jnlpba","task":"Named Entity Recognition (NER)","dataset":"JNLPBA","model":"BioDistilBERT","rank_in_archive_order":10,"of":17,"metrics":{"F1":"79.1"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI-disease","model":"CompactBioBERT","rank_in_archive_order":10,"of":26,"metrics":{"F1":"88.67"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI-disease","model":"DistilBioBERT","rank_in_archive_order":14,"of":26,"metrics":{"F1":"87.93"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI-disease","model":"BioDistilBERT","rank_in_archive_order":18,"of":26,"metrics":{"F1":"87.61"},"uses_additional_data":false},{"leaderboard":"/sota/named-entity-recognition-ner-on-ncbi-disease","task":"Named Entity Recognition (NER)","dataset":"NCBI-disease","model":"BioMobileBERT","rank_in_archive_order":20,"of":26,"metrics":{"F1":"87.21"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"nlpie-research/compact-biomedical-transformers","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"983c22e64ed7a782c5229cfac2de66c8f8bb0038ecfc5e7a8cfa6c4742b03a09","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}