{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/breaking-free-transformer-models-task","title":"Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs","arxiv_id":"2401.16638","date":"2024-01-30","proceeding":null,"authors":["Stepan Tytarenko","Mohammad Ruhul Amin"],"abstract":"Fine-tuning large pre-trained language models (LLMs) on particular datasets is a commonly employed strategy in Natural Language Processing (NLP) classification tasks. However, this approach usually results in a loss of models generalizability. In this paper, we present a framework that allows for maintaining generalizability, and enhances the performance on the downstream task by utilizing task-specific context attribution. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space, referred to as context attribution in this paper. The specific concept operator is optimized during the supervised learning stage via novel loss functions. The proposed framework demonstrates that context attribution of the text representation for each task objective can improve the capacity of the discriminator function and thus achieve better performance for the classification task. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability. Specifically, for the non-fine-tuned BERT on the HateXplain dataset, we observe 8% improvement in accuracy and 10% improvement in F1-score. Whereas for the IMDB dataset, fine-tuned state-of-the-art XLNet is outperformed by 1% for both accuracy and F1-score. Furthermore, in an out-of-domain cross-dataset test, DistilBERT fine-tuned on the IMDB dataset in conjunction with the proposed model improves the F1-score on the HateXplain dataset by 7%. For the Social Media Attributions dataset of YouTube comments, we observe 5.2% increase in F1-metric. The proposed framework is implemented with PyTorch and provided open-source on GitHub.","url_abs":"https://arxiv.org/abs/2401.16638v1","url_pdf":"https://arxiv.org/pdf/2401.16638v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"breaking-free-transformer-models-task","repo_url":"https://github.com/stepantita/space-model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"sentiment-classification","task_name":"Sentiment Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"zero-shot-text-classification","task_name":"Zero-Shot Text Classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"distilbert","method_name":"DistilBERT"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"},{"method_slug":"xlnet","method_name":"XLNet"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/sentiment-analysis-on-imdb","task":"Sentiment Analysis","dataset":"IMDb","model":"Space-XLNet","rank_in_archive_order":20,"of":49,"metrics":{"Accuracy":"94.88"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-imdb-movie-reviews-1","task":"Sentiment Analysis","dataset":"IMDb Movie Reviews","model":"Space-XLNet","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy (2 classes)":"0.9488","F1 Macro":"0.9487"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-imdb-movie-reviews-1","task":"Sentiment Analysis","dataset":"IMDb Movie Reviews","model":"Space-DistilBERT","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy (2 classes)":"0.8322","F1 Macro":"0.8320"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-hatexplain-1","task":"Text Classification","dataset":"HateXplain","model":"Space-XLNet","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy (2 classes)":"0.8798","F1 Macro":"0.8797"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-hatexplain-1","task":"Text Classification","dataset":"HateXplain","model":"XLNet","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy (2 classes)":"0.8160","F1 Macro":"0.8156"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-hatexplain-1","task":"Text Classification","dataset":"HateXplain","model":"Space-BERT","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy (2 classes)":"0.8110","F1 Macro":"0.8108"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-hatexplain-1","task":"Text Classification","dataset":"HateXplain","model":"BERT-base","rank_in_archive_order":4,"of":4,"metrics":{"Accuracy (2 classes)":"0.6588","F1 Macro":"0.6555"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-imdb-movie-reviews-1","task":"Text Classification","dataset":"IMDb Movie Reviews","model":"XLNet","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy (2 classes)":"0.9387"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-imdb-movie-reviews-1","task":"Text Classification","dataset":"IMDb Movie Reviews","model":"Space-XLNet","rank_in_archive_order":3,"of":3,"metrics":{"F1 Macro":"0.9487"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-social-media","task":"Text Classification","dataset":"Social media attributions of YouTube comments","model":"Space-BERT","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy (2 classes)":"0.8309","F1 Macro":"0.8006"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-social-media","task":"Text Classification","dataset":"Social media attributions of YouTube comments","model":"BERT-base","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy (2 classes)":"0.8220","F1 Macro":"0.7484"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"stepantita/space-model","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b95201f8e51823e84513c70edfd39bd441583065d6d56f69f4298003923038e8","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}