{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/trans-blstm-transformer-with-bidirectional","title":"TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding","arxiv_id":"2003.07000","date":"2020-03-16","proceeding":null,"authors":["Zhiheng Huang","Peng Xu","Davis Liang","Ajay Mishra","Bing Xiang"],"abstract":"Bidirectional Encoder Representations from Transformers (BERT) has recently achieved state-of-the-art performance on a broad range of NLP tasks including sentence classification, machine translation, and question answering. The BERT model architecture is derived primarily from the transformer. Prior to the transformer era, bidirectional Long Short-Term Memory (BLSTM) has been the dominant modeling architecture for neural machine translation and question answering. In this paper, we investigate how these two modeling techniques can be combined to create a more powerful model architecture. We propose a new architecture denoted as Transformer with BLSTM (TRANS-BLSTM) which has a BLSTM layer integrated to each transformer block, leading to a joint modeling framework for transformer and BLSTM. We show that TRANS-BLSTM models consistently lead to improvements in accuracy compared to BERT baselines in GLUE and SQuAD 1.1 experiments. Our TRANS-BLSTM model obtains an F1 score of 94.01% on the SQuAD 1.1 development dataset, which is comparable to the state-of-the-art result.","url_abs":"https://arxiv.org/abs/2003.07000v1","url_pdf":"https://arxiv.org/pdf/2003.07000v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"paraphrase-identification","task_name":"Paraphrase Identification"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-classification","task_name":"Sentence Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"TRANS-BLSTM","rank_in_archive_order":18,"of":43,"metrics":{"Accuracy":"94.08%"},"uses_additional_data":false},{"leaderboard":"/sota/paraphrase-identification-on-quora-question","task":"Paraphrase Identification","dataset":"Quora Question Pairs","model":"TRANS-BLSTM","rank_in_archive_order":24,"of":31,"metrics":{"Accuracy":"88.28"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-glue-cola","task":"Text Classification","dataset":"GLUE COLA","model":"TRANS-BLSTM","rank_in_archive_order":1,"of":1,"metrics":{},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-glue-mrpc","task":"Text Classification","dataset":"GLUE MRPC","model":"TRANS-BLSTM","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"90.45"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-glue-rte","task":"Text Classification","dataset":"GLUE RTE","model":"TRANS-BLSTM","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"79.78"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-glue-sst2","task":"Text Classification","dataset":"GLUE SST2","model":"TRANS-BLSTM","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"94.38"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-glue-stsb","task":"Text Classification","dataset":"GLUE STSB","model":"TRANS-BLSTM","rank_in_archive_order":1,"of":1,"metrics":{},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2003.07000","atlas_url":"https://app.syntology.ai/?focus=2003.07000","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2003.07000"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2003.07000\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2003.07000"},"call":"get_concepts_for_paper(arxiv_id=\"2003.07000\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"40b21fc5a2e96b9e907b27bde8efb3df6c142eb8c03d1643fad237c5ba9c3a56","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}