{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/etc","slug":"etc","name":"ETC","full_name":"Extended Transformer Construction","full_name_withheld":false,"description_markdown":"**Extended Transformer Construction**, or **ETC**, is an extension of the [Transformer](https://paperswithcode.com/method/transformer) architecture with a new attention mechanism that extends the original in two main ways: (1) it allows scaling up the input length from 512 to several thousands; and (2) it can ingesting structured inputs instead of just linear sequences. The key ideas that enable ETC to achieve these are a new [global-local attention mechanism](https://paperswithcode.com/method/global-local-attention), coupled with [relative position encodings](https://paperswithcode.com/method/relative-position-encodings). ETC also allows lifting weights from existing [BERT](https://paperswithcode.com/method/bert) models, saving computational resources while training.","description_state":"present","introduced_year":null,"introduced_by":{"title":"ETC: Encoding Long and Structured Inputs in Transformers","paper":"/paper/etc-encoding-long-and-structured-data-in","first_author":"Joshua Ainslie","n_authors":10,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/etc-encoding-long-and-structured-data-in"},"source":{"url":"https://arxiv.org/abs/2004.08483v5","title":"ETC: Encoding Long and Structured Inputs in Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":36,"archive_num_papers":36,"papers_newest_first":[{"paper":null,"slug":"two-player-zero-sum-games-with-bandit","title":"Two-Player Zero-Sum Games with Bandit Feedback","date":"2025-06-17","arxiv_id":"2506.14518","n_code_links":0,"syntology":null},{"paper":null,"slug":"koopman-based-event-triggered-control-from","title":"Koopman-Based Event-Triggered Control from Data","date":"2025-04-19","arxiv_id":"2504.14334","n_code_links":0,"syntology":null},{"paper":null,"slug":"performance-barrier-event-triggered-pde","title":"Performance-Barrier Event-Triggered PDE Control of Traffic Flow","date":"2025-01-01","arxiv_id":"2501.00722","n_code_links":0,"syntology":null},{"paper":null,"slug":"automated-toll-management-system-using-rfid","title":"Automated Toll Management System Using RFID and Image Processing","date":"2024-12-02","arxiv_id":"2412.01728","n_code_links":0,"syntology":null},{"paper":null,"slug":"hierarchical-event-triggered-systems-safe","title":"Hierarchical Event-Triggered Systems: Safe Learning of Quasi-Optimal Deadline Policies","date":"2024-09-15","arxiv_id":"2409.09812","n_code_links":0,"syntology":null},{"paper":null,"slug":"performance-barrier-event-triggered-control","title":"Performance-Barrier Event-Triggered Control of a Class of Reaction-Diffusion PDEs","date":"2024-07-11","arxiv_id":"2407.08178","n_code_links":0,"syntology":null},{"paper":null,"slug":"contextual-dynamic-pricing-algorithms","title":"Contextual Dynamic Pricing: Algorithms, Optimality, and Local Differential Privacy Constraints","date":"2024-06-04","arxiv_id":"2406.02424","n_code_links":0,"syntology":null},{"paper":null,"slug":"event-triggered-robust-cooperative-output","title":"Event-Triggered Robust Cooperative Output Regulation for a Class of Linear Multi-Agent Systems with an Unknown Exosystem","date":"2024-03-01","arxiv_id":"2403.00645","n_code_links":0,"syntology":null},{"paper":null,"slug":"replication-proof-bandit-mechanism-design","title":"Replication-proof Bandit Mechanism Design with Bayesian Agents","date":"2023-12-28","arxiv_id":"2312.16896","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-based-scheduling-for-information","title":"Learning-based Scheduling for Information Accuracy and Freshness in Wireless Networks","date":"2023-10-24","arxiv_id":"2310.15705","n_code_links":0,"syntology":null},{"paper":null,"slug":"listen-to-minority-encrypted-traffic","title":"Listen to Minority: Encrypted Traffic Classification for Class Imbalance with Contrastive Pre-Training","date":"2023-08-31","arxiv_id":"2308.16453","n_code_links":0,"syntology":null},{"paper":null,"slug":"high-dimensional-contextual-bandit-problem","title":"High-dimensional Contextual Bandit Problem without Sparsity","date":"2023-06-19","arxiv_id":"2306.11017","n_code_links":0,"syntology":null},{"paper":null,"slug":"permutation-decision-trees","title":"Permutation Decision Trees","date":"2023-06-05","arxiv_id":"2306.02617","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-improved-heart-disease-prediction-using","title":"An Improved Heart Disease Prediction Using Stacked Ensemble Method","date":"2023-04-12","arxiv_id":"2304.06015","n_code_links":0,"syntology":null},{"paper":null,"slug":"asynchronous-event-triggered-control-for-non","title":"Asynchronous Event-Triggered Control for Non-Linear Systems","date":"2022-11-25","arxiv_id":"2211.13846","n_code_links":0,"syntology":null},{"paper":null,"slug":"littlebird-efficient-faster-longer","title":"LittleBird: Efficient Faster & Longer Transformer for Question Answering","date":"2022-10-21","arxiv_id":"2210.11870","n_code_links":0,"syntology":null},{"paper":null,"slug":"short-text-pre-training-with-extended-token","title":"Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding","date":"2022-10-08","arxiv_id":"2210.03915","n_code_links":0,"syntology":null},{"paper":null,"slug":"embed-to-control-partially-observed-systems","title":"Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency","date":"2022-05-26","arxiv_id":"2205.13476","n_code_links":0,"syntology":null},{"paper":"/paper/thompson-sampling-for-bandit-learning-in","slug":"thompson-sampling-for-bandit-learning-in","title":"Thompson Sampling for Bandit Learning in Matching Markets","date":"2022-04-26","arxiv_id":"2204.12048","n_code_links":1,"syntology":null},{"paper":null,"slug":"etcetera-beyond-event-triggered-control","title":"ETCetera: beyond Event-Triggered Control","date":"2022-03-03","arxiv_id":"2203.01623","n_code_links":0,"syntology":null},{"paper":null,"slug":"formal-analysis-of-the-sampling-behaviour-of","title":"Formal Analysis of the Sampling Behaviour of Stochastic Event-Triggered Control","date":"2022-02-21","arxiv_id":"2202.10178","n_code_links":0,"syntology":null},{"paper":null,"slug":"chaos-and-order-in-event-triggered-control","title":"Chaos and order in event-triggered control","date":"2022-01-12","arxiv_id":"2201.04462","n_code_links":0,"syntology":null},{"paper":null,"slug":"detecting-extratropical-cyclones-of-the","title":"Detecting Extratropical Cyclones of the Northern Hemisphere with Single Shot Detector","date":"2021-12-01","arxiv_id":"2112.01283","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-countable-armed-bandit-with-vanishing","title":"Bandits with Dynamic Arm-acquisition Costs","date":"2021-10-23","arxiv_id":"2110.12118","n_code_links":0,"syntology":null},{"paper":null,"slug":"computing-the-average-inter-sample-time-of","title":"Computing the average inter-sample time of event-triggered control using quantitative automata","date":"2021-09-29","arxiv_id":"2109.14391","n_code_links":0,"syntology":null},{"paper":null,"slug":"uncertainties-and-output-feedback-in-rollout","title":"Uncertainties and output feedback in rollout event-triggered control","date":"2021-08-20","arxiv_id":"2108.09125","n_code_links":0,"syntology":null},{"paper":null,"slug":"rollout-event-triggered-control-reconciling","title":"Rollout event-triggered control: reconciling event- and time-triggered control","date":"2021-08-06","arxiv_id":"2108.02994","n_code_links":0,"syntology":null},{"paper":null,"slug":"debiasing-samples-from-online-learning-using","title":"Debiasing Samples from Online Learning Using Bootstrap","date":"2021-07-31","arxiv_id":"2108.00236","n_code_links":0,"syntology":null},{"paper":null,"slug":"model-selection-for-generic-contextual","title":"Model Selection for Generic Contextual Bandits","date":"2021-07-07","arxiv_id":"2107.03455","n_code_links":0,"syntology":null},{"paper":null,"slug":"abstracting-the-sampling-behaviour-of","title":"Abstracting the Sampling Behaviour of Stochastic Linear Periodic Event-Triggered Control Systems","date":"2021-03-25","arxiv_id":"2103.13839","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-wireless-control-bus-enabling-efficient","title":"The Wireless Control Bus: Enabling Efficient Multi-hop Event-Triggered Control with Concurrent Transmissions","date":"2021-01-26","arxiv_id":"2101.10961","n_code_links":0,"syntology":null},{"paper":"/paper/informer-transformer-likes-informed-attention","slug":"informer-transformer-likes-informed-attention","title":"RealFormer: Transformer Likes Residual Attention","date":"2020-12-21","arxiv_id":"2012.11747","n_code_links":5,"syntology":null},{"paper":null,"slug":"traffic-abstractions-of-nonlinear-event","title":"Abstracting the Traffic of Nonlinear Event-Triggered Control Systems","date":"2020-10-23","arxiv_id":"2010.12341","n_code_links":0,"syntology":null},{"paper":null,"slug":"machine-learning-in-event-triggered-control","title":"Machine Learning in Event-Triggered Control: Recent Advances and Open Issues","date":"2020-09-27","arxiv_id":"2009.12783","n_code_links":0,"syntology":null},{"paper":null,"slug":"region-based-self-triggered-control-for","title":"Region-Based Self-Triggered Control for Perturbed and Uncertain Nonlinear Systems","date":"2020-05-01","arxiv_id":"2005.00473","n_code_links":0,"syntology":null},{"paper":"/paper/etc-encoding-long-and-structured-data-in","slug":"etc-encoding-long-and-structured-data-in","title":"ETC: Encoding Long and Structured Inputs in Transformers","date":"2020-04-17","arxiv_id":"2004.08483","n_code_links":2,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["google-research/google-research"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}}],"papers_shown":36,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":36,"with_a_code_link":3,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":36,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/method/etc/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/etc/papers/ran/1","json":"/method/etc/papers/ran/1.json","rows":1}],"tasks":[{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/multi-armed-bandits","name":"Multi-Armed Bandits","papers":2},{"task":"/task/question-answering","name":"Question Answering","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/thompson-sampling","name":"Thompson Sampling","papers":2},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":2},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":"/task/disease-prediction","name":"Disease Prediction","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/management","name":"Management","papers":1},{"task":"/task/masked-language-modeling","name":"Masked Language Modeling","papers":1},{"task":"/task/model-predictive-control","name":"Model Predictive Control","papers":1},{"task":"/task/model-selection","name":"Model Selection","papers":1},{"task":"/task/natural-language-inference","name":"Natural Language Inference","papers":1},{"task":"/task/natural-questions","name":"Natural Questions","papers":1}],"tasks_shown":20,"n_tasks":38,"usage_by_year":[{"year":"2020","papers":5},{"year":"2021","papers":9},{"year":"2022","papers":8},{"year":"2023","papers":6},{"year":"2024","papers":5},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/etc"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=ETC","read_at":"2026-10-05T00:55:44+00:00","query":"ETC","kind":"paper","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"2004.08483","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"unresolved_bearers","may_be_wrong_sense":false,"bearers_found":7,"served":null,"leading_other_bearer":{"arxiv_id":"2004.08483","title":"ETC: Encoding Long and Structured Inputs in Transformers","citations":58,"named_by":["title_prefix"],"bearer_name":"ETC: Encoding Long and Structured Inputs in Transformers","ask_with":"have('2004.08483')"},"n_other_bearers_listed":5,"note":"Nothing resolved to 'ETC' by name, but 7 thing(s) here DECLARE that name -- most cited: the paper 'ETC: Encoding Long and Structured Inputs in Transformers' (arXiv:2004.08483, 58 citations). None of them is a well-cited bearer of the name, so this is still a coverage gap for any better-known sense -- they are listed so you can judge, not as a substitute."},"name_competition_text":"Nothing resolved to this name, but 7 entries in Syntology's graph carry it; the most cited is the paper “ETC: Encoding Long and Structured Inputs in Transformers” (arXiv:2004.08483, 58 citations). None of them is well cited, so for any better known sense of the name this is still a gap in coverage; they are listed so you can judge, not as a substitute.","near_matches":["HD-ETC","OMS-ETC","EBHD-ETC"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). Other entries in Syntology's graph carry this name; the lookup's note below says what that means for this name.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"42e61747390fbb884e282b8500351acd1fc64a2e060087470f0f2a2763c3a33a","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}