{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/gpt-2","slug":"gpt-2","name":"GPT-2","full_name":"GPT-2","full_name_withheld":false,"description_markdown":"**GPT-2** is a [Transformer](https://paperswithcode.com/methods/category/transformers) architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a [WebText dataset](https://paperswithcode.com/dataset/webtext) - text from 45 million website links. It largely follows the previous [GPT](https://paperswithcode.com/method/gpt) architecture with some modifications:\r\n\r\n- [Layer normalization](https://paperswithcode.com/method/layer-normalization) is moved to the input of each sub-block, similar to a\r\npre-activation residual network and an additional layer normalization was added after the final self-attention block. \r\n\r\n- A modified initialization which accounts for the accumulation on the residual path with model depth\r\nis used. Weights of residual layers are scaled at initialization by a factor of $1/\\sqrt{N}$ where $N$ is the number of residual layers. \r\n\r\n- The vocabulary is expanded to 50,257. The context size is expanded from 512 to 1024 tokens and\r\na larger batch size of 512 is used.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Language Models are Unsupervised Multitask Learners","paper":"/paper/language-models-are-unsupervised-multitask","first_author":"Alec Radford","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/language-models-are-unsupervised-multitask"},"source":{"url":"https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf","title":"Language Models are Unsupervised Multitask Learners","url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Autoregressive Transformers","url":"/methods/category/autoregressive-transformers","pwc_aliases":[]},{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":768,"archive_num_papers":768,"papers_newest_first":[{"paper":"/paper/the-automated-llm-speedrunning-benchmark","slug":"the-automated-llm-speedrunning-benchmark","title":"The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements","date":"2025-06-27","arxiv_id":"2506.22419","n_code_links":1,"syntology":null},{"paper":null,"slug":"m2beamllm-multimodal-sensing-empowered-mmwave","title":"M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models","date":"2025-06-17","arxiv_id":"2506.14532","n_code_links":0,"syntology":null},{"paper":"/paper/decomposing-mlp-activations-into","slug":"decomposing-mlp-activations-into","title":"Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization","date":"2025-06-12","arxiv_id":"2506.10920","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-novel-lightweight-transformer-with-edge","title":"A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning","date":"2025-06-11","arxiv_id":"2506.09429","n_code_links":0,"syntology":null},{"paper":null,"slug":"quality-diversity-red-teaming-automated","title":"Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models","date":"2025-06-08","arxiv_id":"2506.07121","n_code_links":0,"syntology":null},{"paper":null,"slug":"dissecting-bias-in-llms-a-mechanistic","title":"Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective","date":"2025-06-05","arxiv_id":"2506.05166","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-exploratory-framework-for-future-seti","title":"An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models","date":"2025-06-03","arxiv_id":"2506.02730","n_code_links":0,"syntology":null},{"paper":null,"slug":"rethinking-the-effects-of-data-contamination","title":"Rethinking the effects of data contamination in Code Intelligence","date":"2025-06-03","arxiv_id":"2506.02791","n_code_links":0,"syntology":null},{"paper":"/paper/how-neural-networks-organize-concepts","slug":"how-neural-networks-organize-concepts","title":"How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability","date":"2025-06-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"power-of-two-pot-weights-in-large-language","title":"Power-of-Two (PoT) Weights in Large Language Models (LLMs)","date":"2025-05-31","arxiv_id":"2506.00315","n_code_links":0,"syntology":null},{"paper":null,"slug":"matryoshka-model-learning-for-improved","title":"Matryoshka Model Learning for Improved Elastic Student Models","date":"2025-05-29","arxiv_id":"2505.23337","n_code_links":0,"syntology":null},{"paper":null,"slug":"privacy-preserving-chest-x-ray-report","title":"Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2","date":"2025-05-27","arxiv_id":"2505.21715","n_code_links":0,"syntology":null},{"paper":null,"slug":"anveshana-a-new-benchmark-dataset-for-cross","title":"Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents","date":"2025-05-26","arxiv_id":"2505.19494","n_code_links":0,"syntology":null},{"paper":null,"slug":"conversational-lexicography-querying","title":"Conversational Lexicography: Querying Lexicographic Data on Knowledge Graphs with SPARQL through Natural Language","date":"2025-05-26","arxiv_id":"2505.19971","n_code_links":0,"syntology":null},{"paper":null,"slug":"eslm-risk-averse-selective-language-modeling","title":"ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining","date":"2025-05-26","arxiv_id":"2505.19893","n_code_links":0,"syntology":null},{"paper":null,"slug":"strong-membership-inference-attacks-on","title":"Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models","date":"2025-05-24","arxiv_id":"2505.18773","n_code_links":0,"syntology":null},{"paper":"/paper/adams-momentum-itself-can-be-a-normalizer-for","slug":"adams-momentum-itself-can-be-a-normalizer-for","title":"AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training","date":"2025-05-22","arxiv_id":"2505.16363","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["pku-huzhang/AdamS"],"state":"official: no sample here; runs from other or unrecorded repositories","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["found_in_text"]}}},{"paper":null,"slug":"the-polar-express-optimal-matrix-sign-methods","title":"The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm","date":"2025-05-22","arxiv_id":"2505.16932","n_code_links":0,"syntology":null},{"paper":"/paper/breaking-bad-tokens-detoxification-of-llms","slug":"breaking-bad-tokens-detoxification-of-llms","title":"Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders","date":"2025-05-20","arxiv_id":"2505.14536","n_code_links":0,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":null,"slug":"out-of-distribution-generalization-of-in","title":"Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective","date":"2025-05-20","arxiv_id":"2505.14808","n_code_links":0,"syntology":null},{"paper":null,"slug":"scaling-laws-for-state-dynamics-in-large","title":"Scaling Laws for State Dynamics in Large Language Models","date":"2025-05-20","arxiv_id":"2505.14892","n_code_links":0,"syntology":null},{"paper":"/paper/vesselgpt-autoregressive-modeling-of-vascular","slug":"vesselgpt-autoregressive-modeling-of-vascular","title":"VesselGPT: Autoregressive Modeling of Vascular Geometry","date":"2025-05-19","arxiv_id":"2505.13318","n_code_links":1,"syntology":null},{"paper":null,"slug":"can-an-easy-to-hard-curriculum-make-reasoning","title":"Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2","date":"2025-05-16","arxiv_id":"2505.11643","n_code_links":0,"syntology":null},{"paper":null,"slug":"optimal-control-for-transformer-architectures","title":"Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency","date":"2025-05-16","arxiv_id":"2505.13499","n_code_links":0,"syntology":null},{"paper":"/paper/are-sparse-autoencoders-useful-for-java","slug":"are-sparse-autoencoders-useful-for-java","title":"Are Sparse Autoencoders Useful for Java Function Bug Detection?","date":"2025-05-15","arxiv_id":"2505.10375","n_code_links":1,"syntology":null},{"paper":null,"slug":"memorization-compression-cycles-improve","title":"Memorization-Compression Cycles Improve Generalization","date":"2025-05-13","arxiv_id":"2505.08727","n_code_links":0,"syntology":null},{"paper":"/paper/probability-consistency-in-large-language","slug":"probability-consistency-in-large-language","title":"Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies","date":"2025-05-13","arxiv_id":"2505.08739","n_code_links":1,"syntology":null},{"paper":"/paper/llm-e-guess-can-llms-capabilities-advance","slug":"llm-e-guess-can-llms-capabilities-advance","title":"LLM-e Guess: Can LLMs Capabilities Advance Without Hardware Progress?","date":"2025-05-07","arxiv_id":"2505.04075","n_code_links":1,"syntology":null},{"paper":"/paper/unidetox-universal-detoxification-of-large","slug":"unidetox-universal-detoxification-of-large","title":"UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation","date":"2025-04-29","arxiv_id":"2504.20500","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["EminLU/UniDetox"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"vexp-a-low-cost-risc-v-isa-extension-for","title":"VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers","date":"2025-04-15","arxiv_id":"2504.11227","n_code_links":0,"syntology":null},{"paper":null,"slug":"conceptformer-towards-efficient-use-of","title":"ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models","date":"2025-04-10","arxiv_id":"2504.07624","n_code_links":0,"syntology":null},{"paper":null,"slug":"2506-10005","title":"Multimodal Cinematic Video Synthesis Using Text-to-Image and Audio Generation Models","date":"2025-04-06","arxiv_id":"2506.10005","n_code_links":0,"syntology":null},{"paper":"/paper/localized-definitions-and-distributed","slug":"localized-definitions-and-distributed","title":"Localized Definitions and Distributed Reasoning: A Proof-of-Concept Mechanistic Interpretability Study via Activation Patching","date":"2025-04-03","arxiv_id":"2504.02976","n_code_links":1,"syntology":null},{"paper":null,"slug":"analysis-of-an-idealized-stochastic-polyak","title":"Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation","date":"2025-04-02","arxiv_id":"2504.01898","n_code_links":0,"syntology":null},{"paper":"/paper/towards-interpretable-soft-prompts","slug":"towards-interpretable-soft-prompts","title":"Towards Interpretable Soft Prompts","date":"2025-04-02","arxiv_id":"2504.02144","n_code_links":1,"syntology":null},{"paper":null,"slug":"accelerating-high-efficiency-organic","title":"Accelerating High-Efficiency Organic Photovoltaic Discovery via Pretrained Graph Neural Networks and Generative Reinforcement Learning","date":"2025-03-31","arxiv_id":"2503.23766","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-realization-of-tones-in-spontaneous","title":"The realization of tones in spontaneous spoken Taiwan Mandarin: a corpus-based survey and theory-driven computational modeling","date":"2025-03-29","arxiv_id":"2503.23163","n_code_links":0,"syntology":null},{"paper":null,"slug":"gpt-meets-graphs-and-kan-splines-testing","title":"GPT Meets Graphs and KAN Splines: Testing Novel Frameworks on Multitask Fine-Tuned GPT-2 with LoRA","date":"2025-03-25","arxiv_id":"2504.10490","n_code_links":0,"syntology":null},{"paper":null,"slug":"graplus-graph-based-placement-using-semantics","title":"GraPLUS: Graph-based Placement Using Semantics for Image Composition","date":"2025-03-20","arxiv_id":"2503.15761","n_code_links":0,"syntology":null},{"paper":"/paper/vlog-video-language-models-by-generative","slug":"vlog-video-language-models-by-generative","title":"VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary","date":"2025-03-12","arxiv_id":"2503.09402","n_code_links":1,"syntology":{"ran":7,"of":10,"n_ran_checked":5,"n_instrument":2,"unverified":3,"pointer_only":10,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified","official":{"repos":["showlab/vlog"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"interpretable-and-robust-dialogue-state","title":"Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs","date":"2025-03-11","arxiv_id":"2503.08857","n_code_links":0,"syntology":null},{"paper":"/paper/implicit-reasoning-in-transformers-is","slug":"implicit-reasoning-in-transformers-is","title":"Implicit Reasoning in Transformers is Reasoning through Shortcuts","date":"2025-03-10","arxiv_id":"2503.07604","n_code_links":1,"syntology":{"ran":7,"of":7,"n_ran_checked":6,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["TianheL/LM-Implicit-Reasoning"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"improving-cognitive-diagnostics-in-pathology","title":"Improving cognitive diagnostics in pathology: a deep learning approach for augmenting perceptional understanding of histopathology images","date":"2025-03-10","arxiv_id":"2503.06894","n_code_links":0,"syntology":null},{"paper":"/paper/the-box-is-in-the-pen-evaluating-commonsense-1","slug":"the-box-is-in-the-pen-evaluating-commonsense-1","title":"The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation","date":"2025-03-05","arxiv_id":"2503.03308","n_code_links":1,"syntology":null},{"paper":null,"slug":"effectively-steer-llm-to-follow-preference","title":"Effectively Steer LLM To Follow Preference via Building Confident Directions","date":"2025-03-04","arxiv_id":"2503.02989","n_code_links":0,"syntology":null},{"paper":null,"slug":"weak-to-strong-generalization-even-in-random","title":"Weak-to-Strong Generalization Even in Random Feature Networks, Provably","date":"2025-03-04","arxiv_id":"2503.02877","n_code_links":0,"syntology":null},{"paper":null,"slug":"attention-condensation-via-sparsity-induced","title":"Attention Condensation via Sparsity Induced Regularized Training","date":"2025-03-03","arxiv_id":"2503.01564","n_code_links":0,"syntology":null},{"paper":"/paper/codi-compressing-chain-of-thought-into","slug":"codi-compressing-chain-of-thought-into","title":"CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation","date":"2025-02-28","arxiv_id":"2502.21074","n_code_links":1,"syntology":{"ran":0,"of":2,"n_ran_checked":0,"n_instrument":0,"unverified":2,"pointer_only":2,"phrase":"0 ran · 2 unverified","official":null}},{"paper":"/paper/seismollm-advancing-seismic-monitoring-via","slug":"seismollm-advancing-seismic-monitoring-via","title":"SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Model","date":"2025-02-27","arxiv_id":"2502.19960","n_code_links":1,"syntology":{"ran":3,"of":6,"n_ran_checked":3,"n_instrument":0,"unverified":3,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","official":{"repos":["StarMoonWang/SeisMoLLM"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"the-sharpness-disparity-principle-in","title":"The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training","date":"2025-02-26","arxiv_id":"2502.19002","n_code_links":0,"syntology":null},{"paper":"/paper/a-close-look-at-decomposition-based-xai","slug":"a-close-look-at-decomposition-based-xai","title":"A Close Look at Decomposition-based XAI-Methods for Transformer Language Models","date":"2025-02-21","arxiv_id":"2502.15886","n_code_links":2,"syntology":null},{"paper":"/paper/bp-gpt-auditory-neural-decoding-using-fmri","slug":"bp-gpt-auditory-neural-decoding-using-fmri","title":"BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM","date":"2025-02-21","arxiv_id":"2502.15172","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":0,"n_instrument":3,"unverified":0,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","official":{"repos":["1994cxy/bp-gpt"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"entropy-uid-a-method-for-optimizing","title":"Entropy-UID: A Method for Optimizing Information Density","date":"2025-02-20","arxiv_id":"2502.14366","n_code_links":0,"syntology":null},{"paper":"/paper/pitvqa-vector-matrix-low-rank-adaptation-for","slug":"pitvqa-vector-matrix-low-rank-adaptation-for","title":"PitVQA++: Vector Matrix-Low-Rank Adaptation for Open-Ended Visual Question Answering in Pituitary Surgery","date":"2025-02-19","arxiv_id":"2502.14149","n_code_links":1,"syntology":null},{"paper":"/paper/adasplash-adaptive-sparse-flash-attention","slug":"adasplash-adaptive-sparse-flash-attention","title":"AdaSplash: Adaptive Sparse Flash Attention","date":"2025-02-17","arxiv_id":"2502.12082","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":0,"n_instrument":2,"unverified":1,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":{"repos":["deep-spin/adasplash"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"ai-generated-text-detection-with-a-gltr-based","title":"AI-generated Text Detection with a GLTR-based Approach","date":"2025-02-17","arxiv_id":"2502.12064","n_code_links":0,"syntology":null},{"paper":null,"slug":"mechanistic-unveiling-of-transformer-circuits","title":"Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning","date":"2025-02-13","arxiv_id":"2502.09022","n_code_links":0,"syntology":null},{"paper":"/paper/intar-inter-task-auto-reconfigurable","slug":"intar-inter-task-auto-reconfigurable","title":"InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs","date":"2025-02-12","arxiv_id":"2502.08807","n_code_links":1,"syntology":null},{"paper":null,"slug":"ynote-a-novel-music-notation-for-fine-tuning","title":"YNote: A Novel Music Notation for Fine-Tuning LLMs in Music Generation","date":"2025-02-12","arxiv_id":"2502.10467","n_code_links":0,"syntology":null},{"paper":null,"slug":"find-central-dogma-again","title":"Find Central Dogma Again: Leveraging Multilingual Transfer in Large Language Models","date":"2025-02-10","arxiv_id":"2502.06253","n_code_links":0,"syntology":null},{"paper":null,"slug":"emergence-of-episodic-memory-in-transformers","title":"Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training","date":"2025-02-09","arxiv_id":"2502.06902","n_code_links":0,"syntology":null},{"paper":null,"slug":"eap-gp-mitigating-saturation-effect-in","title":"EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification","date":"2025-02-07","arxiv_id":"2502.06852","n_code_links":0,"syntology":null},{"paper":null,"slug":"aligning-human-and-machine-attention-for","title":"Aligning Human and Machine Attention for Enhanced Supervised Learning","date":"2025-02-04","arxiv_id":"2502.06811","n_code_links":0,"syntology":null},{"paper":"/paper/harmonic-loss-trains-interpretable-ai-models","slug":"harmonic-loss-trains-interpretable-ai-models","title":"Harmonic Loss Trains Interpretable AI Models","date":"2025-02-03","arxiv_id":"2502.01628","n_code_links":1,"syntology":null},{"paper":"/paper/learnable-polynomial-trigonometric-and","slug":"learnable-polynomial-trigonometric-and","title":"Polynomial, trigonometric, and tropical activations","date":"2025-02-03","arxiv_id":"2502.01247","n_code_links":1,"syntology":null},{"paper":null,"slug":"libra-measuring-bias-of-large-language-model","title":"LIBRA: Measuring Bias of Large Language Model from a Local Context","date":"2025-02-02","arxiv_id":"2502.01679","n_code_links":0,"syntology":null},{"paper":null,"slug":"fast-solvers-for-discrete-diffusion-models","title":"Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms","date":"2025-02-01","arxiv_id":"2502.00234","n_code_links":0,"syntology":null},{"paper":null,"slug":"alphaadam-asynchronous-masked-optimization","title":"AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates","date":"2025-01-30","arxiv_id":"2501.18094","n_code_links":0,"syntology":null},{"paper":null,"slug":"structure-development-in-list-sorting","title":"Structure Development in List-Sorting Transformers","date":"2025-01-30","arxiv_id":"2501.18666","n_code_links":0,"syntology":null},{"paper":null,"slug":"weight-based-analysis-of-detokenization-in","title":"Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference","date":"2025-01-27","arxiv_id":"2501.15754","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-attempt-to-unraveling-token-prediction","title":"An Attempt to Unraveling Token Prediction Refinement and Identifying Essential Layers of Large Language Models","date":"2025-01-25","arxiv_id":"2501.15054","n_code_links":0,"syntology":null},{"paper":null,"slug":"focus-first-order-concentrated-updating","title":"FOCUS: First Order Concentrated Updating Scheme","date":"2025-01-21","arxiv_id":"2501.12243","n_code_links":0,"syntology":null},{"paper":null,"slug":"vision-language-models-for-automated-chest-x","title":"Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2","date":"2025-01-21","arxiv_id":"2501.12356","n_code_links":0,"syntology":null},{"paper":null,"slug":"fsmoe-a-flexible-and-scalable-training-system","title":"FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models","date":"2025-01-18","arxiv_id":"2501.10714","n_code_links":0,"syntology":null},{"paper":null,"slug":"investigating-energy-efficiency-and","title":"Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings","date":"2025-01-14","arxiv_id":"2501.08219","n_code_links":0,"syntology":null},{"paper":"/paper/finerweb-10bt-refining-web-data-with-llm","slug":"finerweb-10bt-refining-web-data-with-llm","title":"FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering","date":"2025-01-13","arxiv_id":"2501.07314","n_code_links":1,"syntology":null},{"paper":"/paper/decoding-fmri-data-into-captions-using-prefix","slug":"decoding-fmri-data-into-captions-using-prefix","title":"Decoding fMRI Data into Captions using Prefix Language Modeling","date":"2025-01-05","arxiv_id":"2501.02570","n_code_links":1,"syntology":null},{"paper":"/paper/crrg-clip-automatic-generation-of-chest","slug":"crrg-clip-automatic-generation-of-chest","title":"CRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs","date":"2024-12-31","arxiv_id":"2501.01989","n_code_links":1,"syntology":null},{"paper":null,"slug":"why-are-positional-encodings-nonessential-for","title":"Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph","date":"2024-12-31","arxiv_id":"2501.00659","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-adversarial-robustness-of-language-models","title":"On Adversarial Robustness of Language Models in Transfer Learning","date":"2024-12-29","arxiv_id":"2501.00066","n_code_links":0,"syntology":null},{"paper":null,"slug":"assessing-text-classification-methods-for","title":"Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms","date":"2024-12-27","arxiv_id":"2412.19928","n_code_links":0,"syntology":null},{"paper":null,"slug":"bridging-auditory-perception-and-language","title":"Bridging Auditory Perception and Language Comprehension through MEG-Driven Encoding Models","date":"2024-12-22","arxiv_id":"2501.03246","n_code_links":0,"syntology":null},{"paper":"/paper/psychadapter-adapting-llm-transformers-to","slug":"psychadapter-adapting-llm-transformers-to","title":"PsychAdapter: Adapting LLM Transformers to Reflect Traits, Personality and Mental Health","date":"2024-12-22","arxiv_id":"2412.16882","n_code_links":1,"syntology":null},{"paper":"/paper/no-more-adam-learning-rate-scaling-at","slug":"no-more-adam-learning-rate-scaling-at","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","date":"2024-12-16","arxiv_id":"2412.11768","n_code_links":1,"syntology":null},{"paper":null,"slug":"priority-aware-model-distributed-inference-at","title":"Priority-Aware Model-Distributed Inference at Edge Networks","date":"2024-12-16","arxiv_id":"2412.12371","n_code_links":0,"syntology":null},{"paper":null,"slug":"gpt-2-through-the-lens-of-vector-symbolic","title":"GPT-2 Through the Lens of Vector Symbolic Architectures","date":"2024-12-10","arxiv_id":"2412.07947","n_code_links":0,"syntology":null},{"paper":"/paper/superficial-consciousness-hypothesis-for","slug":"superficial-consciousness-hypothesis-for","title":"Superficial Consciousness Hypothesis for Autoregressive Transformers","date":"2024-12-10","arxiv_id":"2412.07278","n_code_links":1,"syntology":null},{"paper":null,"slug":"towards-predictive-communication-with-brain","title":"Towards Predictive Communication with Brain-Computer Interfaces integrating Large Language Models","date":"2024-12-10","arxiv_id":"2412.07355","n_code_links":0,"syntology":null},{"paper":"/paper/batchtopk-sparse-autoencoders","slug":"batchtopk-sparse-autoencoders","title":"BatchTopK Sparse Autoencoders","date":"2024-12-09","arxiv_id":"2412.06410","n_code_links":2,"syntology":{"ran":2,"of":3,"n_ran_checked":2,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["bartbussmann/batchtopk"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/dp-2stage-adapting-language-models-as","slug":"dp-2stage-adapting-language-models-as","title":"DP-2Stage: Adapting Language Models as Differentially Private Tabular Data Generators","date":"2024-12-03","arxiv_id":"2412.02467","n_code_links":1,"syntology":null},{"paper":null,"slug":"impact-of-data-snooping-on-deep-learning","title":"Impact of Data Snooping on Deep Learning Models for Locating Vulnerabilities in Lifted Code","date":"2024-12-03","arxiv_id":"2412.02048","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-asymptotic-behavior-of-attention-in","title":"The Asymptotic Behavior of Attention in Transformers","date":"2024-12-03","arxiv_id":"2412.02682","n_code_links":0,"syntology":null},{"paper":null,"slug":"su-roberta-a-semi-supervised-approach-to","title":"Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models","date":"2024-12-02","arxiv_id":"2412.01353","n_code_links":0,"syntology":null},{"paper":null,"slug":"tokenizing-3d-molecule-structure-with","title":"Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates","date":"2024-12-02","arxiv_id":"2412.01564","n_code_links":0,"syntology":null},{"paper":"/paper/streamlining-prediction-in-bayesian-deep","slug":"streamlining-prediction-in-bayesian-deep","title":"Streamlining Prediction in Bayesian Deep Learning","date":"2024-11-27","arxiv_id":"2411.18425","n_code_links":1,"syntology":{"ran":8,"of":10,"n_ran_checked":8,"n_instrument":0,"unverified":2,"pointer_only":0,"phrase":"8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","official":{"repos":["aaltoml/suq"],"state":"official (archive's flag): 8 ran","n_ran":8,"n_constructed":0,"n_ran_no_instrument_failure":8,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/clover-constrained-learning-with-orthonormal","slug":"clover-constrained-learning-with-orthonormal","title":"CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning","date":"2024-11-26","arxiv_id":"2411.17426","n_code_links":1,"syntology":null},{"paper":"/paper/distributed-sign-momentum-with-local-steps","slug":"distributed-sign-momentum-with-local-steps","title":"Distributed Sign Momentum with Local Steps for Training Transformers","date":"2024-11-26","arxiv_id":"2411.17866","n_code_links":1,"syntology":null},{"paper":"/paper/pretrained-llm-adapted-with-lora-as-a","slug":"pretrained-llm-adapted-with-lora-as-a","title":"Pretrained LLM Adapted with LoRA as a Decision Transformer for Offline RL in Quantitative Trading","date":"2024-11-26","arxiv_id":"2411.17900","n_code_links":1,"syntology":null},{"paper":null,"slug":"adaptive-circuit-behavior-and-generalization","title":"Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability","date":"2024-11-25","arxiv_id":"2411.16105","n_code_links":0,"syntology":null},{"paper":"/paper/development-of-pre-trained-transformer-based","slug":"development-of-pre-trained-transformer-based","title":"Development of Pre-Trained Transformer-based Models for the Nepali Language","date":"2024-11-24","arxiv_id":"2411.15734","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":768,"with_a_code_link":339,"where_syntology_ran_a_sample":125,"not_listed_spam_title":0,"listed":768,"listed_where_code_ran":125,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":98,"every_run_a_failure_of_syntologys_instrument":27,"listed_with_a_run_with_no_instrument_failure":98,"listed_every_run_a_failure_of_syntologys_instrument":27,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":125,"papers_where_code_ran_first_page":"/method/gpt-2/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":8,"url":"/method/gpt-2/papers/2","json":"/method/gpt-2/papers/2.json","rows":100},{"order":"archive","page":3,"of":8,"url":"/method/gpt-2/papers/3","json":"/method/gpt-2/papers/3.json","rows":100},{"order":"archive","page":4,"of":8,"url":"/method/gpt-2/papers/4","json":"/method/gpt-2/papers/4.json","rows":100},{"order":"archive","page":5,"of":8,"url":"/method/gpt-2/papers/5","json":"/method/gpt-2/papers/5.json","rows":100},{"order":"archive","page":6,"of":8,"url":"/method/gpt-2/papers/6","json":"/method/gpt-2/papers/6.json","rows":100},{"order":"archive","page":7,"of":8,"url":"/method/gpt-2/papers/7","json":"/method/gpt-2/papers/7.json","rows":100},{"order":"archive","page":8,"of":8,"url":"/method/gpt-2/papers/8","json":"/method/gpt-2/papers/8.json","rows":68},{"order":"ran","page":1,"of":2,"url":"/method/gpt-2/papers/ran/1","json":"/method/gpt-2/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":2,"url":"/method/gpt-2/papers/ran/2","json":"/method/gpt-2/papers/ran/2.json","rows":25}],"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":240},{"task":"/task/language-modeling","name":"Language Modeling","papers":183},{"task":"/task/text-generation","name":"Text Generation","papers":128},{"task":"/task/sentence","name":"Sentence","papers":49},{"task":"/task/question-answering","name":"Question Answering","papers":40},{"task":"/task/decoder","name":"Decoder","papers":39},{"task":"/task/large-language-model","name":"Large Language Model","papers":30},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":28},{"task":null,"name":"GPU","papers":26},{"task":"/task/retrieval","name":"Retrieval","papers":23},{"task":"/task/diversity","name":"Diversity","papers":20},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":20},{"task":"/task/in-context-learning","name":"In-Context Learning","papers":19},{"task":"/task/model","name":"model","papers":19},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":18},{"task":"/task/text-classification","name":"Text Classification","papers":17},{"task":"/task/translation","name":"Translation","papers":17},{"task":"/task/response-generation","name":"Response Generation","papers":16},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":16},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":15}],"tasks_shown":20,"n_tasks":398,"usage_by_year":[{"year":"2019","papers":31},{"year":"2020","papers":110},{"year":"2021","papers":119},{"year":"2022","papers":102},{"year":"2023","papers":138},{"year":"2024","papers":191},{"year":"2025","papers":77}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/gpt-2"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=GPT-2","read_at":"2026-10-05T00:57:09+00:00","query":"GPT-2","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":null,"repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'GPT-2' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["GPT","GPT-D","H-GPT","RF-GPT","MS-GPT","GPT-RE","VA-GPT","GPT-ST"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"663065230ce4b8fdb624af3b634ebb1e39a09b0730c3882cb39dc820be41cc83","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}