Browse State-of-the-Art › Lip Reading
Lip Reading
49 papers with code · 3 benchmarks · 6 datasets archive 2025-07-28
Lip Reading is a task to infer the speech content in a video by using only the visual information, especially the lip movements. It has many crucial applications in practice, such as assisting audio-based speech recognition, biometric authentication and aiding hearing-impaired people.
Source: Mutual Information Maximization for Effective Lip Reading
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
6 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 153 · with a code link: 49 · where Syntology ran a sample: 17 (14 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (17 of 153 tagged: 14 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 153, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
6 Sep 2018 4 repositories listedThe goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
-
12 Mar 2017 4 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 1 pointer-only (licence)We propose an end-to-end deep learning architecture for word-level visual speech recognition.
-
12 Feb 2021 3 repositories listedIn this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
-
9 Mar 2023 2 repositories listed Syntology community repositories only · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 3 pointer-only (licence)However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on…
-
5 Jan 2022 2 repositories listedThe lip-reading WER is further reduced to 26.
-
4 Dec 2020 2 repositories listedBiometric systems based on Machine learning and Deep learning are being extensively used as authentication mechanisms in resource-constrained environments like smartphones and other small computing devices.
-
23 Jan 2020 2 repositories listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
-
16 Oct 2018 2 repositories listedIt has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
-
4 May 2025 1 repository listedConducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and…
-
11 Mar 2025 1 repository listedAutomatic CS Recognition (ACSR) refers to the AI-driven process of automatically recognizing hand gestures and lip movements in CS, converting them into text.
-
2 Sep 2024 1 repository listedTo address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality.
-
9 May 2024 1 repository listedThe article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
-
18 Apr 2024 1 repository listedThen we design a spatio-temporal fusion module based on temporal granularity alignment, where the global spatial features extracted from event frames, together with the local relative spatial and temporal features…
-
23 Feb 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 12 harvested samples) · 12 pointer-only (licence)In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
-
11 Dec 2023 1 repository listedOur method, which we call NEUral Text to ARticulate Talk (NEUTART), is a talking face generator that uses a joint audiovisual feature space, as well as speech-informed 3D facial reconstructions and a lip-reading loss…
-
23 Nov 2023 1 repository listedThe Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years.
-
8 Oct 2023 1 repository listedFor deep layers where both the speaker's features and the speech content features are all expressed well, we introduce the speaker-adaptive features to learn for suppressing the speech content irrelevant noise for…
-
19 Jun 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)To enhance the visual accuracy of generated lip movement while reducing the dependence on labeled data, we propose a novel framework SelfTalk, by involving self-supervision in a cross-modals network system to learn 3D…
-
10 Jun 2023 1 repository listedWe demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and…
-
5 Jun 2023 1 repository listedCued Speech (CS) is a multi-modal visual coding system combining lip reading with several hand cues at the phonetic level to make the spoken language visible to the hearing impaired.
-
5 Jun 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples)We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier.
-
29 Mar 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results.
-
31 Jan 2023 1 repository listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 2 pointer-only (licence)Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality.
-
16 Jan 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed.
-
4 Jan 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks.
-
7 Nov 2022 1 repository listedDeepfake technology has advanced a lot, but it is a double-sided sword for the community.
-
20 Sep 2022 1 repository listedThe powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the…
-
3 Sep 2022 1 repository listedIn this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
-
9 Aug 2022 1 repository listedIn this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding.
-
4 Apr 2022 1 repository listedWith the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input…
-
4 Apr 2022 1 repository listed Syntology 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
24 Feb 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
29 Oct 2021 1 repository listed
-
14 Jun 2021 1 repository listed
-
7 Apr 2021 1 repository listed
-
15 Nov 2020 1 repository listed
-
29 Sep 2020 1 repository listed
-
2 Sep 2020 1 repository listed
-
17 May 2020 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 9 harvested samples) · 1 pointer-only (licence)
-
8 May 2020 1 repository listed
-
13 Mar 2020 1 repository listed
-
12 Mar 2020 1 repository listed
-
6 Mar 2020 1 repository listed
-
26 Nov 2019 1 repository listed
-
20 Jul 2018 1 repository listed Syntology 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 2 pointer-only (licence)
-
23 May 2018 1 repository listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
26 Oct 2017 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
2 Sep 2017 1 repository listed
-
3 Aug 2017 1 repository listed
-
VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis8 Jul 2025 0 repositories listed
-
SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer7 May 2025 0 repositories listed
-
Development and evaluation of a deep learning algorithm for German word recognition from lip movements22 Apr 2025 0 repositories listed
-
Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides21 Apr 2025 0 repositories listed
-
VALLR: Visual ASR Language Model for Lip Reading27 Mar 2025 0 repositories listed
-
Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction23 Jan 2025 0 repositories listed
-
GLaM-Sign: Greek Language Multimodal Lip Reading with Integrated Sign Language Accessibility9 Jan 2025 0 repositories listed
-
LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition8 Jan 2025 0 repositories listed
-
Spatio-temporal Transformers for Action Unit Classification with Event Cameras29 Oct 2024 0 repositories listed
-
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective29 Sep 2024 0 repositories listed
-
Neuromorphic Facial Analysis with Cross-Modal Supervision16 Sep 2024 0 repositories listed
-
RAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views9 Sep 2024 0 repositories listed
-
Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert1 Jul 2024 0 repositories listed
-
Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach2 Jun 2024 0 repositories listed
-
Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model30 Apr 2024 0 repositories listed
-
Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder8 Apr 2024 0 repositories listed
-
Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization24 Mar 2024 0 repositories listed
-
Cross-Attention Fusion of Visual and Geometric Features for Large Vocabulary Arabic Lipreading18 Feb 2024 0 repositories listed
-
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition31 Jan 2024 0 repositories listed
-
Exploring Lip Segmentation Techniques in Computer Vision: A Comparative Analysis20 Nov 2023 0 repositories listed
-
DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation8 Nov 2023 0 repositories listed
-
End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibition7 Oct 2023 0 repositories listed
-
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge18 Aug 2023 0 repositories listed
-
11 Aug 2023 0 repositories listed Syntology 17 ran (of which 0 constructed an object rather than computing a result; 16 with no instrument failure: 4 honoured, 1 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 19 harvested samples) · 19 pointer-only (licence)
-
Leveraging Visemes for Better Visual Speech Representation and Lip Reading19 Jul 2023 0 repositories listed
-
Emotional Speech-Driven Animation with Content-Emotion Disentanglement15 Jun 2023 0 repositories listed
-
Deep Learning-based Spatio Temporal Facial Feature Visual Speech Recognition30 Apr 2023 0 repositories listed
-
PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors11 Apr 2023 0 repositories listed
-
Word-level Persian Lipreading Dataset8 Apr 2023 0 repositories listed
-
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision30 Mar 2023 0 repositories listed
-
A large-scale multimodal dataset of human speech recognition15 Mar 2023 0 repositories listed
-
17 Feb 2023 0 repositories listed
-
A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset21 Jan 2023 0 repositories listed
-
Speech Driven Video Editing via an Audio-Conditioned Diffusion Model10 Jan 2023 0 repositories listed
-
Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss20 Nov 2022 0 repositories listed
-
Streaming Audio-Visual Speech Recognition with Alignment Regularization3 Nov 2022 0 repositories listed
-
Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings31 Oct 2022 0 repositories listed
-
A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids24 Oct 2022 0 repositories listed
-
Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation24 Oct 2022 0 repositories listed
-
VCSE: Time-Domain Visual-Contextual Speaker Extraction Network9 Oct 2022 0 repositories listed
-
29 Aug 2022 0 repositories listed
-
Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale21 Aug 2022 0 repositories listed
-
Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models5 Jun 2022 0 repositories listed
-
Learning Speaker-specific Lip-to-Speech Generation4 Jun 2022 0 repositories listed
-
RUSAVIC Corpus: Russian Audio-Visual Speech in Cars1 Jun 2022 0 repositories listed
-
Expression-preserving face frontalization improves visually assisted speech processing6 Apr 2022 0 repositories listed
-
27 Feb 2022 0 repositories listed
-
Multi-Grained Spatio-Temporal Features Perceived Network for Event-Based Lip-Reading1 Jan 2022 0 repositories listed
-
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading9 Dec 2021 0 repositories listed
-
8 Dec 2021 0 repositories listed
-
Contrastive Learning of Global and Local Video Representations1 Dec 2021 0 repositories listed
Syntology lines on 17 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections