{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/semantically-meaningful-metrics-for-norwegian","title":"Semantically Meaningful Metrics for Norwegian ASR Systems","arxiv_id":null,"date":"2022-09-03","proceeding":"Interspeech 2022 9","authors":["Janine Rugayan","Torbjørn Svendsen","Giampiero Salvi"],"abstract":"Evaluation metrics are important for quanitfying the performance of Automatic Speech Recognition (ASR) systems. However, the widely used word error rate (WER) captures errors at the word-level only and weighs each error equally, which makes it insufficient to discern ASR system performance for downstream tasks such as Natural Language Understanding (NLU) or information retrieval. We explore in this paper a more robust and discriminative evaluation metric for Norwegian ASR systems through the use of semantic information modeled by a transformer-based language model. We propose Aligned Semantic Distance (ASD) which employs dynamic programming to quantify the similarity between the reference and hypothesis text. First, embedding vectors are generated using the NorBERT model. Afterwards, the minimum global distance of the optimal alignment between these vectors is obtained and normalized by the sequence length of the reference embedding vector. In addition, we present results using Semantic Distance (SemDist), and compare them with ASD. Results show that for the same WER, ASD and SemDist values can vary significantly, thus, exemplifying that not all recognition errors can be considered equally important. We investigate the resulting data, and present examples which demonstrate the nuances of both metrics in evaluating various transcription errors.","url_abs":"https://www.isca-archive.org/interspeech_2022/rugayan22_interspeech.html","url_pdf":"https://www.isca-archive.org/interspeech_2022/rugayan22_interspeech.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semantically-meaningful-metrics-for-norwegian","repo_url":"https://github.com/janinerugayan/aligned-semantic-distance","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"janinerugayan/aligned-semantic-distance","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"6b28c262f03e9917d74b04894aec6a6aef5908fc8b308adc92b5d56fad3ebcb7","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}