Papers › Benchmarking Natural Language Understanding Services for building Conversational Agents

Benchmarking Natural Language Understanding Services for building Conversational Agents

13 Mar 2019arXiv:1903.05566archive 2025-07-28

Xingkun Liu, Arash Eshghi, Pawel Swietojanski, Verena Rieser

We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

xliuhw/NLU-Evaluation-Data officialmentioned in papermentioned on GitHubCC-BY-4.0Syntology: not harvested report
Lackel/DNA mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery (arXiv:2310.10151; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result); nothing checks that they implement this paper's method. report
PolyAI-LDN/polyai-models mentioned on GitHubtfSyntology: not harvested report
PolyAI-LDN/task-specific-datasets mentioned on GitHubCC-BY-4.0Syntology: not harvested report
amazon-science/intent-aware-encoder mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Efficient Intent Detection with Dual Sentence Encoders (arXiv:2003.04807; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
haodeqi/BenchmarkingIntentDetection mentioned on GitHubSyntology: not harvested report
lackel/hierarchical_weighted_scl mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical... (arXiv:2210.07733; that paper's own run record: 0 ran · 1 unverified); nothing checks that they implement this paper's method. report
lackel/sdc mentioned on GitHubpytorchSyntology: not harvested report
RasaHQ/rasa Apache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational... (arXiv:1910.00912; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

BenchmarkingGeneral ClassificationIntent ClassificationNatural Language Understandingintent-classification

Datasets

Introduced by this paper, per the archive.

HWU64

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections