Papers › HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

20 Jun 2024arXiv:2406.14341archive 2025-07-28

Ivan Karpukhin, Foma Shipilov, Andrey Savchenko

Accurately forecasting multiple future events within a given time horizon is crucial for finance, retail, social networks, and healthcare applications. Event timing and labels are typically modeled using Marked Temporal Point Processes (MTPP), with evaluations often focused on next-event prediction quality. While some studies have extended evaluations to a fixed number of future events, we demonstrate that this approach leads to inaccuracies in handling false positives and false negatives. To address these issues, we propose a novel evaluation method inspired by object detection techniques from computer vision. Specifically, we introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP, which overcomes the limitations of existing long-horizon evaluation metrics. Our extensive experiments demonstrate that models with strong next-event prediction accuracy can yield poor long-horizon forecasts and vice versa, indicating that specialized methods are needed for each task. To support further research, we release HoTPP, the first benchmark designed explicitly for evaluating long-horizon MTPP predictions. HoTPP includes large-scale datasets with up to 43 million events and provides optimized procedures for both autoregressive and parallel inference, paving the way for future advancements in the field.

PaperPDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

ivan-chai/hotpp-benchmark officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences (arXiv:2210.01753; that paper's own run record: 18 ran (of which 6 constructed an object rather than computing a result; 17 with no instrument failure: 0 honoured, 0 violated, 17 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
ivan-chai/torch-linear-assignment mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

BenchmarkingPoint ProcessesTime Series Forecasting

Datasets

Introduced by this paper, per the archive.

AgeGroup Transactions MTPP

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Point Processes AgeGroup Transactions MTPP RMTPP Accuracy (%) 34.15 #2 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP RMTPP MAE 0.749 #2 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP RMTPP OTD 6.88 #2 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP RMTPP T-mAP 6.69 #2 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP IFTPP Accuracy (%) 34.08 #3 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP IFTPP MAE 0.693 #3 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP IFTPP OTD 6.90 #3 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP IFTPP T-mAP 5.88 #3 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP NHP Accuracy (%) 35.43 #4 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP NHP MAE 0.696 #4 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP NHP OTD 6.97 #4 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP NHP T-mAP 5.61 #4 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP ODE-RNN Accuracy (%) 35.6 #5 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP ODE-RNN MAE 0.695 #5 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP ODE-RNN OTD 6.97 #5 of 5 Archive leaderboard report
Point Processes AgeGroup Transactions MTPP ODE-RNN T-mAP 5.52 #5 of 5 Archive leaderboard report
Point Processes Amazon MTPP NHP Accuracy (%) 11.06 #2 of 4 Archive leaderboard report
Point Processes Amazon MTPP NHP MAE 0.449 #2 of 4 Archive leaderboard report
Point Processes Amazon MTPP NHP OTD 9.02 #2 of 4 Archive leaderboard report
Point Processes Amazon MTPP NHP T-mAP 26.29 #2 of 4 Archive leaderboard report
Point Processes Amazon MTPP IFTPP Accuracy (%) 35.73 #3 of 4 Archive leaderboard report
Point Processes Amazon MTPP IFTPP MAE 0.242 #3 of 4 Archive leaderboard report
Point Processes Amazon MTPP IFTPP OTD 6.52 #3 of 4 Archive leaderboard report
Point Processes Amazon MTPP IFTPP T-mAP 22.56 #3 of 4 Archive leaderboard report
Point Processes Amazon MTPP RMTPP Accuracy (%) 35.76 #4 of 4 Archive leaderboard report
Point Processes Amazon MTPP RMTPP MAE 0.294 #4 of 4 Archive leaderboard report
Point Processes Amazon MTPP RMTPP OTD 6.57 #4 of 4 Archive leaderboard report
Point Processes Amazon MTPP RMTPP T-mAP 20.06 #4 of 4 Archive leaderboard report
Point Processes Retweet MTPP ODE-RNN Accuracy (%) 59.95 #2 of 6 Archive leaderboard report
Point Processes Retweet MTPP ODE-RNN MAE 18.38 #2 of 6 Archive leaderboard report
Point Processes Retweet MTPP ODE-RNN OTD 165.3 #2 of 6 Archive leaderboard report
Point Processes Retweet MTPP ODE-RNN T-mAP 48.81 #2 of 6 Archive leaderboard report
Point Processes Retweet MTPP NHP Accuracy (%) 60.08 #3 of 6 Archive leaderboard report
Point Processes Retweet MTPP NHP MAE 18.42 #3 of 6 Archive leaderboard report
Point Processes Retweet MTPP NHP OTD 165.8 #3 of 6 Archive leaderboard report
Point Processes Retweet MTPP NHP T-mAP 45.07 #3 of 6 Archive leaderboard report
Point Processes Retweet MTPP RMTPP Accuracy (%) 60.07 #4 of 6 Archive leaderboard report
Point Processes Retweet MTPP RMTPP MAE 18.45 #4 of 6 Archive leaderboard report
Point Processes Retweet MTPP RMTPP OTD 166.7 #4 of 6 Archive leaderboard report
Point Processes Retweet MTPP RMTPP T-mAP 44.74 #4 of 6 Archive leaderboard report
Point Processes Retweet MTPP IFTPP Accuracy (%) 59.95 #5 of 6 Archive leaderboard report
Point Processes Retweet MTPP IFTPP MAE 18.27 #5 of 6 Archive leaderboard report
Point Processes Retweet MTPP IFTPP OTD 172.7 #5 of 6 Archive leaderboard report
Point Processes Retweet MTPP IFTPP T-mAP 31.75 #5 of 6 Archive leaderboard report
Point Processes Retweet MTPP AttNHP Accuracy (%) 60.03 #6 of 6 Archive leaderboard report
Point Processes Retweet MTPP AttNHP MAE 18.39 #6 of 6 Archive leaderboard report
Point Processes Retweet MTPP AttNHP OTD 171.6 #6 of 6 Archive leaderboard report
Point Processes Retweet MTPP AttNHP T-mAP 25.85 #6 of 6 Archive leaderboard report
Point Processes StackOverflow MTPP RMTPP Accuracy (%) 45.43 #2 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP RMTPP MAE 0.701 #2 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP RMTPP OTD 13.17 #2 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP RMTPP T-mAP 12.72 #2 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP IFTPP Accuracy (%) 45.41 #3 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP IFTPP MAE 0.641 #3 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP IFTPP OTD 13.64 #3 of 3 Archive leaderboard report
Point Processes StackOverflow MTPP IFTPP T-mAP 8.31 #3 of 3 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections