Submission Format — Trajectory Prediction

The prediction task, the coordinate frame, the trajectory shape and the metrics are documented in the View of Delft prediction development kit — read docs/benchmark_instructions.md first, and tutorials/prediction_tutorial.ipynb for a worked example. This page covers only what those do not: how the individual predictions are packed into one file, which tokens to submit, and what this server does with them.

The devkit README points at an eval.ai leaderboard. Submissions go here instead.

1. One file, one JSON list

The devkit describes a single prediction object — instance, sample, prediction, probabilities. A submission to this server is all of them in one .json file, as a JSON list. Not a zip, not one file per agent, and not an object keyed by token: the file begins with [.

[
  {
    "instance": "uii9qtdsknv4zpy1f9cbttn1ruw1d1fy",
    "sample": "xjgtzfqg2602w4xl9yubvpgh9kpf65by",
    "prediction": [[[593142.69, 5763561.22], ... 30 points ...], ... up to 25 modes ...],
    "probabilities": [0.31, 0.22, ...]
  },
  ... 454 entries ...
]

If you built your predictions with the devkit's Prediction class, that is json.dump([p.serialize() for p in predictions], f). Every coordinate and every probability must be a finite number — NaN, an infinity and an overflowing decimal such as 1e400 are rejected.

2. Which tokens to submit

Exactly one entry for every test token, and each one only once — all 454 of them. A missing token, a repeated token, or a token that is not in the test split is rejected before evaluation starts, and the message names which.

get_prediction_challenge_split() cannot give you this list. It raises ValueError for "test"; it only accepts mini_train, mini_val, train, train_val and val. Read the token list out of the released archive instead:

import json

with open('vod_official/v1.0-test/prediction_scenes.json') as f:
    scenes = json.load(f)

tokens = [t for scene_tokens in scenes.values() for t in scene_tokens]
print(len(tokens))                       # 454
instance, sample = tokens[0].split('_')  # the two fields of an entry

3. Reading your leaderboard row

Each metric is reported for Aggregate — all 454 tokens — and separately for Vehicle, Cyclist and Pedestrian. You do not label the class of an entry; it is taken from the annotation of the token.

If you submit fewer than 10 modes, the higher columns repeat the lower ones. A 3‑mode submission reports its best‑of‑3 in the _5 and _10 columns too, so those columns look better than a genuine 5‑ or 10‑mode result would. Fewer modes is a legitimate submission and is accepted; just read the columns accordingly when comparing rows.

Probabilities are used only to rank your modes for the top‑k metrics. They are not scored and need not sum to 1.

4. Example submission

example_prediction.json is a correctly built submission: 454 entries, 10 modes of 30 points each.

The trajectories in it are randomly generated placeholders scattered around a single fixed point. They are not predictions and they are not annotations — no ground truth is contained in or derivable from this file. Uploading it as-is is a way to confirm that your file is accepted: it evaluates without errors and scores a displacement error of several hundred metres. If your own file is built the same way but with your real predictions, it will evaluate too.

5. Before you submit

You get 4 submissions per month. The devkit ships the same evaluation code this server runs and the released archive includes the validation split, so reproduce your metrics locally first — get_prediction_challenge_split("val") does work. The metric names it reports are the ones shown on the leaderboard.

6. If the submission is rejected

Anything that can be checked without the ground truth is rejected on the submission page itself, naming the entry and the token. Everything else is reported per submission under My Submissions. The usual causes:

What you seeCause
The submission must be a JSON list with one entry per test token The file is a JSON object rather than a list. See section 1.
This is a zip archive; upload the .json file itself You zipped the file. Upload the .json as it is.
The submission must hold exactly one entry for each of the 454 test tokens Tokens are missing, repeated, or are not test tokens — commonly a val split submitted by mistake. The message lists examples of each. See section 2.
every mode must hold 30 points, one per timestep Your horizon is not the 3 seconds at 10 Hz the task asks for, or you wrote one point per mode instead of a trajectory.
every point must be [x, y], two finite numbers Points carry a third coordinate, or are bare numbers rather than pairs.
at most 25 are accepted or one probability per mode is required Too many modes, or probabilities is not the same length as prediction.
It evaluates, but the displacement errors are in the hundreds of metres Almost certainly the wrong coordinate frame. Predictions are in the global frame, so an x is around 593000 and a y around 5763500; an agent-local or ego-relative trajectory is shaped correctly, passes every check, and simply scores badly. Check that your first predicted point is close to the agent's current position.

7. Submit

When your local numbers look right, upload your file on the submission page.