Submission Format — Trajectory Prediction
The prediction task, the coordinate frame, the trajectory shape and the metrics are documented in the View of Delft prediction development kit — read docs/benchmark_instructions.md first, and tutorials/prediction_tutorial.ipynb for a worked example. This page covers only what those do not: how the individual predictions are packed into one file, which tokens to submit, and what this server does with them.
The devkit README points at an eval.ai leaderboard. Submissions go here instead.
1. One file, one JSON list
The devkit describes a single prediction object — instance,
sample, prediction, probabilities. A submission to this
server is all of them in one .json file, as a JSON list. Not a zip,
not one file per agent, and not an object keyed by token: the file begins with [.
[
{
"instance": "uii9qtdsknv4zpy1f9cbttn1ruw1d1fy",
"sample": "xjgtzfqg2602w4xl9yubvpgh9kpf65by",
"prediction": [[[593142.69, 5763561.22], ... 30 points ...], ... up to 25 modes ...],
"probabilities": [0.31, 0.22, ...]
},
... 454 entries ...
]
If you built your predictions with the devkit's Prediction class, that is
json.dump([p.serialize() for p in predictions], f). Every coordinate and every
probability must be a finite number — NaN, an infinity and an overflowing
decimal such as 1e400 are rejected.
2. Which tokens to submit
Exactly one entry for every test token, and each one only once — all 454 of them. A missing token, a repeated token, or a token that is not in the test split is rejected before evaluation starts, and the message names which.
get_prediction_challenge_split() cannot give you this list. It
raises ValueError for "test"; it only accepts
mini_train, mini_val, train, train_val and
val. Read the token list out of the released archive instead:
import json
with open('vod_official/v1.0-test/prediction_scenes.json') as f:
scenes = json.load(f)
tokens = [t for scene_tokens in scenes.values() for t in scene_tokens]
print(len(tokens)) # 454
instance, sample = tokens[0].split('_') # the two fields of an entry
3. Reading your leaderboard row
Each metric is reported for Aggregate — all 454 tokens — and separately
for Vehicle, Cyclist and Pedestrian. You do not label the
class of an entry; it is taken from the annotation of the token.
If you submit fewer than 10 modes, the higher columns repeat the lower ones. A
3‑mode submission reports its best‑of‑3 in the _5 and
_10 columns too, so those columns look better than a genuine 5‑ or
10‑mode result would. Fewer modes is a legitimate submission and is accepted; just read
the columns accordingly when comparing rows.
Probabilities are used only to rank your modes for the top‑k metrics. They are not scored and need not sum to 1.
4. Example submission
example_prediction.json is a correctly built submission: 454 entries, 10 modes of 30 points each.
The trajectories in it are randomly generated placeholders scattered around a single fixed point. They are not predictions and they are not annotations — no ground truth is contained in or derivable from this file. Uploading it as-is is a way to confirm that your file is accepted: it evaluates without errors and scores a displacement error of several hundred metres. If your own file is built the same way but with your real predictions, it will evaluate too.
5. Before you submit
You get 4 submissions per month. The devkit ships the same evaluation code
this server runs and the released archive includes the validation split, so reproduce your
metrics locally first — get_prediction_challenge_split("val") does work. The
metric names it reports are the ones shown on the
leaderboard.
6. If the submission is rejected
Anything that can be checked without the ground truth is rejected on the submission page itself, naming the entry and the token. Everything else is reported per submission under My Submissions. The usual causes:
| What you see | Cause |
|---|---|
The submission must be a JSON list with one entry per test token |
The file is a JSON object rather than a list. See section 1. |
This is a zip archive; upload the .json file itself |
You zipped the file. Upload the .json as it is. |
The submission must hold exactly one entry for each of the 454 test
tokens |
Tokens are missing, repeated, or are not test tokens — commonly a val split submitted by mistake. The message lists examples of each. See section 2. |
every mode must hold 30 points, one per timestep |
Your horizon is not the 3 seconds at 10 Hz the task asks for, or you wrote one point per mode instead of a trajectory. |
every point must be [x, y], two finite numbers |
Points carry a third coordinate, or are bare numbers rather than pairs. |
at most 25 are accepted or
one probability per mode is required |
Too many modes, or probabilities is not the same length as
prediction. |
| It evaluates, but the displacement errors are in the hundreds of metres | Almost certainly the wrong coordinate frame. Predictions are in the global frame, so
an x is around 593000 and a y around
5763500; an agent-local or ego-relative trajectory is shaped correctly,
passes every check, and simply scores badly. Check that your first predicted point
is close to the agent's current position. |
7. Submit
When your local numbers look right, upload your file on the submission page.