Submission Format — Object Detection

This page describes how to package a submission for this benchmark. The dataset, the label format and the evaluation code are documented in the View of Delft development kit and are not repeated here — only what is specific to this server is.

1. The archive

One zip, containing one .txt file per test frame, flat at the root of the archive. There must be no containing folder and nothing else in the archive:

submission.zip
├── 02532.txt
├── 02533.txt
├── 02534.txt
├── ...
└── 09517.txt

Zipping the folder instead of its contents is the most common mistake, and the evaluator cannot recover from it. Build the archive from inside the directory:

cd my_predictions/
zip ../submission.zip *.txt

Every .txt in the archive is read as a frame, so do not add a README, a config, or anything else. If you zip on macOS, check with unzip -l submission.zip that no __MACOSX/ or ._ entries were added, and strip them with zip -d submission.zip '__MACOSX/*' if they were.

Include a file for every frame id listed in lidar/ImageSets/test.txt — all 2247 of them. The file name is the frame id plus .txt, keeping the five digits and the leading zeros. If your detector finds nothing in a frame, include an empty file for it rather than leaving it out. Confirm the count before uploading:

unzip -l submission.zip | tail -1     # must report 2247 files

The frame set is checked before evaluation starts. An archive that does not hold exactly the 2247 test frames is rejected and does not reach the leaderboard; the message under My Submissions names what is missing or extra.

2. File contents

Each file holds one detection per line, in the same format as a View of Delft label file — see docs/ANNOTATION.md in the devkit for the fields, their order and their units. Fields are separated by single spaces.

A submission deviates from a label file in three ways:

One extra column at the end: your confidence score. It is required. A file without it is still accepted, but every detection then scores 0, which collapses the average precision for the whole submission.
Car, Pedestrian and Cyclist are the classes that are scored. Lines with any other class are read but do not contribute to the reported metrics.
The truncated and occluded fields are not used for detection scoring. Write -1 for both.

The example archive below contains concrete files in exactly this format.

3. Example submission

example_submission.zip is a correctly packaged submission: 2247 files, flat at the archive root, one per test frame.

The boxes in it are randomly generated placeholders. They are not predictions and they are not annotations — no ground truth is contained in or derivable from this file. Uploading it as-is is a way to confirm that your packaging is accepted: it evaluates without errors and scores approximately zero. If your own archive is built the same way but with your real detections, it will evaluate too.

4. Check your detector before submitting

You get 12 submissions per month. The devkit ships the same evaluation code this server runs, and the dataset includes the validation labels, so you can reproduce the metrics locally first. See 4_evaluation.ipynb for a worked example and docs/GETTING_STARTED.md for installing the devkit. The metric names it reports are the ones shown on the leaderboard.

5. If the evaluation fails

The evaluator's output is shown per submission under My Submissions. The usual causes:

What you see	Cause
`The .txt files must be at the root of the zip, not inside a folder`	Your `.txt` files are inside a folder rather than at the root of the archive. See section 1. macOS `__MACOSX` entries produce this too.
`The submitted files do not match the test split`	Frames are missing, or the archive contains a `.txt` whose name is not a test frame id — a README, a config, or predictions for train/val frames. The message lists examples of both. See section 1.
`IndexError: list index out of range` or `could not convert string to float`	A line does not parse — commonly comma-separated instead of space-separated values, or a non-label file that got zipped in.
Evaluation succeeds but every metric is 0	The score column is missing, so all detections rank equally. See section 2.

6. Submit

When your local numbers look right, upload your zip on the submission page.