Quick Start to the Tasks

There is a WeChat blog post from the community (many thanks for the sharing!) on how they tackle our tasks.

• WeChat article (original in Chinese)
• Google translated link

Please feel free to have a look to get some inspirations :)

How to Submit Your Algorithm Docker?

Input/Output Interface is All We Need

For example, the "Dummy" algorithms submitted to the validation phases by the organizers are built and exported from the "TopCoW_Algo_Submission" repo. All we have changed are the track and task in the process.py file.

How do We Evaluate Your Predictions?

FAQ

1. How do I match the MRA CTA pair during test phase?

The grand-challenge algorithm environment is designed such that only one set of input is provided to the algorithm container. So in our challenge, that one set of input is a pair of CT and MR from a patient, and we have taken care of the matching for you.

And please refer to the "Input/Output Interface is All We Need" part of our GitHub repo 👉 TopCoW_Algo_Submission 🦾.

2. Is the region of interest (ROI) provided during test phase as well?

The ROIs of the test set are NOT available to the participants, but are used by our evaluation to calculate the metrics.

The segmentation results will be evaluated only for the CoW region of interest (ROI). (Please refer to the annotation page for more info.) We will not assess the segmentation performance on the peripheral and further downstream vessels outside the CoW ROI. Participants should focus on segmenting the CoW vessel components necessary to characterize the CoW angio-architecture.

Sidenote: Validation set released will have the whole image and the accompanying ROIs.

3. Is the input during test phase the whole image or the ROI view?

The input available for your algorithm during test phase is the pair of whole images.

For each test-case, we present to your algorithm whole brain image of both CT and MR modalities from that patient, i.e. the CTA-MRA pair, irrespective of whether your algorithm uses both modalities or not. In case you only need one of them, simply ignore the other modality input.

4. How do you weight the metrics and how will the final ranking be done?

For the MICCAI in-person event and its results annoucement, we will simply use an equal weighting of Dice, clDice and Betti number errors. Please visit our GitHub repo 👉 TopCoW_Eval_Metrics 📐 for our evaluation metrics for the tasks. (We will broadcast in the forum and webpages and github readme accordingly if we update any metrics.)

The ranking to be annouced for the event and awards will thus be based on the leaderboard displayed on grand-challenge. The leaderboard uses equal weights for each column (for example when you see "mean position", it is the mean of several columns' positions/ranks), and 'rank then average'.

We will summarize the results and do post-challenge analysis. Additional metrics and more advanced ranking analysis will be introduced.

5. Can I take part in just one track and one task?

Yes, definitely. You are welcome to submit to any track or task of your preference. There are 4 rankings in the end.

Submissions to multi-class segmentation tasks are automatically evaluated by us for binary segmentation performance. (For example, in the metrics output from our evaluation on multi-class task, you can see the results for merged-binary segmentation. And we will insert them manually to the binary-task's leaderboard afterwards.)

6. Submission Confidentiality and Availability

According to grand-challenge, "Challenge organisers do not have access to the algorithms. Challenge organisers only have access to the algorithms logs and predictions for the cases in the challenges test/training archive. Challenge organisers cannot use the algorithm on any other cases. Algorithm owners can see who has access to use their algorithm in the Users tab on their algorithm page. No one is ever given access to the algorithms container image."

Having said that, please contact us with your team's information and contact details, so we can reach out to you for the MICCAI event and follow-up publications. (We organizers cannot see your profile's email due to GDPR.)