Basic Information

  1st place: $1500 accuracy divided by speed

  2nd place: $1,000 most accurate, speed within the top five

  3rd place: $1,000 fastest, accuracy within the top five

Note: A submitted solution must have BOTH  better accuracy and speed than the sample solution (Team ‘sample-solution’ in the leaderboard) to be considered for an award.

_____________________________________________________________________________________________________________________________

Overview

Disasters like floods and earthquakes threaten human safety, infrastructure, and natural systems. Every year, disasters kill an average of 60,000 people, affect 200 million and cause $150 billion (USD) billion in damage. A timely and accurate understanding of the environment plays a key role in disaster preparedness, assessment and response. Recently, unmanned aerial vehicles (UAV) with inexpensive sensors have emerged as a practical tool to collect situational imagery from disaster areas that can be hard-to-reach for humans. However, UAVs are equipped with energy-constrained supplies and low-compute devices, which limit the ability to perform automatic on-device analysis. This adds to on-board system latency, resulting in longer response times for disaster relief efforts.. Therefore, achieving effective on-device computer vision with low power consumption and low latency remains a significant challenge.

To promote the community’s interest and progress toward an efficient and effective understanding of disaster scenes on UAV-based edge devices, we propose the On-device Disaster Scene Parsing Competition. Participants will devise models to improve semantic segmentation on an edge device (NVIDIA Jetson Nano 2GB Developer Kit) with a new disaster-scene dataset containing 1,700 samples collected by UAVs. The submitted models will be automatically benchmarked by evaluating their accuracy and execution time. We hope our competition clarifies the challenges of UAV-view scene understanding with edge computing and spurs innovations for practical applications.

_____________________________________________________________________________________________________________________________

Winners

  1. ModelTC - Score: 75.608, Accuracy: 0.51162, Time: 6.8ms. The team members are Yiru Wang, Xin Jin, Zhiwei Dong, Yifan Pu, Yongqiang Yao, Bo Li, Ruihao Gong, Haoran Wang, Xianglong Liu, Gao Huang, and Wei Wu. The first four made equal contributions, and Ruihao Gong was the project leader.
  2. AidgetRock - Score: 36.972, Accuracy: 0.55421, Time: 15ms. The team members are Zifu Wan, Xinwang Chen, Ning Liu, Ziyi Zhang, Dongping Liu, Ruijie Shan, Zhengping Che, Fachao Zhang, Xiaofeng Mou, and Jian Tang. The project team leader was Ning Liu.
  3. ENOT - Score: 8.974, Accuracy: 0.60089, Time: 67ms. The team members are Alexander Goncharenko, Max Chuprov, Andrey Sherbin, Sergey Alyamkin, and Ivan Malofeev.

_____________________________________________________________________________________________________________________________

Task and Dataset

The on-device disaster scene parsing task is to segment and parse images of UAV-view disaster scenes (a) into different regions associated with semantic categories related to disasters (b).

We have collected a novel UAV-view disaster scene dataset consisting of 1,720 images collected from various disaster scenes. We will release 1,120 of the images as well as their ground-truth labels for training/validation, while reserving the rest for benchmarking submissions on our server. Images in the training test and the testing set are resized to 512X512 resolution, and densely annotated with these 14 categories:

0. background

5. debris/mud//rock flow

10. person

1. avalanche

6. fire/flare

11. pyroclastic_flow

2. building_undamaged

7. flood/water/river/sea

12. road/railway/bridge

3. building_damaged

8. ice_jam_flow

13. vehicle

4. cracks/fissure/subsidence

9. lava_flow

The training and validation data can be downloaded from here.

_____________________________________________________________________________________________________________________________

Hardware

For this competition, we will be evaluating models on an NVIDIA Jetson Nano 2GB, which is a low-cost option in the NVIDIA Jetson product line. We think this device has the right balance of low power, affordability, and robust support for Linux and the ML stack in a remote execution setting.
Information about the NVIDIA Jetson Nano 2GB Developer Kit is available here.

In the scenario that you are unable to purchase the above device (say, the device is out of stock),  we recommend that you create your solution on a Linux system with the same dependencies as the evaluation system.

____________________________________________________________________________________________________________________________

Evaluation

The evaluation of the models is done as follows:

Overall ranking: The leaderboard is ranked based on the score, which is calculated as a function of the model’s accuracy and execution time. Detailed description as follows:

Accuracy: We use the average of Dice Coefficient (as defined here) calculated over all the test images to evaluate the accuracy of your model. Lager value of Accuracy (represented as “Accuracy” in the leaderboard ) is desirable.

Execution Time : Execution time is calculated as the average of inference time of your model over all the test images. Smaller value of Execution Time (represented as “Time” in the leaderboard) is desirable.

Overview of the Evaluation Score:

Let n be the total number of test images on which your model is evaluated.

Let TP, FP, and FN be the numbers of true positive, false positive, and false negative pixels, respectively, as predicted by your model. The final score is calculated as a function of the accuracy of the model and execution time.

The equations for the evaluation are given below:

The Evaluation Script was released with the sample solution. This  includes the dice calculations performed. The evaluation script may give better insight to model evaluation.

_____________________________________________________________________________________________________________________________

Frequently Asked Questions (FAQ)

  1. Q: Will the background class (id: 0) be counted in the accuracy evaluation?

A: Yes. We will compute accuracy over all 14 categories.

  1. Q: Where are images collected from?

A: The dataset is collected from public sources on the Internet.

  1. Q: Can I use external data to train the model?

A:  Participants can use any data to train their model. We do not have any restrictions on how the participants build their solutions. The only restriction is that they must open-source their solutions and declare all the datasets used.

  1. Q: Will all testing images be used for grading?

A: Not necessarily. It is possible that some images are ignored. A set of 600 images will be used for selecting winners. All teams will be evaluated using the same sets of images and ground-truth masks.

  1. Q: Will you provide data for training and validation?

A: Yes. A set of 1120 images will be released for model training and validation.

  1. Q: How are the winners selected?

A: The submitted models will be individually evaluated and ranked for accuracy, and latency. The winners will be selected based on their two rankings.

  1. Q: Can my codes communicate with a cloud during evaluation?

A: No. The evaluation will be terminated if finding the model try to send/receive data outside the server.

  1. Q:  Can I log into the evaluation server and install my software?

A: No. Participants are only allowed to submit their codes and models. Participants should specificize the software dependency in the environment setup phase.

  1. Q: What happens after running one solution?

A: The test environment is reset so that all solutions start from the same initial state.

  1. Q: Will the computation be performed while a UAV is flying?

A: No. The data has been prepared in advance and will be given as a file. This is necessary to ensure fairness: All solutions are evaluated using the same data. Also, this allows the referee system to run 24 hours to accommodate many contestants.

  1. Q: Do I have to use the Jetson Nano 2GB Developer Kit?

A: Yes. This is an online competition, and it is necessary to have the same hardware to compare the solutions fairly.  See the Hardware section above for our recommendation.

  1. Q: What software framework will be used?

A: PyTorch. This is an online competition, and it is necessary to have the same software framework.

  1. Q: What happens if my program takes shorter?

        A: When your program finishes, the evaluation server will send the results to the leaderboard.

  1. Q: What happens if my program takes longer than the allowed time?

A: If your program takes too long, your solution is disqualified. The referee system has to evaluate many solutions. Thus, the referee system must not allow any single solution to take too long.

  1. Q: How will the limit of execution time be measured?

A: The execution time will be measured by the referee system. It will not be measured by a human pressing a stopwatch.

  1. Q: What’s the resolution for evaluation?

A: 512×512 for both input and output.

  1. Q: Will the testing data for selecting winners be hidden?

A: Yes. The testing data will be different from the training data and will not be publicly available.

  1. Q: Will the challenge provide a leaderboard?

A:  Yes.

  1. Q: Will you provide a sample solution?

A: Yes it can be found here

  1. Q: Who can participate?

A: Anyone in the world can participate as long as this person is not on the restricted list of Embargoed and Sanctioned Countries by the US government. The restrictions are needed because the organizers reside in the US.

  1. Q: Is the competition restricted to students?

A:  The competition is open to anyone (again, with the restrictions by the relevant laws).

  1. Q: Are there any rules about conflict of interest?

A: The organizing committee members are prohibited from joining any team that enters the competition. The sponsoring organizations are allowed to participate and ranked but are not allowed to receive cash prizes.

  1. Q: What is the registration fee?

A:Zero.

  1. Q: Is there any requirement for open-source?

A: Yes. The challenge aims to promote innovation and exchange new ideas. Thus, winners must open-source before receiving the cash prizes.

  1. Q: Will anyone check the source code?

A: Yes. The organizers will check reproducibility and readability.

  1. Q:What is the long-term goal of this competition?
    A: Eventually, sophisticated computer vision should run on UAVs directly. UAVs have limited energy, so low-power computing is important

  1. Q: The winners will be announced at 2023 Embedded Systems Week. Will the conference be on-site?

A: That is the plan. We will follow the instructions by Embedded Systems Week 2023.

  1. Q: Where can I sign the agreement form?

A: You should sign the agreement form when submitting your solution. It is posted on the profile page after you sign in with your registered account. Each account only needs to sign it once. The agreement form is a legal binding document, so please read it carefully.

  1. Q: How can we register as a team?

A: One team only needs one registered account. You can indicate your team members when signing the agreement form.

  1. Q: How often can we submit our solutions?

A: Each team can submit at most once every 24 hours. The latest submission will be used for evaluation if a team submits more than one solution within 24 hours.

  1. Q: Can you give more information on the Leaderboard and Evaluation?

A: Please see section evaluation above.

  1.  What model files in formats such as .pth/.onxx etc. will be supported? Is libtorch inference supported?

A:  We have released a preliminary sample solution.

Due to hardware limitations, libtorch will not be supported at this time.  For a list of supported libraries, tools, and APIs, please see the NVIDIA JetPack SDK 4.6.3 information page. Should any support be added, we will notify everyone.

  1. Q: What is the awards planned for the winners?

A: Cash awards are planned for the winners!

1. $1500: accuracy divided by time.

2. $1000: most accurate, time within the top five.

3. $1000: fastest, accuracy within the top five.

_____________________________________________________________________________________________________________________________

Organizers

Name

Organization

Contact Information

Benjamin Chase Boardley

Student, Purdue University

bboardle@purdue.edu

Ping Hu

Student, Boston University

pinghu@bu.edu

Yung-Hsiang Lu

Professor, Purdue University

yunglu@purdue.edu

Gowri Ramshankar

Student, Purdue University

gramshan@purdue.edu

Kate Saenko

Professor, Boston University

saenko@bu.edu

Nicholas Synovic

Student, Loyola University Chicago

nsynovic@luc.edu

George K. Thiruvathukal

Professor, Loyola University Chicago

gkt@cs.luc.edu

Oscar Yanek

Student, Loyola University Chicago

oyanek@luc.edu

____________________________________________________________________________________________________________________________