SAR + AI/ML for dark vessel detection: the xView3 challenge

As troubling, dark vessels are the enabling factor in a range of illicit activities, from transporting illegal drugs, weapons, and unauthorized cargo (such as sanctioned crude oil and stolen Ukrainian grain) to human smuggling. After recently revisiting the results of the 2021 xView3 Challenge, we decided to share more about it, as the contest does such a great job of highlighting the problems and potential solutions around the timely issue of dark vessel detection.

Dark vessels, also known as ships involved in IUU (Illegal, Unreported, and Unregulated) fishing activity, mask their locations by operating with Automatic Identification Systems (AIS) turned to the “off” position. Spoofing, the intentional transmission of false position reports, is another method of concealing movements from authorities.

However, IUU fishing has a more significant impact. These activities in the waters of developing nations can result in over-harvesting and degraded fish stocks, leading to food and societal insecurity, a rise in crime, international skirmishes on the high seas, and an overall reduction in regional stability.

Recognizing the threat IUU activity presents to the global food supply, in August 2021, two organizations with radically different missions joined forces to help tackle this global problem and risk to international food security: The Defense Innovation Unit (DIU), the US Military’s commercial technology accelerator, and Global Fishing Watch (GFW) teamed up to launch xView3, a dark vessel detection challenge.

The need to act on IUU fishing is urgent; according to a 2020 study from the University of British Columbia, between eight and 14 million metric tons of fish are caught annually, leading to economic losses approaching $50 billion.

During the three-month event, over 1,900 registrants from 67 countries submitted machine learning models to identify and differentiate between fishing, non-fishing, and fixed maritime infrastructure. The contest’s objective was to determine the top five algorithms capable of detecting and characterizing dark vessels engaged in IUU fishing activity. Synthetic Aperture Radar (SAR) imagery was used as a baseline for this purpose.

SAR offers wide-area coverage using an active sensor unaffected by weather. In addition, SAR images have a high spatial resolution, which helps monitor ship movements. Measuring a vessel’s SAR signal traveling time and scattering effect on the target’s radar reflection can provide vessel type, location, direction, speed, and size.

Complicating technical factors

The use of SAR for global monitoring is limited to satellite overpass rates. Continuously collected, massive amounts of maritime SAR imagery requires time-intensive analysis, primarily conducted by humans.

Additionally, a target’s image resolution, sensor incidence angle, radar cross-section, and ship build materials are often complex to determine without confirmation from a vessel’s radio transponder signal or AIS (​​Automatic Identification System: the primary vessel tracking and communication system around the globe). In the case of dark vessel operations, these tracking signals are unavailable, as the signal is turned off.

Detecting IUU fishing activity calls for data beyond location and ship size estimations. Unfortunately, the lack of validated, global SAR marine datasets to train AI/ML detection and identification algorithms, as well as the lack of technical capability to both process the imagery and match historical AIS signals to views, presents a barrier to high-speed SAR image processing, an issue xView3 had to address for the challenge.

To overcome the hurdle, contest technical staff built a multi-dimensional database of 1000 open-source maritime SAR images from the European Space Agency’s Sentinel-1 satellite constellation. Imagery from regions with high levels of legal and IUU fishing activity, near and off-shore, was selected for the contest. Labeled training data was derived from a combination of GFW’s historical detection model for maritime objects shown in the imagery, the output of its unique AIS analytics, and human visual detections.

SAR imagery was used for the contest due to the effects of weather on imagery obtained with passive electro-optical (EO) sensors. Accurate dark vessel detection and identification rely on the target’s reflected electromagnetic and infrared radiation. Therefore, frequent weather events at sea, such as cloud cover, haze, and storms, can influence the effectiveness of passive EO sensors by degrading a ship’s reflected radiation profile.

Contest procedures and leaderboards

For registrants, the first step required downloading the xView3 AI/ML – artificial intelligence (AI) and machine learning (ML) – training, development, and validation datasets from the contest website. Then, when solutions were ready, entrants downloaded another public file containing only SAR imagery without matching AIS data.

Participants would then run their AI/ML algorithm with the SAR imagery file. Results were uploaded to the contest portal in CSV format and graded instantly. A Leaderboard displayed rankings in real-time as entries were submitted and scored. Contestants were allowed three entries per day. Code in the upper fifty percentile was eligible for further verification to determine if the results were reproducible.

Contest submissions were closed at the end of November 2021. On January 31, 2022, the DIU and GFW announced the xView3 winners. $150,000.00 in prize money went to the Five top-scoring entrants.

The winning entries explained

The first place $37,000.00 prize went to BloodAxe, who submitted algorithms 76 times during the contest. “My first Leaderboard scores were 0.44, which gradually went up to 0.60+; the contest took two months of work.” They used two NVidia RTX GPUs to train their deep learning model.

Selim_sef, a computer vision programmer, came in second, winning $27,000.00. Commenting on their efforts, they said the experimentation took some time, and “a large part of it did not lead to improvements.” While they did not say how many algorithms they submitted, they spent one month refining their entry.

Tumen took third place with an $18,000.00 prize. Skylight and PRIOR at the Allen Institute for AI split the $11,000.00 fourth place award. Fifth place went to Kohei, who won the  $7,000.00 prize.

For IUU detection, one size does not fit all

Capturing and transmitting SAR imagery with the subsequent processing and analysis necessary to extract valuable data comes at a cost. Developing nations affected by IUU and other interested entities may not have the funding or infrastructure to monitor maritime activities using SAR. For such circumstances, non-SAR solutions are available. Service providers using AIS tracking, historical data, and predictive modeling based on vessel size, characteristics, and movement, can provide alternatives to SAR imagery for maritime IUU detection and identification.

Developing speed and scale is crucial in fighting illegal activity.  “The machine learning models will help our US and international partners identify potential IUU fishing activity at a speed and scale that would be practically impossible for human analysts,” according to Jared Dunnmon, DIU AI/ML Portfolio Director.