Predicting port berth arrivals
How Clearmetal created a machine learning engine to provide port alerts 31 hours faster to help optimize operations and costs.
Fusing EDI and Satellite AIS Data
How ClearMetal Provides Port Alerts 31 Hours Faster
Executive Summary
Across the industry, all EDI data for ships in port is incomplete and often late. Historically, the industry never even receives EDI messages indicating a vessel being at berth for ~10% of all container ships.
In addition, for the ~90% of trips that do have an arrival update, there is latency in EDI data in excess of 36 hours, meaning that a ship could be at port for 1.5 days before anyone knows that it has arrived.
In order to fix this pervasive problem, we applied AIS data to map the berth locations of every port in the world and triangulate that with EDI to improve our understanding of port and near-port transit. This has increased our arrival completeness to ~96%, and has reduced our latency by 86% to ~5 hours. This creates a ripple effect on accurately predicting near-port activities like container availability, demurrage & detention, drayage, deconsolidation, and other components that impact margins, personnel efficiency, and gross revenue.
Challenges
The industry has struggled with the complexity of berth level detail and visibility into when a ship is actually berthed for years. Our objective was to unravel this conundrum for our customers. We began by conducting a deep analysis of the EDI messages we receive, during which we discovered that up to 10% of ships don’t transmit an EDI message when they arrive in port. For the remaining voyages, there was on average, a 36-hour delay in identifying when a ship was at berth.
Thus, today, in the best case scenario, shippers receive an update indicating that a ship has arrived at a port 1.5 days after the event; in the worst case scenario, they do not get notified at all. This lack of transparency creates significant uncertainty in container availability, which in turn causes an exponential increase in costs across the whole supply chain, including excess drayage fees, increased costs in demurrage, and higher cost of capital for inventory on-hand.
Adding to the challenge, each port in the world is different: different in how they are run, different in how they report key milestones such as “Available for Pickup,” and of course, different in location, shape, size, and geographical features. This last challenge has a dramatic effect on reporting when a ship is actually at berth vs. stopped and waiting for a berth to open up.
The latency of a day and a half in predicting transport exceptions is unacceptable. To remain relevant, shippers must continuously fine-tune their supply chain by proactively managing transit exceptions and continuously improving on-time performance for their orders The up and downstream effects create value through increases in revenue, improved cost of capital, and greater productivity of personnel.
Our quest to innovate and transform supply chain visibility led us to seek out non-traditional datasets that we could use to improve our data quality. This search led us to AIS data and Spire.
Results, Return on Investment and Future Plans
Understanding where every berth around the world is will lead us to further leverage our machine learning and AI on top of historical data to give berth specific predictions:
Carriers who only use specific berths may experience delays if they are full despite all other berths being open. Distances or difficulties in reaching specific berths have effects on predictions and availability. Port performance and port congestion can be quantified based on the number of berths vs. the number of ships waiting to be unloaded and the time between berth cycles.
The impact of mapping berths with AIS and combining that with EDI drives incredible value for our customers by accurately predicting near-port activities such as: Container availability, demurrage & detention, drayage, deconsolidation, and other components that impact margins, personnel efficiency, and gross revenue. The industry has given up on many of these when it comes to having an accurate prediction that could be leveraged to be more strategic. Detention & Demurrage for example, which can cost millions in fees each year is often thought of as just the cost of doing business and fees are simply factored into every shipment. We’ve seen reductions in those fees of 20-50% resulting in a saving that far surpasses the cost of our visibility solution.
How Spire Helped
EDI and AIS are complementary data sources. EDI provides context about a container, such as whether it has been unloaded from a ship, and AIS provides a more accurate location with substantially less latency. Using EDI and AIS to fact check each other gives us the best of both worlds. We can provide a highly accurate current position of a ship and also leverage it to fill in gaps in time that EDI always has and cannot catch up to solving. While satellite data can tell us quickly that a ship has stopped, it can’t tell us why. Knowing this our data science team devised a unique approach to identifying berth calls much earlier by using AIS data to accurately map the berths at every port around the world and fill in the missing 10% of “ship at port” EDI messages and reduce the nearly 1.5-day latency.
After dissecting the EDI data, we then analyzed over 2 billion AIS messages to define when a vessel is considered to be at a stop. Vessel speed is one of the pieces of data that is sent with every AIS message, but not all vessel stops are the same. Container vessels might stop for a number of reasons. For example, if a specific berth location is already occupied by another vessel, the vessel would have to wait its turn, and then move into the location. Another example is a pilot station, where the container ships wait for a new captain to take control of the vessel. If we only use stops and assume a vessel is at berth if it is within a certain distance of the port, then we might interpret these “waiting” stops as berth stops, and inaccurately report that a vessel has arrived at its destination port.
We observed that vessel stops are clustered very densely at berth locations, but as we move further away from ports, these stops become less clustered and more spread out. So given all vessel stops, if we can find locations where multiple vessels have stopped in the same spot, we can be confident that these are berth locations. We iterated on this idea, tuning the parameters of our unsupervised clustering algorithm until we converged on a solution. The analysis and resulting algorithm that was built to isolate the clusters revealed an incredibly accurate map of berths at ports around the world. The accuracy of this solution far surpasses the industry standard of using an arbitrary latitude longitude area around the port and triggering EDI alerts off of a ship’s location.
Learn more:
Spire Global
Clearmetal