Maritime survey and engineering industry trends: #5 – Deep learning technology

EIVA sees five decisive trends resulting from new and improved technologies. In this last piece of the series, our CEO, Jeppe Nielsen, dives into the very interesting benefits brought to the industry by deep learning technology.

Author: EIVA CEO Jeppe Nielsen

What is deep learning?

One of the monumental trends seen in the IT world in recent years is machine learning. Simply put, machine learning is an application of artificial intelligence. It focuses on the development of algorithms that can interpret data and learn from the data in order to apply this knowledge to make informed decisions – that is, the machine is trained using data. 

Deep learning is a subfield of machine learning, taking the technology to the next level by making the software able to learn by itself – as opposed to being told how to make a prediction or decision, which is the case for the machine learning model.

(Learn more at https://www.zendesk.com/blog/machine-learning-and-deep-learning/ and https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/)

Looking at Gartner Group’s 2017 Hype Cycle chart, machine learning and deep learning are at their expectation peak and are expected to reach mainstream adoption in 2-5 years.

Neither machine learning, nor deep learning is a new discipline. The underlying technology of neural networks has been around for at least 25 years. However, it is in recent years that it has truly become productive due to gaining access to massive computing power, the development in affordable graphics/GPU performance and big data cloud infrastructures, and the massive amount of data available.

How can the maritime industry benefit from deep learning?

Deep learning is finding its way into many industries – including subsea and offshore segments. As it makes it possible to let software automatically recognise and localise objects in various types of data, typically photos and video. Deep learning has the potential to save companies specialising in for example pipeline inspections, habitat mapping or UXO detection countless hours of manual work – just to mention a few possible applications. 

At EIVA, we wish to be part of driving this initiative as we constantly strive to provide the most efficient data processing and data interpretation platform for subsea-related data. Consequently, we established a dedicated software development team with engineers specialising in machine learning, machine vision and deep learning earlier this year. The team are now seeing the first results of their work, which are very promising. 

Integrating deep learning in the NaviSuite software products

• Classification: Identifying what is in a data set
• Segmentation: Marking where the identified objects are in the data set

NaviSuite Deep Learning is a black box software product that works in combination with the eventing features in NaviPac, the NaviSuite product for navigation and positioning, and NaviModel, which is dedicated to 3D/4D modelling and visualisation. 

You can choose to apply it in three different ways, that is, as:

• A cloud service hosted at EIVA. This means that NaviModel and NaviPac can utilise the cloud service for classification and pixel-by-pixel segmentation of images and videos. This is typically useful for data processing staff working onshore with good internet connections.
• A rack server allowing you to add the segmentation and classification functionality to your own network, so that data traffic is kept locally. This is useful for onboard vessel data processing.
• An onboard computer where the segmentation and classification functionality is running on a small, high-speed computer board suitable for integration into AUV (autonomous underwater vehicle) and USV (unmanned surface vehicle) equipment. This makes it possible to let the AUV/USV change mission based on objects detected via the deep learning software.

Training NaviSuite Deep learning with millions of data samples

NaviSuite Deep Learning is based on a neural network model, which is trained on millions of various data samples from various providers, typically NaviSuite customers owning subsea assets. The resulting neural network consists of a set of parameters performing the actual object classification and segmentation. 

High performance

NaviSuite Deep Learning is very fast. The rack server is able to classify more than 500 FPS, that is, process some 500 images per second, where objects are identified and events are generated. Comparing this with a typical video stream of 30-50 FPS, a single NaviSuite Deep Learning rack server is easily able to process multiple video streams in real time and annotate each frame in the video with the objects that are detected. The onboard electronics version is able to perform 15-20 FPS, ie it can be used real time on video feeds.

Imagine the following usage scenarios:

• An ROV pilot will automatically have a video tagged with objects observed in the file, thus giving assistance to spotting objects of interest that the operator may otherwise miss. The tagged objects will be marked as events for later examination by the eventing data processor.
• A video eventing data processor inspecting recorded videos can scroll through the video, and each video frame is tagged with objects found as the video is viewed – or the software can go through the entire video and tag all frames while the data processor is doing initial quality control of the video.  
• An automated workflow, where the NaviSuite Workflow Manager automatically processes all videos, so they are ready for the data processor to inspect and/or modify the results, thereby vastly reducing the data processing time.
• Carrying out eventing across multiple sources, ie looking at video, sonar, laser and other sources to improve the verification of the objects identified.
• An end client who has received a deliverable with a video and a list of anomaly can quickly check this – and check historical data for occurrences of specific anomaly.

For operations such as subsea pipeline inspections, NaviSuite Deep Learning can identify many different object types, including (but not limited to): pipes, anodes, joints, damages, boulders and debris.

For habitat mapping operations, we are looking for eelgrass, sand, rocks, gravel, muzzle banks, etc. 

Initially, the deep learning software is working on video and still image sources, but we are currently gathering enough material to use it on intensity data (sidescan sonar and backscatter), terrain data (slope), and other sources.

Integration with NaviPac or NaviModel

When using the NaviSuite Deep Learning software from within NaviPac or NaviModel, the classification software is working with the eventing and video functionalities of these products.

• Events are generated based on the objects found in the video
• Events can be added in real time or processed in the background on the entire video/image source
• Events present in consecutive images are grouped as one long event
• The operator can filter which events to be seen
• The operator can edit/delete events

Pricing

NaviSuite Deep Learning is available on an annual subscription basis. Subscriptions for rack server and cloud usage are priced at €9.995,- / year including 24/7 help desk support and all updates.

Rack server and onboard units are purchased in addition to the subscription. Retail price for the rack unit is € 5,000,- and onboard PCBs € 1,500,-.

Do you want to know more?