Featuring Technology

  1. HOME
  2. Featuring Technology : SoftNeuro

Fast AI Inference Engine “SoftNeuro®”

World-class AI technology created by masters of image processing

Kawaguchi, Majima, Kuroda, Nishino, Hilaga, Matsuo, Hirasaki, Ogawa, Kurihara

"SoftNeuro®" operates in multiple environments, utilizing learning results that have been obtained through a variety of deep learning frameworks.
During this roundtable, the development team discussed the development history and possible future directions.

What is the background of development and how did it come about?

Hilaga
The in-house Deep Learning inference module (ConvNet) that was developed for the Morpho Scene Classifier™ scene recognition technology wasn't reaully built for customization, so we had to rebuild it, right?
Kuroda
It is true that one of the motivations for development was that it wasn't very well-organized. Another motivation was that we developed ConvNet between 2014 and 2016, and towards the end of that period, my boss pointed out that it could be a little faster, so we did some experiments at hand and found that there was indeed a data structure that made it faster.
Hilaga
ConvNet was implemented with the motivation that it would be nice to be able to do Deep Learning on embedded devices, but there were problems with the software architecture.
Kawaguchi
Was it implemented in C at the time? And the learning side?
Kuroda
It was implemented in C. On the learning side, there were deepnet, Theano, Caffe, etc. deepnet is a minor framework that has been around since early days.
Hilaga
At the time, we didn’t quite know which learning framework to choose.
Nishion
Caffe wasn't around back then in the first place.

What are the advantages of SoftNeuro?

Hilaga
It's easy to use, it's fast and it can be used anywhere. Using this framework doesn't require you to know anything about Deep Learning.
I was aiming for a very easy-to-use inference framework similar to Keras, and I personally wanted something like that.
Another advantage of SoftNeuro is that it can import from various frameworks and run fast on various architectures.
Also, SoftNeuro is both flexible and fast because of the separation of the layer and its execution pattern, which is a concept of routine. I think this is a new feature not found in other inference engines.
Nishino
And, you can perform tuning.

Please tell us how your development was organized in the first half of the deveopment period (~2018)?

Matsuo
It was my first assinged task after my OJT.
At first, I could only load and execute VGG from memory, but by adding importers, various layers, etc, it became a library that worked properly.
There were very few inference engines at the time, but Intel's inference engine was fast. To beat them, we improved speed of SoftNeuro as a whole by applying new ideas at the algorithm and HW level. This was a lot of fun.
Kawaguchi
Where did you come up with the idea of tuning and speeding up on a per device basis?
Kuroda
Thas was Hilaga-san’s idea. Since we implemented various Convolution patterns, we also needed a mechanism to select the most appropriate one.
Nishino
We thought we had a chance to patent it if we could speed up the inference on a per device basis. And then we did, and we got the patent as well.
Kawaguchi
Was it because you had experiences in performance tuning with embedded systems and had the know-how’s?
Hilaga
Yes. It has the effect of adjusting the parallel granularity and cache size.
Kuroda
We created a smartphone demo upon request for GTC Japan 2017 showcase. We also introduced SoftNeuro in INTERNATIONAL TECHNICAL EXHIBITION ON IMAGE TECHNOLOGY AND EQUIPMENT and Embedded Vision Summit the following year. Other companies had segmentation demos at trade shows in 2018, but I believe none had it working on smartphones.
Hilaga
Kurihara-san, what kind of difficulties did you have?
Kurihara
I was the only one working on productization of the engine and tools at the time. There were no documentations... no convert tools... I had to do a lot of organizing and development.
Hilaga-san's code was easy to read and came with documentation, so working with that was not impossible despite the amount of code. But productization became harder and harder as OpenCL implementation, FP16, and quantization came along.

Please tell us about your current development team.

Majima
When I joined the team, the framework of SoftNeuro was already established, but it was still going through a transformtion phase with many variations being added. It badly needed a proper testing and seirous organizing.
Our short-term challenge is speeding up the relase cycle while keeping their quality.
Hirasaki
Unit test is now more enhanced and CI using Jenkins is also introduced.
Productization efficiency has greatly improved since. Time-consuming Dynamic code analysis can now be done in one shot.
There are more and more routines and importers, but I think we can guarantee and improve the quality of our products by enhancing our testing.
Hilaga
Beginning ver. 4, python plug-in and pytest which has direct connection to the library were introduced and we can all agree that this made things much easier. The code is only about 200,000 lines, so I don't think it's so huge compared to Linux and such.

What are some examples of applications of SoftNeuro?

Matsuo
Our Deep Learning product portfolio has expanded between 2018-2019. For example, Segmentation, Detection, Tracking, etc.
Also, we are recently shifting our focus towards real-world applications, such asTV-related products utilizing tracking technology (e.g. Face Tracker). I have a feeling that SoftNeuro is entering a new phase. It's expanding through products.
Hilaga
It's handy when GPU is not there for use like when using a laptop.
Nishion
Also, it's used in other companies' products.
Kawaguchi
We develop both the inference engine and the importer, which is a great advantange allowing speedy implementatin cycle of both learning and inference.
Hilaga
There seems to be an advantage as well that we keep inference in mind when building the learning system.
Ogawa
From an engineer's perspective, SoftNeuro is easy to embed, and was easy to implement in Unity, etc. With the increase in the latest research in Pytorch, it will be even easier to use if we can make the conversion to ONNX and other systems smoother.
Kawaguchi
The implementation to Vieureka in OJT was also very smooth and easy for new employees. They did it in about two to three months.

What is the future direction of SoftNeuro?

Matsuo
Recently, the performance of competing products are improving, so I'd really like to focus on making SoftNeuro run faster. I'm also interested in developing underlying technologies to speed up the development process.
Kawaguchi
I want SoftNeuro to be running on all smartphones If the inference speed is overwhelming, you would use SoftNeuro. We'll need to support it to run on a variety of processors, though.
Hilaga
I'd like to be able to license the software for developers. Right now we're releasing it as a B2B product, but we'd like to try to expand into B2C. For example, I'm thinking about a framework for downloading software from the web.
Ogawa
In addition to SoftNeuro itself, improving importers is also essential for wider use of SoftNeuro. Like making execusion available without converting the model file.
Hirasaki
Since tuning is a unique feature, I'm interested in a mechanism that can be extended to dtune itself at the time of execution. Making the engine fast can easily cause degration in the result... so a system which ensures accuracy while improving the speed is something we would love to have.