Image Enhancement by AI Based Segmentation and Pixel Filtering “Morpho Semantic Filtering™”

AI-evolved Image processing, allowing everyone to easily take impressive quality photo

“Morpho Semantic Filtering” is an image enhancement software that uses AI based segmentation and pixel filtering.
In 2020, it won the Best AI Software/Algorithm Award at a Vision Product of the Year Awards 2020 sponsored by the Edge AI’s international-industry organization.
During this roundtable, the project team discussed the development history and possible future directions.

What is the background to the development?

Abe: We needed a new product to showcase at Qualcomm's Tech Summit 2019. That's how the planning of this product started.
Kawaguchi: The idea for the product spawned during a brain-storming session among Morpho engineers. Conventional image processing works uniformly with the entire image. On the other hand, photograph professionals usually perform retouching per area. So, we thought using AI to segment the image in several areas and applying retouching automatically might be interesting.
Abe: The first version was developed for portraits, but we learned through prospect customer interviews that support for various scenes are in demand. We needed to be able to handle certain subjects, outdoors and indoors scenes, and food. As a starter, we began the current development by supporting outdoor scenes.
Yoshida: I do retouching as a hobby, so personally, I thought it's great that AI creates the masks automatically. Even now, prior to development completion, highly accurate masks are generated. I have high hopes for the product.

What was the hardest part in the development?

Kawaguchi: It has to be planning of annotation specification for the learning data. Trying to label the natural world is fairly complicated, but targeting edge AI requires minimization of AI from computational resource perspective and the amount of retained data will be limited.
Abe: In order to plan specification, labels needed to be minimal with these two perspectives in mind; whether the area can be learned by AI and whether it is effective for retouching.
Yoshida: Also, there were required specifications that we didn't realize until we started learning and retouching, which caused specifications to keep changing.
Abe: Right, like snow or ice. If the color tones are corrected based on incorrect specification, it ends up looking "not right" as a photo. So we did some relabeling.
Kawaguchi: Doing annotation all over again is best if avoided as it's a resource-consuming task.
Yoshida: This might be an issue that all edge AI engineers face.
Kawaguchi: Another difficulty we had was accomplishing accuracy with segmentation in the detailed area. Learning and inference is usually performed with base images that are resized to smaller size. This causes detail in the image to be lost.
Yoshida: Here's an example. The areas between leaves and branches are small individually, but when you have a cluster of them, we would like it to be considered as "sky". Correct masks need to be generated. Otherwise, it'll produce a weird looking photo as a result of retouching further down the process.
Abe: Looking back, developing a function that algorithmically optimizes region-specific masks was inevitable.
Kawaguchi: The inferences from AI are not perfect, and it makes sense to optimize them with algorithms from refinement functions. It is also beneficial that it can collaborate with the retouching process which takes place later on.
Abe: The implementation of the refinement function is where Morpho's strength, image processing, is utilized at its best.
Yoshida: And the retouch process. Ultimately, the impression of a photo is subjectively decided. How to achieve refinement that intrigues users emotionally will be an everlasting challenge.
Abe: Current specifications allow you to freely operate the filter functions available, but at the same time, we are thinking about adding Morpho-original presets.
Kawaguchi: I would like to bring it to a level that smartphone manufacturers would want to use as a preset.

Please tell us how the team is organized.

Abe: The development team consists of seven engineers. However, this project is a task force, and it has sales and marketing members as well. We work with members with different roles on a day-to-day basis to accomplish product development that takes customer needs into account.
Kawaguchi: Also, the annotation work is outsourced.
Abe: As a side note, when data creation was outsourced, it was very difficult to get the operator to understand the specification description and its intention. As I mentioned earlier, specifications are not completely covered for all scenes, so we needed to update and explain the specifications repeatedly through trial and error.
Kawaguchi: The working speed and quality are amazing, though.
Abe: That's right. Vendors overseas have plenty of human resources and handle tasks at speeds not possible domestically. This was really helpful.

What are the use cases and future advancement plans?

Abe: For now, we are targeting the smartphone market. First step is providing a function that is limited to outdoor scenes. The use case for that, for example, would be a function that becomes available only when outdoor mode is selected in the camera app.
Yoshida: My dream world would be where everyone can easily get retouched photos with edge devices like smartphones.
Abe: We are still in the process of development for covering all scenes, so it'd be great if we can start by adding support to specific subjects such as indoor scenes and food in the future.
Yoshida: Scene determination will become a requirement. The retouching method will change for daytime and night for outdoors and even whether the subject is there or not.
Kawaguchi: And then comes the support for video. Whether the target market is smartphone or not, people’s purpose of camera shooting is expected to shift to video. If video segmentation accuracy is improved, all markets will become the target. I'd love to work to reach that.
Abe: In the medical field, you can imagine a variety of use cases. Like accurately determining the type and location of the organ from the body image, or clipping a specific subject and replacing the background in video production.