Ultra High Resolution Real Time Image Enhancement By Regions

An end-to-end AI algorithm that can automatically enhance the low-quality photos/videos locally on the edge devices (PC, phone and edge devices with gpu) without network connection (up to  (but not limited to) 7000 x 5000 pixels) in real time.

Project scope

With the high penetration of mobile phones, photo shooting has become much more convenient. As it is so effortless, amateur travellers usually take lots of photos during their trips, shots with poor exposure and incorrect settings are inevitable taken. To restore those precious moments, various enhancer apps are created to take care of the touch-up, however, most of these apps take heavy loading time to fix those unsatisfactory photos.

 

Today, people carve for interactions across social media, the long wait delays the sharing with friends on various platforms. It is believed that an end-to-end one-touch photos/videos enhancement application is in high demand.

Background

We have built a deep learning neural network that can automatically touch-up the low-quality photos/ videos locally on the devices, e.g. smartphones without using the cellular network or cloud system. The automatic enhancement is an end-to-end process that is completed in less than a second by just a tap.

There are different image enhancement Apps on the market that improve the quality of images or edit them automatically through advanced artificial intelligence already. But the drawback of these apps requires users to send their images to cloud servers. This increases the risk of privacy issues. The reason for this approach is because normally images/videos deep-learning networks are computationally heavy and require strong GPU servers.

 

We have developed specific deep learning architectures which can run on smartphone CPUs with general GPUs. With this technology, we can enhance users’ images and videos locally on their smartphones without any connection to the internet, and hence make sure the user’s data is under control.

AI deep learning model

Our AI model learns the way to adjust your image by smaller regions and each region applies the unique setting to make sure your image looks nicely in all parts. After the local adjustment, the AI model will combine all parts of the image and combine as an image with a global adjustment. There are three sample images in the result session, the left side is the original image, and the right side is the outcome generated by our AI model. Those images show that our AI model is capable of adjusting a very wide range of bad images, from extreme under/over-exposed images, light flare, low contrast and much more.

 

Image processing is a very heavy computational task due to the resolutions for the image. Therefore, many mobile applications will limit the size of an image to full HD (1920*1080), or higher-end paid applications might support up to 2K (2480*1080) resolution on mobile phones, or the image will be compressed/resized before being processed which will lose so much detail information. The higher the image size, the more resources (RAM memory) and processing time are required. Our AI model is not only capable of enhancing picture and video nicely, but also overcomes the limitations of both traditional AI applications on servers and applications on mobile(edge) devices. Our solution takes the advantage of the GPU (Graphics Processing Unit, which is natively integrated with modern mobile devices) able to support up to 8K (7680*4320) images (even higher on high-end devices) without any distortion, compression or even quality drop under a fraction of a second.

Video processing is even more challenging compared to image processing. Since the smoothness in terms of lighting and color changes directly impact the quality of the video. Flickering becomes an issue as each frame (image) being processed from the video has different conditions of the lighting, colors and tones; the color and lighting might dramatically change between frames. Therefore, we implemented a special architecture for our AI model to suppress the flickering issue while having video quality enhanced with no compression, close to real-time performance (around 30fps) with up to 8K resolution.

Our proprietary training data

- We have collected millions of professional photos for the training dataset

- The model covers super high-resolution photos up to but not limited to 7000x5000 pixels

- We have applied a special lighting flow control algorithm, which successfully eliminates the common flickering problem in the video auto enhancement program.