Night Photos Improvement With AI

 

With neural networks and machine learning we can do almost everything, especially with the use of new technologies and new powerful computers within everyone's reach, which can train a network like this in a couple of hours and get good results as you can see below.

Architecture

This is a conditional generative adversarial network (cGAN), and this network is specifically based on the Pix2Pix Network Architecture, which is made by two sub-networks, the first is a convolutional-deconvolutional network with an hourglass shape, also known as Generator Networks, the second is a discriminator network and receives the output of the first network.

This first network gets the input image and his work transforms it into the target image, this process consists first in extract all the image information passing it through convolutional layers, these layers extract all valuable information from the image and then pass it to the deconvolutional layers, here is where the image is rebuilt but this time with the light improvement, but this does not guarantee   a good result and that is why we need a second network. 

The second network is pretty simple, this is trained to determine if an image is real or fake, and this is why this architecture is adversarial, both networks compete, the first tries to generate an image good enough to fool the discriminator, meanwhile this last tries to rigorously detect fake images.

So the cGAN architecture is generated joining these both networks as you can si below, where generator network is shown with G and discriminator network is shown with D.

Training

In order to train this network I use 7000 images, extracted from various videos that contain night scenes, I split frames into two 900x900 images and then rotated one of them to have more variety in the dataset. Also, most of the image information, like people or cars, was in the center of the frame, so dataset images are spliced.

Then these images were edited, reducing brightness, adding contrast, and also adding noise, these edited images were the inputs, and non-edited images were the targets.

Results

This model was trained for 25 epochs with 7000 images and then tested with 200 images, generating good results, but a low-quality image of just 256x256, this could be solved using the newest architecture of Pix2Pix called Pix2PixHD that can generate images up to 2K, anyway these are some of this network generated images:

Try it by yourself

You can try it by yourself by downloading the pre-trained model and running it with TensorFlow, you can do it with the next code.

Model: https://drive.google.com/file/d/1cbPuh0sfDz45KdsjKH1jxYHqiK1Y1mWw/view?usp=sharing

These are all the libraries you need:

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
from PIL import Image
sys.modules['Image'] = Image 

Load Model:

#Load Models
generator = tf.keras.models.load_model('yourModelRoute/generator_model.h5', compile=False)

Functions to load image an generate prediction:

#Functions to predict a single image
@tf.function()
#Load Image
def load_single_image(filename):
    inimg = tf.cast(tf.image.decode_jpeg(tf.io.read_file(filename)), tf.float32)[..., :3]
    inimg = tf.image.resize(inimg, [256,256])
    inimg = (inimg / 127.5 - 1)

    inimg = inimg[np.newaxis,...]

    return inimg

def generate_prediction(model, test_input, save_filename=False, display_imgs=True):

    prediction = model(test_input, training=True)

    if save_filename:

    tf.keras.preprocessing.image.save_img('yourOutputDirectory/'+save_filename+'.jpg', prediction[0,...])

    plt.imshow(prediction[0] * 0.5 + 0.5)

    plt.show()

Generate prediction:

#Prediction
generate_prediction(generator,load_single_image('yourImage.JPG'),False, True)

 

 

References:

Pix2Pix Paper: https://arxiv.org/pdf/1711.11585.pdf

Hourglass-like Model: https://medium.com/@sunnerli/simple-introduction-about-hourglass-like-model-11ee7c30138