This project is part of the Nanodegree em Deep Learning Foundation taught by Udacity. The source code for running this project is available in my repository on GitHub.

In this project, I’ll classify images from the CIFAR-10 dataset, processes them, then train a convolutional neural network on all the samples. The images need to be normalized and the labels need to be one-hot encoded. I’ll get to apply build a convolutional, max pooling, dropout, and fully connected layers.

Step 1: Get the data

Download the CIFAR-10 dataset for python.

from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import problem_unittests as tests
import tarfile

cifar10_dataset_folder_path = 'cifar-10-batches-py'

floyd_cifar10_location = '/input/cifar-10/python.tar.gz'
if isfile(floyd_cifar10_location):
    tar_gz_path = floyd_cifar10_location
else:
    tar_gz_path = 'cifar-10-python.tar.gz'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(tar_gz_path):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:
        urlretrieve(
            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',
            tar_gz_path,
            pbar.hook)

if not isdir(cifar10_dataset_folder_path):
    with tarfile.open(tar_gz_path) as tar:
        tar.extractall()
        tar.close()

tests.test_folder_path(cifar10_dataset_folder_path)

Step 2: Explore the data

The dataset is broken into batches to prevent your machine from running out of memory. The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc..

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

batch_id = 3
sample_id = 7
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)

Expected outcome:
Image classification found a cat

Step 3: Implement preprocess functions

Normalize

Implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1. The return object should be the same shape as x.

def normalize(x):
    maximum = np.max(x)
    minimum = np.min(x)
    return (x - minimum) / (maximum - minimum)

tests.test_normalize(normalize)

One-hot encode

Just like the previous code cell, i’ll be implementing a function for preprocessing. This time, I’ll implement the one_hot_encode function.The input, x, are a list of labels and the function to return the list of labels as One-Hot encoded Numpy array. The one-hot encoding function return the same encoding for each value between each call to one_hot_encode.

def one_hot_encode(x):
    nx = np.max(x) + 1
    return np.eye(nx)[x]

tests.test_one_hot_encode(one_hot_encode)

Step 4: Build the network

For the neural network, I’ll build each layer into a function. The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability.

To this I Implement the following functions:

  • neural_net_image_input
    • Set the shape using image_shape with batch size set to None.
    • Name the TensorFlow placeholder “x” using the TensorFlow name parameter in the TF Placeholder.
  • neural_net_label_input
    • Set the shape using n_classes with batch size set to None.
    • Name the TensorFlow placeholder “y” using the TensorFlow name parameter in the TF Placeholder.
  • neural_net_keep_prob_input
    • Name the TensorFlow placeholder “keep_prob” using the TensorFlow name parameter in the TF Placeholder.

These names will be used at the end of the project to load your saved model.

import tensorflow as tf

def neural_net_image_input(image_shape):
    return tf.placeholder(
        tf.float32,
        [None, image_shape[0], image_shape[1], 3],
        name='x'
    )

def neural_net_label_input(n_classes):
    return tf.placeholder(
        tf.float32,
        [None, n_classes],
        name='y'
    )

def neural_net_keep_prob_input():
    return tf.placeholder(tf.float32, name='keep_prob')

tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)

Step 5: Convolution and max pooling layer

Convolution layers have a lot of success with images. I implement the function conv2d_maxpool to apply convolution then max pooling:

  • Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor
  • Apply a convolution to x_tensor using weight and conv_strides
    • We recommend you use same padding, but you’re welcome to use any padding
  • Add bias
  • Add a nonlinear activation to the convolution
  • Apply Max Pooling using pool_ksize and pool_strides
    • I recommend to use same padding, but is possible to use any padding
def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    input_depth = x_tensor.get_shape().as_list()[-1]
    
    W = tf.Variable(tf.random_normal(
        [conv_ksize[0], conv_ksize[1], input_depth, conv_num_outputs],
        stddev=0.1
    )) 
    
    b = tf.Variable(tf.zeros(conv_num_outputs))
    
    conv = tf.nn.conv2d(x_tensor, W, [1, conv_strides[0], conv_strides[1], 1], 'SAME') + b
    conv = tf.nn.relu(conv)

    return tf.nn.max_pool(
        conv,
        [1, pool_ksize[0], pool_ksize[1], 1],
        [1, pool_strides[0], pool_strides[1], 1],
        'SAME'
    )

tests.test_con_pool(conv2d_maxpool)

Step 6: Flatten layer

I implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor. The output will be the shape Batch Size, Flattened Image Size.

def flatten(x_tensor):
    shape = x_tensor.get_shape().as_list()
    return tf.reshape(x_tensor, [-1, np.prod(shape[1:])])

tests.test_flatten(flatten)

Step 7: Fully-Connected layer

I implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs).

def fully_conn(x_tensor, num_outputs):
    shape = x_tensor.get_shape().as_list()
    
    W = tf.Variable(tf.random_normal([shape[-1], num_outputs], stddev=0.1))
    b = tf.Variable(tf.zeros(num_outputs)) + 0.11
    
    return tf.nn.relu(tf.add(tf.matmul(x_tensor, W), b))

tests.test_fully_conn(fully_conn)

Step 8: Output layer

I implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs).

def output(x_tensor, num_outputs):
    shape = x_tensor.get_shape().as_list()
    
    W = tf.Variable(tf.random_normal([shape[-1], num_outputs]))
    b = tf.Variable(tf.zeros(num_outputs))
    
    return tf.add(tf.matmul(x_tensor, W), b)

tests.test_output(output)

Step 9: Create convolutional model

I implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits.

def conv_net(x, keep_prob):
    tmp = conv2d_maxpool(x, 64, [3, 3], [1, 1], [3, 3], [2, 2])
    tf.nn.dropout(tmp, keep_prob=keep_prob)

    tmp = flatten(tmp)
    tmp = fully_conn(tmp, 384)
    tf.nn.dropout(tmp, keep_prob=keep_prob)

    tmp = fully_conn(tmp, 192)
    tf.nn.dropout(tmp, keep_prob=keep_prob)
    
    return output(tmp, 10)

tf.reset_default_graph()

x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)

keep_prob = neural_net_keep_prob_input()

logits = conv_net(x, keep_prob)
logits = tf.identity(logits, name='logits')

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)

Step 10: Train the neural network

I implement the function train_neural_network to do a single optimization. The optimization use optimizer to optimize in session with a feed_dict of the following:

  • x for image input
  • y for labels
  • keep_prob for keep probability for dropout

This function will be called for each batch, so tf.global_variables_initializer() has already been called.

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    session.run(optimizer, feed_dict={x: feature_batch, y: label_batch, keep_prob: keep_probability})

tests.test_train_nn(train_neural_network)

Step 11: Show stats

I implement the function print_stats to print loss and validation accuracy and use the global variables valid_features and valid_labels to calculate validation accuracy. I use a keep probability of 1.0 to calculate the loss and validation accuracy.

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    global valid_features, valid_labels
    
    validation_accuracy = session.run(
        accuracy,
        feed_dict={
            x: valid_features,
            y: valid_labels,
            keep_prob: 1.0,
        }
    )
    
    cost = session.run(
        cost,
        feed_dict={
            x: feature_batch,
            y: label_batch,
            keep_prob: 1.0,
        }
    )
    
    print('Cost = {0} - Validation Accuracy = {1}'.format(cost, validation_accuracy))

Step 12: Hyperparameters

Tune the following parameters:

epochs = 50
batch_size = 1024
keep_probability = 0.7

Step 13: Train on a single CIFAR-10 batch

Instead of training the neural network on all the CIFAR-10 batches of data, I use a single batch to save time while you iterate on the model to get a better accuracy. Once the final validation accuracy hit 50% or greater, I run the model on all the data in the next section.

print('Checking the Training on a Single Batch...')

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)

Step 14: Fully train the model

Now that I got a good accuracy with a single CIFAR-10 batch.

save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    
    for epoch in range(epochs):
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)
            
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)

Step 15: Test model

I test my model against the test dataset and this will be your final accuracy. I should have an accuracy greater than 50%.

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')
        
        test_batch_acc_total = 0
        test_batch_count = 0
        
        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)

test_model()

Expected outcome:

Softmax Predictions

Conclusion

I obtained as a result the precision of the test estimated at 0.6157106995582581.

Running the project

Downloading files needed to run this project on my GitHub. There you also have additional information on preparing the environment to run it.