MNIST classification

Current events on this problem
Keywords
MNIST_simplified_starting_kit

# Paris Saclay Center for Data ScienceÂ¶

## RAMP MNIST handwritten digit classificationÂ¶

Mehdi Cherti (CNRS), Balázs Kégl (CNRS)

## DataÂ¶

The goal of this RAMP is to classify correctly handwritten digits. For each submission, you will have to provide an image classifier (versus the original setup that required a transformer and a batch classifier). The images are usually big so loading them into the memory at once may be impossible. The image classifier therefore will access them through an img_loader function which can load one image at a time.

## HintsÂ¶

Setting up an AWS instance is easy, just follow this tutorial.

For learning the nuts and bolts of convolutional nets, we suggest that you follow Andrej Karpathy’s excellent course.

In [1]:
import os
import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import AxesGrid
from matplotlib import cm

%matplotlib inline

pd.set_option('display.max_rows', 500)

# The dataÂ¶

If the images are not yet in data/imgs, change the type of the net cell to "Code" and run it.

In [2]:
X_df = df['id']
y_df = df['class']
X = X_df.values
y = y_df.values

The class distribution is balanced.

In [3]:
labels_counts_df = df.groupby('class').count()
labels_counts_df
Out[3]:
id
class
0 4742
1 5370
2 4795
3 4914
4 4675
5 4337
6 4709
7 4998
8 4707
9 4753

It is worthwhile to look at some image panels, grouped by label.

In [4]:
nb_rows = 4
nb_cols = 4
nb_elements = nb_rows * nb_cols
label = 8

df_given_label = df[df['class']==label]

subsample = np.random.choice(df_given_label['id'], replace=False, size=nb_elements)

fig = plt.figure(figsize=(4, 4))
grid = AxesGrid(fig, 111, # similar to subplot(141)
nrows_ncols = (nb_rows, nb_cols),
label_mode = "1",
)
for i, image_id in enumerate(subsample):
filename = 'data/imgs/{}'.format(image_id)
im = grid[i].imshow(image, cmap='Greys', interpolation='nearest')
grid[i].axis('off')
plt.tight_layout()
/Users/kegl/anaconda/lib/python2.7/site-packages/matplotlib/figure.py:1999: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
warnings.warn("This figure includes Axes that are not compatible "

All images have size 28 $\times$ 28.

In [5]:
image.shape
Out[5]:
(28, 28)
In [6]:
n_subsample = 1000
shapes = np.empty((n_subsample, 2))
for i, image_id in enumerate(X_df[:n_subsample]):
filename = 'data/imgs/{}'.format(image_id)
shapes[i] = image.shape
In [7]:
shapes_df = pd.DataFrame(shapes, columns=['height', 'width'])
shapes_df['count'] = 0
shapes_df.groupby(['height', 'width']).count().sort_values('count', ascending=False)
Out[7]:
count
height width
28.0 28.0 1000
In [8]:
shapes_df['height'].hist()
Out[8]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c165c6090>

# Image preprocessingÂ¶

In the first workflow element image_preprocessor.py you can resize, crop, or rotate the images. This is an important step. Neural nets need standard-size images defined by the dimension of the input layer. MNIST images are centered and resized, so these operations are unlikely to be useful but rotation may help.

In [9]:
filename = 'data/imgs/{}'.format(X_df[161])
plt.imshow(image, cmap='Greys', interpolation='nearest')
plt.show()

Here we resize the images to different resolutions, then blow them up so the difference can be visible.

In [10]:
from skimage.transform import resize

nb_rows = 1
nb_cols = 2
nb_elements = nb_rows * nb_cols

fig = plt.figure(figsize=(4, 4))
grid = AxesGrid(fig, 111, # similar to subplot(141)
nrows_ncols = (nb_rows, nb_cols),
label_mode = "1",
)
grid[0].imshow(
resize(resize(image, (16, 16)), (224, 224), order=0),
cmap='Greys', interpolation='nearest')
grid[0].axis('off')
grid[1].imshow(
resize(resize(image, (8, 8)), (224, 224), order=0),
cmap='Greys', interpolation='nearest')
grid[1].axis('off')
plt.tight_layout()
/Users/kegl/anaconda/lib/python2.7/site-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.
warn("The default mode, 'constant', will be changed to 'reflect' in "

Here we rotate the image. Explore options in skimage.

In [11]:
from skimage.transform import rotate

nb_rows = 1
nb_cols = 4
nb_elements = nb_rows * nb_cols

fig = plt.figure(figsize=(10, 10))
grid = AxesGrid(fig, 111, # similar to subplot(141)
nrows_ncols = (nb_rows, nb_cols),
label_mode = "1",
)
grid[0].imshow(rotate(image, 30), cmap='Greys', interpolation='nearest')
grid[0].axis('off')
grid[1].imshow(rotate(image, 45), cmap='Greys', interpolation='nearest')
grid[1].axis('off')
grid[2].imshow(rotate(image, 60), cmap='Greys', interpolation='nearest')
grid[2].axis('off')
grid[3].imshow(rotate(image, 75), cmap='Greys', interpolation='nearest')
grid[3].axis('off')
plt.tight_layout()

All these tansformations should be implemented in the transform function found in the image_preprocessor workflow element that you will submit.

## The pipelineÂ¶

For submitting at the RAMP site, you will have to write a single ImageClassifier class implementing a fit and a predict_proba function.

Note that the following code cells are not executed in the notebook. The notebook saves their contents in the file specified in the first line of the cell, so you can edit your submission before running the local test below and submitting it at the RAMP site.

### The starting kit image classifierÂ¶

The starting kit implements a simple keras neural net. Since MNIST is a small set of small images, we can actually load them into the memory. MNIST contains well-centered and aligned images so _transform only needs to scale the pixels into [0, 1].

In [18]:
%%file submissions/starting_kit/image_classifier.py
import numpy as np

from keras.models import Model
from keras.layers import Input
from keras.layers import Dense
from keras.layers import Flatten
from keras.optimizers import SGD

class ImageClassifier(object):

def __init__(self):
inp = Input((28, 28, 1))
x = Flatten(name='flatten')(inp)
x = Dense(100, activation='relu', name='fc1')(x)
out = Dense(10, activation='softmax', name='predictions')(x)
self.model = Model(inp, out)
self.model.compile(
loss='categorical_crossentropy',
optimizer=SGD(lr=1e-4),
metrics=['accuracy'])

def _transform(self, x):
# adding channel dimension at the last position
x = np.expand_dims(x, axis=-1)
# bringing input between 0 and 1
x = x / 255.
return x

# load the full data into memory
# make a 4D tensor:
# number of images x width x height x number of channels
X = np.zeros((nb, 28, 28, 1))
# one-hot encoding of the labels to set NN target
Y = np.zeros((nb, 10))
for i in range(nb):
X[i] = self._transform(x)
# since labels are [0, ..., 9], label is the same as label index
Y[i, y] = 1
self.model.fit(X, Y, batch_size=32, validation_split=0.1, epochs=1)

X = np.zeros((nb, 28, 28, 1))
for i in range(nb):
return self.model.predict(X)
Overwriting submissions/starting_kit/image_classifier.py

### A simple keras convnetÂ¶

In [19]:
%%file submissions/keras_convnet/image_classifier.py
import numpy as np

from keras.models import Model
from keras.layers import Input
from keras.layers import Dense
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten

class ImageClassifier(object):

def __init__(self):
inp = Input((28, 28, 1))
# Block 1
x = Conv2D(
name='block1_conv1')(inp)
x = Conv2D(
name='block1_conv2')(x)
x = MaxPooling2D(
(2, 2), strides=(2, 2),
name='block1_pool')(x)
# dense
x = Flatten(
name='flatten')(x)
x = Dense(
512, activation='relu',
name='fc1')(x)
out = Dense(10, activation='softmax', name='predictions')(x)
self.model = Model(inp, out)
self.model.compile(
loss='categorical_crossentropy',
metrics=['accuracy'])

def _transform(self, x):
# adding channel dimension at the last position
x = np.expand_dims(x, axis=-1)
# bringing input between 0 and 1
x = x / 255.
return x

# load the full data into memory
# make a 4D tensor:
# number of images x width x height x number of channels
X = np.zeros((nb, 28, 28, 1))
# one-hot encoding of the labels to set NN target
Y = np.zeros((nb, 10))
for i in range(nb):
X[i] = self._transform(x)
# since labels are [0, ..., 9], label is the same as label index
Y[i, y] = 1
self.model.fit(X, Y, batch_size=32, validation_split=0.1, epochs=1)

X = np.zeros((nb, 28, 28, 1))
for i in range(nb):
return self.model.predict(X)
Overwriting submissions/keras_convnet/image_classifier.py

### The same keras convnet using generatorsÂ¶

In [21]:
%%file submissions/keras_generator/image_classifier.py
import numpy as np

from keras.models import Model
from keras.layers import Input
from keras.layers import Dense
from keras.layers import Flatten

from rampwf.workflows.image_classifier import get_nb_minibatches

class ImageClassifier(object):
def __init__(self):
inp = Input((28, 28, 1))
x = Flatten(name='flatten')(inp)
x = Dense(100, activation='relu', name='fc1')(x)
out = Dense(10, activation='softmax', name='predictions')(x)
self.model = Model(inp, out)
self.model.compile(
loss='categorical_crossentropy',
metrics=['accuracy'])
self.batch_size = 64

np.random.seed(24)
nb_train = int(nb * 0.9)
nb_valid = nb - nb_train
indices = np.arange(nb)
np.random.shuffle(indices)
ind_train = indices[0:nb_train]
ind_valid = indices[nb_train:]

gen_train = self._build_train_generator(
indices=ind_train,
batch_size=self.batch_size,
shuffle=True
)
gen_valid = self._build_train_generator(
indices=ind_valid,
batch_size=self.batch_size,
shuffle=True
)
self.model.fit_generator(
gen_train,
steps_per_epoch=get_nb_minibatches(nb_train, self.batch_size),
epochs=1,
max_queue_size=16,
workers=1,
use_multiprocessing=True,
validation_data=gen_valid,
validation_steps=get_nb_minibatches(nb_valid, self.batch_size),
verbose=1
)

return self.model.predict_generator(
gen_test,
steps=get_nb_minibatches(nb_test, self.batch_size),
max_queue_size=16,
workers=1,
use_multiprocessing=True,
verbose=0
)

shuffle=False):
indices = indices.copy()
nb = len(indices)
X = np.zeros((batch_size, 28, 28, 1))
Y = np.zeros((batch_size, 10))
while True:
if shuffle:
np.random.shuffle(indices)
for start in range(0, nb, batch_size):
stop = min(start + batch_size, nb)
# load the next minibatch in memory.
# The size of the minibatch is (stop - start),
# which is batch_size for the all except the last
# minibatch, which can either be batch_size if
# nb is a multiple of batch_size, or nb % batch_size.
bs = stop - start
Y[:] = 0
for i, img_index in enumerate(indices[start:stop]):
x = self._transform(x)
X[i] = x
Y[i, y] = 1
yield X[:bs], Y[:bs]

X = np.zeros((batch_size, 28, 28, 1))
while True:
for start in range(0, nb, batch_size):
stop = min(start + batch_size, nb)
# load the next minibatch in memory.
# The size of the minibatch is (stop - start),
# which is batch_size for the all except the last
# minibatch, which can either be batch_size if
# nb is a multiple of batch_size, or nb % batch_size.
bs = stop - start
for i, img_index in enumerate(range(start, stop)):
x = self._transform(x)
X[i] = x
yield X[:bs]

def _transform(self, x):
x = np.expand_dims(x, axis=-1)
x = x / 255.
return x
Overwriting submissions/keras_generator/image_classifier.py

### A simple pytorch convnetÂ¶

In [10]:
%%file submissions/pytorch_convnet/image_classifier.py
from __future__ import division
import time
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

is_cuda = torch.cuda.is_available()

def _make_variable(X):
variable = Variable(torch.from_numpy(X))
if is_cuda:
variable = variable.cuda()
return variable

def _flatten(x):
return x.view(x.size(0), -1)

class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.block1 = nn.Sequential(
nn.ReLU(inplace=True),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.fc = nn.Sequential(
nn.Linear(32 * 14 * 14, 512),
nn.ReLU(True),
nn.Linear(512, 10),
)
self._initialize_weights()

def forward(self, x):
x = self.block1(x)
x = _flatten(x)
x = self.fc(x)
return x

def _initialize_weights(self):
# Source: https://github.com/pytorch/vision/blob/master/torchvision/
# models/vgg.py
for m in self.modules():
if isinstance(m, nn.Conv2d):
n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2. / n))
if m.bias is not None:
m.bias.data.zero_()
elif isinstance(m, nn.Linear):
n = m.weight.size(1)
m.weight.data.normal_(0, 0.01)
m.bias.data.zero_()

class ImageClassifier(object):

def __init__(self):
self.net = Net()
if is_cuda:
self.net = self.net.cuda()

def _transform(self, x):
# adding channel dimension at the first position
x = np.expand_dims(x, axis=0)
# bringing input between 0 and 1
x = x / 255.
return x

def _get_acc(self, y_pred, y_true):
y_pred = y_pred.cpu().data.numpy().argmax(axis=1)
y_true = y_true.cpu().data.numpy()
return (y_pred == y_true)

n_minibatch_images = len(indexes)
X = np.zeros((n_minibatch_images, 1, 28, 28), dtype=np.float32)
# one-hot encoding of the labels to set NN target
y = np.zeros(n_minibatch_images, dtype=np.int)
X[i] = self._transform(x)
# since labels are [0, ..., 9], label is the same as label index
X = _make_variable(X)
y = _make_variable(y)
return X, y

n_minibatch_images = len(indexes)
X = np.zeros((n_minibatch_images, 1, 28, 28), dtype=np.float32)
X[i] = self._transform(x)
X = _make_variable(X)
return X

validation_split = 0.1
batch_size = 32
nb_epochs = 1
lr = 1e-4
criterion = nn.CrossEntropyLoss().cuda()
if is_cuda:
criterion = criterion.cuda()

for epoch in range(nb_epochs):
t0 = time.time()
self.net.train()  # train mode
nb_trained = 0
train_loss = []
train_acc = []
n_images = len(img_loader) * (1 - validation_split)
i = 0
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
# zero-out the gradients because they accumulate by default
y_pred = self.net(X)
loss = criterion(y_pred, y)
optimizer.step()  # update params

# Loss and accuracy
train_acc.extend(self._get_acc(y_pred, y))
train_loss.append(loss.data[0])
nb_trained += X.size(0)
if nb_updates % 100 == 0:
print(
'Epoch [{}/{}], [trained {}/{}], avg_loss: {:.4f}'
', avg_train_acc: {:.4f}'.format(
epoch + 1, nb_epochs, nb_trained, n_images,
np.mean(train_loss), np.mean(train_acc)))

self.net.eval()  # eval mode
valid_acc = []
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
y_pred = self.net(X)
valid_acc.extend(self._get_acc(y_pred, y))

delta_t = time.time() - t0
print('Finished epoch {}'.format(epoch + 1))
print('Time spent : {:.4f}'.format(delta_t))
print('Train acc : {:.4f}'.format(np.mean(train_acc)))
print('Valid acc : {:.4f}'.format(np.mean(valid_acc)))

# We need to batch load also at test time
batch_size = 32
i = 0
y_proba = np.empty((n_images, 10))
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
y_proba[indexes] = nn.Softmax()(self.net(X)).cpu().data.numpy()
return y_proba
Overwriting submissions/pytorch_convnet/image_classifier.py

In [20]:
%%file submissions/pytorch_convnet_parallel/image_classifier.py
from __future__ import division
import time
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

is_cuda = torch.cuda.is_available()

def _make_variable(X):
variable = Variable(torch.from_numpy(X))
if is_cuda:
variable = variable.cuda()
return variable

def _flatten(x):
return x.view(x.size(0), -1)

class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.block1 = nn.Sequential(
nn.ReLU(inplace=True),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.fc = nn.Sequential(
nn.Linear(32 * 14 * 14, 512),
nn.ReLU(True),
nn.Linear(512, 10),
)
self._initialize_weights()

def forward(self, x):
x = self.block1(x)
x = _flatten(x)
x = self.fc(x)
return x

def _initialize_weights(self):
# Source: https://github.com/pytorch/vision/blob/master/torchvision/
# models/vgg.py
for m in self.modules():
if isinstance(m, nn.Conv2d):
n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2. / n))
if m.bias is not None:
m.bias.data.zero_()
elif isinstance(m, nn.Linear):
n = m.weight.size(1)
m.weight.data.normal_(0, 0.01)
m.bias.data.zero_()

class ImageClassifier(object):

def __init__(self):
self.net = Net()
if is_cuda:
self.net = self.net.cuda()

def _transform(self, x):
# adding channel dimension at the first position
x = np.expand_dims(x, axis=0)
# bringing input between 0 and 1
x = x / 255.
return x

def _get_acc(self, y_pred, y_true):
y_pred = y_pred.cpu().data.numpy().argmax(axis=1)
y_true = y_true.cpu().data.numpy()
return (y_pred == y_true)

transforms = [{'name': 'rotate', 'l_angle': -30, 'u_angle': 30}]
X = np.array([self._transform(x) for x in X], dtype=np.float32)
X = _make_variable(X)
y = np.array(y)
y = _make_variable(y)
return X, y

X = np.array([self._transform(x) for x in X], dtype=np.float32)
X = _make_variable(X)
return X

validation_split = 0.1
batch_size = 32
nb_epochs = 1
lr = 1e-4
criterion = nn.CrossEntropyLoss().cuda()
if is_cuda:
criterion = criterion.cuda()

for epoch in range(nb_epochs):
t0 = time.time()
self.net.train()  # train mode
nb_trained = 0
train_loss = []
train_acc = []
n_images = len(img_loader) * (1 - validation_split)
i = 0
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
# zero-out the gradients because they accumulate by default
y_pred = self.net(X)
loss = criterion(y_pred, y)
optimizer.step()  # update params

# Loss and accuracy
train_acc.extend(self._get_acc(y_pred, y))
train_loss.append(loss.data[0])
nb_trained += X.size(0)
if nb_updates % 100 == 0:
print(
'Epoch [{}/{}], [trained {}/{}], avg_loss: {:.4f}'
', avg_train_acc: {:.4f}'.format(
epoch + 1, nb_epochs, nb_trained, n_images,
np.mean(train_loss), np.mean(train_acc)))

self.net.eval()  # eval mode
valid_acc = []
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
y_pred = self.net(X)
valid_acc.extend(self._get_acc(y_pred, y))

delta_t = time.time() - t0
print('Finished epoch {}'.format(epoch + 1))
print('Time spent : {:.4f}'.format(delta_t))
print('Train acc : {:.4f}'.format(np.mean(train_acc)))
print('Valid acc : {:.4f}'.format(np.mean(valid_acc)))

batch_size = 32
i = 0
y_proba = np.empty((n_images, 10))
while i < n_images:
indexes = range(i, min(i + batch_size, n_images))
i += len(indexes)
y_proba[indexes] = nn.Softmax()(self.net(X)).cpu().data.numpy()
return y_proba
Overwriting submissions/pytorch_convnet_parallel/image_classifier.py

## Local testing (before submission)Â¶

It is important that you test your submission files before submitting them. For this we provide a unit test. Note that the test runs on your files in submissions/starting_kit, not on the classes defined in the cells of this notebook.

First pip install ramp-workflow or install it from the github repo. Make sure that the python file image_classifier.py is in the submissions/starting_kit folder, and the data train.csv and test.csv are in data. If you haven't yet, download the images by executing python download_data.py. Then run

ramp_test_submission

If it runs and print training and test errors on each fold, then you can submit the code.

In [1]:
!ramp_test_submission
Testing MNIST classification
Reading train and test files from ./data ...
Training ./submissions/starting_kit ...
Using TensorFlow backend.
Train on 34560 samples, validate on 3840 samples
Epoch 1/1
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
34560/34560 [==============================] - 3s - loss: 2.3441 - acc: 0.1071 - val_loss: 2.2978 - val_acc: 0.1289
CV fold 0
train acc = 0.132
valid acc = 0.132
test acc = 0.129
train nll = 2.294
valid nll = 2.294
test nll = 2.293
----------------------------
train acc = 0.132 ± 0.0
train nll = 2.294 ± 0.0
valid acc = 0.132 ± 0.0
valid nll = 2.294 ± 0.0
test acc = 0.129 ± 0.0
test nll = 2.293 ± 0.0

## Submitting to ramp.studioÂ¶

Once you found a good feature extractor and classifier, you can submit them to ramp.studio. First, if it is your first time using RAMP, sign up, otherwise log in. Then find an open event on the particular problem, for example, the event MNIST for this RAMP. Sign up for the event. Both signups are controled by RAMP administrators, so there can be a delay between asking for signup and being able to submit.

Once your signup request is accepted, you can go to your sandbox and copy-paste (or upload) image_preprocessor.py and batch_classifier.py from submissions/starting_kit. Save it, rename it, then submit it. The submission is trained and tested on our backend in the same way as ramp_test_submission does it locally. While your submission is waiting in the queue and being trained, you can find it in the "New submissions (pending training)" table in my submissions. Once it is trained, you get a mail, and your submission shows up on the public leaderboard. If there is an error (despite having tested your submission locally with ramp_test_submission), it will show up in the "Failed submissions" table in my submissions. You can click on the error to see part of the trace.

After submission, do not forget to give credits to the previous submissions you reused or integrated into your submission.

The data set we use at the backend is usually different from what you find in the starting kit, so the score may be different.

The usual way to work with RAMP is to explore solutions, add feature transformations, select models, perhaps do some AutoML/hyperopt, etc., locally, and checking them with ramp_test_submission. The script prints mean cross-validation scores

----------------------------
train acc = 0.132 ± 0.0
train nll = 2.294 ± 0.0
valid acc = 0.132 ± 0.0
valid nll = 2.294 ± 0.0
test acc = 0.129 ± 0.0
test nll = 2.293 ± 0.0

The official score in this RAMP (the first score column after "historical contributivity" on the leaderboard) is balanced accuracy aka macro-averaged recall, so the line that is relevant in the output of ramp_test_submission is valid acc = 0.132 ± 0.0. When the score is good enough, you can submit it at the RAMP.