Classification Accuracy Calculation#

Introduction#

We will use the Neural Compiler and the MXA to verify the accuracy of the MXA chip against the MLPerf™ Inference Benchmark Suite. In this tutorial we will use the resnet50-v1.5 tensorflow model from MLCommons Repository and the Imagenet2012 Validation dataset.

Note

This tutorial assumes a four chip solution is correctly connected.

The following section provides a step-by-step explanation of the necessary steps to run the application successfully.

1. Download and Compile the Model#

The resnet50-v1.5 tensorflow model is available on the MLCommons GitHub Page. The model can be directly downloaded with this link resnet50-v1.5.pb.

API

from memryx import NeuralCompiler
nc = NeuralCompiler(num_chips=4, models='resnet50_v1.pb', verbose=1, dfp_fname = 'resnet50_v1', autocrop=True)
dfp = nc.run()

CLI Tool

In your command line, you need to type,

mx_nc -v -m resnet50_v1.pb --autocrop -c 4

This will produce a DFP file ready to be used by the accelerator. In your Python code, you need to point the dfp variable to the generated file path,

dfp = 'resnet50_v1.dfp'

Python Pre-Compiled DFP

In your Python code, you need to point the dfp variable to the generated file path,

dfp = 'resnet50_v1.dfp'

2. Download ImageNet Dataset#

To access the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) dataset, sign up and create an account on the ImageNet Org website. After agreeing to the terms and conditions for using the images, the following steps can be used to download the ImageNet2012 Validation Dataset:

From the project folder, navigate to the following directory to store the ImageNet Validation dataset.

cd assets/ImageNet2012_valdata
mkdir images && cd images

Download and extract the ImageNet Validation images into ImageNet2012_valdata/images. Then, look at the number of images to verify that 50000 images were extracted.

wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
tar xvf ILSVRC2012_img_val.tar
rm ILSVRC2012_img_val.tar

3. Image loading And Preprocessing#

Now, let’s write some Python code to load the images into numpy arrays and preprocess the images. This will prepare them for inferencing. The pre-processing code has been recycled from the MlCommons GitHub page. Below is how we import the libraries, load the data and pre-process it.:

import os, sys, subprocess, tarfile, glob, argparse
import cv2
import numpy as np

self.model_path   = model_path
self.input_shape  = (224,224,3)
self.dataset_path = dataset_path

def load_images_and_labels(self, start, end):
    """
    Loads images and their respective labels
    """
    images = []
    labels = []
    for fname in self.image_paths[start:end]:
        img = cv2.imread(fname)
        preprocessed_img = pre_process_vgg(img, self.input_shape)
        preprocessed_img = np.expand_dims(preprocessed_img, 0)
        images.append(preprocessed_img)
        # get ground_truth labels!
        # ILSVRC2012_val_xxxxxxxx -> int(xxxxxxxx) - 1 ; as numbering starts from 1
        label = self.ground_truth[int(os.path.split(fname)[1].split('_')[2].split('.')[0]) - 1]
        labels.append(label)
    return images, labels

########################################
# Helper functions for Pre-processing
#######################################
# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def center_crop(img, out_height, out_width):
    height, width, _ = img.shape
    left = int((width - out_width) / 2)
    right = int((width + out_width) / 2)
    top = int((height - out_height) / 2)
    bottom = int((height + out_height) / 2)
    img = img[top:bottom, left:right]
    return img

# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def resize_with_aspectratio(img, out_height, out_width, scale=87.5, inter_pol=cv2.INTER_LINEAR):
    height, width, _ = img.shape
    new_height = int(100. * out_height / scale)
    new_width = int(100. * out_width / scale)
    if height > width:
        w = new_width
        h = int(new_height * height / width)
    else:
        h = new_height
        w = int(new_width * width / height)
    img = cv2.resize(img, (w, h), interpolation=inter_pol)
    return img

# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def pre_process_vgg(img, dims=None, need_transpose=False):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    output_height, output_width, _ = dims
    cv2_interpol = cv2.INTER_AREA
    img = resize_with_aspectratio(img, output_height, output_width, inter_pol=cv2_interpol)
    img = center_crop(img, output_height, output_width)
    img = np.asarray(img, dtype='float32')

    # normalize image
    means = np.array([123.68, 116.78, 103.94], dtype=np.float32)
    img -= means

    # transpose if needed
    if need_transpose:
        img = img.transpose([2, 0, 1])
    return img

images, labels = self.load_images_and_labels(start=start, end=end)

The load_images_and_labels function will load the 50000 ImageNet2012 validation images along with their corresponding labels.

4. Run inference#

Next, let’s run inference on the MXA so we can compare the results with the MLPerf Benchmark! The Async API is the straightforward way to best utilize the MX3. You need only to connect an input and an output function, and the API will handle the threading and the data streaming under the hood.

from tqdm import tqdm
from memryx import NeuralCompiler, AsyncAccl

def get_frame():
    return next(img_iter, None)

def process_output(*outputs):
   mxa_outputs.extend(outputs[0])
accl = AsyncAccl(dfp_path)

accl.connect_input(get_frame)
accl.connect_output(process_output)
accl.wait()

5. Predictions#

The output from the MXA is passed to the postprocess function to get predictions:

def postprocess(self, outputs):
    outputs = outputs[:,1:]
    preds = decode_predictions(outputs, top=5)
    return preds

mxa_outputs = np.stack([np.squeeze(arr) for arr in mxa_outputs])
mxa_predictions = self.postprocess(mxa_outputs)

Finally, let’s compare the prediction results to the ground truth.

def get_accuracy(self, predictions, ground_truth):
    top1, top5, total = 0, 0, len(predictions)
    for i,pred in enumerate(predictions):
        gt = ground_truth[i]

        classes = [guess[0] for guess in pred]
        if gt in classes:
            top5 += 1
        if gt == classes[0]:
            top1 += 1

    print("Top 1: ({}/{})  {:.2f} % ".format(top1, total, top1/total*100))
    print("Top 5: ({}/{})  {:.2f} % ".format(top5, total, top5/total*100))

self.get_accuracy(predictions, groundtruth)

After running the script, the obtained Top-1 accuracy = 76.23% which is comparable to the accuracy from the MLPerf Benchmark Accuracy which is 76.456%

6. Summary#

This tutorial demonstrates how to replicate the MLPerf Benchmark Accuracy with MXA chip and the ImageNet Dataset. The full script is available for download. Please refer to the README.md file for guidance on how to run the provided code.