Classification Accuracy Calculation#

Introduction#

We will use the Neural Compiler and the MXA to verify the accuracy of the MXA chip against the MLPerf™ Inference Benchmark Suite. In this tutorial we will use the resnet50-v1.5 tensorflow model from MLCommons Repository and the Imagenet2012 Validation dataset

Note

This tutorial assumes a four chip solution is correctly connected.

Download and Compile the Model#

The resnet50-v1.5 tensorflow model is available on the MLCommons GitHub Page. The model can be directly downloaded with this link resnet50-v1.5.pb.

from memryx import NeuralCompiler
nc = NeuralCompiler(num_chips=4, models='resnet50_v1.pb', verbose=1, dfp_fname = 'resnet50_v1', autocrop=True)
dfp = nc.run()

In your command line, you need to type,

mx_nc -v -m resnet50_v1.pb --autocrop -c 4

This will produce a DFP file ready to be used by the accelerator. In your Python code, you need to point the dfp variable to the generated file path,

dfp = 'resnet50_v1.dfp'

In your Python code, you need to point the dfp variable to the generated file path,

dfp = 'resnet50_v1.dfp'

Note

You can use the pre-compiled dfp from the resnet50_v1.zip and skip the compilation step. Please, make sure to unzip the file and include the resnet50_v1.dfp in your working folder.

Download ImageNet Dataset#

To access the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) dataset, sign up and create an account on the ImageNet Org website. After agreeing to the terms and conditions for using the images, the following steps can be used to download the ImageNet2012 Validation Dataset

  1. Create a directory to store the ImageNet Validation dataset.

mkdir ImageNet2012_valdata && cd ImageNet2012_valdata
mkdir images && cd images                               # now in ImageNet2012_valdata/images

  1. Download and extract the ImageNet Validation images into ImageNet2012_valdata/images. Then count the number of images to verify that 50000 images were extracted.

wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
tar xvf ILSVRC2012_img_val.tar
ls -1 *.JPEG | wc -l  # should return 50000

  1. You can download the ground_truth file from the ground_truth to this tutorial. Please, make sure to include it in your working folder.

Image loading And Preprocessing#

Now, let’s write some python code to load the images into a numpy arrays and preprocess the images. This will prepare them for inferencing. The pre-processing code has been recycled from the MlCommons GitHub page. In a python file add the following lines:

import cv2, glob, os
import numpy as np

input_shape = (224,224,3)
# Make sure that path correctly points to the downloaded dataset
imagenet_path = 'ImageNet2012_valdata/images'
# # ground truth annotations path
ground_truth_path = 'ground_truth'
################################################################################
# Loading Imagenet Dataset And Preprocessing
################################################################################
def load_images_and_labels():

    with open(ground_truth_path) as f:
        ground_truth = f.read().split('\n')
    image_paths = glob.glob(imagenet_path+'/*.JPEG')
    images = []
    labels = []
    for fname in image_paths:
        img = cv2.imread(fname)
        preprocessed_img = pre_process_vgg(img, input_shape)
        images.append(preprocessed_img)
        # get ground_truth labels!
        # ILSVRC2012_val_xxxxxxxx -> int(xxxxxxxx) - 1 ; as numbering starts from 1
        label = ground_truth[int(os.path.split(fname)[1].split('_')[2].split('.')[0]) - 1]
        labels.append(label)
    return images, labels
########################################
# Helper functions for Pre-processing
#######################################
# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def center_crop(img, out_height, out_width):
    height, width, _ = img.shape
    left = int((width - out_width) / 2)
    right = int((width + out_width) / 2)
    top = int((height - out_height) / 2)
    bottom = int((height + out_height) / 2)
    img = img[top:bottom, left:right]
    return img

# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def resize_with_aspectratio(img, out_height, out_width, scale=87.5, inter_pol=cv2.INTER_LINEAR):
    height, width, _ = img.shape
    new_height = int(100. * out_height / scale)
    new_width = int(100. * out_width / scale)
    if height > width:
        w = new_width
        h = int(new_height * height / width)
    else:
        h = new_height
        w = int(new_width * width / height)
    img = cv2.resize(img, (w, h), interpolation=inter_pol)
    return img

# from MlCommons GitHub MLCommons/inference/vision/classification_and_detection/python/dataset.py
def pre_process_vgg(img, dims=None, need_transpose=False):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    output_height, output_width, _ = dims
    cv2_interpol = cv2.INTER_AREA
    img = resize_with_aspectratio(img, output_height, output_width, inter_pol=cv2_interpol)
    img = center_crop(img, output_height, output_width)
    img = np.asarray(img, dtype='float32')

    # normalize image
    means = np.array([123.68, 116.78, 103.94], dtype=np.float32)
    img -= means

    # transpose if needed
    if need_transpose:
        img = img.transpose([2, 0, 1])
    return img

#################################################################################
# load data and corresponding ground_truth labels
################################################################################
images, labels = load_images_and_labels()

The load_images_and_labels function will load the 50000 ImageNet2012 validation images along with their corresponding labels.

Note

We use the OpenCV library to load the JPEG images. You can install this library by running pip install opencv-python on the command line.

Run inference#

Next, let’s run inference on the MXA so we can compare the results with the MLPerf Benchmark! The Async API is the straightforward way to best utilize the MX3. You need only to connect an input and an output function, and the API will handle the threading and the data streaming under the hood.

################################################################################
# Async API (Pipelined)
################################################################################
from memryx import AsyncAccl
# MXA run
img_iter = iter(images)
mxa_outputs = []
def get_frame():
    return next(img_iter, None)

def process_output(*outputs):
    mxa_outputs.append(np.squeeze(outputs[0], 0))

accl = AsyncAccl(dfp='resnet50_v1.dfp')

accl.connect_input(get_frame)
accl.connect_output(process_output)
accl.wait()

Predictions#

Let’s pass the output from the MXA through the postprocess function to get the predictions.

def postprocess(outputs):
    from tensorflow.keras.applications.resnet50 import decode_predictions
    outputs = outputs[:,1:]
    preds = decode_predictions(outputs, top=5)
    return preds

mxa_outputs = np.stack([np.squeeze(arr) for arr in mxa_outputs])
mxa_predictions = postprocess(mxa_outputs)

Finally, let’s compare the prediction results to the ground truth.

def get_accuracy(predictions, ground_truth):
    top1, top5, total = 0, 0, len(predictions)
    for i,pred in enumerate(predictions):
        gt = ground_truth[i]

        classes = [guess[0] for guess in pred]
        if gt in classes:
            top5 += 1
        if gt == classes[0]:
            top1 += 1

    print("Top 1: ({}/{})  {:.2f} % ".format(top1, total, top1/total*100))
    print("Top 5: ({}/{})  {:.2f} % ".format(top5, total, top5/total*100))
print(f"Accuracy from MXA:")
get_accuracy(mxa_predictions, labels)

After running the script, the obtained Top-1 accuracy = 76.23% which is comparable to the accuracy from the MLPerf Benchmark Accuracy which is 76.456%

Summary#

This tutorial demonstrates how to replicate the MLPerf Benchmark Accuracy with MXA chip and the ImageNet Dataset. The full script is available for mlperf_resnet50v1_5.py

See also

On this page