Docker

Important

Be sure to read the documentation on MXA-Manager and Shared / Local modes before configuring for Docker!

Introduction

Note

Keep in mind this tutorial is for running inference on an MXA from within a container. You do not need to do any of this if you only wish to put the NeuralCompiler in a container!

Both Shared and Local mode can be used from within Docker container(s) by connecting to the mxa-manager running on the host. Although TCP/IP networking can be used to connect, UNIX Domain Sockets are an easier and more secure option, and will be the focus of this tutorial.

This tutorial will cover:

1. Runtime installation within Docker (without a duplicate mxa-manager)

2. How to share the UNIX socket file with the container

3. How to use benchmark tools or Python/C++ applications from within Docker

Runtime Installation

The runtime installation inside a container is similar, except the mxa-manager shouldn’t run inside the container and should instead be installed only on the host.

On the Host

Both mxa-manager and memx-accl are needed to be installed on the host:

sudo apt install memx-accl mxa-manager

In the Container

In the container, you don’t need the manager service. You can skip its installation with the --no-install-recommends flag when installing memx-accl

sudo apt install --no-install-recommends memx-accl

Connect UNIX Socket

Assuming you are using the default path in mxa-manager.conf, you simply need to mount the host’s folder as a volume in the container.

If using the docker run command, you can do this with the -v flag:

docker run -v /run/mxa_manager:/run/mxa_manager my_container

If using a Docker Compose file, you can do this with the volumes section:

services:
  my_service:
    image: my_container
    volumes:
      - /run/mxa_manager:/run/mxa_manager

Usage

Once set up, usage within the container is the same as on the host, with a few considerations.

–privileged

If you plan to use Local Mode in the container, you will need to run the container with the --privileged flag in order for the application to directly use the host MXA hardware.

Important

--privileged is not needed for Shared Mode (which is the default), as the mxa_manager on the host handles all hardware access.

Use Benchmark Tools

It’s important to note that the benchmark tools are designed to run in Local Mode by default, which would need the --privileged flag as described above.

So be use to use the -s flag to run them in Shared Mode if you do not want to run the container with --privileged.

Shared Mode

mx_bench -f 500 -d MyModel.dfp -s

or

acclBench -f 500 -d MyModel.dfp -s

Local Mode

mx_bench -f 500 -d MyModel.dfp

or

acclBench -f 500 -d MyModel.dfp

Use in Applications

You can use the Python or C++ APIs in your application as you normally would, since the default address path in the constructors (/run/mxa_manager/) is the same.

accl = AsyncAccl("my_model.dfp")

MxAccl* accl = new MxAccl("my_model.dfp");