Docker#

Important

Please be sure to read the Shared mode tutorial first so you understand the “Shared” and “Local” terminology used throughout this tutorial, as well as when to use each mode.

Introduction#

Note

Keep in mind this tutorial is for running inference on an MXA from within a container. You do not need to do any of this if you only wish to put the SDK Tools in a container!

Both Shared and Local mode can be used from within Docker container(s) by connecting to the mx_server running on the host over a virtual network interface.

In Shared mode, data will be sent between the clients inside Docker and the server on the host, supporting multiple concurrent applications/processes/containers. Meanwhile in Local mode, the server is used only for managing permissions for MXA devices, and the application communicates directly with the MXA. See the Shared Mode Tutorial above for more explanation on Shared vs. Local.

This tutorial will cover:

  1. C++ Runtime installation within Docker

  2. How to configure the virtual interfaces

  3. How to use benchmark tools or Python/C++ applications from within Docker

Runtime Installation#

The runtime installation inside a container is exactly the same as installation on the host PC. Be sure to install the runtimes (driver and accl) on both the host OS and the container.

Configure Net Interfaces#

The approach is different depending on your Docker version.

For Local Mode (the default), you need to give the Docker container privileged access to the host’s devices via the --privileged=true flag.

New Docker (20.10.0+)

If you are using a recent version of Docker, you can simply add a flag to run:

--add-host host.docker.internal:host-gateway

For example,

docker run --privileged=true --add-host host.docker.internal:host-gateway -t -i ubuntu:noble /bin/bash

Now host.docker.internal will point to the host. No further network configuration is necessary.

Old Docker

If the above flag does not work in your version of Docker, you’ll have to manually configure network interfaces.

You have 2 options: host networking, or virtual interface connections.

Host Networking

If host/container separation is not important for your use case, you can simply have the container directly access the host’s network stack.

For example,

docker run --privileged=true --network="host" -t -i ubuntu:noble /bin/bash

Now the 127.0.0.1 default argument to the MxAccl constructor will point to the host. No further configuration is necessary.

Please note that this can have security implications, because any port opened in your docker container will now be opened directly on the host as well.

Virtual Interfaces

The default for Docker is for a virtual network interface to be created on the host, typically called docker0, through which data is transferred between host and container. We need to edit mx_server’s configuration to accept connections over any interface, not just localhost, so that traffic from docker0 is accepted.

Edit /etc/memryx/mx_server.conf in your favorite text editor, and replace the LISTEN_ADDRESS with 0.0.0.0:

E.g., replace this:

LISTEN_ADDRESS="127.0.0.1"

With this:

LISTEN_ADDRESS="0.0.0.0"

And restart the service:

sudo systemctl restart memx-accl.service

mx_server will now listen for connections on any interface. Please note that this includes ethernet/wifi interfaces, so set your firewall rules accordingly for security.

Now you can run the container without any network-specific flags:

docker run --privileged=true -t -i ubuntu:noble /bin/bash

For Shared Mode, containers do not need to use --privileged=true, since interaction with the MXA is over network packets sent to the host’s mx_server process.

New Docker (20.10.0+)

If you are using a recent version of Docker, you can simply add a flag to run:

--add-host host.docker.internal:host-gateway

For example,

docker run --add-host host.docker.internal:host-gateway -t -i ubuntu:noble /bin/bash

Now host.docker.internal will point to the host. No further network configuration is necessary.

Old Docker

If the above flag does not work in your version of Docker, you’ll have to manually configure network interfaces.

You have 2 options: host networking, or virtual interface connections.

Host Networking

If host/container separation is not important for your use case, you can simply have the container directly access the host’s network stack.

For example,

docker run --network="host" -t -i ubuntu:noble /bin/bash

Now the 127.0.0.1 default argument to the MxAccl constructor will point to the host. No further configuration is necessary.

Please note that this can have security implications, because any port opened in your docker container will now be opened directly on the host as well.

Virtual Interfaces

The default for Docker is for a virtual network interface to be created on the host, typically called docker0, through which data is transferred between host and container. We need to edit mx_server’s configuration to accept connections over any interface, not just localhost, so that traffic from docker0 is accepted.

Edit /etc/memryx/mx_server.conf in your favorite text editor, and replace the LISTEN_ADDRESS with 0.0.0.0:

E.g., replace this:

LISTEN_ADDRESS="127.0.0.1"

With this:

LISTEN_ADDRESS="0.0.0.0"

And restart the service:

sudo systemctl restart memx-accl.service

mx_server will now listen for connections on any interface. Please note that this includes ethernet/wifi interfaces, so set your firewall rules accordingly for security.

Now you can run the container without any network-specific flags:

docker run -t -i ubuntu:noble /bin/bash

Usage#

Just like applications running directly on the host, in Docker containers C++ can use either Shared or Local mode, while Python can only use Local.

Use the Right Address#

In all the examples below, we will need to use the correct address to connect to the host’s mx_server. We shall use $ADDR in all examples below to represent this value.

$ADDR is found differently depending on the configuration method you used above.

Recent Docker

$ADDR = host.docker.internal

Old Docker –> Host Networking

$ADDR = localhost

Old Docker –> Virtual Interfaces

First, start your docker container and access a shell (/bin/bash) inside it. Run this command to get the virtual interface’s address:

ip route

In the output, look for the line that starts with “default”, e.g.:

default via 172.17.42.1 dev enp5s0 proto dhcp src 172.17.1.192 metric 100

We want the IP after the via keyword, which in this case is 172.17.42.1.

Use this IP for your applications, i.e. $ADDR = 172.17.42.1

Use Benchmark Tools#

To use mx_bench from a container, simply supply your $ADDR address to the --mxserver_addr argument:

mx_bench --mxserver_addr $ADDR -f 500 -d MyModel.dfp

Similarly, to use acclBench from a container, give $ADDR to the --server_addr/-s argument:

acclBench -s $ADDR -f 500 -d MyModel.dfp

acclBench is capable of using Shared Mode, allowing for multiple instances of it to run in parallel. To do so, just add the --shared_mode/-r flag and the $ADDR address to --server_addr/-s:

acclBench -s $ADDR -f 500 -d MyModel.dfp -r

Use in Applications#

In Local mode, just change the mx_server address in the C++ or Python APIs to $ADDR.

For Python:

accl = AsyncAccl(dfp="my_model.dfp", mxserver_addr=$ADDR)

For C++ (note the first argument is use_shared=false, to indicate Local mode):

MxAccl* accl = new MxAccl(false, $ADDR);
accl->connect_dfp("my_model.dfp");

For Shared mode, which is only supported for C++ applications, simply use:

MxAccl* accl = new MxAccl(true, $ADDR);
accl->connect_dfp("my_model.dfp");

This is the same as the Local mode except for ‘true’ instead of ‘false’.

Your client should now be able to successfully connect to the mx_server process running on the host! Please note the considerations regarding performance and “same DFP” detection when using Shared Mode apply whether run inside a container or on the host.

On this page