Docker#
Important
Please be sure to read the Shared mode tutorial first so you understand the “Shared” and “Local” terminology used throughout this tutorial, as well as when to use each mode.
Introduction#
Note
Keep in mind this tutorial is for running inference on an MXA from within a container. You do not need to do any of this if you only wish to put the SDK Tools in a container!
Both Shared and Local mode can be used from within Docker container(s) by connecting to the mxa-manager running on the host over a virtual network interface.
In Shared mode, data will be sent between the clients inside Docker and the mxa-manager on the host, supporting multiple concurrent applications/processes/containers. Meanwhile in Local mode, the mxa-manager is used only for managing permissions for MXA devices, and the application communicates directly with the MXA. See the Shared Mode Tutorial above for more explanation on Shared vs. Local.
This tutorial will cover:
C++ Runtime installation within Docker
How to configure the virtual interfaces
How to use benchmark tools or Python/C++ applications from within Docker
Runtime Installation#
The runtime installation inside a container is similar, except the mxa-manager shouldn’t run inside the container and should instead be installed only on the host.
On the Host#
Only mxa-manager is actually needed on the host. The memx-accl package is optional if all your clients will come from within Docker containers.
sudo apt install mxa-manager
In the Container#
In the container, you don’t need the manager service. You can skip its installation with the --no-install-recommends flag when installing memx-accl
sudo apt install --no-install-recommends memx-accl
Configure Net Interfaces#
The approach is different depending on your Docker version.
For Local Mode (the default), you need to give the Docker container privileged access to the host’s devices via the --privileged=true flag.
Gateway Method
If you are using a recent version of Docker, you can simply add a flag to run:
--add-host host.docker.internal:host-gateway
For example,
docker run --privileged=true --add-host host.docker.internal:host-gateway -t -i ubuntu:noble /bin/bash
Now host.docker.internal will point to the host. No further network configuration should be necessary.
Network Methods / Old Dockers
If the above flag does not work in your version of Docker, you’ll have to manually configure network interfaces.
You have 2 options: host networking, or virtual interface connections.
Host Networking
If host/container separation is not important for your use case, you can simply have the container directly access the host’s network stack.
For example,
docker run --privileged=true --network="host" -t -i ubuntu:noble /bin/bash
Now the 127.0.0.1 default argument to the MxAccl constructor will point to the host. No further configuration is necessary.
Please note that this can have security implications, because any port opened in your docker container will now be opened directly on the host as well.
Virtual Interfaces
The default for Docker is for a virtual network interface to be created on the host, typically called docker0, through which data is transferred between host and container. We need to edit mxa_manager’s configuration to accept connections over any interface, not just localhost, so that traffic from docker0 is accepted.
Edit /etc/memryx/mxa_manager.conf in your favorite text editor, and replace the LISTEN_ADDRESS with 0.0.0.0:
E.g., replace this:
LISTEN_ADDRESS="127.0.0.1"
With this:
LISTEN_ADDRESS="0.0.0.0"
And restart the service:
sudo systemctl restart mxa-manager.service
mxa_manager will now listen for connections on any interface. Please note that this includes ethernet/wifi interfaces, so set your firewall rules accordingly for security.
Now you can run the container without any network-specific flags:
docker run --privileged=true -t -i ubuntu:noble /bin/bash
For Shared Mode, containers do not need to use --privileged=true, since interaction with the MXA is over network packets sent to the host’s mxa_manager process.
Gateway Method
If you are using a recent version of Docker, you can simply add a flag to run:
--add-host host.docker.internal:host-gateway
For example,
docker run --add-host host.docker.internal:host-gateway -t -i ubuntu:noble /bin/bash
Now host.docker.internal will point to the host. No further network configuration is necessary.
Network Methods / Old Dockers
If the above flag does not work in your version of Docker, you’ll have to manually configure network interfaces.
You have 2 options: host networking, or virtual interface connections.
Host Networking
If host/container separation is not important for your use case, you can simply have the container directly access the host’s network stack.
For example,
docker run --network="host" -t -i ubuntu:noble /bin/bash
Now the 127.0.0.1 default argument to the MxAccl constructor will point to the host. No further configuration is necessary.
Please note that this can have security implications, because any port opened in your docker container will now be opened directly on the host as well.
Virtual Interfaces
The default for Docker is for a virtual network interface to be created on the host, typically called docker0, through which data is transferred between host and container. We need to edit mxa_manager’s configuration to accept connections over any interface, not just localhost, so that traffic from docker0 is accepted.
Edit /etc/memryx/mxa_manager.conf in your favorite text editor, and replace the LISTEN_ADDRESS with 0.0.0.0:
E.g., replace this:
LISTEN_ADDRESS="127.0.0.1"
With this:
LISTEN_ADDRESS="0.0.0.0"
And restart the service:
sudo systemctl restart mxa-manager.service
mxa_manager will now listen for connections on any interface. Please note that this includes ethernet/wifi interfaces, so set your firewall rules accordingly for security.
Now you can run the container without any network-specific flags:
docker run -t -i ubuntu:noble /bin/bash
Usage#
Just like applications running directly on the host, in Docker containers C++ can use either Shared or Local mode, while Python can only use Local.
Use the Right Address#
In all the examples below, we will need to use the correct address to connect to the host’s mxa_manager. We shall use $ADDR in all examples below to represent this value.
$ADDR is found differently depending on the configuration method you used above.
Recent Docker
$ADDR = host.docker.internal
Old Docker –> Host Networking
$ADDR = localhost
Old Docker –> Virtual Interfaces
First, start your docker container and access a shell (/bin/bash) inside it. Run this command to get the virtual interface’s address:
ip route
In the output, look for the line that starts with “default”, e.g.:
default via 172.17.42.1 dev enp5s0 proto dhcp src 172.17.1.192 metric 100
We want the IP after the via keyword, which in this case is 172.17.42.1.
Use this IP for your applications, i.e. $ADDR = 172.17.42.1
Use Benchmark Tools#
To use mx_bench from a container, simply supply your $ADDR address to the --mxserver_addr argument:
mx_bench --mxserver_addr $ADDR -f 500 -d MyModel.dfp
Similarly, to use acclBench from a container, give $ADDR to the --server_addr/-s argument:
acclBench -s $ADDR -f 500 -d MyModel.dfp
acclBench is capable of using Shared Mode, allowing for multiple instances of it to run in parallel. To do so, just add the --shared_mode/-r flag and the $ADDR address to --server_addr/-s:
acclBench -s $ADDR -f 500 -d MyModel.dfp -r
Use in Applications#
In Local mode, just change the mxa_manager address in the C++ or Python APIs to $ADDR.
For Python:
accl = AsyncAccl(dfp="my_model.dfp", mxserver_addr=$ADDR)
For C++ (note the first argument is use_shared=false, to indicate Local mode):
MxAccl* accl = new MxAccl(false, $ADDR);
accl->connect_dfp("my_model.dfp");
For Shared mode, which is only supported for C++ applications, simply use:
MxAccl* accl = new MxAccl(true, $ADDR);
accl->connect_dfp("my_model.dfp");
This is the same as the Local mode except for ‘true’ instead of ‘false’.
Your client should now be able to successfully connect to the mxa_manager process running on the host! Please note the considerations regarding performance and “same DFP” detection when using Shared Mode apply whether run inside a container or on the host.