Inference Over IP Networks#

Important

Shared Mode functionality is currently in a Feature Preview state. The APIs will not change much going forward, but there are internal optimizations that are still being made. On most systems, we do not currently recommend exceeding 4 client applications/containers at a time.

Introduction#

One of the “bonus” features of mx_server and Shared Mode is that connections between client and server do not have to be on the same physical machine. One computer can have MXA(s) connected to it, and one or more clients can request to use them in Shared Mode, via typical TCP/IP networking over ethernet, etc.

This tutorial explains how to set up such a system.

Warning

This feature is not the intended way to use the MXA, and will have performance gaps versus running on the same physical machine. Data is sent uncompressed between client and server, so network traffic congestion can be quite severe.

However, minimal changes are required for programs to function over the network, so this tutorial is provided as an “interesting experiment” users may try.

Runtime Installation#

Both the server and the client(s) need to have the runtime installed. No special steps are necessary.

Configure Server#

Edit Address Config#

You can configure the mx_server daemon to listen either on all interfaces, or a specific network interface. As network configurations can vary greatly based on your infrastructure, we will assume you know which you want to use.

To listen on all interfaces, use 0.0.0.0 in the following config.

To listen on a specific interface, use its IP, such as 192.168.1.1 in the following config.

Edit /etc/memryx/mx_server.conf in your favorite text editor, and replace the LISTEN_ADDRESS with your chosen IP. E.g., replace this:

LISTEN_ADDRESS="127.0.0.1"

With this:

LISTEN_ADDRESS="0.0.0.0"

Edit Port Config (optional)#

By default, mx_server will use ports 50051, 50052, and 50053. You can configure the base port in the config file, then the server will use the base, base+1, and base+2 ports. For example, a base port of 1000 will use ports 1000, 1001, and 1002.

To change this config, edit the /etc/memryx/mx_server.conf file:

# Replace with your desired base port
BASE_PORT=50051

Update Firewall Rules#

Lastly, make sure your firewall allows incoming connections on your chosen ports. This is very OS and environment dependent, but if you’re using ufw on Ubuntu, the commands would be the following for the default ports:

sudo ufw allow 50051
sudo ufw allow 50052
sudo ufw allow 50053

Restart the Service#

Now that everything’s configured, restart the service:

sudo systemctl restart memx-accl.service

Connect Clients#

In your client applications, now we simply specify the IP (+port) of the server when setting up your MxAccl or MxAcclMT object, like you would for Docker.

For example, if your server is running on a computer with IP 192.168.1.100 and the default ports, use:

MxAccl* accl = new MxAccl(true, "192.168.1.100");
accl->connect_dfp("my_model.dfp");
...

And if you want to specify the base port as 1000 as well, replace the first line with:

MxAccl* accl = new MxAccl(true, "192.168.1.100", 1000);

Summary#

Congrats! You are now running inference remotely over the network, with one server and one or more clients. As previously noted, the performance will be low and network traffic quite heavy, but this feature might enable some niche applications such as mini MXA server farms.

On this page