Accelerator API (Python-wrapped MxAccl)#

Auto-Threading Mode

class MxAccl#

Python binding of the MxAccl class (auto-threading)

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: memryx.mxapi.MxAccl, dfp_path: os.PathLike, device_ids_to_use: list[int] = [0], use_model_shape: Annotated[list[bool], FixedSize(2)] = [True, True], local_mode: bool = False, sched_options: memryx.mxapi.SchedulerOptions = <memryx.mxapi.SchedulerOptions object at 0x7fa8be441d30>, client_options: memryx.mxapi.ClientOptions = <memryx.mxapi.ClientOptions object at 0x7fa8c0bd2530>, server_addr: str = ‘/run/mxa_manager/’, server_port_base: int = 10000, ignore_server: bool = False) -> None
__init__(self: memryx.mxapi.MxAccl, dfp_bytes: bytes, dfp_bytes_size: int, device_ids_to_use: list[int] = [0], use_model_shape: Annotated[list[bool], FixedSize(2)] = [True, True], local_mode: bool = False, sched_options: memryx.mxapi.SchedulerOptions = <memryx.mxapi.SchedulerOptions object at 0x7fa8be441ff0>, client_options: memryx.mxapi.ClientOptions = <memryx.mxapi.ClientOptions object at 0x7fa8be441fb0>, server_addr: str = ‘/run/mxa_manager/’, server_port_base: int = 10000, ignore_server_: bool = False) -> None

Initializes the MxAccl object by loading a DFP from a file (1st version) or from raw bytes (2nd version).

Parameters:

dfp_pathstr: Path to the DFP file.
dfp_bytesbytes: DFP as raw bytes object.
dfp_bytes_sizeint: Size of the DFP bytes.
device_ids_to_useList[int], optional: List of device IDs to use. Default is [0].
use_model_shapeTuple[bool, bool], optional: Whether to use model shape for input and output feature maps. Default is (True, True).
local_modebool, optional: Whether to run in local mode without connecting to server. Default is False.
sched_optionsSchedulerOptions, optional: Scheduler options for this client. Default is SchedulerOptions().
client_optionsClientOptions, optional: Client options for this client. Default is ClientOptions().
server_addrstr, optional: Server address to connect to (e.g., “/run/mxa_manager/”). Default is “/run/mxa_manager/”.
server_port_baseint, optional: Base port number for server connection. Default is 10000.
ignore_serverbool, optional: Whether to ignore server connection and run without it (DANGER). Default is False.

connect_post_model(self: memryx.mxapi.MxAccl, post_model_path: os.PathLike, model_id: int = 0, post_size_list: list[int] = []) → None#

Connects a post-processing model to the specified model ID.

Parameters:

post_model_pathstr: The file path to the post-processing model.
model_idint, optional: The ID of the model to connect to. Default is 0.

connect_pre_model(self: memryx.mxapi.MxAccl, pre_model_path: os.PathLike, model_id: int = 0) → None#

Connects a pre-processing model to the specified model ID.

Parameters:

pre_model_pathstr: The file path to the pre-processing model.
model_idint, optional: The ID of the model to connect to. Default is 0.

connect_stream(self: memryx.mxapi.MxAccl, in_callback: object, out_callback: object, stream_id: int, model_id: int = 0) → None#

Connects a stream to a model using the specified input and output callbacks functions.

This method registers a data stream for the given model and binds it to both input and output callback functions. Streams uniquely identified by stream_id can be connected to the same model. This function must be called before start().

The provided input callback function should have the following signature, where the return value is either a list of numpy arrays or None.

`python def in_callback(stream_id: int) -> List[np.ndarray]: `

And the output callback function should have the following signature:

`python def out_callback(ofmaps: List[np.ndarray], stream_id: int) -> None: `

Parameters:

in_callbackCallable[[int], Optional[List[np.ndarray]]]: A Python callable that takes a stream ID as input and returns a list of numpy arrays to the MXA, or None to indicate the end of the stream.
out_callbackCallable[[List[np.ndarray], int], None]: A Python callable that takes a list of numpy arrays (ofmaps from the MXA) and a stream ID as input.
stream_idint: An integer identifier for the stream. This can be used to differentiate multiple streams connected to the same model.
model_idint, optional: The ID of the model to connect the stream to. Default is 0.

start(self: memryx.mxapi.MxAccl, model_id: int = -1) → None#

Starts the execution of the specified model, or all models in the DFP if model_id is -1.

Parameters:

model_idint, optional: The ID of the model to start. If -1, starts all models in the DFP. Default is -1.

wait(self: memryx.mxapi.MxAccl, model_id: int = -1) → None#

Blocks until all streams for the specified model(s) have completed execution.

Parameters:

model_idint, optional: The ID of the model to wait for. If -1, waits for all models in the DFP. Default is -1.

stop(self: memryx.mxapi.MxAccl, model_id: int = -1) → None#

Stops the execution of the specified model, or all models in the DFP if model_id is -1.

Parameters:

model_idint, optional: The ID of the model to stop. If -1, stops all models in the DFP. Default is -1.

set_operating_frequency(self: memryx.mxapi.MxAccl, device_id: int, freq: int) → bool#

Sets the operating frequency of the specified device.

Parameters:

device_idint: The ID of the device to set the frequency for.
freqint: The desired operating frequency in MHz. Must be a valid 25MHz step between 200 and 1000.

Returns:

bool: True if the frequency was successfully set, False otherwise.

get_power(self: memryx.mxapi.MxAccl, device_id: int) → float#

Gets the current power consumption of the specified device in mW.

Parameters:

device_idint: The ID of the device to query.

Returns:

float: The current power consumption in milliwatts (mW). Returns -1 if power data is unavailable.

can_get_power_consumption(self: memryx.mxapi.MxAccl, device_id: int) → bool#

Checks if the given device is able to report power consumption data.

Parameters:

device_idint: The ID of the device to check.

Returns:

bool: True if power consumption data can be retrieved for the device, False otherwise.

get_max_temperature(self: memryx.mxapi.MxAccl, device_id: int) → float#

Gets the current maximum temperature among all chips on the specified device in Celsius.

Parameters:

device_idint: The ID of the device to query.

Returns:

float: The current maximum temperature in Celsius. Returns -1 if temperature data is unavailable.

get_chip_temperatures(self: memryx.mxapi.MxAccl, device_id: int) → numpy.ndarray[numpy.float32]#

Gets the current temperature of each chip on the specified device in Celsius.

Parameters:

device_idint: The ID of the device to query.

Returns:

List[float]: A list of current temperatures in Celsius for each chip on the device.

Manual Threading Mode

class MxAcclMT#

Python binding of the MxAcclMT class (manual-threading)

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: memryx.mxapi.MxAcclMT, dfp_path: os.PathLike, device_ids_to_use: list[int] = [0], use_model_shape: Annotated[list[bool], FixedSize(2)] = [True, True], local_mode: bool = False, sched_options: memryx.mxapi.SchedulerOptions = <memryx.mxapi.SchedulerOptions object at 0x7fa8be442af0>, client_options: memryx.mxapi.ClientOptions = <memryx.mxapi.ClientOptions object at 0x7fa8be442ab0>, server_addr: str = ‘/run/mxa_manager/’, server_port_base: int = 10000, ignore_server: bool = False) -> None
__init__(self: memryx.mxapi.MxAcclMT, dfp_bytes: bytes, dfp_bytes_size: int, device_ids_to_use: list[int] = [0], use_model_shape: Annotated[list[bool], FixedSize(2)] = [True, True], local_mode: bool = False, sched_options: memryx.mxapi.SchedulerOptions = <memryx.mxapi.SchedulerOptions object at 0x7fa8be442db0>, client_options: memryx.mxapi.ClientOptions = <memryx.mxapi.ClientOptions object at 0x7fa8be442d70>, server_addr: str = ‘/run/mxa_manager/’, server_port_base: int = 10000, ignore_server_: bool = False) -> None

connect_pre_model(self: memryx.mxapi.MxAcclMT, pre_model_path: os.PathLike, model_id: int = 0) → None#

Connects a pre-processing model to the specified model ID.

Parameters:

pre_model_pathstr: The file path to the pre-processing model.
model_idint, optional: The ID of the model to connect to. Default is 0.

connect_post_model(self: memryx.mxapi.MxAcclMT, post_model_path: os.PathLike, model_id: int = 0, post_size_list: list[int] = []) → None#

Connects a post-processing model to the specified model ID.

Parameters:

post_model_pathstr: The file path to the post-processing model.
model_idint, optional: The ID of the model to connect to. Default is 0.

send_input(self: memryx.mxapi.MxAcclMT, in_data: object, model_id: int, stream_id: int, timeout: int = 0) → None#

Sends input data to the accelerator for the specified model and stream.

Parameters:

in_dataList[np.ndarray] or np.ndarray: The input feature maps to send.
model_idint: The ID of the model to send the input to.
stream_idint: The ID of the stream to send the input to.
timeoutint, optional: The maximum time to wait for the input to be sent, in milliseconds. Default is 0 (no timeout).

Returns:

bool: True if the input was successfully sent, False if there was a timeout or error.

receive_output(self: memryx.mxapi.MxAcclMT, model_id: int, stream_id: int, timeout: int = 0) → list[numpy.ndarray[numpy.float32]]#

Receives output data from the accelerator for the specified model and stream.

Parameters:

model_idint: The ID of the model to receive the output from.
stream_idint: The ID of the stream to receive the output from.
timeoutint, optional: The maximum time to wait for the output to be received, in milliseconds. Default is 0 (no timeout).

Returns:

List[np.ndarray]: The output feature maps received from the accelerator. Returns None if there was a timeout or error.

run(self: memryx.mxapi.MxAcclMT, in_data: numpy.ndarray[numpy.float32], model_id: int, stream_id: int, timeout: int = 0) → list[numpy.ndarray[numpy.float32]]#

Runs inference on a batch of inputs and returns the resulting the outputs.

WARNING! While convenient for simple use cases, this method has much worse performance than using send_input and receive_output in separate threads. Separate threads are necessary to fully utilize the MXA’s pipelined architecture. Real-time applications should never use this function.

Parameters:

in_dataList[np.ndarray] or np.ndarray: The input feature maps to send.
model_idint: The ID of the model to run inference on.
stream_idint: The ID of the stream to run inference on.
timeoutint, optional: The maximum time to wait for the inference to complete, in milliseconds. Default is 0 (no timeout).

Returns:

List[np.ndarray]: The output feature maps received from the accelerator. Returns None if there was a timeout or error.

set_operating_frequency(self: memryx.mxapi.MxAcclMT, device_id: int, freq: int) → bool#

Sets the operating frequency of the specified device.

Parameters:

device_idint: The ID of the device to set the frequency for.
freqint: The desired operating frequency in MHz. Must be a valid 25MHz step between 200 and 1000.

Returns:

bool: True if the frequency was successfully set, False otherwise.

get_power(self: memryx.mxapi.MxAcclMT, device_id: int) → float#

Gets the current power consumption of the specified device in mW.

Parameters:

device_idint: The ID of the device to query.

Returns:

float: The current power consumption in milliwatts (mW). Returns -1 if power data is unavailable.

can_get_power_consumption(self: memryx.mxapi.MxAcclMT, device_id: int) → bool#

Checks if the given device is able to report power consumption data.

Parameters:

device_idint: The ID of the device to check.

Returns:

bool: True if power consumption data can be retrieved for the device, False otherwise.

get_max_temperature(self: memryx.mxapi.MxAcclMT, device_id: int) → float#

Gets the current maximum temperature among all chips on the specified device in Celsius.

Parameters:

device_idint: The ID of the device to query.

Returns:

float: The current maximum temperature in Celsius. Returns -1 if temperature data is unavailable.

get_chip_temperatures(self: memryx.mxapi.MxAcclMT, device_id: int) → numpy.ndarray[numpy.float32]#

Gets the current temperature of each chip on the specified device in Celsius.

Parameters:

device_idint: The ID of the device to query.

Returns:

List[float]: A list of current temperatures in Celsius for each chip on the device.

Scheduler and Client Options#

class SchedulerOptions#

SchedulerOptions encapsulates various options for the scheduler, including frame and time limits, queue sizes, and autoclock settings.

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: memryx.mxapi.SchedulerOptions) -> None
__init__(self: memryx.mxapi.SchedulerOptions, frame_limit: int = 20, time_limit: int = 250, ifmap_queue_size: int = 16, ofmap_queue_size: int = 21, autoclock_enabled: bool = False, autoclock_power_limit_mw: int = 11500, autoclock_check_fps_saturation: bool = False, autoclock_sample_interval_ms: int = 50, autoclock_num_samples: int = 6) -> None

property frame_limit#: The number of frames to process before the DFP is rescheduled. Default: 20

property time_limit#: The maximum idle time (in milliseconds) to wait for new input before forcing the DFP to be rescheduled. Default: 250 ms

property ifmap_queue_size#: Capacity of the input feature map (ifmap) queue used by the DFP, shared by all clients. Default: 16

property ofmap_queue_size#: Capacity of the per-client output feature map (ofmap) queues. Default: 21 (frame_limit + 1)

property autoclock_enabled#: Enable automatic frequency up/down-clocking when starting DFP, if supposed by the target MX3 hardware. Default: false

property autoclock_power_limit_mw#: Power limit in milliwatts. Autoclocking will increase frequency until this power limit is reached. Range: [1000, 14700]. Default: 11500 mW

property autoclock_check_fps_saturation#: Experimental: If true, autoclocking will stop increasing frequency, even if there’s still power headroom, if FPS has not increased over the last 3 frequency steps. Default: false

property autoclock_sample_interval_ms#: Interval in milliseconds between power samples while testing autoclocking steps. Default: 50 ms

property autoclock_num_samples#: Number of power samples to collect (and find the maximum of) before making a decision to stay at the current frequency or increase it. Default: 6

class ClientOptions#

Configures client-side execution behavior, such as FPS smoothing.

__init__(*args, **kwargs)#

Overloaded function.

__init__(self: memryx.mxapi.ClientOptions) -> None
__init__(self: memryx.mxapi.ClientOptions, smoothing: bool = False, fps_target: float = 0.0) -> None

property smoothing#: If true, enables frame rate smoothing to avoid sudden FPS “jumps”, at the cost of increased in-to-out latency. Default: false

property fps_target#: When smoothing is enabled, the target FPS to smooth towards. mxa-manager will automatically adjust the delay between frames to try to achieve this target FPS.

Advanced Control#

class Client#

Python binding of the MX::Runtime::Client class, which can be used to query information from mxa-manager (such as power, temperature, pressure, etc.) separately from an MxAccl/MxAcclMT instance.

init_connection(self: memryx.mxapi.Client, server_address: str = '/run/mxa_manager/', base_port: int = 10000) → bool#

Initializes connection to mxa-manager. Must be called before any other Client method.

Parameters:

server_addressstr, optional: Address of the mxa-manager server (default: “/run/mxa_manager/”). For local UNIX socket connection, use the format “/path/to/socket”. For TCP connection, use the format “ip_address”.
base_portint, optional: Base port number (TCP) or numeric filename (UNIX) for connecting to mxa-manager (default: 10000).

Returns:

bool: True if connection is successfully established, False otherwise.

end_connection(self: memryx.mxapi.Client) → bool#

Ends connection to mxa-manager. Should be called to clean up resources when Client is no longer needed.

Returns:

bool: True if connection is successfully closed, False otherwise.

set_power_mode(self: memryx.mxapi.Client, device_id: int, freq_mhz: int) → bool#

Sets the operating frequency of the specified device.

Parameters:

device_idint: ID of the device to set the frequency on.
freq_mhzint: Desired operating frequency in MHz. Must be a valid 25MHz step between 200 and 1000.

Returns:

bool: True if the frequency was successfully set, False otherwise (e.g. invalid frequency or device_id).

get_avg_max_temp(self: memryx.mxapi.Client, device_id: int) → float#

Gets the rolling average of the maximum temperatures across all chips on the device.

Parameters:

device_idint: ID of the device to query.

Returns:

float: Average of the maximum temperatures across all chips on the device, in degrees Celsius, over the past HW_MONITOR_INFO (set in /etc/memryx/mxa_manager.conf, default 500ms).

get_inst_max_temp(self: memryx.mxapi.Client, device_id: int) → float#

Gets the current maximum temperature across all chips on the device.

Parameters:

device_idint: ID of the device to query.

Returns:

float: Current maximum temperature across all chips on the device, in degrees Celsius.

get_avg_power(self: memryx.mxapi.Client, device_id: int) → float#

Gets the rolling average power consumption of the device, if supported by the device.

Parameters:

device_idint: ID of the device to query.

Returns:

float: Rolling average power consumption of the device in milliwatts (mW) over the past HW_MONITOR_INFO (set in /etc/memryx/mxa_manager.conf, default 500ms), if supported by the device. If not supported, returns -1.

get_inst_power(self: memryx.mxapi.Client, device_id: int) → float#

Gets the current power consumption of the device, if supported by the device.

Parameters:

device_idint: ID of the device to query.

Returns:

float: Current power consumption of the device in milliwatts (mW), if supported by the device. If not supported, returns -1.

get_pressure(self: memryx.mxapi.Client, device_id: int) → str#

Gets the current “pressure” level of the device, hinting to the user how many more streams could be assigned to the device before it becomes overloaded.

Parameters:

device_idint: ID of the device to query.

Returns:

str: Current pressure level of the device, returned as a string with possible values: “low”, “medium”, “high”, “full”. See C++ API docs for details on what these levels mean.

get_device_infos(self: memryx.mxapi.Client) → list[memryx.mxapi.device_info_t]#

Gets information about all available MemryX devices from mxa-manager.

Returns:

List[device_info_t]: A list of device_info_t objects, each containing info about that MXA device.

property my_client_id#: The unique client ID assigned by mxa-manager to this Client instance after a successful call to init_connection.

class device_info_t#

Struct with info about a MemryX device, retrieved from mxa-manager.

property chip_count#: Number of chips in the device.

property can_get_power_data#: Whether power data can be retrieved from the device. If false, power will always be reported as -1.

property is_usb#: Whether the device is connected via USB.

property freqs#: Frequencies in MHz of each chip on the device. The length of this list should match chip_count.

property volt#: Voltage in mV of the device. This applies to all chips on the device.

property current_config#: DEPRECATED Current ‘group’ configuration of the device.

property num_groups#: DEPRECATED Number of ‘groups’ in the device.

property chips_per_group#: DEPRECATED Number of chips per group.