Driver API

Warning

The Driver API is for extremely advanced use-cases only! All users should use the Accelerator API instead.

Please refer to this page only if explicitly told to do so by MemryX customer engineering teams.

Terminology

The terms ‘model context’ and ‘MXA device group’ are introduced in the driver.

/**
* +------------+     PCIE       +------+     +------+
* |            |--------------->| MX3  |-----| MX3  |
* |            |                +------+     +------+
* |            |                (MXA device group 0)
* |            |
* |    Host    |
* |            |     PCIE       +------+     +------+
* |            |--------------->| MX3  |-----| MX3  |
* |            |                +------+     +------+
* |            |                (MXA device group 1)
* +------------+
*/

• MXA device group (group_id):

  Hardware device (e.g., MX3) connected to host PC. No matter how many devices are cascaded (1x, 2x, 4x, 8x) in a chain, there is always one device node per chain on the host PC. We called the cascaded/chained devices an MXA device group, and we used hardware pins to assign different group IDs to each MXA device group. The group ID is later used for the driver to access the target MXA device group. When multi-threading, locks apply at the MXA device group level.

• Model context (model_id):

  Model context is a software resource (model’s parameters and buffers) stored within the driver on the host PC. A specific MXA device group (hardware resource) is required to be given to run inference to create a new model context. However, the model context is not limited to the first bound MXA device group and can be re-configured to different MXA device groups later.

C/C++ API

Constants

MEMX_DEVICE_CASCADE_PLUS

MemryX device MX3:Cascade+. Constant value which should be referenced only and not be modified manually.

Options

Model download options

MEMX_DOWNLOAD_TYPE_WTMEM

Option of memx_download_model() to download weight memory only to device. Can be used together with MEMX_DOWNLOAD_TYPE_MODEL.

MEMX_DOWNLOAD_TYPE_MODEL

Option of memx_download_model() to download model only to device. Can be used together with MEMX_DOWNLOAD_TYPE_WTMEM.

MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL

Option of memx_download_model() to download both weight memory and model to device. The same effect as using MEMX_DOWNLOAD_TYPE_WTMEM and MEMX_DOWNLOAD_TYPE_MODEL together.

Macros

memx_status_no_error(_status_)

Helper macro to check if there is no error occurs.

memx_status_error(_status_)

Helper macro to check if there is any error occurs.

Functions

memx_status memx_lock(uint8_t group_id)

Acquires mutex lock from driver. To be noticed, this function will not actually lock any operation but only acquire the unique mutex within driver. Lock is only required in multi-threading or multi-processing case to avoid driver access conflict when multiple processes trying to re-configure the same hardware device. Also, this function will block until mutex is acquired successfully. See memx_trylock() for non-blocking mutex lock.

// lock MPU device group 0
memx_lock(0);
printf("lock success.\n");
// always remember to release lock finally
memx_unlock(0);

See also

Advanced Driver Usage: Example 3: Runtime Model Swap

Parameters:

group_id – MPU device group ID

Returns:

0 on success, otherwise error code

memx_status memx_trylock(uint8_t group_id)

Acquires mutex lock from driver. Apart from memx_lock(), this function returns immediately if fail to acquire lock.

// do something only if lock MPU device group 0 successfully
if (memx_trylock(0) == 0) {
  printf("lock success.\n");
  // always remember to release lock finally
  memx_unlock(0);
} else {
  printf("lock failure.\n");
}

Parameters:

group_id – MPU device group ID

Returns:

0 on success, otherwise error code

memx_status memx_unlock(uint8_t group_id)

Releases the mutex lock acquired from driver. Be careful and not to unexpectedly release mutex which is not locked by current process. Uses memx_lock() and memx_trylock() to acquire lock first.

Parameters:

group_id – MPU device group ID

Returns:

0 on success, otherwise error code

memx_status memx_open(uint8_t model_id, uint8_t group_id, float chip_gen)

Open new model context with specified MPU device driver and MPU device group binding to it. Internal reference count increases if the same model ID is re-used and opened multiple times. All operations later using this model ID will use binded MPU device driver and MPU device group. MPU device group binded can later be re-configured using memx_reconfigure() while MPU device driver may not be re-configured once model context is created. Currently, at most 32 model contexts (model ID = 0 ~ 31) can be stored within driver.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  printf("open success.\n");
} else {
  printf("open failure.\n");
}
// always remember to close device finally
memx_close(0);

See also

Classification (Driver APIs): Basic Inference

Parameters:

• model_id – model ID

• group_id – MPU device group ID

• chip_gen – Deprecated, user don’t care. device chip_gen was detected by driver

Returns:

0 on success, otherwise error code

memx_status memx_close(uint8_t model_id)

Close current connected MPU device group and destory model context stored in driver. If model context is opened multiple times, calling close will decrease internal reference count one each time. Only after reference count goes down to zero will model context finally be released. So make sure to use equivalent times of open and close to clean-up driver resources. Otherwise, it is safe to call close model ID which is not opened or is closed already. See memx_open() for more information.

Parameters:

model_id – model ID

Returns:

0 on success, otherwise error code

memx_status memx_reconfigure(uint8_t model_id, uint8_t group_id)

Runtime re-configure MPU device group binding to specified model ID. Be sure to finish all on-going data transfer first before calling this function.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // re-configure to bind MPU device group 1 to model 0
  status = memx_reconfigure(0, 1);
}
// always remember to close device finally
memx_close(0);

Parameters:

• model_id – model ID

• group_id – MPU device group ID

Returns:

0 on success, otherwise error code

memx_status memx_config_mpu_group(uint8_t group_id, uint8_t mpu_group_config)

Config MPU group on the device group, support MEMX_MPU_GROUP_CONFIG_*.

Parameters:

• group_id – device group ID

• mpu_group_config – MPU group config

Returns:

0 on success, otherwise error code

memx_status memx_get_chip_gen(uint8_t model_id, uint8_t *chip_gen)

Read back the chip gen for the currently active model.

Parameters:

• model_id – model ID

• chip_gen – chip gen of device (MEMX_MPU_CHIP_GEN_CASCADE, MEMX_MPU_CHIP_GEN_CASCADE_PLUS)

Returns:

0 on success, otherwise error code

memx_status memx_download_model(uint8_t model_id, const char *file_path, uint8_t model_idx, int32_t type)

Download weight memory and model to device based on given download type selection. If both weight memory and model download are required, weight memory download will be performed before model download. After model download is completed, input and output feature map shape will be configured to driver to allocate resources automatically. After model download and before data can be transferred to device, memx_set_stream_enable() must be called to change driver internal state from idle to data-transfer.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // download weight memory and 1st model within DFP file to device
  status = memx_download_model(0, "model.dfp", 0,
    MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL);
}
// always remember to close device finally
memx_close(0);

// download weight memory and 1st model within DFP file to device separately
// same effect as example using type MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // download weight memory to device, model index within DFP will be ignored
  status = memx_download_model(0, "model.dfp", 0, MEMX_DOWNLOAD_TYPE_WTMEM);
}
if (status == 0) {
  // download 1st model within DFP file to device
  status = memx_download_model(0, "model.dfp", 0, MEMX_DOWNLOAD_TYPE_MODEL);
}
// always remember to close device finally
memx_close(0);

Parameters:

• model_id – model ID

• file_path – DFP file path

• model_idx – model index within DFP file, give ‘0’ if there is only one model compiled in DFP file. model index will be ignored if download type selection is weight memory only

• type – 0: ignored, 1: weight memory only, 2: model only, 3: both weight memory and model

Returns:

0 on success, otherwise error code

memx_status memx_set_stream_enable(uint8_t model_id, int32_t wait)

Enable all input and output data to be transferred to device of specified model. When model context is first initialized, all data transfers are blocked (but model download is allowed) to avoid data being transferred to device before model download is finished.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // download weight memory and 1st model within DFP file to device
  status = memx_download_model(0, "model.dfp", 0,
    MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL);
}
if (status == 0) {
  // enable all data being transferred to device
  status = memx_set_stream_enable(0, 0);
}
// always remember to close device finally
memx_close(0);

Parameters:

• model_id – model ID

• wait – wait until driver state change from idle to data-transfer complete

Returns:

0 on success, otherwise error code

memx_status memx_set_stream_disable(uint8_t model_id, int32_t wait)

Disable all input and output data to be transferred to device of specified model. This function is used to guarantee no data from specified model will be transferred to device in case multiple models are sharing the the same hardware resource and want to transfer data in the same time. Use memx_set_stream_enable() to resume data transfer of specified model.

Parameters:

• model_id – model ID

• wait – wait until driver state change from data-transfer to idle complete

Returns:

0 on success, otherwise error code

memx_status memx_get_ifmap_size(uint8_t model_id, uint8_t flow_id, int32_t *height, int32_t *width, int32_t *z, int32_t *channel_number, int32_t *format)

Read back input feature map shape configured of specific flow (port). Returns failure with variables set to all zeros in case given flow is not configured.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // download weight memory and 1st model within DFP file to device
  status = memx_download_model(0, "model.dfp", 0,
    MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL);
}
if (status == 0) {
  // read back flow 0 (port 0) input feature map shape for debug information
  int32_t height, width, z, channel_number, format;
  memx_get_ifmap_size(0, 0, &height, &width, &z, &channel_number, &format);
  // for example: shape = (224, 224, 1, 3), format = 1
  printf("shape = (%d, %d, %d, %d), format = %d\n",
    height, width, z, channel_number, format);
}
// always remember to close device finally
memx_close(0);

Parameters:

• model_id – model ID

• flow_id – input flow (port) ID

• height – input feature map height

• width – input feature map width

• z – input feature map z

• channel_number – input feature map channel number

• format – input feature map format

Returns:

0 on success, otherwise error code

memx_status memx_get_ofmap_size(uint8_t model_id, uint8_t flow_id, int32_t *height, int32_t *width, int32_t *z, int32_t *channel_number, int32_t *format)

Read back output feature map shape configured of specific flow (port). Returns failure with variables set to all zeros in case given flow is not configured.

// bind MPU device group 0 as MX3 to model 0
memx_status status = memx_open(0, 0, MEMX_DEVICE_CASCADE_PLUS);
if (status == 0) {
  // download weight memory and 1st model within DFP file to device
  status = memx_download_model(0, "model.dfp", 0,
    MEMX_DOWNLOAD_TYPE_WTMEM_AND_MODEL);
}
if (status == 0) {
  // read back flow 0 (port 0) input feature map shape for debug information
  int32_t height, width, z, channel_number, format;
  memx_get_ofmap_size(0, 0, &height, &width, &z, &channel_number, &format);
  // for example: shape = (224, 224, 1, 3), format = 1
  printf("shape = (%d, %d, %d, %d), format = %d\n",
    height, width, z, channel_number, format);
}
// always remember to close device finally
memx_close(0);

Parameters:

• model_id – model ID

• flow_id – output flow (port) ID

• height – output feature map height

• width – output feature map width

• z – output feature map z

• channel_number – output feature map channel number

• format – output feature map format

Returns:

0 on success, otherwise error code

memx_status memx_stream_ifmap(uint8_t model_id, uint8_t flow_id, void *ifmap, int32_t timeout)

Data channel write input feature map of specified flow (port) to device. This function must be called after memx_download_model().

memx_status status = MEMX_STATUS_OK;
if (status == 0) {
  // write data to model 0 flow 0 (port 0) to run inference
  status = memx_stream_ifmap(0, 0, ifmap, 200);
}

Parameters:

• model_id – model ID

• flow_id – input flow (port) ID

• ifmap – input feature map (frame)

• timeout – milliseconds timeout, ‘0’ indicates infinite

Returns:

0 on success, otherwise error code

memx_status memx_stream_ofmap(uint8_t model_id, uint8_t flow_id, void *ofmap, int32_t timeout)

Data channel read output feature map of specified flow (port) from device. This function must be called after memx_download_model().

memx_status status = MEMX_STATUS_OK;
if (status == 0) {
  // read data from model 0 flow 0 (port 0) as inference result
  status = memx_stream_ofmap(0, 0, ofmap, 200);
}

Parameters:

• model_id – model ID

• flow_id – output flow (port) ID

• ofmap – output feature map (frame)

• timeout – milliseconds timeout, ‘0’ indicates infinite

Returns:

0 on success, otherwise error code

Python API

Functions

class mxa

MemryX driver.

The interface to the MemryX driver. This class wraps MemryX driver C API to Python.

lock(group_id: int)

Acquires mutex lock from driver. To be noticed, this function will not actually lock any operation but only acquire the unique mutex within driver. In multi-threads programming, user can use this lock to prevent driver access conflict. Also, this function will block until mutex is acquired.

Parameters:

group_idint

MPU device group ID

unlock(group_id: int)

Releases the mutex lock acquired from driver. Be careful and not to unexpectedly release mutex which is not locked by current process (thread).

Parameters:

group_idint

MPU device group ID

open(model_id: int, group_id: int, chip_gen: int)

Open device with interface auto-detected. Currently only one device connected to host is supported within this version implementation. Even in case chips are cascaded to compose data flow, there is only one device can be seen within host.

Parameters:

model_idint

Model ID

group_idint

MPU device group ID

chip_genint

Target system Casade = 3

close(model_id: int)

Close current connected device and interface in use. This function returns always no error even if device is not opened.

Parameters:

model_idint

Model ID

download(model_id: int, file_path: str, model_idx: int, type: int)

Download weight memory and model to device based on given download type selection. If both weight memory and model download are required, weight memory download will be performed before model download. Also after model is downloaded, input and output feature map shape will be configured to driver automatically.

Parameters:

model_idint

Model ID

file_pathstring

DFP file path

model_idxint

Which rgcfg to use (used in model swapping). Set to 0 if only 1 model.

typeint

Target model type (any of the following)

• mxa.download_type_wtmem / 1

• mxa.download_type_model / 2

• mxa.download_type_wtmem_and_model / 3

set_stream_enable(model_id: int, wait: int)

Enable all input and output data flows of this MPU context to interface driver.

Parameters:

model_idint

Model ID

waitbool

Wait until state changed or not

set_stream_disable(model_id: int, wait: int)

Disable all input and output data flows of this MPU context to interface driver.

Parameters:

model_idint

Model ID

waitbool

Wait until state changed or not

stream_ifmap(model_id: int, flow_id: int, ifmap: ndarray, timeout: int = 0)

Data channel write frame to device.

Parameters:

model_idint

Model ID

flow_idint

Input flow (port) ID

ifmapnp.ndarray

Input feature map (frame)

timeoutint

Milliseconds timeout, ‘0’ indicates infinite

stream_ofmap(model_id: int, flow_id: int, ofmap: ndarray, timeout: int = 0)

Stream data out from device.

Parameters:

model_idint

Model ID

flow_idint

Output flow (port) ID

ofmapnp.ndarray

Output feature map (frame).

timeoutint

Milliseconds timeout, ‘0’ indicates infinite

See also

• Accelerator API

• Tutorials

• Basic Inference using Accelerator Tutorial

• Basic Inference using Driver Tutorial

• Advanced Driver Usage APIs