Skip to main content

Models

Overview​

This comprehensive guide empowers you to seamlessly deploy your trained TensorFlow/Keras models directly onto your Barbara Edge Nodes. By leveraging the processing power at the edge, you can enable real-time inference and unlock the full potential of your AI applications.

This guide will equip you with the knowledge to:

  • Prepare your TensorFlow/Keras model for deployment on edge devices.
  • Deploy your model to your desired edge node(s) using the Barbara platform.

By following these steps, you'll be able to harness the power of AI directly on your edge devices, enabling faster decision-making, improved efficiency, and innovative edge applications.

Prerequisites​

  • Your AI models must be saved in the SavedModel format. Refer to the Tensorflow Documentation for instructions on saving your models.
  • Ensure your Barbara Edge Node meets the hardware requirements. Only nodes with x86 architecture and AVX-enabled kernels are compatible with model serving. Check the node's specification sheet in the Nodes Hardware section for details.

Ready to get started?​

This guide includes a downloadable demo model to help you follow the deployment process step-by-step on your Barbara Edge Node. Download the demo model and save it to your computer to proceed.

Upload your model to the Barbara Panel's Library​

Create a New Model​

In the Barbara Panel platform, head to the "Library" view.

Library page

Library page

Click the "NEW" button to initiate the "Select Library Item" modal.

Select Item

Select Item

Select "Model" from the options and click "NEW" again. In the "New Model" dialog provide other model details:

  1. Enter a name (e.g., "half_plus_two").
  2. Add a logo image (optional)
  3. Include pictures and descriptions (optional).

Finally click "CREATE" to save your new model.

New Model

New Model

Now, your new model is available in the Library view, but it still has no versions. Let's add one.

New model created

New model created

Add Model Version​

  1. Click the "+" button in the model's row to open the "New version" dialog.
  2. Select your previously downloaded demo model (or just select any other SavedModel format model).
  3. Assign a version number
  4. Add the release notes (optional).
  5. Finally, click "CREATE" to finalize the version upload.

New version dialog

New version dialog

Your model is now uploaded to the Barbara Panel's library and it is ready for deployment to compatible nodes.

Deploy your model to a Barbara Edge Node​

In the Barbara Panel edge nodes view, choose a compatible node and navigate to its details view.

Add a Model Card​

Click "ADD CARD" and select "Model" from the options to launch the model wizard.

Add Card

Add Card

Select your Model in the Model Wizard​

In the model wizard, select the previously uploaded model "half_plus_two" and its version.

Select Model

Select Model

note

Do you want to save time? Keep checked following options in "Advance":

  • Run automatically after install
  • Activate logs for this application

Finally click "ADD VARIABLES" to go to next step.

Configure Variables in the Model Wizard​

Did you know...

gRPC is an open-source framework enabling efficient, bi-directional communication between services in distributed systems. It leverages HTTP/2 and Protocol Buffers for platform-agnostic, structured data exchange, ideal for real-time applications and microservices.

In this step you must define the following parameters:

  • gRPC Port Number: Define the port for gRPC communication (default: 9083).
  • REST API Port Number: Define the port for API communication (default: 9084).
  • Model Name: Set the name of the model (e.g. "half_plus_two). This will be a part of the URL we will use later for requesting predictions to the model once served.
warning

Are you serving two or more models in the same node? In that case, their gRPC/REST API port numbers must always be different!!

Adding Variables

Adding Variables

Deploy the Model​

Finally click "SEND" to initiate the deployment process. A new card will appear in the node's details view.

Model Card

Model Card

Once the building process finishes, your model will be live and ready to receive prediction requests.

Get Predictions from Your Deployed Model​

  1. Open a Terminal: Access a terminal window to send prediction requests to the model's API REST port.
  2. Craft the Predictin Request:
    • Use the curl command to make a POST request to the model's endpoint.
    • Structure the request as follows:
curl -d '{"instances": [YOUR_INPUT_DATA]}' -X POST http://IP_NODE:API_REST_PORT/v1/models/MODEL_NAME:predict

  1. Replace the placeholders with your specific values:

    • YOUR_INPUT_DATA: Input data required by your model (e.g. a vector of numbers in this example).
    • IP_NODE: IP address of your node (use the default gateway IP if on the same LAN, or the VPN if accessing remotely).
    • API_REST_PORT: REST API port configured during deployment (9084 in this example).

  2. Send the Request: Execute the crafted curl command to initiate the prediction request. In our example, the final request is:

curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://10.30.0.181:9084/v1/models/half_plus_two:predict
  1. Receive the Results: The model will process the input data and return the predictions, displayed directly in your terminal.

Predictions results

Predictions results