RVC Real-time Transmission v1.0

Based on tuyaos_robot_rvc_sdk, this topic describes the process and APIs for transmitting real-time maps and routes of laser robot vacuums, as well as the coordinate system of maps.

Overview

Laser robot vacuums support three types of files:

• Full map file
• Cleaning route file
• AI-based object recognition file

How it works

Transmit real-time files

Data type

OBJECT_POINT_S

/**
 * @brief Structure for coordinate points
 */
typedef struct {
    int x; // X value of the coordinate point
    int y; // Y value of the coordinate point
} OBJECT_POINT_S;

AI_OBJECTS_TYPE_E

/**
 * @brief Names of recognized objects
 */
typedef enum {
    OBJECTS_TYPE_WIRE = 0, // Wire
    OBJECTS_TYPE_SHOES, // Shoes
    OBJECTS_TYPE_SOCKS, // Socks
    OBJECTS_TYPE_TOYS, // Toys
    OBJECTS_TYPE_CHAIR, // Chair
    OBJECTS_TYPE_TABLE, // Table
    OBJECTS_TYPE_TRASH_CAN, // Trash can
    OBJECTS_TYPE_POTTED_PLANT, // Potted plant
    OBJECTS_TYPE_OTHER, // Other
    OBJECTS_TYPE_MAX,
} AI_OBJECTS_TYPE_E;

OBJECT_THUMBNAILS_PARAM_S

/**
 * @brief thumbnail param structure
 */
typedef struct {
    OBJECT_POINT_S points; // Points of objects
    AI_OBJECTS_TYPE_E object_type; // Recognized objects
    unsigned char accuracy; // Recognition accuracy, report FF if not used
} OBJECT_THUMBNAILS_PARAM_S;

RVC_AI_OBJECT_THUMBNAILS_INFO_S

/**
 * @brief AI thumbnail data structure
 */
typedef struct {
    int map_id; // Map ID
    int object_num; // Number of objects
    OBJECT_THUMBNAILS_PARAM_S* param; // Points of objects, type, and accuracy
} RVC_AI_OBJECT_THUMBNAILS_INFO_S;

RVC_MAP_HEAD_INFO_S

/**
 * @brief Parameters for the map protocol header of the vacuum cleaner
 */
typedef struct {
    unsigned short map_id;           // Map ID
    int status;                      // Map status, 0 indicates unstable, 1 indicates stable
    int height;                      // Height of the map in pixels
    int width;                       // Width of the map in pixels
    int resolution;                  // Map resolution (width/length of each grid), reported as original value, unit: meters
    POINT_COOR_S origin;             // Machine origin X/Y coordinates, given relative to the screen coordinate system, reported as original value
    POINT_COOR_S charge_point;       // Charging station X/Y coordinates, given relative to the screen coordinate system, reported as original value
    int charge_angle;                // Orientation of the charging station
} RVC_MAP_HEAD_INFO_S;

PATH_POINT_S

typedef struct {
    int type;                //Path point type
    POINT_COOR_S coordinates;  //Path coordinate array, given with reference to the machine coordinate system
} PATH_POINT_S;

RVC_PATH_INFO_S

/**
 * @brief Parameters for the path protocol of the robot vacuum cleaner
 */
typedef struct {
    unsigned short path_id;           // Path ID: Roadmap identification for a complete sweep
    int type;                        //0x01 full path (normal mode), 0x02 full path (complex mode), 0x03 navigation path
    int count;                       // Total number of waypoints (full)
    int direction;                   // Device heading angle
    PATH_POINT_S*path_point;        //Path coordinate array, given with reference to the machine coordinate system, the original value is reported
} RVC_PATH_INFO_S;

RVC_PIXEL_TYPE_E

/**
 * @brief Pixel point types
 */
typedef enum {
    RVC_PIXEL_BLANK=0,     // Free area, laser scanning points of unpartitioned rooms.
    RVC_PIXEL_OBS,       // Obstacle
    RVC_PIXEL_CARPET,   // Carpet area
    RVC_PIXEL_UNKNOW,   // Unknown area, the background area of the map rectangular canvas
    RVC_PIXEL_MAX,      // Pad data, no use
} RVC_PIXEL_TYPE_E;  

RVC_POINT_S

/**
 * @brief Point in the corresponding coordinate system
 */
typedef  struct {
    float x;    // X-axis coordinate in the corresponding coordinate system, unit: meters
    float y;    // Y-axis coordinate in the corresponding coordinate system, unit: meters
    float z;    // Z-axis coordinate in the corresponding coordinate system, unit: meters
}RVC_POINT_S;

ROOM_POLYGON_S

/**
 * @brief Polygon parameters
 */
typedef  struct {
    RVC_POINT_S* points;   // Polygon points
    int points_size;                    // Number of points
}ROOM_POLYGON_S;

RVC_ROOM_POLYGON_S

/**
 * @brief Area polygon parameters
 */
typedef struct {
    int id;                                                 // Area ID
    ROOM_POLYGON_S polygon;      // Polygon
} RVC_ROOM_POLYGON_S;

RVC_MAP_DATA_S

/**
 * @brief Map data structure
 */
typedef struct{
    RVC_MAP_HEAD_INFO_S header;  //Parameters for the map protocol header of the robot vacuum cleaner
    unsigned char** map_buff;  //Pixel data on the map
    int room_polygons_count;   //Number of room zones
    RVC_ROOM_POLYGON_S* room_polygons_data;  //Area polygon parameters
}RVC_MAP_DATA_S;

APIs

Initialize streaming media service

Initialize the streaming media service. Real-time maps and routes can be transmitted as expected only if you call this interface.

/**
 * @brief Real-time map and route transmission callback
 */
typedef INT_T (*RVC_TRANS_EVENT_CB)(IN CONST int onoff);

/***********************************************************
 *@Function: tuya_sdk_media_server_init
 *@brief  Initialize media server
 *@param[in] handler The status callback for real-time data transmission,
             with statuses including: start and end.
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/

OPERATE_RET ty_rvc_server_init(RVC_TRANS_EVENT_CB handler);

Parameters

Parameter	Description
handler	The transmission status of real-time maps and routes. You can determine whether data reporting is started based on the status value.

Report real-time map data

Send structured data of real-time maps to the mobile app for display. For more information, see Device angle.

/***********************************************************
 *@function: ty_rvc_rt_map_send_v2
 *@brief:  Send real-time structured map data to the app
 *@param[in] map_info: map data, including fixed header data, pixel data,
                       the number of room zones, and boundary grid coordinate point data.
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ** *********************************************************/
OPERATE_RET ty_rvc_rt_map_send_v2(RVC_MAP_DATA_S* map_info)

Parameters

Parameter	Description
map_info	The map data. For more information, see the RVC_MAP_DATA_S struct.

For more information about the above request parameters, see Map and route data structure.

Report real-time route data

Send structured data of real-time routes to the mobile app for display.

/***********************************************************
 *@function: ty_rvc_rt_cleanpath_send
 *@brief Send real-time cleanpath data to app
 *@param[in] path_info: realtime path data which is need to be upload
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/
OPERATE_RET ty_rvc_rt_cleanpath_send_v2(IN RVC_PATH_INFO_S* path_info);

Parameters

Parameter	Description
path_info	For more information about real-time route data, see Map and route data structure.

Report thumbnail data

Send the AI thumbnail file data to the mobile app. You only need to pass the data to the TuyaOS SDK as required by the RVC_AI_OBJECT_THUMBNAILS_INFO_S struct. The TuyaOS SDK will internally transform the data before transmitting it to the mobile app.

/***********************************************************
 *@function: ty_rvc_rt_ai_thumbnails_send
 *@brief Send realtime ai thumbnails to app
 *@param[in] object_info: ai object thumbnails param
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/
OPERATE_RET ty_rvc_rt_ai_thumbnails_send(IN RVC_AI_OBJECT_THUMBNAILS_INFO_S* object_info);

Parameters

Parameter	Description
object_info	The content of the thumbnail data.

Register a callback to get ​​original HD image file name

Register a callback to get the AI ​​original HD image file name. Based on the file name sent by the mobile app, you notify the application through the sets_handler callback. The application retrieves the specified original HD image data according to the file name.

/**
 * @brief The callback for AI HD image data. ai_name is the name of an AI ​​HD image file sent by the mobile app.
 */
typedef INT_T (*RVC_AI_DATA_LOAD_CB)(CHAR_T *ai_name);

/***********************************************************
 *@function: ty_rvc_ai_original_init
 *@brief Callback function for get AI HD name
 *@param[in] sets_handler: set unpload AI HD name cb
 *@return OPERATE_RET: 0 for success, other error codes indicate failure
 ***********************************************************/
OPERATE_RET ty_rvc_ai_original_init(RVC_AI_DATA_LOAD_CB sets_handler);

Report ​​original HD images

Send the original HD image data to the mobile app. Based on the file name sent by the mobile app, you notify the application through the sets_handler callback. The application retrieves the specified original HD image data according to the file name and passes it to the TuyaOS SDK. The TuyaOS SDK then sends it to the mobile app. You need to pass the data content, length, and format of the original HD image to the TuyaOS SDK. The TuyaOS SDK will transform the data before transmitting it to the mobile app.

/***********************************************************
 *@Function: ty_rvc_ai_original_send
 *@brief send ai original to app
 *@param[in] file_type: ai original types include jpg, png, gif, svg, bmp, ico
 *@param[in] file_buff: ai original buff
 *@param[in] file_len: ai original len
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/
OPERATE_RET ty_rvc_ai_original_send(IN CHAR_T* file_type, IN CHAR_T* file_buff, IN INT_T file_len);

Parameters

Parameter	Description
file_type	The following six types of HD images are currently supported: JPG, PNG, GIF, SVG, BMP, and ICO. To support other image types, contact the Tuya project manager.
file_buff	The content of the specified original HD image data.
file_len	The length of the specified original HD image data.

Map and route data structure

Map data

Map data RVC_MAP_DATA_S consists of the following categories:

• The fixed header data of the map. It is composed of members of the RVC_MAP_HEAD_INFO_S struct.

  No.	Field	Data type	Description
  1	map_id	UINT16_T	The ID of a cleaning map. If the same map is just updated, the ID remains unchanged. If a map is lost or otherwise needs to be rebuilt, the ID is incremented. It is necessary to store the ID from before shutdown so that it can continue to be used after powering on and restarting.
  2	status	UINT32_T	Used for UI interactions on app panels. 0x00: The map is in an unstable state. For example, the map is incomplete and is still being built. 0x01: The map is in a stable state. The map is built and has a complete environment.
  3	height	UINT32_T	The pixel height of a map. If there is no map, this parameter is 0.
  4	width	UINT32_T	The pixel width of a map. If there is no map, this parameter is 0.
  5	resolution	UINT32_T	The resolution of a map (the width and length of each grid), reported in original value, in centimeters.
  6	origin	UINT32_T	The X/Y coordinates of the device origin, with reference to the screen coordinate system. The original values ​​are reported. For more information, see Map coordinate system.
  7	charge_point	UINT32_T	The X/Y coordinates of a charging station, with reference to the screen coordinate system. The original values ​​are reported. If the X and Y coordinates of the charging station are both equal to 0, the charging station will not be displayed. For more information, see Map coordinate system.
  8	charge_angle	UINT32_T	The device’s heading angle at the current point. For more information, see Device angle.

• All pixel data in the map is represented by the map_buff member. For pixel values, refer to the definitions of the RVC_PIXEL_TYPE_E enumeration values.
  Based on the pixel width and height of the map, fill the specified pixel enumeration value at the top-left corner R(0,0) within its area using a two-dimensional scanning method. 1: The current pixel is an obstacle. 2: The current pixel is a carpet.

• The outer boundary information of the specified area in a map is represented by the RVC_ROOM_POLYGON_S member. It refers to the coordinates of the boundary points of the room partition.
  As shown in the following figure:

  
  Each boundary grid coordinate point shall be filled into the memory space sequentially, starting from the top-left corner according to the screen coordinate system, and finally returning to the starting point. Avoid skipping or fitting any points in between, as this might cause errors on the panel.

• Map reporting routine

  int main_test(void)
  {
      RVC_MAP_DATA_S rvc_map_info = { 0 };
      int i = 0;  int j = 0;
      /* Assign values to the fixed bytes of the map protocol header */
      rvc_map_info.header.map_id = 0;   //The map ID
      rvc_map_info.header.status = 1; //0 indicates that the map is unstable, and 1 indicates that the map is stable
      rvc_map_info.header.width = 121; //Pixel width of the map
      rvc_map_info.header.height = 152;  //Pixel height of the map    
      rvc_map_info.header.origin.x = 564; //X coordinate of the device origin
      rvc_map_info.header.origin.y = 1357; //Y coordinate of the device origin
      rvc_map_info.header.resolution = 5; //Resolution              
      rvc_map_info.header.charge_point.x = 0; //X coordinate of the charging station
      rvc_map_info.header.charge_point.y = 0; //Y coordinate of the charging station
      rvc_map_info.header.charge_angle = 0; //Angle of the charging station
      //Call functions to allocate memory
      rvc_map_info.map_buff = NULL;
      char strFileInfo_buff[]; //Your original pixel data to be filled in the following two-dimensional array
      //Request a two-dimensional array of memory for temp_map_buf
      for (i = 0; i < rvc_map_info.header.height; i++) {
          for (j = 0; j < rvc_map_info.header.width; j++) {
              rvc_map_info.map_buff[i][j] =     strFileInfo_buff[rvc_map_info.header.width* i + j];
          }
      }
      char map_buf_info[]; //Provide information of room outline coordinate points
      if (map_buf_info!= NULL) {    // Check if the data is valid
          int room_count = 2; // Provide the number of room partitions
          RVC_ROOM_POLYGON_S *tmp_room_polygon = NULL;   
          int tmp_points_size = 0;
          int offset_len = 0; //Data offset
          int all_points_size = 0; //The number of all coordinates
          for (i = 0; i < room_count; i++) { //Traverse the room outline information
                all_points_size += xxxx; //Provide the number of boundary point coordinates in all rooms
          }
          tmp_room_polygon = (RVC_ROOM_POLYGON_S*)Malloc(room_count * sizeof(RVC_ROOM_POLYGON_S) + all_points_size * sizeof(RVC_POINT_S));  //Request the memory space for the struct. Note that the polygon member in the struct is a pointer, and you need to assign a value to the pointer object.
          rvc_map_info.room_polygons_data = tmp_room_polygon;
          float tmp_origin_x = (float)map_header.origin.x /1.0f;    //Get the coordinates of the origin
          float tmp_origin_y = (float)map_header.origin.y /1.0f;
          RVC_POINT_S* points_memory =(RVC_POINT_S*)(tmp_room_polygon + room_count);  //Coordinate point data points to the secondary pointer
          for ( i = 0; i < room_count; i++) { //Traverse the room outline information
              tmp_room_polygon->id = (map_buf_info[4 + 8 * i ] << 24) | (map_buf_info[3 + 8 * i] << 16) | (map_buf_info[2 + 8 * i] << 8) | map_buf_info[1 + 8 * i];
              tmp_room_polygon->polygon.points_size = (map_buf_info[8 + 8 * i] << 24) | (map_buf_info[7 + 8 * i] << 16) | (map_buf_info[6 + 8 * i] << 8) | map_buf_info[5 + 8 * i];
              PR_DEBUG("room id :%d, points size :%d", tmp_room_polygon->id, tmp_room_polygon->polygon.points_size);
              offset_len = 1 + 8 * room_count + tmp_points_size; //Data offset
              tmp_room_polygon->polygon.points = points_memory;
              for (j = 0; j < tmp_room_polygon->polygon.points_size; j++) {      //The coordinates are reported to the panel based on the screen coordinate system. You can adjust it according to your own map algorithm.
                  memcpy(&tmp_room_polygon->polygon.points[j].x, map_buf_info.buff + (offset_len + j * 12), sizeof(RVC_POINT_S));
                  tmp_room_polygon->polygon.points[j].x = tmp_room_polygon->polygon.points[j].x * 100 / 5 + (tmp_origin_x);   //Based on the resolution and origin coordinates of the world coordinate system, calculate the screen coordinate data and report it to the panel for display
                  tmp_room_polygon->polygon.points[j].y = (tmp_origin_y) - (tmp_room_polygon->polygon.points[j].y * 100 / 5) ;
                  tmp_room_polygon->polygon.points[j].z = tmp_room_polygon->polygon.points[j].z * 100 / 5;  
                // PR_DEBUG("point[%d] x:%f,y:%f, z:%f", j, tmp_room_polygon->polygon.points[j].x, tmp_room_polygon->polygon.points[j].y,tmp_room_polygon->polygon.points[j].z);
               }
               tmp_points_size += (tmp_room_polygon->polygon.points_size * sizeof(RVC_POINT_S));   //Add the number of coordinate points in the room
               points_memory += tmp_room_polygon->polygon.points_size;   //Address offset of secondary pointer
               tmp_room_polygon++;         //Get the outline information of the next room
           }
           rvc_map_info.room_polygons_count = room_count;
           int  ret = ty_rvc_rt_map_send_v2(&rvc_map_info);   //Report map data
           if(rvc_map_info.room_polygons_data != NULL) {
              Free(rvc_map_info.room_polygons_data);
           }
      }
        //The requested memory must be released later
      return ret;
  }
  

Route data

For the format of route data, see the RVC_PATH_INFO_S struct.

No.	Field	Data type	Description
1	path_id	UINT16_T	The route map ID of a complete cleaning. This field is reported incrementally at the beginning of each cleaning.
2	type	UINT32_T	The type of route. Valid values: 0x01: Full route (standard mode) 0x02: Full route (complex mode) 0x03: navigation route
3	direction	UINT32_T	The device’s heading angle. For more information, see Device angle.
4	count	UINT32_T	The total number of waypoints in a full route.
5	path_point	PATH_POINT_S	The array size of the route coordinates is count × 8, which can be given by referring to the Robot coordinate system. The reported value = (original value ÷ grid length) × 10.

Route reporting routine

/**
 * @brief Converts the route data
 * @param [out] path_info is converted to RVC_PATH_INFO_S and then output
 */
void __ty_path_data_unpack(RVC_PATH_INFO_S* path_info)
{
    /*The route is based on the robot coordinate system. The output coordinate raw data is of the Float type. Generally, the raw data will be converted in the algorithm stage. The route coordinate transmission value = (route coordinate raw value ÷ 0.05) * 10
    The algorithm is as follows:
    path_point.x = htons(((int16_t)((path.poses[i].pose.position.x)  / 0.05 * 10));
    path_point.y = htons(((int16_t)((path.poses[i].pose.position.y)  / 0.05 * 10);
    Transfer path_point.x and path_point.y to the path_buf cache and give it to the application.
    */
    char path_buf[1000] = {0};  //Route data obtained from the algorithm

    path_info->path_id = 0;  //Report incrementally at the beginning of each cleaning
    path_info->path_type = 3;  //Navigation route
    path_info->direction = 88;  //Device heading angle
    path_info->point_count =  10;    //Total number of waypoints is 10
    path_info->path_points = Malloc(path_info->point_count * sizeof(PATH_POINT_S));  //Request the array of route coordinates

    char* dest_data = Malloc(path_info->point_count *4);       
    if(dest_data == NULL){
        PR_ERR("malloc failed");
        return;
    }
    memset(dest_data, 0,path_info->point_count *4);  
    memcpy(dest_data, &path_buf[0], path_info->point_count *4);     //Copy the original value of the route to memory
    for(int i = 0; i < path_info->point_count;i++){
        path_info->path_points[i].coordinates.x = ((dest_data[4*i] << 8) & 0xff00) | (dest_data[4*i+1] & 0xff);   //Convert the device's 2-byte valid coordinate data to int type (x, y)
        path_info->path_points[i].coordinates.y = (dest_data[4*i+2] << 8) | (dest_data[4*i+3]);
        path_info->path_points[i].type = 0;   //Fill in the cleaning waypoints according to the actual waypoint type
    }
    Free(dest_data);
}

/**
 * @brief  Example of reporting the route
 * @param
 */
void test_main(void)
{
    OPERATE_RET ret = OPRT_OK;
    while (1) {
  	RVC_PATH_INFO_S path_data_proper = {0};
	__ty_path_data_unpack(&path_data_proper);
	ret = ty_rvc_rt_cleanpath_send_v2(&path_data_proper); //Report the route
	if(path_data_proper.path_points != NULL) {
  	   Free(path_data_proper.path_points);
	}
      	sleep(5);
    }
}

Map coordinate system

When the robot vacuum performs a new cleaning task, it starts from the initial position and builds a map while cleaning to form a cleaning map and cleaning route. As the map gradually expands and the route continuously accumulates, the process involves the world coordinate system, the robot coordinate system, and the screen coordinate system displayed on the mobile app. Their relationship is as follows:

World coordinate system

The world coordinate system refers to the actual physical coordinate system, that is, the plane coordinate system of the room. Each time the robot vacuum cleans, a certain point in the world coordinate system is used as the starting point. The position of this point in the world coordinate system remains unchanged during this cleaning process. The world coordinate system helps users reference and understand the robot coordinate system.

Robot coordinate system

The robot coordinate system is established with the starting point as the origin and referenced to the world coordinate system. During the cleaning process, the origin is used as a reference to get the coordinate values ​​of the waypoints.

Screen coordinate system

The screen coordinate system is used for mapping on mobile apps, with the top left corner of the screen R(0,0) as the origin of this coordinate system.

Conversion between map coordinates

Map data

• The map formed after cleaning is an irregular polygon. For ease of processing, the circumscribed rectangle of the polygon is taken as the robot’s map. The map data is an array in the form of UINT8_T. The total number of map pixels is Map_width × Map_height. The first data point in the map array is UINT8_T0, which is the top left corner R(0,0).
• The robot origin is O(x,y), where x/y is the number of pixels relative to R. As cleaning progresses, the map width and height will change, so the position of the origin O in the map array will also change (the x/y values ​​change).

Route data

Route data is a collection of traversed location points, whose coordinates are INT32 values. The current location point is P(x,y), where x/y is the value of point P relative to point O.

Area data

The coordinate data of the cleaning area, no-go area frame, virtual wall line, and point and go are INT32 values. The coordinate point is M(x,y), where x/y is the value of point M relative to point O, based on the robot coordinate system.

Device angle

• The device direction (front collision orientation) takes the world coordinate as the origin, with the angle increasing counterclockwise.
• The angle range is 0° to 360° (0x00 to 0x168).
• 0° represents the X-axis direction of the world coordinate system.