Audio and Video Capabilities

This topic describes how to integrate the live video preview and audio talk features of laser robot vacuums.

Integration process

Audio and video features depend on basic services such as networking and time. Therefore, you need to wait until MQTT is connected and the time is synchronized before pushing audio and video stream data.

The demo included in the development kit provides a typical startup procedure for a laser robot vacuum. The following diagram shows the process:

APIs

Initialize the media abstraction layer

/**
 * @brief Initialize Tuya SDK for media adapter
 *
 * @param[in] p_media_adatper_info: TUYA_ROBOT_SDK_MEDIA_ADAPTER_S
 * @param[in] p_media_infos: DEVICE_ROBOT_MEDIA_INFO_T
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_sdk_media_adapter_init(TUYA_ROBOT_SDK_MEDIA_ADAPTER_S *p_media_adatper_info, DEVICE_ROBOT_MEDIA_INFO_T *p_media_infos);

Register the callback for the audio and video data sent from the mobile app to the device, and the callback for the device to get snapshot data and send it back to the mobile app. Input the audio and video codec configuration parameters.

Initialize the ring buffer

/***********************************************************
 *@function: ty_rvc_media_ring_buffer_init
 *@brief  initialize media ring buffer init
 *@param[in] pMediaInfo: DEVICE_SDK_MEDIA_INFO_S
 *@param[in] max_buffer_seconds: buffer timer 6~10S
 *@param[in] channel:  media channel
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/
OPERATE_RET ty_rvc_media_ring_buffer_init(CONST DEVICE_SDK_MEDIA_INFO_S* pMediaInfo, INT_T max_buffer_seconds, INT_T channel);

Initialize a specific stream of a specific video channel. This operation also creates buffers and initializes nodes. For more information, see Audio and Video Ring Buffer.

Register audio and video P2P event callbacks

/***********************************************************
 *@function: ty_rvc_media_av_event_init
 *@brief:  sdk audio and video registration callback int
 *@param  [in] event_cb: audio and video stream trans cb
 *@return [*]OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/
OPERATE_RET ty_rvc_media_av_event_init(RVC_AV_STREAM_EVENT_CB event_cb);

Register audio and video P2P event callbacks to transmit events such as live video preview and audio talk to the device, allowing users to handle the events. For specific events, refer to the tuya_sweeper_av_event_cb function in the demo.

The following diagram shows the process of some events:

Initialize the device media service

/**
 * @brief real time map and path trans cb
 */
typedef INT_T (*RVC_TRANS_EVENT_CB)(IN CONST int onoff);

/***********************************************************
 *@Function: ty_rvc_server_init
 *@brief  initialize media server init
 *@param[in] handler The status callback for real-time data transmission,
             with statuses including: start and end.
 *@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 ***********************************************************/

OPERATE_RET ty_rvc_server_init(RVC_TRANS_EVENT_CB handler);

Initialize the media service. After starting this interface, you can stream audio and video data as expected.

Example

/**
 * @brief Create an audio and video streaming task
 * @param  
 * @return [*]
 */
void tuya_av_start(void)
{
    PR_DEBUG("create av task!!!\n");
    int ret = 0;
    ret = tal_vi_init(NULL, TY_DSP_MEDIA_VI_MAX);     //Initialize the underlying TKL hardware video input
    if (0 != ret) {
        PR_ERR("tal_vi_init failed,%d\n", ret);
        return ;
    }
    ret = tal_venc_init(0, NULL, TY_DSP_MEDIA_VENC_MAX);   //Initialize the underlying TKL video encoding
    if (0 != ret) {
        PR_ERR("tal_venc_init failed,%d\n", ret);
        return ;
    }
    ret = tal_ai_init(-1, TY_DSP_MEDIA_AUDIO_MAX);     //Initialize the underlying TKL hardware audio input
    if (0 != ret) {
        PR_ERR("tal_ai_init failed,%d\n", ret);
        return ;
    }
    /*Create a thread*/
    pthread_t main_pid = -1;
    pthread_t sub_pid = -1;
    pthread_t audio_pid = -1;
    pthread_create(&main_pid, NULL, __video_main,NULL);
    pthread_create(&sub_pid, NULL, __video_sub,NULL);
    pthread_create(&audio_pid, NULL, __audio_main,NULL);

    return ;
}

The above example from the laser robot vacuum demo shows how to implement simple audio and video streaming based on the x86 platform. When developing audio and video features, you need to configure parameters to suit your hardware platform. For more information, see Video Input and Audio Input and Output.

Learn more

• For basic knowledge and functional introduction of audio and video, see Live Preview and Two-Way Talk.
• If you need to implement SD card storage and cloud storage for your robot vacuum devices, refer to Cloud Storage and Local Storage.