Media Streaming

Background

Integrating with streaming components allows you to add streaming media capability to your IoT products. The real-time streaming protocol (RTSP) is used to establish and control the connection between the streaming server and the client (such as a mobile app) for real-time media data transfer.

The RTSP service is used for two purposes:

• Connection: The client can establish a TCP connection with end devices for media transfer if there is no internet access.
• Media streaming: The client of media servers can issue commands such as play, record, and pause, to facilitate real-time control of the media streaming.

Application scenarios

This solution uses the BK7231U module as the microcontroller. The BK7231U comes with Bluetooth 5.1 and Wi-Fi 802.11n combo connectivity. The streaming component on the module is connected to the mobile app through the RTSP protocol.

• This solution is ideal for developing products like beauty and personal care devices such as camera built-in earwax removers, blackhead removers, and oral irrigators as well as accessories for outdoor use.
• Support integration with features such as OTA updates and media streaming.
• Due to the constraints in hardware, this solution applies to products with compact designs that do not require advanced and complicated features.

Things to note

• If you develop earwax removers, contact the project manager to request the specific control panel.
• Only the OEM app supports the RTSP service.
• Currently, the SDK is built for the BK7231U module only. Contact the project manager to request the SDK.
• Check if your SDK has the two files tuya_svc_lan_rtsp.c and tuya_svc_lan_rtsp.h, on which your project will be dependent.

Interaction

Applications interact with the SDK

Render video frames

The buffer size depends on the frame format. For example, VGA quality (640 x 480 pixel) video output requires a buffer of more than 30 KB.

Control device using mobile app

APIs

Register the streaming control callback

#include "tuya_svc_lan_rtsp.h"
/**
 * @brief register stream src to rtsp
 * 
 * @param[in] src stream info
 *
 * @return OPRT_OK when success. For other errors, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_svc_lan_reg_rtsp_stream_src(rtsp_stream_src_t src);

Streaming control struct

typedef struct {
    int (*get_sample_rate)(RTSP_MEDIA_TYPE_E type);
    /* `get_frame` is deprecated. Make sure to use the interface `get_frame_fragment`. */
    int (*get_frame)(int user_id, RTSP_MEDIA_TYPE_E type, char** buf, int *plen, uint64_t *pts); // Returns 0 on success. The I-frame (the first frame) is required.
    int (*get_frame_fragment)(int user_id, rtsp_frame_fragment_t* pframe);
    RTP_CODEC_E (*get_codec)(RTSP_MEDIA_TYPE_E type);
    int (*start)(); // Returns `user_id`.
    int (*stop)(int user_id);
    int (*get_name)(char* buf, int* buf_len);
} rtsp_stream_src_t;

Data for use in RTSP

Enumerate types of RTSP streaming media

typedef enum {
    RTSP_MEDIA_TYPE_INVALID = -1,
    RTSP_MEDIA_TYPE_VIDEO = 0,
    RTSP_MEDIA_TYPE_AUDIO,
    RTSP_MEDIA_TYPE_NUM,
} RTSP_MEDIA_TYPE_E;

Frame format of streaming media

typedef struct {
    RTSP_MEDIA_TYPE_E         type;  
    char*                     buf;   
    int                       len;   
    uint64_t                  pts;
    uint8_t                   is_eof;
    uint8_t                   fragment_id; 
} rtsp_frame_fragment_t;

Sample code

The following code provides an example of implementing the RTSP media streaming feature on the adaptation layer. You can port the RTSP code provided by the BK7231U platform. The code for the camera feature is for reference only, which should be implemented based on your choice of camera.

// The libraries included.
#include "ty_video_lib.h"             // Camera-related file, which can be changed based on your choice of camera.
#include "tuya_hal_network.h"         // Networking APIs.
#include "tuya_hal_semaphore.h"       // The semaphore.
#include "tuya_svc_lan_rtsp.h"        // Contains the RTSP, INIT, UNINIT, and REGISTER APIs.
#include "rtsp_server.h"              /* Contains the RTSP callback struct, frame struct, and RTSP APIs, which can be used to create a local server when debugging with VLC. Do not use the APIs in this file when you debug with the mobile app. */

typedef struct {
    int pos[RTSP_MEDIA_TYPE_NUM];
    uint64_t last_pts[RTSP_MEDIA_TYPE_NUM];
} user_info_s;
user_info_s* users[5] = {0};           // The RTSP component can save up to 5 streams.
SEM_HANDLE sem_handle;                 // The semaphore for reading images from the camera.
SEM_HANDLE sem_handle_rtsp;            // The semaphore for RTSP streaming.
struct video_pkg *vb_pkg;              // The data structure used to control camera.
UCHAR_T test_tmp_buf[16384];           // Saves full images. Each image is 16 KB in size.

struct video_pkg{
    BYTE_T *pkg_bufs;       /* The data */
    UINT_T actual_len;      /* The actual length of the data */
    BYTE_T pkg_id;          /* The serial number of an image */
};

// Get the file format.
RTP_CODEC_E get_codec(RTSP_MEDIA_TYPE_E type)
{
    switch (type){
    case RTSP_MEDIA_TYPE_VIDEO:
        return RTP_CODEC_JPEG;
    case RTSP_MEDIA_TYPE_AUDIO:
        return RTP_CODEC_INVALID;
    default:
        return 0;
    }
}

// The sampling rate differs depending on video types.
int get_sample_rate(RTSP_MEDIA_TYPE_E type)
{
    switch (type){
    case RTSP_MEDIA_TYPE_VIDEO:
        return 90000;
    case RTSP_MEDIA_TYPE_AUDIO:
        return 8000;
    default:
        return 8000;
    }
}

// Initialize the user_id. RTSP supports multi streams.
int start()
{
    int i;
    for(i = 0; i < 5; i++){
        if (NULL == users[i]){
            users[i] = malloc(sizeof(user_info_s));
            users[i]->last_pts[RTSP_MEDIA_TYPE_VIDEO] = 0;
            users[i]->last_pts[RTSP_MEDIA_TYPE_AUDIO] = 0;
            users[i]->pos[RTSP_MEDIA_TYPE_VIDEO] = 0;
            users[i]->pos[RTSP_MEDIA_TYPE_AUDIO] = 0;
            return i;
        }
    }
    return -1;
}
// Free the user_id.
int stop(int user_id)
{
    if (users[user_id]){
        free(users[user_id]);
        users[user_id] = NULL;
    }
    return 0;
}

Get the stream name that is used by the RTSP to pull streams.
int get_name(char* buf, int* buf_len)
{
    char name[] = "stream_0";
    memcpy(buf, name, sizeof(name));

    *buf_len = sizeof(name);

    return 0;
}

/* Get the image frame.
 * IN:  user_id, pframe->type
 * OUT: pframe->buf
 */
STATIC INT_T __get_frame_fragment_jpeg(INT_T user_id, rtsp_frame_fragment_t* pframe)
{
    user_info_s* pinfo = users[user_id];
	RTSP_MEDIA_TYPE_E type = pframe->type;
    INT_T pos = pinfo->pos[type];
    INT_T start = -1, end = -1;
    INT_T media_len;
    CONST INT_T frame_rate[2] = {28, 50};
	UINT_T time_now;

	time_now = rtos_get_time();//uni_time_get_posix_ms();

    if (time_now - pinfo->last_pts[type] < 1000 / frame_rate[type]) {
        return -1;
    }

	if (pframe == NULL || user_id >=5 || user_id < 0) {
		PR_ERR("param invalid");
		return -1;
	}
	
	tuya_os_adapt_semaphore_wait(sem_handle_rtsp);

	memset(test_tmp_buf, 0, sizeof(test_tmp_buf));
	memcpy(test_tmp_buf, vb_pkg->pkg_bufs, vb_pkg->actual_len - 5);
	media_len = vb_pkg->actual_len - 5;

	tuya_os_adapt_semaphore_post(sem_handle);
	

    if (type == RTSP_MEDIA_TYPE_VIDEO){
        PR_DEBUG("frame len:%d", media_len);
        pframe->buf = test_tmp_buf;
    }
    else {
    	return -1;      
    }

	pframe->len = media_len;
    pframe->pts = time_now;
    pframe->is_eof = 1;
    
    pinfo->last_pts[type] = time_now;    
    
    pinfo->pos[type] = media_len;

    if (pinfo->pos[type] > 0){
        pinfo->pos[type] = 0;
    }

    return 0;
}

/* Initialize the RTSP application, which can be called after device_init. */
VOID user_rtsp_init()
{
    CHAR_T url[256] = {0};
    int url_len = 256;
    rtsp_stream_src_t src = {0};	
    
    /* Register the RTSP control function. */
    src.get_codec = __get_codec;
    //src.get_frame = __get_frame_jpeg;    // This interface is deprecated.
    src.get_frame_fragment = __get_frame_fragment_jpeg;
    src.get_sample_rate = __get_sample_rate;
    src.start = __start;
    src.stop = __stop;
    src.get_name = __get_name;
    tuya_svc_lan_reg_rtsp_stream_src(src);       
  	
    /* Camera feature APIs, implemented as needed. */
    video_device *soc_video_device = NULL;
    struct video_pix_format v_format;
    struct vbq_config video_buff_q_config;
    
    /* Configure the tuya_config.h file. Then, call tuya_soc_camera_init to build the system and create nodes. */
    tuya_soc_camera_init();
    soc_video_device = tuya_video_dev_find("ty_video");
    if(soc_video_device == NULL){
        PR_DEBUG("soc_video_device is null");
        return OPRT_COM_ERROR;
    }

    /* Initialize video_device. */
    tuya_video_dev_open(soc_video_device);

    /* Set the format. */
    v_format.p_type = QVGA_320_240;
    v_format.pixelformat = V4L2_PIX_FMT_JPEG;
    v_format.field = V4L2_FIELD_ANY;
    v_format.colorspace = V4L2_COLORSPACE_JPEG;
    tuya_video_dev_ctl(soc_video_device,CMD_SET_FMT,&v_format);
    
    /* Set the buffer. */
    video_buff_q_config.video_pkg_buffer_len = 16 * 1024;
    video_buff_q_config.video_pkg_num = 2;
    tuya_video_dev_ctl(soc_video_device,CMD_STREAM_SET_BUFF,&video_buff_q_config);
    
    /* Start */
    tuya_video_dev_ctl(soc_video_device,CMD_STREAM_START,NULL);

	tuya_os_adapt_semaphore_create_init(&sem_handle, 0, 1);
	tuya_os_adapt_semaphore_create_init(&sem_handle_rtsp, 0, 1);

    while(1)
    {    
        /* Read images from camera and wait for the RTSP to fetch them. */
        tuya_video_dev_read(soc_video_device,&vb_pkg);        
		tuya_os_adapt_semaphore_post(sem_handle_rtsp);

        /* Free the buffer that has been read.  */  
        tuya_os_adapt_semaphore_wait(sem_handle);        
        tuya_video_dev_ctl(soc_video_device, CMD_STREAM_DEQUEEU_BUFF,NULL);
    }
}

SDK download

This SDK is not generally available. Contact the project manager and request the SDK.

SDK directory

+-- software
¦   +-- IoTOS2.3.4_ty_iot_sdk_2.3.4_bk7231u_1.1.0    # TuyaOS IoT SDK
¦   ¦   +-- apps	 				   # The developer guides with samples.
¦   ¦   +-- sdk 	 				   # The header files and libraries.
¦   ¦   +-- platforms	 			  # The library from the chip platform and toolkit.
¦   ¦   +-- CHANGELOG.md 	 		 # The version history.
¦   ¦   +-- README.md 	 			  # The introduction to developing with the SDK.
¦   ¦   +-- build_app.sh 	 		 # The script for compilation.
¦   +-- Tuya IoTOS IoT SDK x.x.x Release Notes .pdf   	# The release notes of the SDK.
+-- pc
¦   +-- tools 	 					    # The development toolkit.
+-- hardware
¦   +-- board 	 					    # The materials of the module and open source hardware.
¦   +-- chip   					    # The materials of the chip.

Get started

After the initialization with device_init, call user_rtsp_init() to register the RTSP application callback.

FAQs