English
English
简体中文
Contact Us
Register
Log In

Media Streaming

Last Updated on : 2022-11-24 09:20:26download

Background

Integrating with streaming components allows you to add streaming media capability to your IoT products. The real-time streaming protocol (RTSP) is used to establish and control the connection between the streaming server and the client (such as a mobile app) for real-time media data transfer.

The RTSP service is used for two purposes:

  • Connection: The client can establish a TCP connection with end devices for media transfer if there is no internet access.
  • Media streaming: The client of media servers can issue commands such as play, record, and pause, to facilitate real-time control of the media streaming.

Application scenarios

This solution uses the BK7231U module as the microcontroller. The BK7231U comes with Bluetooth 5.1 and Wi-Fi 802.11n combo connectivity. The streaming component on the module is connected to the mobile app through the RTSP protocol.

  • This solution is ideal for developing products like beauty and personal care devices such as camera built-in earwax removers, blackhead removers, and oral irrigators as well as accessories for outdoor use.
  • Support integration with features such as OTA updates and media streaming.
  • Due to the constraints in hardware, this solution applies to products with compact designs that do not require advanced and complicated features.

Things to note

  • If you develop earwax removers, contact the project manager to request the specific control panel.
  • Only the OEM app supports the RTSP service.
  • Currently, the SDK is built for the BK7231U module only. Contact the project manager to request the SDK.
  • Check if your SDK has the two files tuya_svc_lan_rtsp.c and tuya_svc_lan_rtsp.h, on which your project will be dependent.

Interaction

Applications interact with the SDK

Media Streaming

Render video frames

Media Streaming

The buffer size depends on the frame format. For example, VGA quality (640 x 480 pixel) video output requires a buffer of more than 30 KB.

Control device using mobile app

Media Streaming

APIs

Register the streaming control callback

Function name tuya_svc_lan_reg_rtsp_stream_src
Description After initialization is completed, the device calls this function to register the streaming control callback in the rtsp_stream_src_t. This way, real-time streaming can be implemented after the RTSP server is connected.
Parameters The pointer to the struct rtsp_stream_src_t
Return value See tuya_error_code.h.
#include "tuya_svc_lan_rtsp.h"
/**
 * @brief register stream src to rtsp
 * 
 * @param[in] src stream info
 *
 * @return OPRT_OK when success. For other errors, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_svc_lan_reg_rtsp_stream_src(rtsp_stream_src_t src);

Streaming control struct

Variables Description
get_sample_rate Get the sampling rate.
get_frame_fragment Transmit image frames to the SDK.
get_codec Get the type of the image frame.
start/stop Start or stop streaming.
get_name Get the name of the streaming, which is stream_0 generally.
typedef struct {
    int (*get_sample_rate)(RTSP_MEDIA_TYPE_E type);
    /* `get_frame` is deprecated. Make sure to use the interface `get_frame_fragment`. */
    int (*get_frame)(int user_id, RTSP_MEDIA_TYPE_E type, char** buf, int *plen, uint64_t *pts); // Returns 0 on success. The I-frame (the first frame) is required.
    int (*get_frame_fragment)(int user_id, rtsp_frame_fragment_t* pframe);
    RTP_CODEC_E (*get_codec)(RTSP_MEDIA_TYPE_E type);
    int (*start)(); // Returns `user_id`.
    int (*stop)(int user_id);
    int (*get_name)(char* buf, int* buf_len);
} rtsp_stream_src_t;

Data for use in RTSP

Enumerate types of RTSP streaming media

typedef enum {
    RTSP_MEDIA_TYPE_INVALID = -1,
    RTSP_MEDIA_TYPE_VIDEO = 0,
    RTSP_MEDIA_TYPE_AUDIO,
    RTSP_MEDIA_TYPE_NUM,
} RTSP_MEDIA_TYPE_E;

Frame format of streaming media

typedef struct {
    RTSP_MEDIA_TYPE_E         type;  
    char*                     buf;   
    int                       len;   
    uint64_t                  pts;
    uint8_t                   is_eof;
    uint8_t                   fragment_id; 
} rtsp_frame_fragment_t;

Sample code

The following code provides an example of implementing the RTSP media streaming feature on the adaptation layer. You can port the RTSP code provided by the BK7231U platform. The code for the camera feature is for reference only, which should be implemented based on your choice of camera.

// The libraries included.
#include "ty_video_lib.h"             // Camera-related file, which can be changed based on your choice of camera.
#include "tuya_hal_network.h"         // Networking APIs.
#include "tuya_hal_semaphore.h"       // The semaphore.
#include "tuya_svc_lan_rtsp.h"        // Contains the RTSP, INIT, UNINIT, and REGISTER APIs.
#include "rtsp_server.h"              /* Contains the RTSP callback struct, frame struct, and RTSP APIs, which can be used to create a local server when debugging with VLC. Do not use the APIs in this file when you debug with the mobile app. */

typedef struct {
    int pos[RTSP_MEDIA_TYPE_NUM];
    uint64_t last_pts[RTSP_MEDIA_TYPE_NUM];
} user_info_s;
user_info_s* users[5] = {0};           // The RTSP component can save up to 5 streams.
SEM_HANDLE sem_handle;                 // The semaphore for reading images from the camera.
SEM_HANDLE sem_handle_rtsp;            // The semaphore for RTSP streaming.
struct video_pkg *vb_pkg;              // The data structure used to control camera.
UCHAR_T test_tmp_buf[16384];           // Saves full images. Each image is 16 KB in size.

struct video_pkg{
    BYTE_T *pkg_bufs;       /* The data */
    UINT_T actual_len;      /* The actual length of the data */
    BYTE_T pkg_id;          /* The serial number of an image */
};

// Get the file format.
RTP_CODEC_E get_codec(RTSP_MEDIA_TYPE_E type)
{
    switch (type){
    case RTSP_MEDIA_TYPE_VIDEO:
        return RTP_CODEC_JPEG;
    case RTSP_MEDIA_TYPE_AUDIO:
        return RTP_CODEC_INVALID;
    default:
        return 0;
    }
}

// The sampling rate differs depending on video types.
int get_sample_rate(RTSP_MEDIA_TYPE_E type)
{
    switch (type){
    case RTSP_MEDIA_TYPE_VIDEO:
        return 90000;
    case RTSP_MEDIA_TYPE_AUDIO:
        return 8000;
    default:
        return 8000;
    }
}

// Initialize the user_id. RTSP supports multi streams.
int start()
{
    int i;
    for(i = 0; i < 5; i++){
        if (NULL == users[i]){
            users[i] = malloc(sizeof(user_info_s));
            users[i]->last_pts[RTSP_MEDIA_TYPE_VIDEO] = 0;
            users[i]->last_pts[RTSP_MEDIA_TYPE_AUDIO] = 0;
            users[i]->pos[RTSP_MEDIA_TYPE_VIDEO] = 0;
            users[i]->pos[RTSP_MEDIA_TYPE_AUDIO] = 0;
            return i;
        }
    }
    return -1;
}
// Free the user_id.
int stop(int user_id)
{
    if (users[user_id]){
        free(users[user_id]);
        users[user_id] = NULL;
    }
    return 0;
}

Get the stream name that is used by the RTSP to pull streams.
int get_name(char* buf, int* buf_len)
{
    char name[] = "stream_0";
    memcpy(buf, name, sizeof(name));

    *buf_len = sizeof(name);

    return 0;
}

/* Get the image frame.
 * IN:  user_id, pframe->type
 * OUT: pframe->buf
 */
STATIC INT_T __get_frame_fragment_jpeg(INT_T user_id, rtsp_frame_fragment_t* pframe)
{
    user_info_s* pinfo = users[user_id];
	RTSP_MEDIA_TYPE_E type = pframe->type;
    INT_T pos = pinfo->pos[type];
    INT_T start = -1, end = -1;
    INT_T media_len;
    CONST INT_T frame_rate[2] = {28, 50};
	UINT_T time_now;

	time_now = rtos_get_time();//uni_time_get_posix_ms();

    if (time_now - pinfo->last_pts[type] < 1000 / frame_rate[type]) {
        return -1;
    }

	if (pframe == NULL || user_id >=5 || user_id < 0) {
		PR_ERR("param invalid");
		return -1;
	}
	
	tuya_os_adapt_semaphore_wait(sem_handle_rtsp);

	memset(test_tmp_buf, 0, sizeof(test_tmp_buf));
	memcpy(test_tmp_buf, vb_pkg->pkg_bufs, vb_pkg->actual_len - 5);
	media_len = vb_pkg->actual_len - 5;

	tuya_os_adapt_semaphore_post(sem_handle);
	

    if (type == RTSP_MEDIA_TYPE_VIDEO){
        PR_DEBUG("frame len:%d", media_len);
        pframe->buf = test_tmp_buf;
    }
    else {
    	return -1;      
    }

	pframe->len = media_len;
    pframe->pts = time_now;
    pframe->is_eof = 1;
    
    pinfo->last_pts[type] = time_now;    
    
    pinfo->pos[type] = media_len;

    if (pinfo->pos[type] > 0){
        pinfo->pos[type] = 0;
    }

    return 0;
}

/* Initialize the RTSP application, which can be called after device_init. */
VOID user_rtsp_init()
{
    CHAR_T url[256] = {0};
    int url_len = 256;
    rtsp_stream_src_t src = {0};	
    
    /* Register the RTSP control function. */
    src.get_codec = __get_codec;
    //src.get_frame = __get_frame_jpeg;    // This interface is deprecated.
    src.get_frame_fragment = __get_frame_fragment_jpeg;
    src.get_sample_rate = __get_sample_rate;
    src.start = __start;
    src.stop = __stop;
    src.get_name = __get_name;
    tuya_svc_lan_reg_rtsp_stream_src(src);       
  	
    /* Camera feature APIs, implemented as needed. */
    video_device *soc_video_device = NULL;
    struct video_pix_format v_format;
    struct vbq_config video_buff_q_config;
    
    /* Configure the tuya_config.h file. Then, call tuya_soc_camera_init to build the system and create nodes. */
    tuya_soc_camera_init();
    soc_video_device = tuya_video_dev_find("ty_video");
    if(soc_video_device == NULL){
        PR_DEBUG("soc_video_device is null");
        return OPRT_COM_ERROR;
    }

    /* Initialize video_device. */
    tuya_video_dev_open(soc_video_device);

    /* Set the format. */
    v_format.p_type = QVGA_320_240;
    v_format.pixelformat = V4L2_PIX_FMT_JPEG;
    v_format.field = V4L2_FIELD_ANY;
    v_format.colorspace = V4L2_COLORSPACE_JPEG;
    tuya_video_dev_ctl(soc_video_device,CMD_SET_FMT,&v_format);
    
    /* Set the buffer. */
    video_buff_q_config.video_pkg_buffer_len = 16 * 1024;
    video_buff_q_config.video_pkg_num = 2;
    tuya_video_dev_ctl(soc_video_device,CMD_STREAM_SET_BUFF,&video_buff_q_config);
    
    /* Start */
    tuya_video_dev_ctl(soc_video_device,CMD_STREAM_START,NULL);

	tuya_os_adapt_semaphore_create_init(&sem_handle, 0, 1);
	tuya_os_adapt_semaphore_create_init(&sem_handle_rtsp, 0, 1);

    while(1)
    {    
        /* Read images from camera and wait for the RTSP to fetch them. */
        tuya_video_dev_read(soc_video_device,&vb_pkg);        
		tuya_os_adapt_semaphore_post(sem_handle_rtsp);

        /* Free the buffer that has been read.  */  
        tuya_os_adapt_semaphore_wait(sem_handle);        
        tuya_video_dev_ctl(soc_video_device, CMD_STREAM_DEQUEEU_BUFF,NULL);
    }
}

SDK download

This SDK is not generally available. Contact the project manager and request the SDK.

SDK directory

+-- software
¦   +-- IoTOS2.3.4_ty_iot_sdk_2.3.4_bk7231u_1.1.0    # TuyaOS IoT SDK
¦   ¦   +-- apps	 				   # The developer guides with samples.
¦   ¦   +-- sdk 	 				   # The header files and libraries.
¦   ¦   +-- platforms	 			  # The library from the chip platform and toolkit.
¦   ¦   +-- CHANGELOG.md 	 		 # The version history.
¦   ¦   +-- README.md 	 			  # The introduction to developing with the SDK.
¦   ¦   +-- build_app.sh 	 		 # The script for compilation.
¦   +-- Tuya IoTOS IoT SDK x.x.x Release Notes .pdf   	# The release notes of the SDK.
+-- pc
¦   +-- tools 	 					    # The development toolkit.
+-- hardware
¦   +-- board 	 					    # The materials of the module and open source hardware.
¦   +-- chip   					    # The materials of the chip.

Get started

After the initialization with device_init, call user_rtsp_init() to register the RTSP application callback.

FAQs

Problem description Troubleshooting
platforms\bk7231t\bk7231t_os\beken378\driver\uart\uart.bk.c\bk_printf.
Insufficient buffer size and no checks for string length result in a buffer overflow.
Change the buffer size to 256 or 1024 bytes. Change the string copy function to vsnprintf(string, sizeof(string) - 1, fmt, ap);.
Failed to create threads after updating the compiler toolchain. Update CFLAGS.
Failed to convert the SSID to BSSIDs, which causes device pairing in EZ mode to fail. Make sure you use the legal channels (1 to 13) on the adaptation layer. The BSSID cannot be empty.
The interface tuya_adapter_wifi_station_get_status is used to return the network status. Its values of network status enum might be different from those defined by the BK7231U platform. Check the obtained IP address.
Failed to pair the device. Check if you have uploaded the firmware file to the Tuya IoT Development Platform. If not, upload it and perform a firmware update on the mobile app.
Key-value (KV) database operation failed. Bluetooth is initialized at 1f4000 that is intended to be assigned to Tuya’s KV database. You can turn off the Bluetooth to fix this issue.