Audio and Video Ring Buffer

The ring buffer, designed based on popular video codecs (I-frame and P-frame in H.264 and H.265) and characteristics of audio and video data, can efficiently handle streams in a lock-free and multiple input multiple output (MIMO) manner.

Background

A ring buffer, also known as circular buffer or cyclic buffer, is a data structure that uses a single, fixed-size buffer to store data in a continuous loop.

The producer writes data to the buffer sequentially. When the buffer is full, new data will overwrite the oldest data.

A ring buffer allows one or more consumers to read data sequentially without interfering with each other and prevents new data from overwriting important data.

Features

• Support multiple independent ring buffers for dedicated services including high definition (HD) channel, standard definition (SD) channel, and audio channel.
• A ring buffer can be written by one producer at a time, but can be read simultaneously by multiple consumers without coordination from the producer. For example, multiple consumers can read from the same HD channel.
• If a service is freezing, the internal frame skipping policy will activate automatically to prevent abnormal data overwrite, encoding errors, and audio and video latency.

Components

svc_ring_buffer

Data structure

Frame node

Audio and video frames are stored in the ring buffer as nodes.

typedef struct
{
  UINT_T index;                   // The node index, which is used cyclically.
  MEDIA_FRAME_TYPE_E type;        // The frame type.
  UCHAR_T *raw_data;              // The pointer to raw data.
  UINT_T size;                    // The size of raw data.
  UINT64_T pts;                   // With timestamp.
  UINT64_T timestamp;             // Decoding timestamp.
  UINT_T seq_no;                  // The sequence number of a frame, which is incrementally added.
  UCHAR_T *extra_data;            // Extra data, not in use.
  UINT_T extra_size;              // The size of extra data, not in use.
  UINT_T seq_sync;                // A global sequence number for audio/video sync.
} RING_BUFFER_NODE_T;

Initial parameters

typedef struct
{
    UINT_T bitrate;            // The bitrate (upper limit) in KB.
    UINT_T fps;                // The frame rate.
    UINT_T max_buffer_seconds; // The maximum buffer duration.
    FUNC_REQUEST_KEY_FRAME_CB request_key_frame_cb; // The I-frame callback, defaulting to empty.
} RING_BUFFER_INIT_PARAM_T;

Parameter description

Parameter	Description
max_buffer_seconds	The maximum buffer duration. It is recommended to set a value greater than the duration of a group of pictures (GOP) but less than 10 seconds. A value of 0 means to use the default of 10 seconds.

Callback parameters

typedef VOID (*FUNC_REQUEST_KEY_FRAME_CB)(INT_T device, INT_T channel, IPC_STREAM_E stream);

This callback is used to trigger an external encoder to quickly generate a key frame (I-frame).

Parameter description

Parameter	Description
device	The sub-device ID. This parameter applies to XVR products only. For IP cameras (IPCs) that do not associate with sub-devices, set this parameter to 0.
channel	The channel ID. For products with multiple cameras or sensors, IDs are numbered beginning with 0.
stream	The stream ID, used to identify different types of streams (such as HD and SD) in the same video channel.

API description

Initialization

Set the initial values for device, channel, and stream. This operation also creates buffers and initializes nodes.

/** @brief initialize one ring buffer for one channel(one device)
 * @param[in]      device           device number
 * @param[in]      channel          channel number in device
 * @param[in]      stream           stream number in channel
 * @param[in]      pparam           initialize parameter
 * @return error code
 * - OPRT_OK        init success
 * - Others         init failed
 */
OPERATE_RET tuya_ipc_ring_buffer_init(INT_T device, INT_T channel, IPC_STREAM_E stream, RING_BUFFER_INIT_PARAM_T* pparam);

Deinitialization

Deallocate device, channel, and stream to free up memory and nodes.

/** @brief Deinitialize one ring buffer for one channel(one device)
 * @param[in]     device   device number
 * @param[in]     channel  channel number in device
 * @param[in]     stream   stream number in channel
 * @return error code
 * - OPRT_OK      uninit success
 * - Others       uninit failed
*/
OPERATE_RET tuya_ipc_ring_buffer_uninit(INT_T device, INT_T channel, IPC_STREAM_E stream);

Create buffer

In the data producer (write) or consumer (read) mode, create a user handle to a specific stream buffer. This handle is used for subsequent data operations.

/** @brief open a new session for read/write
 * @param[in]     device   device number
 * @param[in]     channel channel number in device
 * @param[in]     stream   stream number in channel
 * @param[in]     open_type open type
 * @return user handle
*/
RING_BUFFER_USER_HANDLE_T tuya_ipc_ring_buffer_open(INT_T device, INT_T channel, IPC_STREAM_E stream, RBUF_OPEN_TYPE_E open_type);

Close buffer

Close the buffer that corresponds to the user handle.

/** @brief close session
 *  @warning open/close should be called in pair like file
 *  @param[in]   handle      user handle return by open
 *  @return error code
 * - OPRT_OK      success
 * - Others       failed
*/
OPERATE_RET tuya_ipc_ring_buffer_close(RING_BUFFER_USER_HANDLE_T handle);

Write frame

Add a frame to a specific stream buffer.

/** @brief append new frame into a ring buffer
 * @param[in]   handle       user handle return by open
 * @param[in]   addr         the address of the data to append
 * @param[in]   size         size of the data
 * @param[in]   type         media type
 * @param[in]   pts          time stamp in us
 * @return error code
 * - OPRT_OK      success
 * - Others       failed
*/
OPERATE_RET tuya_ipc_ring_buffer_append_data(RING_BUFFER_USER_HANDLE_T handle, UCHAR_T *addr, UINT_T size, MEDIA_FRAME_TYPE_E type, UINT64_T pts);

Write frame with timestamp

Add a frame with timestamp to a specific stream buffer.

/** @brief append new frame into ring buffer with custom timestamp
 * @param[in]   handle       user handle return by open
 * @param[in]   addr         the address of the data to append
 * @param[in]   size         size of the data
 * @param[in]   type         media type
 * @param[in]   pts          time stamp in us
 * @param[in]   timestamp    time stamp in ms
 * @return error code
 * - OPRT_OK      success
 * - Others       failed
 */
OPERATE_RET tuya_ipc_ring_buffer_append_data_with_timestamp(RING_BUFFER_USER_HANDLE_T handle, UCHAR_T *addr, UINT_T size, MEDIA_FRAME_TYPE_E type, UINT64_T pts, UINT64_T timestamp);

Read frame

Read a frame from a specific device, channel, and stream according to the user handle. The location and status information is updated by the API. The data consumer only needs to call this API as needed to get the latest data.

If the data retrieved is overwritten due to a delay, the API will get the latest matched frame.

/** @brief get next frame from ring buffer, will jump to the latest frame if delayed too much time
 * @param[in]     handle  user handle return by open
 * @param[in]     is_retry whether retry to get the last frame
 * @return the ring buffer node or NULL if there is no newer frame in ring buffer
 */
RING_BUFFER_NODE_T *tuya_ipc_ring_buffer_get_frame(RING_BUFFER_USER_HANDLE_T handle, BOOL_T is_retry);

Find pre-frame

In audio streams, the search is performed backward starting from the latest frame until the frame_num frame is reached. In video streams, the search is performed backward starting from the frame_num frame until the I-frame is reached. This API only returns the node of the corresponding frame.

/** @brief start with the newest frame, find previous frame by count frame_num.
for video channel(device), it will keep find previous frame until it's I frame.
 * @param[in]    handle user handle returned by open
 * @param[in]    frame_num    count frame num backwards
 * @return the ring buffer node or NULL if there is no newer frame in ring buffer
*/
RING_BUFFER_NODE_T *tuya_ipc_ring_buffer_find_pre_by_frame(RING_BUFFER_USER_HANDLE_T handle, UINT_T frame_num);

Find and locate pre-frame

In audio streams, the search is performed backward starting from the latest frame until the frame_num frame is reached. In video streams, the search is performed backward starting from the frame_num frame until the I-frame is reached.

With the information about the pre-frame node, the API will update the node with the status of the corresponding user handle and then call tuya_ipc_ring_buffer_get_frame to get the last frame of this pre-frame.

/** @brief start with the newest frame, find previous frame by count frame_num, and update&anchor user(of userIndex) to this frame node.
for video channel(device), it will keep find previous frame until it's I frame.
 * @param[in]    handle user handle returned by open
 * @param[in]    frame_num    count frame num backwards
 * @return the ring buffer node or NULL if there is no newer frame in ring buffer
*/
RING_BUFFER_NODE_T *tuya_ipc_ring_buffer_get_pre_frame(RING_BUFFER_USER_HANDLE_T handle, UINT_T frame_num);

FAQs

When should I allocate and deallocate memory? How much memory is required?

Contiguous memory is allocated when the ring buffer is initialized. This can avoid the out of memory (OOM) error caused by changes in memory usage when the program is run. The footprint for each stream is: (max_buffer_seconds+1) * bitrate

Can I set the buffer duration to a minimum value to save memory?

In IPC applications, the ring buffer is reused across all the services including live preview, local storage, and cloud storage. A short buffer duration can trigger the frame skipping policy if network latency occurs during uploading videos to the cloud, writing videos to local storage, or live preview.

• Common setting: 10 seconds.
• Recommendation: no less than 6 seconds.

Is there a limit on the size of a single frame?

To avoid anomalies, the maximum size of a video frame should be adjusted proportionally to the video bitrate. The upper limit of a frame is 280 KB at a bitrate of 1.5 Mbit/s.

When will frame skipping be triggered?

When the data consumer lags too far behind the producer, memory and node information will be overwritten. This will trigger frame skipping.

The video skips to the latest I-frame, and the audio skips to the latest frame.