Video Input

This topic describes the complete process of IPC video capture, including video input (VI), video encoding (VENC) initialization, and video streaming.

Description

VI refers to the process of capturing encoded video data. The common video specifications for the IPC:

• 2-channel encoded video
  • High-resolution H.265 stream
  • Low-resolution H.264 stream
• 1-channel image
  Used for alert reporting or secondary recognition.
• 2-channel YUV
  • QR code scan (640 x 360 px)
  • Motion detection (160 x 90 px)

The following figure shows the data stream for the FH8636 platform.

The following figure shows the data stream for the T31 platform.

Implementation

Load drivers and allocate memory

• Some platforms have sensor drivers. For example, the sensor driver for the Anyka platform is loaded during file system initialization.

  insmod /drv/ak_ion.ko
  insmod /drv/ak_uio.ko
  insmod /drv/ak_pcm.ko
  insmod /drv/ak_i2c.ko
  if [ -e /drv/sensor_sc2336.ko ]; then
      insmod /drv/ak_isp.ko
      insmod /drv/sensor_sc2336.ko
  fi
  if [ -e /drv/sensor_sc3336.ko ]; then
      insmod /drv/ak_isp.ko internal_pclk=100000000
      insmod /drv/sensor_sc3336.ko
  fi
  

  The kernel’s board support package (BSP) on the Anyka platform handles media memory allocation.

• Some platforms do not have sensor drivers, such as the Fullhan platform. The code includes the register sequence of the sensor.

  #Allocate memory
  insmod /drv/vmm.ko mmz=anonymous,0,0xA2600000,26624K anony=1
  #0xA2600000 = 0xA0000000 + 0x2600000, 0x2600000 = 38 MB, indicating the system memory. 26624 KB = 26 MB, indicating the size of the media memory zone.
  #Load driver
  insmod /drv/media_process.ko
  insmod /drv/isp.ko
  insmod /drv/enc.ko
  insmod /drv/jpeg.ko
  insmod /drv/bgm.ko
  insmod /drv/nna.ko
  

  How the driver is loaded varies by platform. You only need to focus on initializing the sensor’s register sequence.

Configure video properties

Configure the basic properties of the sensor, including the sensor’s length and width, sensor name, default flipping, and ISP image style.

Video encoding

Configure the properties of the encoder. For more information, see the example in this topic.

As shown in the feature description, there is a coupling between streams. For example, if a substream and QR code scanning use the same channel of the vision processing unit (VPU), their resolutions should both be set to 640 x 360 px.

Video streams can be input actively and passively.

• Active VI: The application layer calls the streaming API to request a stream.
• Passive VI: Register a callback for streaming videos from the underlying layer.

How to

Only one method of VI can be applied to a single chip platform. By default, active input is adopted unless otherwise specified.

Active VI

Passive VI

Passive VI follows the same logic as active VI, except that the callbacks are registered before tal_venc_init. The application layer does not need to call the tal_venc_get_frame to request streaming. The underlying layer starts a thread automatically and calls put_cb to send videos.

Data structure

Configure initial parameters

// vi and isp
typedef struct{
    CHAR_T conf[128];                                             // isp conf path
    CHAR_T conf1[128];                                            // isp conf1 path
    CHAR_T conf2[128];                                            // isp conf2 path
    CHAR_T conf3[128];                                            // isp conf3 path
    CHAR_T version[32];                                           // version num
    CHAR_T name[16];                                              // isp sensor num
    TKL_ISP_DN_SWITCH_CONFIG_T isp_dn_switch_config;              // ADN switch config
    TKL_VI_SENSOR_CBUS_TYPE_E sensor_type;                       // sensor control bus type
    INT32_T              addr;                                    // sensor address
    INT32_T              width;
    INT32_T              height;
    TKL_ISP_FIG_STYLE_CONFIG_T isp_fig_style_day;                 // isp fig style in day
    TKL_ISP_FIG_STYLE_CONFIG_T isp_fig_style_night;               // isp fig style in night
    INT32_T              fps;                                     // sensor fps
}TKL_VI_ISP_CONFIG_T;

typedef struct {
    INT32_T enable;          // 1,enable,0,disable
    TKL_VI_CHN_E chn;        // video input channel
    INT32_T mirror;          // mirror defaults
    INT32_T filp;            // filp defaults
    TKL_VI_ISP_CONFIG_T isp; // isp config
    VOID *pdata;             // reserver data
} TKL_VI_CONFIG_T;

// venc
typedef enum
{
    TKL_VIDEO_PB_FRAME = 0,
    TKL_VIDEO_I_FRAME,
    TKL_VIDEO_TS_FRAME,
    TKL_AUDIO_FRAME,
    TKL_CMD_FRAME,
    TKL_MEDIA_FRAME_MAX,
}TKL_MEDIA_FRAME_TYPE_E;

typedef enum
{
    TKL_CODEC_VIDEO_MPEG4 = 0,
    TKL_CODEC_VIDEO_H263,
    TKL_CODEC_VIDEO_H264,
    TKL_CODEC_VIDEO_MJPEG,
    TKL_CODEC_VIDEO_H265,
    TKL_CODEC_VIDEO_YUV420,
    TKL_CODEC_VIDEO_YUV422,
    TKL_CODEC_VIDEO_MAX = 99,

    TKL_CODEC_AUDIO_ADPCM,
    TKL_CODEC_AUDIO_PCM,
    TKL_CODEC_AUDIO_AAC_RAW,
    TKL_CODEC_AUDIO_AAC_ADTS,
    TKL_CODEC_AUDIO_AAC_LATM,
    TKL_CODEC_AUDIO_G711U,
    TKL_CODEC_AUDIO_G711A,
    TKL_CODEC_AUDIO_G726,
    TKL_CODEC_AUDIO_SPEEX,
    TKL_CODEC_AUDIO_MP3,
    TKL_CODEC_AUDIO_MAX = 199,
    TKL_CODEC_TYPE_MAX
}TKL_MEDIA_CODEC_TYPE_E;

typedef struct
{
    UINT_T left;                                         // The x-coordinate displayed in the top left corner of the OSD.
    UINT_T top;                                          // The y-coordinate displayed in the top left corner of the OSD.
    UINT_T font_w;                                       // The width of the font displayed on the OSD.
    UINT_T font_h;                                       // The height of the font displayed on the OSD.
}TKL_VENC_OSD_CONFIG_T;                                  // Configure the OSD property.

typedef struct {
    UINT_T enable;                    // 1,enable,0,disable
    TKL_VENC_CHN_E chn;               // video encode channel
    TKL_VENC_TYPE_E type;             // stream work type
    TKL_MEDIA_CODEC_TYPE_E codectype; // codec type
    UINT_T fps;                       // fps
    UINT_T gop;                       // I Frame interval
    UINT_T bitrate;                   // bitrate,kbps
    UINT_T width;
    UINT_T height;
    UINT_T min_qp;
    UINT_T max_qp;
    TKL_VENC_DATA_TRANS_MODE_E trans_mode;
    TKL_VENC_PUT_CB put_cb;
    TKL_VENC_OSD_CONFIG_T osd;
} TKL_VENC_CONFIG_T;

#define TAL_VI_CONFIG_T            TKL_VI_CONFIG_T
#define TAL_VI_HARDWARE_SOURCE_T   TKL_VI_HARDWARE_SOURCE_T

Set commands and structs

typedef enum {
    TAL_VI_CMD_MIRROR_FLIP,
    TAL_VI_CMD_ANTIFLICKER,
    TAL_VI_CMD_ILLUMIN,
    TAL_VI_CMD_DAYMODE,
    TAL_VI_CMD_SENSOR_REG,
} TAL_VI_CMD_E;

typedef enum {
    TAL_VENC_CMD_OSD,         // osd
    TAL_VENC_CMD_IDR,         // set IDR
    TAL_VENC_CMD_STREAM_BUFF, // Set the circular cache area of the video memory pool. For example, set two 30K memory
                              // pools for storage
    TAL_VENC_CMD_START,       // venc start
    TAL_VENC_CMD_STOP,        // venc stop
} TAL_VENC_CMD_E;

typedef struct {
    INT32_T enable;
    INT32_T is_dls;
} TKL_VENC_OSD_T;

Data type of streaming request

typedef struct {
    TKL_MEDIA_FRAME_TYPE_E frametype; // bitrate,kbps
    TKL_MEDIA_CODEC_TYPE_E codectype; // codec type
    CHAR_T *pbuf;                     // frame buffer
    UINT_T buf_size;                  // buffer size
    UINT_T used_size;                 // used buffer size
    UINT_T width;                     // frame width
    UINT_T height;                    // frame height
    UINT64_T pts;                     // sdk pts
    UINT64_T timestamp;               // system utc time, unit: ms
    UINT_T seq;                       // frame sequence number
    UINT_T fragment;                  // frame sequence's fragment 
    BYTE_T seq_error;                 // frame sequence is error 
    BYTE_T fragment_is_last;          // frame sequence's fragment is last fragment
} TKL_VENC_FRAME_T;                   // video frame

#define TAL_VENC_FRAME_T       TKL_VENC_FRAME_T

Callbacks

typedef INT_T (*TKL_VENC_PUT_CB)(TKL_VENC_FRAME_T *pframe);

Video input

Initialization

The IPC only supports 1-channel VI currently, so count is 1.

/**
 * @brief vi init
 *
 * @param[in] pconfig: vi config
 * @param[in] count: count of pconfig
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_vi_init(TAL_VI_CONFIG_T *pconfig, INT32_T count);

Set and get parameters

/**
 * @brief vi set
 *
 * @param[in] chn: vi chn
 * @param[in] cmd: cmd
 * @param[in] parg: parg
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_vi_set(INT32_T chn, TAL_VI_CMD_E cmd, VOID *parg);

/**
 * @brief vi get
 *
 * @param[in] chn: vi chn
 * @param[in] cmd: cmd
 * @param[out] parg: parg
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_vi_get(INT32_T chn, TAL_VI_CMD_E cmd, VOID *parg);

Deinitialization

/**
 * @brief vi uninit
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_vi_uninit(VOID);

Video encoding

Initialization

The IPC only supports 1-channel VI currently, so specify vi_chn as 0.

/**
 * @brief video encode init
 *
 * @param[in] vi_chn: vi channel number
 * @param[in] pconfig: venc config
 * @param[in] count: count of pconfig
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_venc_init(INT32_T vi_chn, TAL_VENC_CONFIG_T *pconfig, INT32_T count);

Set and get parameters

/**
 * @brief video set
 *
 * @param[in] vi_chn: vi channel number
 * @param[in] venc_chn: venc chn
 * @param[in] cmd: cmd
 * @param[in] parg: parg
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_venc_set(INT32_T vi_chn, INT32_T venc_chn, TAL_VENC_CMD_E cmd, VOID *parg);

/**
 * @brief video get
 *
 * @param[in] vi_chn: vi channel number
 * @param[in] venc_chn: venc chn
 * @param[in] cmd: cmd
 * @param[out] parg: parg
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_venc_get(INT32_T vi_chn, INT32_T venc_chn, TAL_VENC_CMD_E cmd, VOID *parg);

Request streaming

/**
 * @brief video encode get frame
 *
 * @param[in] vi_chn: vi channel number
 * @param[in] venc_chn: venc channel number
 * @param[out] pframe: output frame
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_venc_get_frame(INT32_T vi_chn, INT32_T venc_chn, TAL_VENC_FRAME_T *pframe);

Deinitialization

/**
 * @brief video encode uninit
 *
 * @param[in] vi_chn: vi channel number
 *
 * @return OPRT_OK on success. Others on error, please refer to tkl_error_code.h
 */
OPERATE_RET tal_venc_uninit(INT32_T vi_chn);

Example

// Configure and initialize the VI parameter:
    TAL_VI_CONFIG_T *pvi = NULL;
    // The IPC has only one VI channel currently.
    pvi = &gMediaInfo.vi[0];
    pvi->enable = true;
    pvi->chn = TKL_VI_0;
    pvi->mirror = ISP_SENSOR_MIRROR;          // The initial flipping.
    pvi->filp = ISP_SENSOR_FILP;
    memcpy(pvi->isp.conf, ISP_SENSOR_CONF, sizeof(pvi->isp.conf));       // ISP configuration file
    memcpy(pvi->isp.version, ISP_FW_VERSION, sizeof(pvi->isp.version));
    memcpy(pvi->isp.name, ISP_SENSOR_NAME, sizeof(pvi->isp.name));
    pvi->isp.width = ISP_SENSOR_WIDTH;
    pvi->isp.height = ISP_SENSOR_HEIGHT;
    memcpy(&pvi->isp.isp_dn_switch_config, &g_Tuya_DN_SWITCH_CONFIG_DAY, sizeof(pvi->isp.isp_dn_switch_config));
    ...
    ret = tal_vi_init(pinfo->vi, 1);
    ...

// Configure the VENC parameter.
    S_DSP_MEDIA_INFO gMediaInfo = {
    /* Mainstream */
    .venc[E_DSP_MEDIA_VIDEO_MAIN].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].chn = TKL_VENC_0,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].type = TKL_VENC_MAIN,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].fps = 20,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].gop = 40,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].bitrate = 1024,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].width = 1920,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].height = 1080,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].codectype = 4,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].min_qp = 26,
    .venc[E_DSP_MEDIA_VIDEO_MAIN].max_qp = 48,

    /* Substream */
    .venc[E_DSP_MEDIA_VIDEO_SUB].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_SUB].chn = TKL_VENC_1,
    .venc[E_DSP_MEDIA_VIDEO_SUB].type = TKL_VENC_SUB,
    .venc[E_DSP_MEDIA_VIDEO_SUB].fps = 20,
    .venc[E_DSP_MEDIA_VIDEO_SUB].gop = 40,
    .venc[E_DSP_MEDIA_VIDEO_SUB].bitrate = 512,
    .venc[E_DSP_MEDIA_VIDEO_SUB].width = 640,
    .venc[E_DSP_MEDIA_VIDEO_SUB].height = 360,
    .venc[E_DSP_MEDIA_VIDEO_SUB].codectype = 2,
    .venc[E_DSP_MEDIA_VIDEO_SUB].min_qp = 26,
    .venc[E_DSP_MEDIA_VIDEO_SUB].max_qp = 48,

    /* The third stream, reserved */
    .venc[E_DSP_MEDIA_VIDEO_3RD].enable = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].chn = TKL_VENC_2,
    .venc[E_DSP_MEDIA_VIDEO_3RD].type = TKL_VENC_3RD,
    .venc[E_DSP_MEDIA_VIDEO_3RD].fps = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].gop = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].bitrate = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].width = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].height = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].codectype = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].min_qp = 0,
    .venc[E_DSP_MEDIA_VIDEO_3RD].max_qp = 0,

    /* The fourth stream, reserved */
    .venc[E_DSP_MEDIA_VIDEO_4TH].enable = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].chn = TKL_VENC_3,
    .venc[E_DSP_MEDIA_VIDEO_4TH].type = TKL_VENC_4TH,
    .venc[E_DSP_MEDIA_VIDEO_4TH].fps = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].gop = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].bitrate = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].width = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].height = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].codectype = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].min_qp = 0,
    .venc[E_DSP_MEDIA_VIDEO_4TH].max_qp = 0,

    /* Image capture */
    .venc[E_DSP_MEDIA_VIDEO_SNAP].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].chn = TKL_VENC_4,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].type = TKL_VENC_SNAP,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].fps = 20,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].gop = 40,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].bitrate = 512,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].width = 640,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].height = 360,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].codectype = 3,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].min_qp = 15,
    .venc[E_DSP_MEDIA_VIDEO_SNAP].max_qp = 15,

    /* Motion detection */
    .venc[E_DSP_MEDIA_VIDEO_MD].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_MD].chn = TKL_VENC_5,
    .venc[E_DSP_MEDIA_VIDEO_MD].type = TKL_VENC_MD,
    .venc[E_DSP_MEDIA_VIDEO_MD].fps = 20,
    .venc[E_DSP_MEDIA_VIDEO_MD].gop = 40,
    .venc[E_DSP_MEDIA_VIDEO_MD].bitrate = 512,
    .venc[E_DSP_MEDIA_VIDEO_MD].width = 320,
    .venc[E_DSP_MEDIA_VIDEO_MD].height = 180,
    .venc[E_DSP_MEDIA_VIDEO_MD].codectype = 5,

    /* Human shape */
    .venc[E_DSP_MEDIA_VIDEO_HD].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_HD].chn = TKL_VENC_6,
    .venc[E_DSP_MEDIA_VIDEO_HD].type = TKL_VENC_HD,
    .venc[E_DSP_MEDIA_VIDEO_HD].fps = 20,
    .venc[E_DSP_MEDIA_VIDEO_HD].gop = 40,
    .venc[E_DSP_MEDIA_VIDEO_HD].bitrate = 512,
    .venc[E_DSP_MEDIA_VIDEO_HD].width = 512,
    .venc[E_DSP_MEDIA_VIDEO_HD].height = 288,
    .venc[E_DSP_MEDIA_VIDEO_HD].codectype = 5,

    /* QR code */
    .venc[E_DSP_MEDIA_VIDEO_QR].enable = 1,
    .venc[E_DSP_MEDIA_VIDEO_QR].chn = TKL_VENC_7,
    .venc[E_DSP_MEDIA_VIDEO_QR].type = TKL_VENC_QR,
    .venc[E_DSP_MEDIA_VIDEO_QR].fps =25,
    .venc[E_DSP_MEDIA_VIDEO_QR].gop = 50,
    .venc[E_DSP_MEDIA_VIDEO_QR].bitrate = 512,
    .venc[E_DSP_MEDIA_VIDEO_QR].width = 640,
    .venc[E_DSP_MEDIA_VIDEO_QR].height = 360,
    .venc[E_DSP_MEDIA_VIDEO_QR].codectype = 5,

// Initialize VENC:
    ret = tal_venc_init(0, pinfo->venc, 8);
    ...

// Active input of the mainstream. It is recommended to allocate 300 KB to the mainstream:
    TAL_VENC_FRAME_T frame = {0};
    frame.pbuf = (UCHAR_T *)malloc(300*1024);
    frame.buf_size = 300*1024;
    ret = tal_venc_get_frame(0, 0, &frame)
    ...

// Get substream:
    frame.pbuf = (UCHAR_T *)malloc(100*1024);
    frame.buf_size = 100*1024;
    ret = tal_venc_get_frame(0, 1, &frame)

// Image capture
    frame.pbuf = (UCHAR_T *)malloc(150*1024);
    frame.buf_size = 150*1024;
    ret = tal_venc_get_frame(0, TKL_VENC_SNAP, &frame)
    ...

// Turn on/off OSD
    TKL_VENC_OSD_T osd = {0};
    osd.enable = 1; // On/off
    osd.is_dls = 0; // Specifies if daylight saving time is in effect.
    ret = tal_venc_set(0, 0, TAL_VENC_CMD_OSD, &osd);
    ...

// Set flipping
    TKL_VI_MIRROR_FLIP_E val = TKL_VI_MIRROR_FLIP_NONE;
    ret =tal_vi_set(0, TAL_VI_CMD_MIRROR_FLIP, &val);
    ...
    val = TKL_VI_MIRROR_FLIP;
    ret = tal_vi_set(0, TAL_VI_CMD_MIRROR_FLIP, &val);

FAQs

Why does VI or VENC initialization fail?

• Check if the driver is fully loaded.
• Check if the media memory is sufficient.