Pet Detection, Profile, and Audio

The Tuya Pet Detection Software Development Kit (SDK) provides features such as pet detection, pet profile management, and MQTT-based pet audio file delivery.

Feature overview

Feature	Description	Optional
Pet detection	Detects and tracks pets based on video frames. Reports results through callbacks.	Core
Pet profile	Binds, unbinds, updates, and queries profiles. The Tuya Kernel Layer (TKL) implements features.	Optional (need_pet_profile)
Pet audio	Downloads or deletes audio files from the cloud via MQTT. Stores files in a specified directory.	Optional (need_audio_download)

Layered architecture and integration points

Application layer (user)
    ↓ Call tuya_pet_detect_init, tuya_pet_detect_update, and tuya_pet_detect_get_pet_list, and register callbacks.
SDK layer (svc_pet_detect_api)
    ↓ Capture frames, schedule detection, synchronize profiles, and manage MQTT files.
Tuya Abstraction Layer (TAL, internal components)
    ↓ Call TKL functions directly.
TKL layer (user-implemented)
    ↓ tkl_npu_pet_algo_init, tkl_npu_pet_detect_track, and so on.
Algorithm and hardware (user)

• Direct use: APIs and types in tuya_pet_detect_api.h.
• Mandatory implementation: TKL layer functions (see tkl_npu_pet_algo.h). Implement them in your project and link them. You do not need to register.

How to integrate

1. Prepare paths and directories
   Allocate and specify model_dir, image_dir, audio_dir in TUYA_PET_DETECT_PATH_T as required.

2. Implement the TKL layer
   Implement at least tkl_npu_pet_algo_init and tkl_npu_pet_detect_track. Implement feature-related TKL functions if pet profile features are required.

3. Call the initialization API
   Call tuya_pet_detect_init() when appropriate during application startup. Pass paths, parameters, callbacks, and optional MQTT parameters.

4. Process callbacks
   Cast result in the detection callback to your custom or demo structure to handle detections and alerts. Process profile operations and MQTT file synchronization as needed.

Path and initialization

Path structure (essential)

path_opt is required and must not be NULL. Pass all paths through one structure. All three fields are pointers (const char *) that can point to strings of any length. A NULL or empty string means the path is not used. The SDK copies non-NULL paths internally during initialization (truncated if the length exceeds 128 characters). You can release or reuse the original buffer after the initialization returns.

#define TUYA_PET_DETECT_PATH_LEN (128) /* Maximum length for internal SDK copying. The input pointer can point to a longer string. */

typedef struct {
    const char *model_dir;  /* Model file directory. Required (non-NULL and non-empty) if need_pet_profile=1. Otherwise, set to NULL. */
    const char *image_dir;  /* Pet face image directory. Required (non-NULL and non-empty) if store_pet_face_image=1. Otherwise, set to NULL. */
    const char *audio_dir;  /* Audio file directory. Required (non-NULL and non-empty) if need_audio_download=1. Otherwise, set to NULL. */
} TUYA_PET_DETECT_PATH_T;

• model_dir: Directory for algorithm model files. Required only if need_pet_profile=1 (non-NULL and non-empty).
• image_dir: Directory for saving pet face images. Required only if store_pet_face_image=1 (valid only when need_pet_profile=1).
• audio_dir: Directory for downloading and deleting pet audio files. Required only if need_audio_download=1. You must store this path (for example, in a global variable) for use in file_sync_cb to scan, report, or play files. The SDK does not provide a path query interface.

Initialization interface

OPERATE_RET tuya_pet_detect_init(
    const TUYA_PET_DETECT_PATH_T        *path_opt,   /* Path configuration. Mandatory and cannot be NULL. */
    const void                          *params,     /* Algorithm configuration. Required if need_pet_profile=1. Otherwise, set to NULL. */
    const TUYA_PET_DETECT_CB_T          *cb,         /* Detection result and profile operation callback. NULL if unused. */
    const TY_SYS_CUSTOM_MQTT_PARAM_T     *mqtt_param, /* Required if need_audio_download=1. Otherwise, set to NULL. */
    const TUYA_PET_DETECT_INIT_OPT_T     *init_opt    /* Initialization options. NULL = all disabled by default. Set bits to 1 to enable features. */
);

Initialization options (init_opt):

typedef struct {
    unsigned int need_pet_profile     : 1;  /* 1 = Enable pet profile (models, profile sync, profile processing, and so on). */
    unsigned int need_audio_download  : 1;  /* 1 = Enable MQTT pet audio download (mqtt_param is required). */
    unsigned int store_pet_face_image : 1;  /* 1 = Download and store pet face images to image_dir. */
} TUYA_PET_DETECT_INIT_OPT_T;

• init_opt == NULL: All features are disabled by default (all bits are 0). Explicitly pass init_opt and set the corresponding bit to 1 to enable pet profiles, audio download, or face image storage.
• Audio download only: Set need_pet_profile = 0 and need_audio_download = 1. Fill only path_opt.audio_dir. params can be NULL. mqtt_param is required.

Feature and mandatory parameter checklist (error prevention):

Enabled feature	Condition	Mandatory parameters
Pet profile	need_pet_profile == 1	path_opt->model_dir is non-NULL and non-empty. params is non-NULL.
Pet face image storage	store_pet_face_image == 1	path_opt->image_dir is non-NULL and non-empty (valid only when need_pet_profile=1)
Audio download	need_audio_download == 1	path_opt->audio_dir is non-NULL and non-empty. mqtt_param is non-NULL.

Parameter validation and fallback:
The SDK validates parameters during initialization based on the table above. The SDK returns an error code (such as OPRT_INVALID_PARM) if mandatory parameters are missing. The SDK stops initialization and does not access invalid pointers. Always check the return value of tuya_pet_detect_init. Use pet detection features only if the return value is OPRT_OK to avoid exceptions from invalid parameters.

Integration example (full features)

#include "tuya_pet_detect_api.h"

/* Store the audio directory for file_sync_cb, playback, and other uses. */
static char s_pet_audio_dir[TUYA_PET_DETECT_PATH_LEN] = {0};

void app_pet_detect_init(const char *base_path)
{
    static char s_model_dir[TUYA_PET_DETECT_PATH_LEN];
    static char s_image_dir[TUYA_PET_DETECT_PATH_LEN];
    static char s_audio_dir[TUYA_PET_DETECT_PATH_LEN];
    snprintf(s_model_dir, sizeof(s_model_dir), "%s/model", base_path);
    snprintf(s_image_dir, sizeof(s_image_dir), "%s/face_image", base_path);
    snprintf(s_audio_dir, sizeof(s_audio_dir), "%s/audio", base_path);
    strncpy(s_pet_audio_dir, s_audio_dir, sizeof(s_pet_audio_dir) - 1);

    TUYA_PET_DETECT_PATH_T path_opt = {
        .model_dir = s_model_dir,
        .image_dir = s_image_dir,
        .audio_dir = s_audio_dir,
    };

    /* Algorithm configuration: The type is agreed between user and SDK. This is a sample structure. */
    TUYA_PET_DETECT_CONFIG_T pet_config = {0};
    pet_config.detect_body_opt = true;
    pet_config.detect_face_opt = true;
    pet_config.track_opt = true;
    /* ... Assign other fields as needed ... */

    TUYA_PET_DETECT_CB_T cb = {
        .pet_detect_result_get_cb = my_detect_result_cb,
        .oper_result_cb           = my_operate_result_cb,
    };

    TY_SYS_CUSTOM_MQTT_PARAM_T mqtt_param = {
        .status_cb    = my_audio_status_cb,
        .file_sync_cb = my_pet_file_sync_cb,
    };

    /* All features disabled by default. Explicitly pass init_opt and set bits for full features. */
    TUYA_PET_DETECT_INIT_OPT_T init_opt = {
        .need_pet_profile = 1,
        .need_audio_download = 1,
        .store_pet_face_image = 1,
    };
    OPERATE_RET ret = tuya_pet_detect_init(&path_opt, &pet_config, &cb, &mqtt_param, &init_opt);
    if (ret != OPRT_OK) {
        /* Process initialization failure. */
    }
}

Callback description

Detection result callback

typedef int (*TUYA_PET_DETECT_GET_DETECT_RESULT_CB)(void *result, int result_count);

• result: The buffer for detection results of the current frame. Cast this pointer to a custom or demo result structure (such as TUYA_PET_DETECT_RESULT_T*) in the callback.
• result_count: The number of targets detected in the current frame.
• Important: The memory pointed to by result might be reused or released by the SDK after the callback returns. Do not hold this pointer outside the callback. Copy the data if you need to retain it.

Profile operation callback

typedef int (*TUYA_PET_DETECT_GET_OPERATE_RESULT_CB)(TUYA_PET_FILE_OPERATE_E operate, int err_code);

This callback triggers after binding, unbinding, updating, or querying a profile. operate indicates the operation type, and err_code == 0 indicates success.

MQTT file status and synchronization (optional)

• status_cb: Triggers when the status of a file download or deletion changes (for example, downloading, success, or failure). eventType distinguishes download and deletion. Use DOWNLOAD_STATUS_* or DELETE_STATUS_* from the documentation for status.
• file_sync_cb: The SDK calls this after a file operation is complete. In the callback, use the stored audio directory (such as s_pet_audio_dir above) to scan files (such as .wav). Report the list to the panel for sync via Data Points (DPs).

Other APIs

Runtime update parameters

OPERATE_RET tuya_pet_detect_update(const void *params);

params must match the type used in tuya_pet_detect_init. This call is passed to tkl_npu_pet_algo_update() in the TKL layer.

Query profile list

OPERATE_RET tuya_pet_detect_get_pet_list(TUYA_PET_FILE_INFO_T *pet_list, OUT int *count);

Pre-allocate the pet_list (recommended length is PET_MEMBER_MAX). The SDK returns the actual number of profiles in count.

TKL implementation checklist

The TKL independent functions provide algorithm capabilities. These are declared in tkl_npu_pet_algo.h. (The header file is provided by the SDK or adaptation layer. You can implement and link the functions).

Mandatory functions (detection only)

Function	Description
tkl_npu_pet_algo_init(const char *model_path, const void *params)	Loads the model and initializes the algorithm. Returns 0 on success.
tkl_npu_pet_detect_track(const void *image, void *result_out, int *result_count)	Performs detection and tracking. Cast image to tkl_pet_image_t*, write results to result_out, and set *result_count.

Optional functions (profile features)

Function	Description
tkl_npu_pet_feature_extract_from_data(...)	Extracts face features from image data (such as JPEG or PNG).
tkl_npu_pet_face_feature_add	Writes features when adding a profile.
tkl_npu_pet_face_feature_update	Updates features when updating a profile.
tkl_npu_pet_face_feature_delete	Removes features when deleting a profile.

Optional functions (general)

Function	Description
tkl_npu_pet_algo_release(void)	Releases resources when the detection thread exits.
tkl_npu_pet_algo_update(const void *params)	Corresponds to tuya_pet_detect_update().
tkl_npu_pet_detect_result_buffer_size(void)	Returns the required byte size for the result buffer. Defaults to 4,096 bytes if not implemented.
tkl_npu_pet_face_feature_query	Queries features by profile ID. Used for validation before deletion.

If a TKL function is not implemented, the SDK uses weak symbol stubs for linking. In this case, detection results will be empty or profile features will be unavailable.

Image and result types

• Input image: After capturing a frame, the TAL layer fills the tkl_pet_image_t structure (including width, height, format, pic_data, and pic_len). Pass this structure as a const void* pointer to tkl_npu_pet_detect_track. Cast the pointer to tkl_pet_image_t* for use.
• Output result: The SDK allocates result_out. Cast it to a custom result structure in the TKL layer and write data. The result structure must match the type cast in the upper-layer callback (refer to TUYA_PET_DETECT_RESULT_T in the demo).

Reference for configuration and result types

• Algorithm configuration (params): The SDK interface layer uses const void *params. The type is agreed between user and SDK. You can use or customize the demo type TUYA_PET_DETECT_CONFIG_T provided in tuya_pet_detect_demo_types.h.
• Detection result (result): Within the callback, cast result to TUYA_PET_DETECT_RESULT_T* (provided in the example) or a custom structure. Ensure the structure layout matches result_out written in the TKL layer.

Notes and FAQs

Paths

You must allocate and pass all three directories via path_opt. Store the audio directory for use in file_sync_cb and playback.

MQTT audio only (no profile)

Set init_opt.need_pet_profile = 0. Fill only audio_dir in path_opt. params can be NULL. mqtt_param is required.

Result pointer lifecycle

The result pointer is valid only within the callback. Do not use it outside the callback. Copy data in the callback for persistence.

TKL registration

You do not need to register. Implement TKL functions and link them. The SDK calls them directly via the TAL layer.

Result buffer size

Defaults to 4,096 bytes. Implement tkl_npu_pet_detect_result_buffer_size() and return the actual byte size if the result structure is larger.