Wukong AI Capabilities

This topic describes how to integrate Wukong AI capabilities into a robot vacuum, focusing on utilizing Tuya’s interfaces to enable intelligent interaction features and audio/video data processing.

Feature description

After integrating Wukong AI capabilities, the robot vacuum can achieve intelligent interaction with users, analyze and process audio and video data, and thus provide an enhanced user experience. The system offers comprehensive AI interaction lifecycle management, including initialization, pre-record configuration, starting interactions, and stopping interactions.

Interfaces and usage

Initialize AI ​​capabilities

tuya_ipc_ai_station_init

Feature description

Initialize the AI station functionality. This interface must be called first before using any AI capabilities.

Interface definition

/**
 * @brief init ai station
 *
 * @param VOID
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_station_init();

Return value

Parameter	Description
OPRT_OK	Initialization succeeded.
Others	An error occurred. For the specific error codes, refer to tuya_error_code.h.

Initialize AI ​​chat

tuya_ipc_ai_chat_init

Feature description

Initialize the AI chat functionality, and configure the event callback function and audio playback callback function. This is the foundational interface for enabling AI chat features.

Interface definition

/**
/**
 * @brief         callback of ai cmd
 * @param[in]     cmd. specifically refer to TUYA_AI_CMD_TYPE_E
 * @param[in]     args. additional data for cmd
 * @return        VOID
 */
typedef VOID (*TUYA_IPC_AI_CMD_CB)(TUYA_AI_CMD_TYPE_E cmd, CONST PVOID_T args);

/**
 * @brief         callback of audio play
 * @param[in]     audio_data. if the AI voice ends, audio_data will be equal to NULL
 * @param[in]     len. len of audio data. if the AI voice ends, len will be equal to 0
 * @return        VOID
 */
typedef VOID (*TUYA_IPC_AI_AUDIO_PLAY_CB)(CONST CHAR_T *audio_data, INT_T len);

/**
 * @brief init ai chat
 *
 * @param[in] event_cb: ai event cb
 * @param[in] audio_cb: audio play cb
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_chat_init(TUYA_IPC_AI_CMD_CB event_cb, TUYA_IPC_AI_AUDIO_PLAY_CB audio_cb);

Parameter description

Parameter	Description
event_cb	The operation commands or status to be handled by you.
audio_cb	The callback function for audio playback.

Return value

Parameter	Description
OPRT_OK	Initialization succeeded.
Others	An error occurred. For the specific error codes, refer to tuya_error_code.h.

Start AI interaction

tuya_ipc_ai_station_start_act

Feature description

Start the current AI interaction session. After calling this interface, the system will begin retrieving audio and video data from the ring buffer and transmitting it to the cloud.

Interface definition

/**
 * @brief start a conversation
 *
 * @param VOID
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_station_start_act();

Return value

Parameter	Description
OPRT_OK	Initialization succeeded.
Others	An error occurred. For the specific error codes, refer to tuya_error_code.h.

Stop AI interaction

tuya_ipc_ai_station_stop_act

Feature description

Stop the current AI interaction session. After the call, the device stops sending data to the cloud and starts receiving data returned by the cloud. The cloud data will be notified to you through TUYA_IPC_AI_CMD_CB event_cb, TUYA_IPC_AI_AUDIO_PLAY_CB audio_cb.

Interface definition

/**
 * @brief stop a conversation
 *
 * @param VOID
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_station_stop_act();

Return value

Parameter	Description
OPRT_OK	Initialization succeeded.
Others	An error occurred. For the specific error codes, refer to tuya_error_code.h.

Set audio pre-record duration

tuya_ipc_ai_set_chat_pre_time

Feature description

Set the audio pre-record duration. After initiating a chat, the system will retrieve historical audio and video data of the specified duration from the ring buffer.

Interface definition

/**

@brief set chat pre time duration
@param[in] chat pre time
@return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
*/
OPERATE_RET tuya_ipc_ai_set_chat_pre_time(INT_T pre_time_ms);

Parameter description

pre_time_ms: Specifies the length of the audio pre-record duration, in milliseconds. For example, setting this parameter to 2000 instructs the system to retrieve audio and video data from the past 2 seconds.

Return value

Parameter	Description
OPRT_OK	Initialization succeeded.
Others	An error occurred. For the specific error codes, refer to tuya_error_code.h.

Scenarios

This interface should be called when a user begins a chat, before executing the start interaction action.

After calling the tuya_ipc_ai_station_start_act interface, Wukong AI retrieves audio and video data from the ring buffer spanning from the current timestamp backward by pre_time_ms milliseconds and transmits this data to the cloud for processing.

Procedure

The following example demonstrates the typical workflow for using the AI capabilities:

#include "tuya_ipc_ai.h"
#include "tuya_error_code.h"

// AI event callback function
void ai_event_callback(AI_EVENT_E event, void *args) {
    switch(event) {
        case E_AI_CMD_CHAT_START:
            /*Start a chat*/
            break;
        case E_AI_CMD_CHAT_FIN:
            /*End the chat*/
            break;
        // Handle other AI events...
    }
}

// Audio playback callback function
void audio_play_callback(const unsigned char *audio_data, unsigned int len) {
    // Implement audio playback logic
}

// Initialize the AI chat functionality
void init_ai_chat() {
    OPERATE_RET ret = tuya_ipc_ai_chat_init(ai_event_callback, audio_play_callback);
    if (ret != OPRT_OK) {
        printf("Failed to initialize AI chat: %d\n", ret);
        return;
    }
    printf("AI chat initialized successfully\n");
}
// Initialize AI features
OPERATE_RET init_ai_features() {
    // Initialize the AI station
    OPERATE_RET ret = tuya_ipc_ai_station_init();
    if (ret != OPRT_OK) {
        printf("Failed to initialize AI station: %d\n", ret);
        return ret;
    }
    OPERATE_RET ret = tuya_ipc_ai_chat_init(ai_event_callback, audio_play_callback);
    if (ret != OPRT_OK) {
        printf("Failed to initialize AI chat: %d\n", ret);
        return;
    }
    // Set a 2-second pre-record duration
    ret = tuya_ipc_ai_set_chat_pre_time(2000);
    if (ret != OPRT_OK) {
        printf("Failed to set chat pre time: %d\n", ret);
        return ret;
    }

    printf("AI features initialized successfully\n");
    return OPRT_OK;
}

// Manage AI interaction sessions
void manage_ai_session() {
    // Start an AI interaction session
    OPERATE_RET ret = tuya_ipc_ai_station_start_act();
    if (ret != OPRT_OK) {
        printf("Failed to start AI interaction: %d\n", ret);
        return;
    }

    // … Execute AI interaction logic …

    // Stop the AI interaction session
    ret = tuya_ipc_ai_station_stop_act();
    if (ret != OPRT_OK) {
        printf("Failed to stop AI interaction: %d\n", ret);
        return;
    }
}
int main() {
    // IoT SDK initialization and device online reporting
    // Initialize the AI functionality
    if (init_ai_features() != OPRT_OK) {
        return -1;
    }
    // Manage the AI session
    manage_ai_session();

    return 0;
}

FAQs

Why is the pre-record time necessary?

The pre-record time ensures the complete beginning part of the user’s voice command is captured, thereby improving recognition accuracy.

Does the AI interaction session support concurrency?

It is recommended to maintain only one active AI interaction session at a time to ensure system stability.

Why am I not receiving audio data from the cloud?

Currently, the cloud only supports PCM and Opus audio formats. The issue might arise if the audio data encoded into the ring buffer does not comply with these format requirements.

By utilizing these interfaces appropriately, you can implement comprehensive AI interaction capabilities in your robot vacuum, making your product more smart and enhancing user experience.