Wukong AI

Based on the TuyaOS Wukong AI Hardware Development Framework, the IPC Development Framework has deeply integrated auditory and visual perception capabilities, delivering multimodal conversational capability. This capability can be deployed across multiple domains, including daily conversations, home security, elderly care, children’s toys, and children’s teaching aids.

Related files

• tuya_ipc_ai_station.h: AI conversation core
• tuya_ipc_ai_player.h: Audio playback (TTS and music)
• tuya_mcp_server.h: MCP tool service

AI conversation

API description

Initialization

/**
 * @brief init ai chat
 *
 * @param[in] event_cb: ai event cb
 * @param[in] audio_cb: audio play cb
 * @param[in] cloud_vad_flag. TRUE: enable cloud vad; FALSE: disable cloud vad
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_chat_init(TUYA_IPC_AI_CMD_CB event_cb, TUYA_IPC_AI_AUDIO_PLAY_CB audio_cb, BOOL_T cloud_vad_flag);

Parameters

Parameter	Description
event_cb	The operation commands or status that you need to handle.
audio_cb	The callback function for audio playback.
cloud_vad_flag	Whether to enable cloud voice activity detection (VAD).

Start sending data

Start an AI conversation and send data. After the call, the audio and video data about 0.5 seconds before the current time will be taken from the ring buffer and sent to the cloud.

/**
 * @brief start a conversation
 *
 * @param VOID
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_station_start_act();

Stop sending data

Stop sending data in the current AI ​​conversation. After the call, the device stops sending data to the cloud and starts receiving data returned by the cloud. You receive cloud data through TUYA_IPC_AI_CMD_CB event_cb and TUYA_IPC_AI_AUDIO_PLAY_CB audio_cb.

/**
 * @brief stop a conversation
 *
 * @param VOID
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_station_stop_act();

Integration description

Single conversation mode

After one start and stop cycle, the system waits for all the cloud data to be returned before initiating the next conversation.

Continuous conversation mode

On-device audio detection algorithm determines whether someone is speaking by analyzing real-time audio inputs. In this case, it is necessary to address scenarios where AI conversations are interrupted and to pay special attention to the effectiveness of the on-device audio detection algorithms.

Music playback

The music playback feature extends the Wukong AI conversation framework and supports music playback control through voice commands. After the cloud recognizes the intent, it sends an E_AI_CMD_MUSIC_CONTROL event, and the device calls the playback API to play music.

API description

Play music

Play music from the specified URL.

/**
 * @brief start music playback
 *
 * @param[in] url: audio stream URL
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_player_start_music(CHAR_T *url);

Stop music

Stop current music playback and clear the playback progress.

/**
 * @brief stop music playback
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_player_stop_music(VOID);

Pause music

Pause current music playback and keep the playback progress.

/**
 * @brief pause music playback
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_player_pause_music(VOID);

Resume music

Resume music playback from the paused position.

/**
 * @brief resume music playback
 *
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_ipc_ai_player_resume_music(VOID);

Implementation

Handle music-related events in the TUYA_IPC_AI_CMD_CB callback:

• E_AI_CMD_MUSIC_CONTROL: Receives a cloud command. args is TUYA_IPC_AI_MUSIC_INFO_T *, which contains action (play, stop, resume, prev, or next), url, and the has_tts flag.
• E_AI_CMD_CHAT_PLAYER_FIN: Signals that TTS playback has finished. If has_tts=TRUE, wait for this event before you start music playback.
• E_AI_CMD_MUSIC_PLAYER_START/FIN/PAUSE: Notifies the music playback status.

For an example, see ty_sdk_ai_chat.c.

Technical notes

When has_tts=TRUE, wait for TTS playback to finish (E_AI_CMD_CHAT_PLAYER_FIN) before you call tuya_ipc_ai_player_start_music(). Otherwise, the two audio streams conflict.

Model Context Protocol (MCP)

MCP allows a large model to call local device tools, such as query status or set volume, during a conversation. The SDK handles protocol parsing and tool scheduling. You only need to register tools and callbacks.

API description

Initialize the service

/**
 * @brief init MCP server
 *
 * @param[in] name: server name
 * @param[in] version: server version
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
OPERATE_RET tuya_mcp_server_init(CONST CHAR_T *name, CONST CHAR_T *version);

Register a tool

Register a tool with the MCP service. Use the macro TUYA_MCP_TOOL_ADD, which supports variadic arguments to append parameter definitions directly.

TUYA_MCP_TOOL_ADD(name, description, callback, user_data, ...properties);

Parameters

Parameter	Description
name	Tool name. The model uses this name to identify and call the tool.
description	Tool description. Directly affects whether the model correctly understands the tool’s purpose.
callback	Callback function that runs when the model calls the tool.
user_data	Custom data that the SDK passes to the callback. Pass NULL if you do not need it.
properties	Tool parameter definitions. Declare them with the following macros.

Parameter definition macros

Macro	Description
MCP_PROP_INT(name, desc)	Integer parameter.
MCP_PROP_INT_DEF(name, desc, def_val)	Integer parameter with a default value.
MCP_PROP_INT_RANGE(name, desc, min, max)	Integer parameter with a value range.
MCP_PROP_INT_DEF_RANGE(name, desc, def_val, min, max)	Integer parameter with a default value and a value range.
MCP_PROP_BOOL(name, desc)	Boolean parameter.
MCP_PROP_BOOL_DEF(name, desc, def_val)	Boolean parameter with a default value.
MCP_PROP_STR(name, desc)	String parameter.
MCP_PROP_STR_DEF(name, desc, def_val)	String parameter with a default value.

Tool callback prototype

/**
 * @param[in] properties: parameters passed by the model
 * @param[out] ret_val: result returned to the model
 * @param[in] user_data: user defined data
 * @return OPRT_OK on success. Others on error, please refer to tuya_error_code.h
 */
typedef OPERATE_RET (*MCP_TOOL_CALLBACK)(
    CONST MCP_PROPERTY_LIST_T *properties,
    MCP_RETURN_VALUE_T *ret_val,
    VOID *user_data);

Destroy the service

Release all registered tools and related resources.

/**
 * @brief destroy MCP server and release all tools
 */
VOID tuya_mcp_server_destroy(VOID);

Implementation example

tuya_mcp_server_init("AI_Ipc", "1.0");

TUYA_MCP_TOOL_ADD("device.info.get", "Get device info.", __get_device_info, NULL);

TUYA_MCP_TOOL_ADD("device.audio.volume_set", "Set volume.",
    __set_volume, NULL,
    MCP_PROP_INT_RANGE("volume", "Volume level (0-100).", 0, 100));

In the callback, use tuya_mcp_return_value_set_json(), set_bool(), set_image(), and similar functions to set the return value. For a complete example, see ty_sdk_ai_mcp_example.c.

Technical notes

The tool’s description directly affects whether the model calls the tool correctly. Write clear English descriptions.

FAQs

Why does the device receive no audio data from the cloud?

Currently, the cloud only supports audio in pcm and opus. The issue might arise if the audio data that you place in the ring buffer does not comply with these format requirements.