AI Capabilities Development

Overview

Against the backdrop of rapid advancements in large model technology, the innovation directions for smart devices continue to expand. The integration of smart devices and AI offers limitless possibilities, such as AI companion robots, AI dolls, AI pets, and other emerging innovative application scenarios.

To help you efficiently implement AI applications, the Tuya Developer Platform offers diverse support, including standardized AI features for different categories, a rich library of smart agent templates, and convenient panel deployment tools. These resources comprehensively ensure the rapid implementation of AI applications across multiple dimensions.

How to implement AI features

Taking AI dolls as an example, their core functionality is an integration of large models, so as to enable personalized character empowerment and deliver smooth, natural conversational experiences.

• Hardware requirements: The device shall have the capability to uplink multimodal data, including the collection and transmission of audio, video, images, text, and other data. For example, to implement AI conversation features, the device must support the hardware configuration of voice reception (microphone) and voice playback (speaker) to ensure smooth human-computer interaction.

• Software requirements: A smart agent shall be deployed on the device to serve as a bridge for interfacing with the large model. The primary responsibility of this agent is to handle the device’s uplink requests, interact with the large model based on the user’s dialogue information, and return the processing results. This way, achieve smart conversation functionalities.

Product AI capabilities

In order to quickly implement product AI scenarios, the platform provides a range of AI capabilities that are common to all business categories. Users can select the required AI capabilities based on product requirements, and the platform will recommend suitable agent templates.

Development procedure

Prerequisites

You have defined product functions and created at least one smart product on the Tuya Developer Platform. For more information, see Create Products.

Configure product agent

1. Log in to the Tuya Developer Platform.

2. In the left-side navigation pane, choose Product > Development. Find the desired product and click Develop in the Operation column. You can configure an AI agent for a product with an AI logo as shown below.

   

   

3. Determine how the product needs to implement AI capabilities and select agents based on the AI ​​functions on the device and panel.

Differences between agent deployment terminals

Develop agents on devices

1. In the Function Definition tab, find Product AI Capabilities and click Add Agent.

   The Product AI Capabilities tab is only visible in product solutions that support AI capabilities. If you do not find this tab on the product development page, it means your solution does not support AI capabilities currently.

   

2. Add agents.

   • Method 1: Recommend Agents based on Capabilities. Select the AI ​​capabilities you want, and then select an agent template. Alternatively, select and bind with an agent that is automatically generated based on the selected AI capabilities, and then click OK.

     If there are no available AI capabilities or recommended agents, refer to Method 3: Select Created Agent under your account, or go to the AI Agent Dev Platform to create an agent first.

     

     

   • Method 2: Select Tuya’s Agent Applications. Tuya provides a range of official agent applications, such as AI pet care miniapp and AI energy saving miniapp. No development is required. As long as the device meets the conditions, the miniapp can be integrated. After selecting an agent application, click Add Now.

     If a product solution is deployed with a Tuya agent application, the agent application will be visible under the solution. Otherwise, the agent application will not be displayed.

     

   • Method 3: Select Created Agent. You can select an agent that has been created and released under your account on the AI Agent Dev Platform, and link it with a product. After selecting an agent, click OK.

     

3. Develop AI hardware. After selecting the agent on the device, proceed to Hardware Development.

   On top of TuyaOS, Tuya provides the Wukong AI Hardware Development Framework designed for the AI hardware domain. v1.0 has been released. You can download the framework through Tuya Wind IDE and develop your own AI products. For more information, see Wukong AI Hardware Development.

   If an SDK with AI features is used in the firmware development, the product will automatically show an AI label.

   

4. Configure device audio parameters.

   If the device supports audio functionality and the agent has been configured, you can configure the audio parameters. After adding the agent on devices, click Configure Audio. Select Audio Format and Timbre to configure device tones, speech speed, and other parameters. And then, click OK.

   

   

Develop agents on panels

1. In the Function Definition tab, find Product AI Capabilities and click Add Agent.

   The Product AI Capabilities tab is only visible in solutions that support AI capabilities. If you do not find this tab on the product development page, it means your solution does not support AI capabilities currently.

2. Add agents.

   This step is the same as adding an agent on the device. For more information, see Develop agents on devices > Add agents.

3. Deploy agents on the panel secondary display.

   Select agents on panels, click Manage Deployment to configure the deployment, and then click OK. After you turn on the switch to show the panel secondary display, you can use the specified agents on the panel.

   

   

   Configure the product panel before you can scan the QR code to try it out. The following figure shows how the deployed agents look like. For more information, see Configure Smart AI Secondary Display.