Prerequisites

Create a panel

You can use the panel miniapp to develop an AI pet device panel based on the Ray framework.

Required conditions

For more information, see Panel MiniApp > Set up environment.

A product defines the data points (DPs) of the associated panel and device. Before you develop a panel, you must create a product, define the required DPs, and then implement these DPs on the panel.

Register and log in to the Tuya Developer Platform and create a product.

  1. In the left-side navigation pane, choose Product > Development > Create.

  1. In the Standard Category tab, select the Small Home Appliances > IPC Pet Feeder.

  1. Select a smart mode and solution, and complete product information.

  1. On the page of Add Standard Function, choose your desired DPs or use the default settings, and click OK.

  1. On the product details page, copy the product ID (PID), and contact your project manager to configure and enable the Product AI Capabilities for this PID.
  2. After confirming this feature has been enabled, you can find the Product AI Capabilities tab in Function Definition.

  1. Under AI Agents on Panels, click Add Agent.

  1. On the Add AI Agents on Panels page, select Select Created Agent tab, choose Pet AI, click OK.

  1. In the section of Selected AI agents, click Details.

  1. Copy the agent ID and save it for later use.

Create panel miniapp on Smart MiniApp Developer Platform

Register and log in to the Smart MiniApp Developer Platform. For more information, see Create panel miniapp.

Create a project based on a template

Open Tuya MiniApp IDE and create a panel miniapp project based on the AI pet template. For more information, see Initialize project.

In the project, configure the agent ID you copied in the previous Create a product > Step 10. The template code is src/pages/Home/index.

// The agent ID. Change it to the agent ID configured for the product.
const AGENT_ID = "xxx";

Add pet

Before using the panel, you need to add your pet's information first, follow the guided process to select the pet's type, breed, gender, activity level, upload a frontal photo, and fill in the pet's nickname, birthday, weight and other information.

Demonstration

Code snippet

<View className={styles.content}>
  {step === 0 && <PetType value={type} onChange={handlePetTypeChange} />}
  {step === 1 && (
    <PetBreed
      petType={type}
      value={breed}
      onChange={handlePetBreedChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 2 && (
    <PetSex
      value={sex}
      onChange={handlePetSexChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 3 && (
    <PetActiveness
      petType={type}
      value={activeness}
      onChange={handlePetActivenessChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 4 && (
    <PetAnalytics
      goNext={handleGoNext}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 5 && (
    <PetInfo
      petType={type}
      breed={breed}
      sex={sex}
      activeness={activeness}
      profile={profile}
    />
  )}
</View>

const handleSave = async () => {
  if (name.trim() === '') {
    ToastInstance({
      context: this,
      message: Strings.getLang('pet_info_name_empty'),
    });
    return;
  }

  if (weight === 0) {
    ToastInstance({
      context: this,
      message: Strings.getLang('weight_not_zero'),
    });
    return;
  }

  try {
    showLoading({
      title: '',
      mask: true,
    });
    const petId = await (dispatch as AppDispatch)(
      addPet({
        petType,
        breedCode: breed,
        sex,
        activeness,
        name: name.trim(),
        avatar: bizUrlRef.current,
        weight: weight * 1000,
        birth: birthday,
        ownerId: getHomeId(),
        timeZone: moment().format('Z'),
        tuyaAppId: getTuyaAppId(),
        idPhotos: profile?.idPhotos,
        features: profile?.features,
      })
    ).unwrap();

    dispatch(fetchPetDetail({ petId, forceUpdate: true }));
    setNavigationBarBack({ type: 'system' });
    navigateBack();

    if (store.getState().global.selectedPetId === -1) {
      dispatch(setSelectedPetId(petId));
    }
  } catch (err) {
    errorToast(err);
  } finally {
    hideLoading();
  }
};

View pet details, update, and delete pets

Click on a pet to view the pet's detailed information. On the details page, you can update the pet's information and delete the pet's records.

Code snippet

// Update pet information
const handleSave = async () => {
  if (name.trim() === '') {
    ToastInstance({
      context: this,
      message: Strings.getLang('pet_info_name_empty'),
    });
    return;
  }

  try {
    showLoading({
      title: '',
      mask: true,
    });
    await (dispatch as AppDispatch)(
      updatePet({
        id: pet.id,
        petType: pet.petType,
        breedCode,
        sex,
        activeness,
        name: name.trim(),
        avatar: bizUrlRef.current,
        weight: weight * 1000,
        birth: birthday,
        rfid: pet.rfid,
        ownerId: getHomeId(),
        timeZone: moment().format('Z'),
        tuyaAppId: getTuyaAppId(),
        idPhotos: profile?.idPhotos,
        features: profile?.features,
      })
    ).unwrap();

    if (pet.id) {
      await (dispatch as AppDispatch)(
        fetchPetDetail({ petId: Number(pet.id), forceUpdate: true })
      ).unwrap();
    }

    navigateBack();
  } catch (err) {
    console.log(err);
  } finally {
    hideLoading();
  }
};

// Delete pet
const handleDelete = async () => {
  try {
    await DialogInstance.confirm({
      context: this,
      title: Strings.getLang('tips'),
      message: Strings.getLang('pet_delete_tips'),
      confirmButtonText: Strings.getLang('confirm'),
      cancelButtonText: Strings.getLang('cancel'),
    });

    showLoading({
      title: '',
      mask: true,
    });
    await (dispatch as AppDispatch)(deletePet(pet.id)).unwrap();
    navigateBack();
  } catch (err) {
    console.log(err);
  } finally {
    hideLoading();
  }
};

Demonstration

Code snippet

export async function pictureQualityDetect(pathUrl: string) {
  const result = await new Promise<{
    imagePath: string;
    lowQuality: boolean;
    lowQualityReason: number;
  }>((resolve, reject) => {
    petsPictureQualityDetectForImage({
      inputImagePath: pathUrl,
      labelAllow: 1,
      objectAreaPercent: 30,
      objectFaceRotationAngle: 40,
      objectFaceSideAngle: 45,
      maximumPictureBrightness: 80,
      minimumPictureBrightness: 0,
      success: res => {
        resolve(res);
      },
      fail: error => {
        reject(error);
      },
    });
  });

  return result;
}

const handleChooseImg = async () => {
    let paths = [];

    try {
      paths = await chooseImage(3, () => enter('analyzing'));
    } catch (err) {
      return;
    }

    // Proceed to the next stage first
    setState({
      analyzingText: Strings.getLang('add_pet_analytics_upload_img'),
    });

    const controller = new AbortController();
    controllerRef.current = controller;
    let tempPaths: Array<{ imagePath: string; lowQuality: boolean; lowQualityReason: number }> = [];
    let pathList: Array<{ imagePath: string; lowQuality: boolean; lowQualityReason: number }> = [];

    if (petType === 'cat') {
      try {
        tempPaths = (await Promise.all(paths.map(url => pictureQualityDetect(url)))).map(d => d);

        pathList = tempPaths.filter(item => !item.lowQuality) as Array<{
          imagePath: string;
          lowQuality: boolean;
          lowQualityReason: number;
        }>;
        const lowQualityList = tempPaths
          .filter(item => item.lowQuality)
          .map(item => item.lowQualityReason) as Array<number>;

        if (pathList.length === 0) {
          const errorText = Array.from(new Set(lowQualityList)).map((lowQualityItem, index) => {
            return `${Strings.getLang(`dsc_${lowQualityItem}`)}`;
          });
          setPicErrorTips(errorText.join('、'));
          setTimeout(() => {
            enter('failed');
          }, 1000);

          return;
        }
      } catch (error) {
        enter('failed');
        return;
      }
    }

    let images: Array<{ imageDisplayUrl: string; objectKey: string }> = [];

    const pathSourceList = petType === 'cat' ? pathList.map(element => element.imagePath) : paths;
    try {
      images = (
        await Promise.all(pathSourceList.map(p => uploadImageCat(p, ANALYTICS_BIZ_TYPE)))
      ).map(d => ({
        imageDisplayUrl: d.publicUrl,
        objectKey: d.cloudKey,
      }));
      if (controller.signal.aborted) {
        return;
      }
    } catch (error) {
      setTimeout(() => {
        enter('failed');
      }, 1000);
      return;
    }

    let idx = 0;
    const tips = [
      Strings.getLang('add_pet_analytics_upload_tip1'),
      Strings.getLang('add_pet_analytics_upload_tip2'),
      Strings.getLang('add_pet_analytics_upload_tip3'),
      Strings.getLang('add_pet_analytics_upload_tip4'),
      Strings.getLang('add_pet_analytics_upload_tip5'),
    ];

    const id = setInterval(() => {
      setState({
        analyzingText: tips[idx++ % tips.length],
      });
    }, 3000);

    try {
      const taskId = await analyzePetFeature({
        ownerId: homeId,
        images,
        miniAppId,
        agentId: AGENT_ID,
      });
      const [infoRes] = await Promise.all([
        loopGetAnalysisResult({
          taskId,
          controller,
          type: AnalysType.Profile,
        }),
      ]);
      if (infoRes) {
        setState({
          similarShow: true,
          petResInfo: infoRes,
        });
      } else if (infoRes) {
        emitter.emit('selectProfile', infoRes);
        enter('success');
      } else {
        // Timeout
        enter('failed');
      }
    } catch (error) {
      enter('failed');
      return;
    } finally {
      clearInterval(id);
    }
  };

// Upload image
export async function uploadImage(filePath: string, bizType: UploadFileBizType) {
  const fileName = parseFileName(filePath);
  const signInfo = await getPetUploadSign({ bizType, fileName });
  const { url, objectKey } = signInfo;

  await uploadFile(url, filePath, fileName);

  return { cloudKey: objectKey };
}

It mainly displays the eating status of pets in the household. When a pet comes to eat, the specific pet will be identified and a eating record will be generated.

Code snippet

const [eatingRecords, setEatingRecords] = useState<IEatingRecord[]>([]);

const getData = async (pageNo: number, isFresh?: boolean) => {
  try {
    const day = dayjs();
    const startOfDay = day.clone().startOf('day').valueOf();
    const endOfDay = day.clone().endOf('day').valueOf();
    setRefreshing(!!isFresh);

    const eatParams = {
      ownerId: homeId,
      uuid: getDevInfo().uuid,
      startTime: startOfDay,
      endTime: endOfDay,
      pageNo,
      pageSize: 100,
    };
    const { pageNo: pageNumber, hasNext, data = [] } = await fetchPetEatingRecordApi(eatParams);
    setRefreshing(false);
    setHasNext(hasNext);
    setCurrentPageNo(pageNumber);

    const formattedEatingRecord = data.map(item => {
      const petName = Array.isArray(item.pets)
        ? item.pets.map(p => {
            return find(pets, { id: p.petId })?.name;
          })
        : Strings.getLang('pet');
      return {
        ...item,
        timeStamp: item.recordTime,
        type: RECORD_DATA_TYPE.feed,
        desc: Strings.formatValue('dsc_feed_eating', petName),
      };
    });

    if (pageNo > 1) {
      setEatingRecords([...eatingRecords, ...formattedEatingRecord]);
    } else {
      setEatingRecords(formattedEatingRecord);
    }
  } catch (error) {
    setRefreshing(false);
    console.log('fetch error: ', error);
  }
};

Supports users in recording and playing audio for specific scenarios, helping to soothe pets, find pets, or engage in fun interactions.

Code snippet

// Initialize recording data
useEffect(() => {
  dispatch(fetchAudios());
}, []);

// Home component
<PageContainer
  show={showVoices}
  customStyle="backgroundColor: transparent"
  position="bottom"
  overlayStyle="background: rgba(0, 0, 0, 0.5);"
  onLeave={() => setShowVoices(false)}
  onAfterEnter={() => setReady(true)}
  onClickOverlay={() => setShowVoices(false)}
>
  <Voices ready={ready} onClose={() => setShowVoices(false)} />
</PageContainer>;

// Recording process

import { getRecorderManager } from '@ray-js/ray';

const recordManager = useRef(getRecorderManager());
const recordFile = useRef<string>();

// (1) Start recording
const handleStart = () => {
  recordManager.current.start({
    frameSize: undefined,
    format: 'wav',
    success: res => {
      recordFile.current = res.tempFilePath;
      setIsRecording(true);
    },
    fail: err => {
      console.log(err);
    },
  });
};

// (2) Finish recording
const handleFinish = () => {
  recordManager.current.stop({
    success: () => {
      setIsRecording(false);
      onRecorded(recordFile.current);
    },
  });
};

// (3) Preview recording

import { getRecorderManager } from '@ray-js/ray';

const audioContext = useRef<ty.CreateInnerAudioContextTask>();
audioContext.current = createInnerAudioContext();

const handleListen = () => {
  if (isPlaying) {
    audioContext.current?.stop?.({
      success: () => {
        setIsPlaying(false);
      },
    });
  } else {
    audioContext.current?.play?.({
      src: file,
      autoplay: true,
      loop: false,
      success: () => {
        setIsPlaying(true);
      },
    });
  }
};

// (4) Save recording
const handleSave = async () => {
  try {
    const {
      data: { inputValue },
    } = await DialogInstance.input({
      context: this,
      title: Strings.getLang('dsc_input_audio_name'),
      overlayStyle: { background: 'transparent' },
      value: '',
      cancelButtonText: Strings.getLang('dsc_cancel'),
      confirmButtonText: Strings.getLang('dsc_confirm'),
      selector: '#smart-dialog-voice',
    });

    if (file) {
      try {
        showLoading({
          title: Strings.getLang('dsc_uploading'),
        });

        const { cloudKey } = await uploadAudio(
          file,
          'pet_media-device',
          'application/octet-stream'
        );

        await fileRelationSave(
          {
            objectKey: cloudKey,
            fileName: `${inputValue}_@_${duration}`,
          },
          devId
        );

        await (dispatch as AppDispatch)(fetchAudios()).unwrap();

        onSave();
      } catch (err) {
        showToast({
          title: Strings.getLang('dsc_save_fail'),
          icon: 'fail',
        });
      } finally {
        hideLoading();
      }
    }
  } catch (err) {
    console.log(err);
  }
};

// (5) Delete recording
import { showLoading, hideLoading, showToast } from '@ray-js/ray';

const [edit, setEdit] = useState(false);
const [selectedIds, setSelectedIds] = useState<string[]>([]);

const handleEdit = async () => {
  if (audiosTotal === 0) return;
  if (edit) {
    try {
      if (selectedIds.length !== 0) {
        showLoading({
          title: '',
        });

        await (dispatch as AppDispatch)(deleteAudios(selectedIds)).unwrap();
      }
      setEdit(false);
      setSelectedIds([]);
    } catch (err) {
      showToast({
        title: Strings.getLang('dsc_delete_failed'),
        icon: 'fail',
      });
    } finally {
      hideLoading();
    }
  } else {
    setEdit(true);
  }
};

Primarily supports users in two-way voice intercom, accompanying pets, and enhancing pets' emotional connections.

Code snippet

// Core component
<IpcPlayer
  objectFit="contain"
  defaultMute={isMute}
  devId={devId}
  onlineStatus={isOnline}
  updateLayout={`${playerLayout}`}
  scalable={false}
  onChangeStreamStatus={handleChangeStreamStatus}
  onCtx={handleCtx}
  onPlayerTap={handlePlayerClick}
  clarity={videoClarityObj[mainDeviceCameraConfig.videoClarity]}
  privateState={dpBasicPrivate ?? false}
  playerStyle={{ borderRadius: 20 }}
/>;

// Export audio and video instance
const handleCtx = (ctx) => {
  dispatch(updateIpcCommon({ playerCtx: ctx }));
};

// Start two-way intercom
export const startTalk = async () => {
  const { isTwoTalking } = store.getState().ipcCommon;
  // Start intercom
  return new Promise((resolve, reject) => {
    const { playerCtx } = store.getState().ipcCommon;

    playerCtx.ctx.startTalk({
      success: () => {
        if (isTwoTalking) showToast("ipc_3s_can_not_donging", "none");
        resolve(true);
      },
      fail: () => {
        showToast();
        reject();
      },
    });
  });
};

// Stop two-way intercom
export const stopTalk = (showErrorToast = true) => {
  return new Promise((resolve, reject) => {
    const { playerCtx } = store.getState().ipcCommon;

    playerCtx.ctx.stopTalk({
      success: () => {
        resolve(false);
      },
      fail: () => {
        showErrorToast && showToast();
        reject();
      },
    });
  });
};