Prerequisites

Create a panel

You can use the panel miniapp to develop an AI pet device panel based on the Ray framework.

Required conditions

For more information, see Panel MiniApp > Set up environment.

A product defines the data points (DPs) of the associated panel and device. Before you develop a panel, you must create a product, define the required DPs, and then implement these DPs on the panel.

Register and log in to the Tuya Developer Platform and create a product.

  1. In the left-side navigation pane, choose Product > Development > Create.

  1. In the Standard Category tab, select the Small Home Appliances > IPC Pet Feeder.

  1. Select a smart mode and solution, and complete product information.

  1. On the page of Add Standard Function, choose your desired DPs or use the default settings, and click OK.

  1. On the product details page, copy the product ID (PID), and contact your project manager to configure and enable the Product AI Capabilities for this PID.
  2. After confirming this feature has been enabled, you can find the Product AI Capabilities tab in Function Definition.

  1. Under AI Agents on Panels, click Add Agent.

  1. On the Add AI Agents on Panels page, select Select Created Agent tab, choose Pet AI, click OK.

  1. In the section of Selected AI agents, click Details.

  1. Copy the agent ID and save it for later use.

Create panel miniapp on Smart MiniApp Developer Platform

Register and log in to the Smart MiniApp Developer Platform. For more information, see Create panel miniapp.

Create a project based on a template

Open Tuya MiniApp IDE and create a panel miniapp project based on the AI pet template. For more information, see Initialize project.

In the project, configure the agent ID you copied in the previous Create a product > Step 10. The template code is src/pages/Home/index.

// The agent ID. Change it to the agent ID configured for the product.
const AGENT_ID = "xxx";

Add pet

Before using the panel, you need to add your pet's information first, follow the guided process to select the pet's type, breed, gender, activity level, upload a frontal photo, and fill in the pet's nickname, birthday, weight and other information.

Demonstration

Code snippet

<View className={styles.content}>
  {step === 0 && <PetType value={type} onChange={handlePetTypeChange} />}
  {step === 1 && (
    <PetBreed
      petType={type}
      value={breed}
      onChange={handlePetBreedChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 2 && (
    <PetSex
      value={sex}
      onChange={handlePetSexChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 3 && (
    <PetActiveness
      petType={type}
      value={activeness}
      onChange={handlePetActivenessChange}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 4 && (
    <PetAnalytics
      goNext={handleGoNext}
      onBack={() => setStep((step) => step - 1)}
    />
  )}
  {step === 5 && (
    <PetInfo
      petType={type}
      breed={breed}
      sex={sex}
      activeness={activeness}
      profile={profile}
    />
  )}
</View>

const handleSave = async () => {
  if (name.trim() === '') {
    ToastInstance({
      context: this,
      message: Strings.getLang('pet_info_name_empty'),
    });
    return;
  }

  if (weight === 0) {
    ToastInstance({
      context: this,
      message: Strings.getLang('weight_not_zero'),
    });
    return;
  }

  try {
    showLoading({
      title: '',
      mask: true,
    });
    const petId = await (dispatch as AppDispatch)(
      addPet({
        petType,
        breedCode: breed,
        sex,
        activeness,
        name: name.trim(),
        avatar: bizUrlRef.current,
        weight: weight * 1000,
        birth: birthday,
        ownerId: getHomeId(),
        timeZone: moment().format('Z'),
        tuyaAppId: getTuyaAppId(),
        idPhotos: profile?.idPhotos,
        features: profile?.features,
      })
    ).unwrap();

    dispatch(fetchPetDetail({ petId, forceUpdate: true }));
    setNavigationBarBack({ type: 'system' });
    navigateBack();

    if (store.getState().global.selectedPetId === -1) {
      dispatch(setSelectedPetId(petId));
    }
  } catch (err) {
    errorToast(err);
  } finally {
    hideLoading();
  }
};

View pet details, update, and delete pets

Click on a pet to view the pet's detailed information. On the details page, you can update the pet's information and delete the pet's records.

Code snippet

// Update pet information
const handleSave = async () => {
  if (name.trim() === '') {
    ToastInstance({
      context: this,
      message: Strings.getLang('pet_info_name_empty'),
    });
    return;
  }

  try {
    showLoading({
      title: '',
      mask: true,
    });
    await (dispatch as AppDispatch)(
      updatePet({
        id: pet.id,
        petType: pet.petType,
        breedCode,
        sex,
        activeness,
        name: name.trim(),
        avatar: bizUrlRef.current,
        weight: weight * 1000,
        birth: birthday,
        rfid: pet.rfid,
        ownerId: getHomeId(),
        timeZone: moment().format('Z'),
        tuyaAppId: getTuyaAppId(),
        idPhotos: profile?.idPhotos,
        features: profile?.features,
      })
    ).unwrap();

    if (pet.id) {
      await (dispatch as AppDispatch)(
        fetchPetDetail({ petId: Number(pet.id), forceUpdate: true })
      ).unwrap();
    }

    navigateBack();
  } catch (err) {
    console.log(err);
  } finally {
    hideLoading();
  }
};

// Delete pet
const handleDelete = async () => {
  try {
    await DialogInstance.confirm({
      context: this,
      title: Strings.getLang('tips'),
      message: Strings.getLang('pet_delete_tips'),
      confirmButtonText: Strings.getLang('confirm'),
      cancelButtonText: Strings.getLang('cancel'),
    });

    showLoading({
      title: '',
      mask: true,
    });
    await (dispatch as AppDispatch)(deletePet(pet.id)).unwrap();
    navigateBack();
  } catch (err) {
    console.log(err);
  } finally {
    hideLoading();
  }
};

Upload pet pictures to the cloud for analysis to identify the pet's body shape, fur color, expression and other appearance features, so that the specific pet can be accurately identified when the pet is eating.

  1. First, request the cloud to obtain the file upload signature and the corresponding objectKey.
  2. Then, upload the picture to the specified server, and pass the objectKey as a parameter to the feature analysis interface after completion. The interface will return a taskId, and then poll the feature analysis result interface to obtain the analysis progress.
    • If the returned analysisResult value is 2, it means that the analysis is successful;
    • If the value is 1, it means that the analysis failed. Display the corresponding analysis information according to the returned result.

Demonstration

Code snippet

const handleChooseImg = async () => {
  let paths = [];
  try {
    paths = await chooseImage(3);
  } catch (err) {
    return;
  }

  // Go to the next stage first
  enter("analyzing");

  setState({
    analyzingText: Strings.getLang("add_pet_analytics_upload_img"),
  });

  const controller = new AbortController();
  controllerRef.current = controller;

  let images: Array<{ objectKey: string }> = [];
  try {
    images = (
      await Promise.all(paths.map((p) => uploadImage(p, "petFeature")))
    ).map((d) => ({
      objectKey: d.cloudKey,
    }));
    if (controller.signal.aborted) {
      return;
    }
  } catch (error) {
    enter("failed");
    return;
  }

  let idx = 0;
  const tips = [
    Strings.getLang("add_pet_analytics_upload_tip1"),
    Strings.getLang("add_pet_analytics_upload_tip2"),
    Strings.getLang("add_pet_analytics_upload_tip3"),
    Strings.getLang("add_pet_analytics_upload_tip4"),
    Strings.getLang("add_pet_analytics_upload_tip5"),
  ];

  const id = setInterval(() => {
    setState({
      analyzingText: tips[idx++ % tips.length],
    });
  }, 3000);

  try {
    const taskId = await aiPetFeature({
      ownerId: homeId,
      images,
      miniAppId,
      agentId: AGENT_ID,
    });

    const [infoRes, similarRes] = await Promise.all([
      loopGetAnalysisResult({
        taskId,
        controller,
        type: AnalysType.Profile,
      }),
      loopGetAnalysisResult({
        taskId,
        controller,
        type: AnalysType.MatchPet,
      }),
    ]);
    if (similarRes?.matchedPets?.length) {
      setState({
        similarShow: true,
        petResInfo: infoRes,
      });
    } else if (infoRes) {
      emitter.emit("selectProfile", infoRes);
      enter("success");
    } else {
      enter("failed");
    }
  } catch (error) {
    enter("failed");
    return;
  } finally {
    clearInterval(id);
  }
};

// Upload image
export async function uploadImage(
  filePath: string,
  bizType: UploadFileBizType
) {
  const fileName = parseFileName(filePath);
  const signInfo = await fetchUploadSign({ bizType, fileName });
  const manager = await getFileSystemManager();
  const data = manager.readFileSync({
    filePath,
    encoding: "base64",
  });
  const { url, objectKey } = signInfo;

  await customFileUpload({ url, file: data?.data });

  return { cloudKey: objectKey };
}

It mainly displays the eating status of pets in the household. When a pet comes to eat, the specific pet will be identified and a eating record will be generated.

Code snippet

const [eatingRecords, setEatingRecords] = useState<IEatingRecord[]>([]);

const getData = async (pageNo: number, isFresh?: boolean) => {
  try {
    const day = dayjs();
    const startOfDay = day.clone().startOf('day').valueOf();
    const endOfDay = day.clone().endOf('day').valueOf();
    setRefreshing(!!isFresh);

    const eatParams = {
      ownerId: homeId,
      uuid: getDevInfo().uuid,
      startTime: startOfDay,
      endTime: endOfDay,
      pageNo,
      pageSize: 100,
    };
    const { pageNo: pageNumber, hasNext, data = [] } = await fetchPetEatingRecordApi(eatParams);
    setRefreshing(false);
    setHasNext(hasNext);
    setCurrentPageNo(pageNumber);

    const formattedEatingRecord = data.map(item => {
      const petName = Array.isArray(item.pets)
        ? item.pets.map(p => {
            return find(pets, { id: p.petId })?.name;
          })
        : Strings.getLang('pet');
      return {
        ...item,
        timeStamp: item.recordTime,
        type: RECORD_DATA_TYPE.feed,
        desc: Strings.formatValue('dsc_feed_eating', petName),
      };
    });

    if (pageNo > 1) {
      setEatingRecords([...eatingRecords, ...formattedEatingRecord]);
    } else {
      setEatingRecords(formattedEatingRecord);
    }
  } catch (error) {
    setRefreshing(false);
    console.log('fetch error: ', error);
  }
};

Supports users in recording and playing audio for specific scenarios, helping to soothe pets, find pets, or engage in fun interactions.

Code snippet

// Initialize recording data
useEffect(() => {
  dispatch(fetchAudios());
}, []);

// Home component
<PageContainer
  show={showVoices}
  customStyle="backgroundColor: transparent"
  position="bottom"
  overlayStyle="background: rgba(0, 0, 0, 0.5);"
  onLeave={() => setShowVoices(false)}
  onAfterEnter={() => setReady(true)}
  onClickOverlay={() => setShowVoices(false)}
>
  <Voices ready={ready} onClose={() => setShowVoices(false)} />
</PageContainer>;

// Recording process

import { getRecorderManager } from '@ray-js/ray';

const recordManager = useRef(getRecorderManager());
const recordFile = useRef<string>();

// (1) Start recording
const handleStart = () => {
  recordManager.current.start({
    frameSize: undefined,
    format: 'wav',
    success: res => {
      recordFile.current = res.tempFilePath;
      setIsRecording(true);
    },
    fail: err => {
      console.log(err);
    },
  });
};

// (2) Finish recording
const handleFinish = () => {
  recordManager.current.stop({
    success: () => {
      setIsRecording(false);
      onRecorded(recordFile.current);
    },
  });
};

// (3) Preview recording

import { getRecorderManager } from '@ray-js/ray';

const audioContext = useRef<ty.CreateInnerAudioContextTask>();
audioContext.current = createInnerAudioContext();

const handleListen = () => {
  if (isPlaying) {
    audioContext.current?.stop?.({
      success: () => {
        setIsPlaying(false);
      },
    });
  } else {
    audioContext.current?.play?.({
      src: file,
      autoplay: true,
      loop: false,
      success: () => {
        setIsPlaying(true);
      },
    });
  }
};

// (4) Save recording
const handleSave = async () => {
  try {
    const {
      data: { inputValue },
    } = await DialogInstance.input({
      context: this,
      title: Strings.getLang('dsc_input_audio_name'),
      overlayStyle: { background: 'transparent' },
      value: '',
      cancelButtonText: Strings.getLang('dsc_cancel'),
      confirmButtonText: Strings.getLang('dsc_confirm'),
      selector: '#smart-dialog-voice',
    });

    if (file) {
      try {
        showLoading({
          title: Strings.getLang('dsc_uploading'),
        });

        const { cloudKey } = await uploadAudio(
          file,
          'pet_media-device',
          'application/octet-stream'
        );

        await fileRelationSave(
          {
            objectKey: cloudKey,
            fileName: `${inputValue}_@_${duration}`,
          },
          devId
        );

        await (dispatch as AppDispatch)(fetchAudios()).unwrap();

        onSave();
      } catch (err) {
        showToast({
          title: Strings.getLang('dsc_save_fail'),
          icon: 'fail',
        });
      } finally {
        hideLoading();
      }
    }
  } catch (err) {
    console.log(err);
  }
};

// (5) Delete recording
import { showLoading, hideLoading, showToast } from '@ray-js/ray';

const [edit, setEdit] = useState(false);
const [selectedIds, setSelectedIds] = useState<string[]>([]);

const handleEdit = async () => {
  if (audiosTotal === 0) return;
  if (edit) {
    try {
      if (selectedIds.length !== 0) {
        showLoading({
          title: '',
        });

        await (dispatch as AppDispatch)(deleteAudios(selectedIds)).unwrap();
      }
      setEdit(false);
      setSelectedIds([]);
    } catch (err) {
      showToast({
        title: Strings.getLang('dsc_delete_failed'),
        icon: 'fail',
      });
    } finally {
      hideLoading();
    }
  } else {
    setEdit(true);
  }
};

Primarily supports users in two-way voice intercom, accompanying pets, and enhancing pets' emotional connections.

Code snippet

// Core component
<IpcPlayer
  objectFit="contain"
  defaultMute={isMute}
  devId={devId}
  onlineStatus={isOnline}
  updateLayout={`${playerLayout}`}
  scalable={false}
  onChangeStreamStatus={handleChangeStreamStatus}
  onCtx={handleCtx}
  onPlayerTap={handlePlayerClick}
  clarity={videoClarityObj[mainDeviceCameraConfig.videoClarity]}
  privateState={dpBasicPrivate ?? false}
  playerStyle={{ borderRadius: 20 }}
/>;

// Export audio and video instance
const handleCtx = (ctx) => {
  dispatch(updateIpcCommon({ playerCtx: ctx }));
};

// Start two-way intercom
export const startTalk = async () => {
  const { isTwoTalking } = store.getState().ipcCommon;
  // Start intercom
  return new Promise((resolve, reject) => {
    const { playerCtx } = store.getState().ipcCommon;

    playerCtx.ctx.startTalk({
      success: () => {
        if (isTwoTalking) showToast("ipc_3s_can_not_donging", "none");
        resolve(true);
      },
      fail: () => {
        showToast();
        reject();
      },
    });
  });
};

// Stop two-way intercom
export const stopTalk = (showErrorToast = true) => {
  return new Promise((resolve, reject) => {
    const { playerCtx } = store.getState().ipcCommon;

    playerCtx.ctx.stopTalk({
      success: () => {
        resolve(false);
      },
      fail: () => {
        showErrorToast && showToast();
        reject();
      },
    });
  });
};