Prerequisites

Development environment

For more information, see Panel MiniApp > Set up environment.

IPC Highlight Moments is an AI-powered video value-added service exclusively developed by the Tuya Developer Platform for all camera-equipped smart devices. How it works: The system automatically identifies preset targets or custom events via AI models, captures highlight footage (such as pets interaction, baby's first steps, sunrises, and sunsets), and easily generates customized vlogs with special effects. This solution comprehensively addresses infant/pet monitoring, lifestyle entertainment, and travel scenarios, enabling users to effortlessly capture highlight moments in their lives.

Features

Create panel miniapp on Smart MiniApp Developer Platform

Register and log in to the Smart MiniApp Developer Platform. For more information, see Create panel miniapp.

Create a project based on the template

Open Tuya MiniApp IDE > Materials, and create a miniapp project based on the IPC Highlight Moments General Template material.

For more information, see Initialize project.

By now, you have completed the initialization of the development template of a panel miniapp. The following section shows the project directories.

├── src
│  	├── api // Aggregate file of all cloud API requests of the panel
│  	├── components
│  	│   ├── GlobalToast // Global toast
│  	│   ├── IconFont // SVG icon container component
│  	│   ├── MusicSelectModal // Background music selection dialog
│  	│   ├── TopBar // Top bar
│  	│   ├── TouchableOpacity // Touchable button component
│  	│   ├── Video // Video container component
│  	│   ├── VideoSeeker // Video slider component
│  	├── constant
│  	│   ├── index.ts // Stores all constant configurations
│  	├── devices // Device model
│  	├── features // Business functions
│  	├── hooks // Hooks
│  	├── i18n // Multilingual settings
│  	├── iconfont // SVG icon
│  	├── pages
│   │   ├── ai-all // Complete album of highlight moments
│   │   ├── ai-video // Edit on-app AI highlight moments
│   │   ├── ai-vision-setting // Edit the details of the highlight moments service
│   │   ├── bind-device // Bind with devices
│   │   ├── device-service-setting // Set the time for smart recognition of highlight moments
│   │   ├── player // Video player
│   │   ├── preview-image // Image preview component
│   │   ├── select-lang // Multilingual setting
│  	├── redux // redux
│   ├── res // Image resources
│   ├── share // Shared resources
│   ├── utils // Common utility functions
│   ├── app.less
│   ├── app.tsx
│   ├── global.config.ts
│   ├── routes.config.ts // Configure routing
│   ├── theme.json // Configure theme
│   ├── variables.less // Less variables
├── types // Define global types

Functional introduction

The details of the highlight moments service mainly include the following five sections:

Get service details

Code snippet

import {
  showLoading,
  hideLoading,
  showToast,
  albumSettingEdit,
  getAlbumSetting,
  albumSettingSave,
} from "@ray-js/ray";

const {
  homeId,
  stockId,
  visionBoxId,
  serviceDetails,
  statusChange,
  settingRef,
  onSaved,
} = props;

import { unstable_batchedUpdates as batchedUpdates } from "@ray-core/ray";
const { data: albumSetting, runAsync: getSetting } = useRequest(
  getAlbumSetting,
  {
    manual: true,
  }
);

useEffect(() => {
  if (visionBoxId && homeId && stockId) {
    showLoading({ title: "" });
    getSetting({
      gid: homeId,
      stockId,
    })
      .then((res) => {
        if (!res) return;
        batchedUpdates(() => {
          pauseChangeStatus.current = true;
          form.setValue("albumName", res.albumName);
          form.setValue("enableStatus", !!res.enableStatus);
          form.setValue("timedCaptureConfig", res.timedCaptureConfig);
          form.setValue("smartCaptureConfig", res.smartCaptureConfig);
          pauseChangeStatus.current = false;
        }, null);
      })
      .finally(() => hideLoading());
  }
}, [visionBoxId, homeId, stockId]);

Set service details

Code snippet

import { useForm, useWatch } from "react-hook-form";

const form = useForm({ mode: "onChange" });
const [timedCaptureConfig, smartCaptureConfig] = useWatch({
  control: form.control,
  name: ["timedCaptureConfig", "smartCaptureConfig"],
});

const handSave = async (): Promise<boolean> => {
  return new Promise((resolve) => {
    if (!serviceDetails?.cloudService?.bindDeviceNum) {
      showToast({
        icon: "none",
        title: I18n.t("aiVisual.requiredDevice.tip"),
      });
      resolve(false);
      return;
    }
    if (loadingRef.current) return;
    form.trigger().then(async (res) => {
      if (res) {
        loadingRef.current = true;
        const values = form.getValues();
        const params: Parameters<typeof albumSettingEdit>[0] = {
          gid: homeId,
          stockId,
          albumId: visionBoxId,
          albumName: values.albumName,
          enableStatus: values.enableStatus ? 1 : 0,
          timedCaptureConfig: values.timedCaptureConfig || [],
          smartCaptureConfig: values.smartCaptureConfig || [],
        };
        showLoading({ title: "" });
        let ret;
        if (visionBoxId) {
          ret = await albumSettingEdit(params);
        } else {
          ret = await albumSettingSave(params);
        }
        hideLoading();
        if (ret) {
          hideLoading();
          const txt = I18n.t("common.save.success");
          showToast({
            title: txt,
            icon: "success",
          });
          changeSettingStatus("saved");
          onSaved && onSaved(visionBoxId || ret);
          resolve(true);
          loadingRef.current = false;
        } else {
          showToast({
            title: I18n.t("common.save.failed"),
            icon: "error",
          });
          resolve(false);
          loadingRef.current = false;
        }
      }
    });
  });
};

Functional introduction

The smart album of highlight moments consists of the following four parts:

Get data of the smart album of highlight moments

Code snippet

import {
  Text,
  View,
  showToast,
  hideLoading,
  showLoading,
  albumVideoDateCount,
} from "@ray-js/ray";

const [aiDatas, setAiDatas] = useState<IAlbumVideoDateCountRes[]>([])
const [todayData, setTodayData] = useState<IAlbumVideoDateCountRes>()
const [loading, setLoading] = useState(true);

const init = () => {
  if (isNaN(Number(timeAlbumId)) || isNaN(Number(stockId))) {
    showToast({ title: "request.param.error" });
    setLoading(false);
    return;
  }
  const param: IAlbumVideoDateCountReq = {
    stockId: Number(stockId), // The stock ID
    gid: homeId, // The home ID
    timeAlbumId: Number(timeAlbumId), // The album ID
  };
  showLoading({ title: "" });
  albumVideoDateCount(param).then((result: IAlbumVideoDateCountRes[]) => {
    hideLoading();
    setLoading(false);
    if (!result || !result.length) return;
    const res = result.map((item) => {
      const fileUrl = item.coverInfo?.fileUrl || "";
      const fileName = fileUrl.split("?")[0].split("/").pop();
      return {
        ...item,
        coverInfo: { ...item.coverInfo, ...{ fileName } },
      };
    });
    if (res.length > 0) {
      if (res[0].type === "today") {
        setTodayData(res[0]);
      }
      const d = res.filter((item) => item.type !== "today");
      setAiDatas(d);
    }
  });
};

Functional introduction

The complete album of highlight moments consists of the following content:

Get data of the complete album of highlight moments

Code snippet

import {
  Button,
  View,
  showToast,
  showModal,
  albumFileDelete,
  albumVideoFileList,
} from "@ray-js/ray";
import { useBatchDecryptImage } from "@/features";
import { deleteImages } from "@ray-js/ray-ipc-decrypt-image";

const { timeAlbumId, stockId, secret, onShow, homeId } = props;
const batchLoadInstance = useBatchDecryptImage(true, 400);

const getPageData = (params) => {
  return new Promise((resolve, reject) => {
    if (isNaN(Number(timeAlbumId)) || isNaN(Number(stockId))) {
      showToast({ title: I18n.t("request.param.error") });
      return;
    }
    const param: IAlbumVideoFileListReq = {
      stockId: Number(stockId), // The stock ID
      gid: homeId, // The home ID
      timeAlbumId: Number(timeAlbumId), // The album ID
      pageNum: params.page,
      // Get data before this time point
      endTime: endTimeRef.current,
      // endTime: 1740386449,
      pageSize: params.pageSize,
    };
    albumVideoFileList(param).then((result: IAlbumVideoFileListRes) => {
      if (result && result.list && result.list.length > 0) {
        const res = result.list.map((item) => {
          const fileUrl = item.coverInfo?.fileUrl || "";
          const fileName = fileUrl.split("?")[0].split("/").pop();
          batchLoadInstance.addOriginData({
            fileUrl,
            decryptKey: secret,
          });
          return {
            ...item,
            coverInfo: { ...item.coverInfo, ...{ fileName } },
          };
        });
        batchLoadInstance.loadAllOriginData();
        resolve({ data: res, total: result.total });
        return;
      }
      resolve({ data: [] });
    });
  });
};

Download, merge, and delete videos from complete album

Code snippet

import {
  Button,
  View,
  showToast,
  showModal,
  albumFileDelete,
  albumVideoFileList,
} from '@ray-js/ray'

import {
  cancelVideoDownload,
  composeVideos,
  getAuthorize,
  imageDownload,
  videoDownload,
} from '@/features/downloadImage'

import { jumpToAiVideoEdit, jumpToPlayer } from '@/features/jump-to-page'

  const [selectedIds, setSelectedIds] = useImmer([])
  const [selectIdIndexMap, setSelectIdIndexMap] = useState<Map<number, number>>(new Map())

  const getSelectDatas = () => {
    const maxCount = selectedIds.length
    let currentCount = 0
    const result = []
    for (let i = 0; i < cloudData.length; i++) {
      if (currentCount >= maxCount) break
      const item = cloudData[i]
      const index = selectedIds.indexOf(item.id)
      if (index >= 0) {
        result[index] = item
        currentCount++
      }
    }
    return result
  }

 const onOperateCompose = async (e, type) => {
    if (type === EOperate.download) {
      isCloseDownload.current = false
      if (selectedIds.length > MAX_DOWNLOAD_SIZE) {
        showToast({
          title: I18n.format('download.count.limit.desc', { time: MAX_DOWNLOAD_SIZE }),
          icon: 'error',
        })
        return
      }
      const selectDatas = getSelectDatas()
      const bl = await getAuthorize()
      if (bl) {
        onShow({ show: true, title: I18n.t('downloading.video'), type: ELoadingType.loading })
        const indexErrs = []
        // After downloading one successfully, download another
        for (let i = 0; i < selectDatas.length; i++) {
          if (isCloseDownload.current) {
            break
          }
          const ele = selectDatas[i]
          if (ele.type === 1 || ele.type === 2) {
            const dl = await videoDownload(deviceId, ele.mediaInfo?.fileUrl, secret)
            if (dl === -1) {
              indexErrs.push(i)
            }
          } else if (ele.type === 0) {
            // If the video does not exist, download the image
            const dl = await imageDownload(deviceId, ele.coverInfo?.fileUrl, secret)
            if (dl === -1) {
              indexErrs.push(i)
            }
          } else {
            indexErrs.push(i) // The cloud returns a type error
          }
        }
        onShow({ show: false, title: I18n.t('download.success'), type: ELoadingType.success })
        if (indexErrs.length > 0) {
          const str = indexErrs.join(',')
          showModal({
            title: '',
            content: str + I18n.t('download.clip.error'),
            showCancel: false,
            confirmText: I18n.t('common.confirm'),
          })
        } else {
          setSelectedIds([])
          setSelectIdIndexMap(null)
        }
      }
    } else if (type === EOperate.compose) {
      if (selectedIds.length > MAX_COMPOSE_SIZE) {
        showToast({
          title: I18n.format('compose.count.limit.desc', { time: MAX_COMPOSE_SIZE }),
          icon: 'error',
        })
        return
      }
      const selectDatas = getSelectDatas()
      const isVideos = isExistSingleImage(selectDatas)
      if (isVideos) {
        showToast({ title: I18n.t('compose.only.media'), icon: 'error' })
        return
      }
      const bl = await getAuthorize()
      if (bl) {
        const mediaInfos = selectDatas.map((item) => {
          return {
            fileUrl: item.mediaInfo?.fileUrl || '',
            key: secret,
          }
        })
        const json = JSON.stringify({ fileInfo: mediaInfos })
        onShow({ show: true, title: I18n.t('composing.video'), type: ELoadingType.loading })
        const result = await composeVideos(deviceId, json)
        if (result === -1) {
          onShow({ show: false, title: I18n.t('compose.fail'), type: ELoadingType.error })
        } else {
          onShow({ show: false, title: I18n.t('compose.success'), type: ELoadingType.success })
          setSelectedIds([])
          setSelectIdIndexMap(null)
          const { path = '', thingfilePath = '' } = result as Record<string, any>
          setTimeout(() => {
            jumpToAiVideoEdit({ videoPath: thingfilePath, videoOriginPath: path })
          }, 0)
        }
      }
    } else if (type === EOperate.delete) {
      if (isNaN(Number(stockId))) {
        showToast({ title: I18n.t('request.param.error') })
        return
      }
      if (selectedIds.length > MAX_DELETE_SIZE) {
        showToast({
          title: I18n.format('delete.count.limit.desc', { time: MAX_DELETE_SIZE }),
          icon: 'error',
        })
        return
      }

      showModal({
        title: '',
        content: I18n.t('module.delete.desc'),
        cancelText: I18n.t('common.cancel'),
        confirmText: I18n.t('aiVisual.delete'),
        success: async ({ confirm, cancel }) => {
          console.log('res===1', confirm, cancel)
          if (confirm) {
            const param: IAlbumFileDeleteReq = {
              stockId: Number(stockId),
              gid: homeId,
              timeAlbumId: Number(timeAlbumId),
              albumRecordIds: selectedIds.join(','),
            }
            const result = await albumFileDelete(param)
            if (result) {
              showToast({ title: I18n.t('aiVisual.delete.success'), icon: 'success' })
              const curCloudDatas = getCurCloudDatas()
              const selectDatas = curCloudDatas.filter((item) => selectedIds.includes(item.id))
              const fileNames = selectDatas.map((item) => item.coverInfo.fileName)
              deleteImages(deviceId, fileNames)
              setDeletedIds([...new Set([...deletedIds, ...selectedIds])])
              setSelectedIds([])
              setSelectIdIndexMap(null)
            } else {
              showToast({ title: I18n.t('aiVisual.delete.fail'), icon: 'error' })
            }
          }
        },
      })
    }
  }

Functional introduction

Video AI editing includes the following features:

For more information, see AI Video Highlight Template.

AI editing with privacy protection

Code snippet

import { ai } from "@ray-js/ray";
import { createTempVideoRoot } from "@/utils";

const {
  objectDetectCreate,
  objectDetectDestroy,
  objectDetectForVideo,
  objectDetectForVideoCancel,
  offVideoObjectDetectProgress,
  onVideoObjectDetectProgress,
} = ai;

// The editing status of this AI video
const [handleState, setHandleState] = useState("idle");
// The processing progress of this AI video stream
const [progressState, setProgressState] = useState(0);

// Register an app AI instance
useEffect(() => {
  objectDetectCreate();

  return () => {
    // Destroy the app AI instance when the page is destroyed
    objectDetectDestroy();
    audioManager.destroy({
      success: (res) => {
        console.log("==destroy2==success", res);
      },
      fail: (error) => {
        console.log("==destroy2==fail", error);
      },
    });
  };
}, []);

// AI video stream processing capabilities
const handleVideoByAI = (
  detectType: number,
  imageEditType: number,
  musicPath = ""
) => {
  const tempVideoPath = createTempVideoRoot();
  onVideoObjectDetectProgress(handleListenerProgress);
  privacyProtectDetectForVideo({
    inputVideoPath: videoOriginSrc,
    outputVideoPath: tempVideoPath,
    detectType,
    musicPath,
    originAudioVolume: volumeObj.video / 100,

    overlayAudioVolume: volumeObj.music / 100,
    imageEditType,
    audioEditType: 3,
    success: ({ path }) => {
      console.log("===path", path, tempVideoPath);
      offVideoObjectDetectProgress(handleListenerProgress);
      setProgressState(0);
      fetchVideoThumbnails({
        filePath: path,
        startTime: 0,
        endTime: 10,
        thumbnailCount: 1,
        thumbnailWidth: 375,
        thumbnailHeight: 212,
        success: (res) => {
          console.log("===fetchVideoThumbnails==res", res);
          setHandleState("success");
          setVideoSrc(path);
          setVideoOriginSrc(path);
          setPosterSrc(res?.thumbnailsPath[0]);
          showToast({
            title: I18n.t("dsc_ai_generates_success"),
            icon: "success",
          });
        },
        fail: ({ errorMsg }) => {
          console.log("==fetchVideoThumbnails==fail==", errorMsg);
        },
      });
    },
    fail: ({ errorMsg }) => {
      console.log("==objectDetectForVideo==fail==", errorMsg);
      offVideoObjectDetectProgress(handleListenerProgress);
      setProgressState(0);
      setHandleState("fail");
      setHandleState("selectSkill");
      setIsShowAISkills(true);
      showToast({
        title: I18n.t("dsc_ai_generates_fail"),
        icon: "error",
      });
    },
  });
};

// Interrupt the generation of AI privacy-preserving video streams
const handleCancelAIProcess = () => {
  objectDetectForVideoCancel({
    success: () => {
      setHandleState("select");
    },
  });
};