Prerequisites

Development environment

For more information, see Panel MiniApp > Set up environment.

Product name: AI dental mirror

Requirement prototype

Create panel miniapp on Smart MiniApp Developer Platform

Register and log in to the Smart MiniApp Developer Platform.

For more information, see Create panel miniapp.

Create a project based on a template

Open Tuya MiniApp IDE and create a panel miniapp project based on the AI ​​dental mirror template.

For more information, see Initialize project.

By now, you have completed the initialization of the development template of a panel miniapp. The following section shows the project directories.

├── src
│  	├── api // Aggregate file of all cloud API requests of the panel
│  	├── components // Common components
│  	├── constant
│  	│   ├── dpCodes.ts // dpCode constant
│  	│   ├── index.ts // Stores all constant configurations
│  	├── devices // Device model
│  	├── hooks // Hooks
│  	├── i18n // Multilingual settings
│  	├── pages
│   │   ├── home // Homepage
│   │   ├── report // Analytics report page
│  	├── redux // redux
│   ├── res // Resources, such as pictures and SVG
│   ├── styles // Global style
│   ├── types // Define global types
│   ├── utils // Common utility methods
│   ├── app.config.ts
│   ├── app.less
│   ├── app.tsx
│   ├── composeLayout.tsx // Handle and listen for the adding, unbinding, and DP changes of sub-devices
│   ├── global.config.ts
│   ├── mixins.less // Less mixins
│   ├── routes.config.ts // Configure routing
│   ├── variables.less // Less variables

Initialize the AI ​​model

Features

The template provides a method for pre-downloading the AI model and simultaneously monitors the download progress to ensure the subsequent AI analytics functions are ready for use.

Code snippet

const handleOralModelDownProgress = (d: any) => {
  dispatch(updateAiModelProgress(d?.progress || 0));
};

useEffect(() => {
  // Initialize the model
  ty.ai.oralDiseaseInit({
    success: () => {
      console.log(`oralDiseaseInit succeeded`);
      showToast({
        title: Strings.getLang("ai_model_download_success"),
        icon: "success",
      });
      dispatch(updateUiState({ aiModel: { init: true, progress: 100 } }));
    },
    fail: () => {
      console.log(`oralDiseaseInit failed`);
      showToast({
        title: Strings.getLang("ai_model_download_fail"),
        icon: "error",
      });
      dispatch(updateUiState({ aiModel: { init: false, progress: 0 } }));
    },
  });
  // Monitor the model download progress
  ty.ai.onOralModelDownProgress(handleOralModelDownProgress);
  return () => {
    ty.ai.offOralModelDownProgress(handleOralModelDownProgress);
  };
}, []);

Take photos or select images from the album and preview them

Features

The template includes methods that allow users to obtain images either by selecting from their photo album or by taking a new photo, and preview them.

Code snippet

  import { chooseImage, previewImage } from "@ray-js/ray";
  const [imageSrc, setImageSrc] = useState<string[]>([]);
  // Choose the image
  const handleChooseImage = () => {
    chooseImage({
      count: 1,
      sizeType: ['original', 'compressed'],
      sourceType: ['album', 'camera'],
      success: res => {
        console.log('chooseImage success', res);
        setImageSrc(res.tempFilePaths);
      },
      fail: res => {
        console.log('chooseImage fail', res);
      },
    });
  };

  // Preview the image
  const previewChooseImage = (urls, current) => () => {
    previewImage({
      urls,
      current,
      success: res => {
        console.log('previewImage success', res);
      },
      fail: res => {
        console.log('previewImage fail', res);
      },
    });
  };

Features

By performing AI analytics on the images, it detects oral diseases, processes the image to show heatmaps, and provides treatment recommendations.

Code snippets for key APIs

import { ai, env } from "@ray-js/ray";

  const handleClickAnalysis = () => {
    if (imageSrc.length === 0) return;
    if (!aiModelInit) {
      showToast({
        title: Strings.getLang('ai_model_not_init_tip'),
        icon: 'error',
      });
      return;
    }
    showLoading({ title: Strings.getLang('analysis_loading') });
    const path = imageSrc[0];
    const outputPath = `${env.USER_DATA_PATH}/aiReport/`;
    ty.ai.oralDiseasePredictionRun({
      inputImagePath: path,
      outImagePath: outputPath,
      success: d => {
        console.log('===oralDiseasePredictionRun===', d);
        if (d?.nonOral) {
          DialogInstance.alert({
            message: Strings.getLang('analysis_nonOral'),
            confirmButtonText: Strings.getLang('analysis_nonOral_confirm'),
          }).then(() => {
            // on close
          });
        } else {
          dispatch(updateAiReport(d));
          navigateTo({
            url: '/pages/report/index',
          });
        }
        hideLoading();
      },
      fail: e => {
        console.log('===oralDiseasePredictionRun err===', e);
        showToast({
          title: e && JSON.stringify(e),
          icon: 'error',
        });
        hideLoading();
      },
    });
  };