Prerequisites

Create a panel

You can utilize the panel miniapp to develop and build an AI Audio Device Panel based on the Ray framework, implementing the following functionalities:

Required conditions

For more information, see Panel MiniApp > Set up environment.

A product defines the data points (DPs) of the associated panel and device. Before you develop a panel, you must create a product, define the required DPs, and then implement these DPs on the panel.

Register and log in to the Tuya Developer Platform and create a product.

  1. In the left-side navigation pane, Go to Product > Development > Create.

  1. Click the Standard Category tab and choose Audio Wearables > AI Headphone.

  1. Follow the prompts to select the smart mode and solution, complete the product information, and then click Create.

  1. On the page of Add Standard Function, you can select DPs based on your requirement and click OK.

Create panel miniapp on Smart MiniApp Developer Platform

Register and log in to the Smart MiniApp Developer Platform. For more information, see Create panel miniapp.

Create a project based on a template

Open Tuya MiniApp IDE and create a panel miniapp project based on the AI headphone template.

For more information, see Initialize project.

Demonstration

Code snippet

  // Start face-to-face translation recording (Pro: recordType=2; non-Pro: recordType=3)
  const startAudioRecording = async (type: 'left' | 'right' | 'auto') => {
    if (!isOnline) {
      ty.showToast({ title: Strings.getLang('device_offline'), icon: 'error' });
      return;
    }
    const audioSourceList = getAudioSourceList(customDevType, recordType);
    const config: any = {
      // Origin. 0: AI Note, 1: AI Translate
      businessType: BusinessType.Note,
      // Multi-source list (replaces the previous single audioSource/recordChannel)
      audioSourceList,
      saveDataWhenError: true,
      // Recording type. 2: face-to-face translation (Pro), 3: face-to-face translation
      recordType,
      controlTimeout: 5,
      dataTimeout: 10,
      // 0: file transcription, 1: real-time transcription
      transferType: 1,
      needTranslate: true,
      // Source/target language (left and right ear swap)
      originalLanguage: isPro ? leftLanguage : type === 'left' ? leftLanguage : rightLanguage,
      targetLanguage: isPro ? rightLanguage : type === 'left' ? rightLanguage : leftLanguage,
      // Agent ID (obtained via atopGetAgentInfo/store)
      agentId,
      ttsEncode: isOpusCelt ? 1 : 0,
      needTts: true,
      needAsr: true,
      // Whether to enable automatic sentence segmentation
      needAutoRecognize: autoSpeek,
      ...(!isPro && {
        // Non-Pro: recording channel 0 Bluetooth LE, 1 BT, 2 micro; f2fChannel 0 left ear, 1 right ear
        recordChannel: isPhone ? 2 : 1,
        f2fChannel: type === 'left' ? 0 : 1,
      }),
      // TTS output config list (generated by audio source)
      ttsConfigList: getTtsConfigList(audioSourceList, recordType),
    };
    await tttStartAudioRecording({ deviceId, config }, true);
    setActiveType(type);
    setInterval(1000);
    lastTimeRef.current = Date.now();
  };

  const handleStartRecord = async (type?: 'left' | 'right' | 'auto') => {
    // Validate benefit balance first
    const balance = await checkBalance(homeId, deviceId);
    if (balance === 0 || balance === 2) return;
    ty.authorize({
      scope: 'scope.record',
      success: () => startAudioRecording(type),
      fail: () => {
        ty.showToast({ title: Strings.getLang('no_record_permisson'), icon: 'error' });
      },
    });
  };

  // Pause/resume/stop (state is synced by the global recording task event)
  await tttPauseRecord(deviceId);
  await tttResumeRecord(deviceId);
  await tttStopRecord(deviceId);

  // Dual-channel ASR: enqueue events; useAsrTransferQueue handles display
  // phase: 0 task, 4 ASR, 5 translation, 6 skill, 7 TTS
  ty.wear.onRecordTransferRealTimeRecognizeStatusUpdateEvent(d => {
    if (!handleTttError(d, true)) return;
    pushEvent(d); // DualChannelASRList/useAsrTransferQueue
  });

  // When re-entering the page, fetch existing real-time results by recordId
  const realTimeResult = await tttGetRecordTransferRealTimeResult({ recordId: task.recordId });
  resetTextList(parseRealTimerResult(realTimeResult));

Demonstration

Code snippet

  // Simultaneous interpretation/real-time transcription share RealTimeRecording;
  // after configuration, the page layer calls tttStartAudioRecording
  const recordingConfig = useMemo(() => {
    const audioSource = getAudioSource(customDevType, currRecordType, isCall);
    const config: any = {
      deviceId,
      config: {
        // Origin. 0: AI Note, 1: AI Translate
        businessType: BusinessType.Note,
        // Audio input source (phone/entry/Pro, etc.)
        audioSource,
        // Recording type. 0: call, 1: conference (simultaneous interpretation uses conference)
        recordType: isCall ? 0 : 1,
        needAsr: true,
        // Simultaneous interpretation enables translation by default
        needTranslate,
        needTts,
        needAmplitude: false,
        originalLanguage: originLanguage,
        agentId,
        ttsEncode: isOpusCelt ? 1 : 0,
      },
    };
    if (needTts) {
      config.config.ttsConfig = {
        // Pass deviceId when the output source is the device; pass empty for BT/MIC
        devId:
          customDevType === DEVTYPES.entry ||
          customDevType === DEVTYPES.phone ||
          customDevType === DEVTYPES.card
            ? ''
            : deviceId,
        ttsOutput: getTtsOutput(audioSource),
        ttsEncode: isOpusCelt ? 2 : customDevType === DEVTYPES.pro ? 1 : 0,
        ttsOutputChannel: 0,
      };
    }
    if (needTranslate) {
      config.config.targetLanguage = translationLanguage;
    }
    return config;
  }, [deviceId, customDevType, needTranslate, originLanguage, translationLanguage, needTts, agentId]);

  // Hot-update parameters when the language changes (no need to stop recording)
  tttUpdateParams(recordingConfig);

  // Unified start at the page layer (includes benefit check, permission, and offline fallback to phone audio source)
  await tttStartAudioRecording(recordingConfig);

  // Pause/resume/stop
  await tttPauseRecord(deviceId);
  await tttResumeRecord(deviceId);
  await tttStopRecord(deviceId);

  // Real-time ASR/translation events: enqueue only; display is handled by useRealTimeTextQueue
  // phase: 0 task, 4 ASR, 5 translation, 6 skill, 7 TTS
  ty.wear.onRecordTransferRealTimeRecognizeStatusUpdateEvent(d => {
    if (d?.errorCode === 10002 || d?.errorCode === '10002') {
      // Device not supported; prompt and return
      return;
    }
    if (!handleTttError(d, true)) return;
    pushEvent(d);
  });

  // Backfill real-time results when a task already exists
  const realTimeResult = await tttGetRecordTransferRealTimeResult({ recordId: task.recordId });
  resetTextList(parseRealTimerResult(realTimeResult));

Demonstration

Code snippet

  // Live recording config (RecordingModes); the recordingMode page uniformly calls tttStartAudioRecording
  const recordingConfig = {
    deviceId,
    config: {
      // Origin. 0: AI Note, 1: AI Translate
      businessType: 0,
      // Audio input source: mapped by device type (phone/entry/proLive, etc.)
      audioSource: getAudioSource(customDevType, currRecordType, isCall),
      // Recording type. 0: call, 1: conference (live recording uses conference)
      recordType: currRecordType,
      needAsr: false,
      needTranslate: false,
      needTts: false,
      // Live recording needs waveform amplitude
      needAmplitude: true,
    },
  };

  // Start: validate online status/permission, then start (fallback to phone audio source when device is offline)
  const handleStartRecord = async (recordingConfig: any) => {
    if (checkAndShowOfflineDialog()) return; // Call mode strictly requires BT; live mode can fall back
    const needPermission =
      customDevType === DEVTYPES.phone ||
      customDevType === DEVTYPES.os ||
      customDevType === DEVTYPES.entry;
    if (needPermission) {
      ty.authorize({
        scope: 'scope.record',
        success: () => startRecordingProcess(recordingConfig),
        fail: () => {
          ty.showToast({ title: Strings.getLang('no_record_permisson'), icon: 'error' });
        },
      });
      return;
    }
    await startRecordingProcess(recordingConfig);
  };

  const startRecordingProcess = async (recordingConfig: any) => {
    await tttStartAudioRecording(recordingConfig);
    setInterval(1000);
    lastTimeRef.current = Date.now();
  };

  // Pause/resume/stop
  await tttPauseRecord(deviceId);
  await tttResumeRecord(deviceId);
  await tttStopRecord(deviceId);
  // After finish, global taskState === FINISH returns to the home page

  // Listen for recording finish events (prompt when the exception code is not 0)
  ty.wear.onRecordTransferFinishEvent(d => {
    if (d.deviceId !== deviceId) return;
    if (d.code !== 0) {
      handleTttError(d, true);
    }
    setInterval(undefined);
  });

Demonstration

Code snippet

// Task type. 0: transcription, 1: summary, 2: translation
enum GenerateType {
  Transcribe,
  Summarize,
  Translate,
}

// Transcription status. 0: not transcribed, 1: transcribing, 2: success, 3: failed
// Summary status. 0: compatible with legacy data, 1: not summarized, 2: summarizing, 3: success, 4: failed

  // 1. Enter detail: fetch file detail by recordId, then load results by status
  getFileDetail({
    recordId,
    amplitudeMaxCount: 100,
  }).then(async res => {
    // transfer === 2: transcription succeeded; fetch transcription result
    if (res?.transfer === TransferStatus.Success) {
      await getTransferResult();
    }
    // summary === 3: summary succeeded; fetch summary result
    if (res?.summary === SummaryStatus.Success) {
      await getSummaryResult();
    }
  });

  // 2. After confirming the template, start transcription/summary (tttTransfer)
  const fetchTransferOrSummary = async () => {
    // If not transcribed yet, always run the transcription task first,
    // whether the entry point is summary or transcription
    let taskType = generateType;
    if (fileDetail?.transfer === TransferStatus.Initial) {
      taskType = GenerateType.Transcribe;
    }

    const params: any = {
      recordTransferId: fileDetail?.recordTransferId,
      // Template ID; pass an empty string for auto
      template: selectedTemplate === 'auto' ? '' : selectedTemplate,
      // Transcription source language
      language: fileDetail?.originalLanguage || language,
      // Summary language
      summaryLang,
      // 0: transcription, 1: summary, 2: translation
      taskType,
      // Whether to distinguish speakers
      enableSpeaker,
    };

    // Translation tasks additionally pass the target language
    if (taskType === GenerateType.Translate) {
      params.transLang = translateLang;
    }

    return tttTransfer(params);
  };

  // 3. Listen for upload progress: after upload completes, mark as transcribing/summarizing locally
  ty.wear.onRecordTransferFileUploadEvent(res => {
    const { fileId, progress, status } = res || {};
    // status: waiting/uploading/completed ...
    if (status === OfflineFileUploadStatus.Completed) {
      updateFileDetailLocal(prev =>
        prev
          ? { ...prev, transfer: TransferStatus.Process, summary: SummaryStatus.Process }
          : prev
      );
    }
  });

  // 4. Listen for partial refresh events: after transcription/summary completes, fetch results
  // (replaces the previous local + cloud dual fetch)
  // Event fields transferStatus/summaryStatus map to detail fields transfer/summary
  const handlePartialRecordUpdate = (res: any) => {
    const currentFile = res?.updateList?.find(
      item => item?.recordId === fileDetailRef.current?.recordId
    );
    if (!currentFile) return;

    const { transferStatus: tStatus, summaryStatus: sStatus } = currentFile;

    if (tStatus === TransferStatus.Success) {
      updateFileDetailLocal(prev =>
        prev ? { ...prev, transfer: tStatus, cloudTranscription: true } : prev
      );
      getTransferResult();
    }
    if (sStatus === SummaryStatus.Success) {
      updateFileDetailLocal(prev => (prev ? { ...prev, summary: sStatus } : prev));
      getSummaryResult();
      refreshFileDetail();
    }
  };

  // 5. Get transcription result
  const fetchRecordTransferResult = async () => {
    const {
      transfer,
      recordId,
      transferType,
      cloudTranscription,
      recordTransferId,
      isFromCloud,
      duration,
    } = fileDetailRef.current;

    // Real-time transcription that has not yet been cloud-transcribed: use the real-time result API
    if (transferType === TransferType.Realtime && !cloudTranscription && !isFromCloud) {
      const realTimeResult = await tttGetRecordTransferRealTimeResult({ recordId });
      return processRealtimeData(realTimeResult);
    }

    // Transcription succeeded/already cloud-transcribed: read locally cached transcription JSON
    if (transfer === TransferStatus.Success || cloudTranscription || isFromCloud) {
      const cloudResult = await tttGetRecordTransferRecognizeResult({
        recordTransferId,
        from: 0, // Local
      });
      const list = cloudResult?.text ? JSON.parse(cloudResult.text) : [];
      return processCloudTransferData(list, duration);
    }
    return [];
  };

  // 6. Get summary result (structured JSON)
  const fetchRecordSummaryResult = async () => {
    const { summary, recordTransferId } = fileDetailRef.current;
    if (summary !== SummaryStatus.Success) return;

    const result = await tttGetRecordTransferSummaryResult({
      recordTransferId,
      from: 0, // Local
    });
    if (!result?.text) return;

    const data = JSON.parse(result.text);
    // data.summary: summary body
    // data.title: summary title (can be written back as the file name)
    // data.outline: chapter outline JSON string
    // data.question: preset questions JSON string
    // data.imageUrl: infographic URL (shown when summaryImageStatus === Complete)
    setSummaryData(data?.summary || '');
  };

{/* 7. Detail tabs: Transcription/Summary/Mind map */}
{activeTab === DetailTabType.Transcribe && (
  <TranscriptContent outlineList={outlineList} updateSummary={updateSummary} />
)}
{activeTab === DetailTabType.Summarize && (
  <SummaryContent
    summary={summaryData}
    updateSummary={updateSummary}
    summaryImageUrl={summaryImageUrl}
  />
)}
{activeTab === DetailTabType.MindMap && <MindMapContent summary={summaryData} />}

{/* Template selection: recommended templates + summary language + distinguish speakers */}
<ChooseTransferTemplate
  show={selectTemplateShow}
  enableSpeaker={enableSpeaker}
  onSpeakerChange={setEnableSpeaker}
  onBottomBtnClick={handleConfirmTemplate}
/>

Code snippet

  // Mode entry: show call recording only for professional devices
  const modes = useMemo(() => {
    const data = [
      // ... other modes
      {
        name: Strings.getLang('mode_title_call_recording'),
        key: RecordMode.phoneRecording,
        icon: 'modeCallIcon',
      },
    ];
    if (deviceList.find(item => item.customDevType === 'pro_version')) {
      return data;
    }
    return data.filter(item => item.key !== RecordMode.phoneRecording);
  }, [deviceList]);

  // Call recording config, then invoke app capabilities to start recording
  const recordingConfig = {
    deviceId,
    config: {
      // Origin. 0: AI Note, 1: AI Translate
      businessType: 0,
      // Professional call audio source
      audioSource: 21,
      // Recording type. 0: call, 1: conference
      recordType: 0,
      needAsr: false,
      needTranslate: false,
      needTts: false,
      needAmplitude: true,
    },
  };

  // Before starting, check device online status and BT connection
  const checkAndShowOfflineDialog = (btStatus?: number) => {
    if (!isOnline) {
      // Prompt to connect headphones
      return true;
    }
    if (btStatus === 0) {
      // Prompt that BT is not connected
      return true;
    }
    return false;
  };

  await tttStartAudioRecording(recordingConfig, true);

  // Listen for call hang-up events (errorCode = 10022) and automatically end recording
  const handleRealTimeRecognizeStatusUpdateEvent = (d: any) => {
    const { errorCode } = d || {};
    if (errorCode !== 10022) return;
    ty.showModal({
      title: Strings.getLang('tip'),
      content: Strings.getLang('phone_call_end_tip'),
      showCancel: false,
      confirmText: Strings.getLang('confirm'),
      success: ({ confirm }) => {
        if (!confirm) return;
        handleStopRecord();
      },
    });
  };
  ty.wear.onRecordTransferRealTimeRecognizeStatusUpdateEvent(
    handleRealTimeRecognizeStatusUpdateEvent
  );

Code snippet

  // Tap the "Import audio" mode to open the system file picker and start importing
  case RecordMode.importAudio: {
    if (isImporting) {
      showToast({ title: Strings.getLang('audio_importing_warn'), icon: 'none' });
      break;
    }
    await tttImportAudio();
    break;
  }

  // Listen for import status events
  // status: 0: not imported, 1: importing, 2: import succeeded, 3: import failed, 4: share-import exception
  const handleFileImportStatusEvent = (res: FileImportStatusResult) => {
    if (res.status === FileImportStatus.Imported) {
      // Refresh the file list; close the import dialog if there are no failed files
      dispatch(updateRecordTransferResultList({ source: 'history_file_imported' }));
      if (!res?.failedFiles?.length) {
        setIsShowImportModal(false);
      }
      setImportInfo(res);
      return;
    }
    setImportInfo(res);
  };
  ty.wear.onFileImportStatusEvent(handleFileImportStatusEvent);

  // Cancel import/retry failed import
  await tttCancelImport();
  await tttRetryImport();

  // Query the current import status (called during page initialization)
  const status = await tttGetAudioImportStatus();

Code snippet

  // Query offline file status on the card device; start BLE transfer if there are pending files
  const status = await getDeviceOfflineFileStatus(deviceId, true);
  if (status?.response?.total > 0) {
    await tttLoadOfflineFile({
      deviceId,
      sessionId: status.sessionId || 0,
      // 1: Bluetooth LE, 2: Wi-Fi fast transfer
      channel: 1,
    });
  }

  // App-level listener for offline file transfer progress
  // (recommended to register once globally to avoid page-switch overwrites)
  // status: 0: not started, 1: downloading, 2: finished
  const handleOfflineFilesProgress = (data: DeviceOfflineFileResult) => {
    const { status, errorCode, response } = data || {};
    // Forward progress to the home/transfer page and update the syncing card info
    if (status === 1) {
      // Transferring
    } else if (status === 2) {
      // Transfer finished; continue detecting other devices or close the prompt
    }
  };
  ty.wear.onOfflineFilesProgressEvent(handleOfflineFilesProgress);

  // Switch transfer channel: Wi-Fi fast transfer <-> BT
  await tttSwitchModeLoadOfflineFile({
    channel: 2, // 2: Wi-Fi fast transfer; 1: Bluetooth LE
    deviceId,
  });

Code snippet

  // Get the cloud sync switch status and initialize sync state
  const data = await tttGetCloudSyncStatus();
  // data.enabled: whether enabled
  // data.syncType: 1 Wi-Fi only; otherwise all networks
  // data.modifyTime: last sync time

  // Enable/disable cloud sync (Highway: m.wearable.sync.switch.save)
  await toggleCloudSync({
    enabled: true,
    syncType: 'wifi', // 'wifi': Wi-Fi only; 'all': all networks
  });

  // Manually sync Note records (upload files + audio, and download cloud Notes)
  await tttSyncNoteRecord();

  // Listen for cloud sync switch changes
  ty.wear.onCloudSyncSwitchStatusEvent(res => {
    // res.enabled/res.syncType/res.modifyTime
  });

  // Listen for file list refresh (including cloud sync upload progress)
  // syncBusinessType: 0 other business (refresh list), 1 upload business (update cloud sync card status only)
  ty.wear.onFileListUpdateStatusEvent(res => {
    if (res?.syncBusinessType === 1) {
      // Update cloud sync card status
    } else {
      // Refresh the recording list
    }
  });

  // Clear local audio cache (cloud copies are retained)
  await tttRemoveFiles({ fileIds: [], isDeleteAll: false });

  // Download cloud audio on demand from the detail page
  await tttDownloadNoteAudio({ fileId, recordId });