#Music Rhythm Feature

Project Overview: This module is applicable to Wi-Fi + BLE protocol RGBIC string light devices, supporting both local music rhythm and cloud music rhythm modes. It can adjust light color and brightness in real-time based on audio signals to achieve synchronization between music and lighting effects.

#🖼️ Preview:

The music rhythm feature listens to device microphone or App audio signals, analyzes audio characteristics (such as decibel values, frequencies, etc.) in real-time, and converts audio characteristics into lighting effects. This feature supports multiple music mode selections, adjustable sensitivity, and both random color and preset color modes, providing users with an immersive music lighting experience.

#Feature List

1. Local Music Rhythm: Captures audio through device microphone for real-time analysis and light control
2. APP Music Rhythm: Obtains audio data through APP audio interface to control lighting effects
3. Music Mode Selection: Supports multiple built-in music modes (Rock, Jazz, Classical, etc.)
4. Sensitivity Adjustment: Adjustable audio sensitivity to control light response level
5. Random Color Mode: Randomly selects colors based on audio for dynamic effects
6. Preset Color Mode: Uses preset color combinations for fixed color schemes
7. Mode Switching: Supports switching between local music and cloud music modes
8. State Synchronization: Music rhythm state synchronizes with device state

#Module Implementation Details

#Implementation Overview

The music rhythm feature uses two different implementation approaches:

• Local Music Rhythm: Uses dreamlightmic_music_data DP point, device captures and processes audio through microphone, panel controls music mode and parameters through DP point
• APP Music Rhythm: Uses music_data DP point, obtains audio data through App's MediaKit audio interface, panel calculates colors in real-time and sends commands

Core design concepts:

• Data Separation: Local and cloud use different DP points and data structures
• Real-time Processing: Cloud mode uses throttling for audio data processing to avoid frequent sending
• State Management: Uses cloud storage to save current music mode and sensitivity settings
• Color Strategy: Supports both random color and preset color strategies

#Core Components

• src/components/music/index.tsx: Main music rhythm component, responsible for mode switching and overall layout
• src/components/music/local/index.tsx: Local music rhythm component
• src/components/music/app/index.tsx: APP music rhythm component
• src/hooks/useLocalMusicInit.ts: Local music initialization and management Hook
• src/standModel/musicModel/LocalMusic/dpParser/localMusic__dreamlightmic_music_data.ts: Local music DP parser
• src/components/music/app/sdk.ts: APP music audio processing SDK
• src/hooks/useStoreSensitivity.ts: Sensitivity storage management Hook
• src/components/random-colors/index.tsx: Random color selection component

Component Responsibilities:

• Music Component: Acts as a container component, manages music mode switching (local/cloud), coordinates sub-components
• LocalMusic Component: Handles local music mode, displays music list and sensitivity adjustment
• AppMusic Component: Handles cloud music mode, registers audio listeners and color calculation
• useLocalMusicInit Hook: Encapsulates local music initialization, playback, and sensitivity adjustment logic
• Audio SDK: Encapsulates MediaKit audio interface, provides audio data callbacks

#Data Flow

sequenceDiagram
    participant User as User
    participant Music as Music Component
    participant Local as Local Music Component
    participant App as Cloud Music Component
    participant Hook as Init Hook
    participant SDK as Audio SDK
    participant DP as DP Protocol
    participant Device as Device

    User->>Music: Select music mode
    Music->>Local: Display local music
    User->>Local: Select music type
    Local->>Hook: Call onPlay
    Hook->>DP: Encode dreamlightmic_music_data
    DP->>Device: Send DP
    Device->>Device: Microphone captures audio
    Device->>User: Display lighting effect

    User->>Music: Switch to cloud music
    Music->>App: Display cloud music
    User->>App: Select music type
    App->>SDK: Register audio listener
    SDK->>SDK: Get audio data
    SDK->>App: Callback audio data
    App->>App: Calculate color and brightness
    App->>DP: Encode music_data
    DP->>Device: Send DP
    Device->>User: Display lighting effect

#Mermaid Flowcharts

Main Feature Flow:

flowchart TD
    A[Enter music rhythm page] --> B{Select music mode}
    B -->|Local music| C[Display local music list]
    B -->|Cloud music| D[Display cloud music list]
    C --> E[User selects music type]
    D --> F[User selects music type]
    E --> G[Set music parameters]
    F --> H[Register audio listener]
    G --> I[Encode dreamlightmic_music_data]
    H --> J[Get audio data]
    J --> K[Calculate color and brightness]
    K --> L[Encode music_data]
    I --> M[Send DP to device]
    L --> M
    M --> N[Device executes lighting effect]
    N --> O[User observes effect]
    O --> P{Continue adjusting?}
    P -->|Yes| Q[Adjust sensitivity/color]
    P -->|No| R[Maintain current state]
    Q --> G
    Q --> K

Data Flow Diagram:

sequenceDiagram
    participant User as User Action
    participant UI as UI Component
    participant Hook as Business Hook
    participant Parser as DP Parser
    participant Device as Device/Audio
    participant Storage as Cloud Storage

    User->>UI: Select music mode
    UI->>Hook: Initialize music data
    Hook->>Storage: Read saved music ID
    Storage-->>Hook: Return music ID
    Hook->>Parser: Build DP data
    Parser->>Device: Send DP command

    Note over Device: Local mode: Device processes audio<br/>Cloud mode: App provides audio

    Device->>Device: Analyze audio characteristics
    Device->>User: Display lighting effect

    User->>UI: Adjust sensitivity
    UI->>Hook: Update sensitivity
    Hook->>Parser: Update DP data
    Parser->>Device: Send update
    Hook->>Storage: Save sensitivity

Component Interaction Diagram:

graph TB
    A[Music Component] -->|Switch mode| B[LocalMusic Component]
    A -->|Switch mode| C[AppMusic Component]
    A -->|Use| D[RandomColors Component]
    B -->|Call| E[useLocalMusicInit]
    C -->|Use| F[Audio SDK]
    E -->|Use| G[useStoreSensitivity]
    E -->|Encode| H[LocalMusicParser]
    F -->|Callback| C
    C -->|Encode| I[MusicDataParser]
    H -->|Send| J[Device]
    I -->|Send| J
    G -->|Save| K[Cloud Storage]

State Transition Diagram:

stateDiagram-v2
    [*] --> Initialization: Page load
    Initialization --> SelectMode: User action
    SelectMode --> LocalMode: Select local
    SelectMode --> CloudMode: Select cloud
    LocalMode --> SelectMusic: Display list
    CloudMode --> SelectMusic: Display list
    SelectMusic --> Playing: Start playing
    Playing --> AdjustParams: User adjusts
    AdjustParams --> Playing: Update params
    Playing --> StopPlaying: User stops
    StopPlaying --> SelectMode: Reselect
    StopPlaying --> [*]: Exit page

#Key Code Snippets

Local Music Playback:

// File: src/hooks/useLocalMusicInit.ts
  const onPlay = (
    active: boolean,
    item: { id: number; colorArr: any[] },
    sensitivity: number,
    isInitDefault = false
  ) => {
    const nextLocalMusicList = {
      ...DefaultLocalMusicData,
      ...(localMusicList || {}),
      brightness: 100,
      sensitivity,
    };
    // Set rhythm color
    nextLocalMusicList.power = active;

    if (item) {
      if (active) {
        actions.work_mode.set('music', {
          ignoreDpDataResponse: isInitDefault ? false : true,
        });
      }

      nextLocalMusicList.id = item.id; // Music mode number
      if (enableRandom) {
        nextLocalMusicList.colors = getArray(item.colorArr);
        nextLocalMusicList.a = 1;
      } else {
        nextLocalMusicList.colors = getArray(selectedRandomColors?.colors); // When random is disabled, use default color palette
        nextLocalMusicList.a = 0;
      }
    }
    structuredActions.dreamlightmic_music_data.set(nextLocalMusicList, {
      ignoreDpDataResponse: isInitDefault ? false : true,
    });
  };

Cloud Music Audio Listener:

// File: src/components/music/app/index.tsx
// Register listener
const onPlay = () => {
  console.log('Register listener');
  onMusic2RgbChange(
    data => {
      if (!data) return;

      // States here change in real-time
      // React state characteristics cause this to always use initial state, so read from Ref

      // Get current color group
      // ------------------------
      const currentId = currentIdRef.current;
      const appItem = getArray(dataSource).find(item => item.id === currentId);
      const mode = appItem?.mode ?? 1;
      let colorList: { hue: number; saturation: number; value: number }[];

      const enableRandom = enableRandomRef.current;
      const selectedRandomColors = selectedRandomItemRef.current;
      if (enableRandom) {
        // When random color is enabled, use HSV from app rhythm callback
        colorList = getArray(appItem?.colorArr) as any;
      } else {
        // When random color is disabled, use preset colors
        colorList = getArray(selectedRandomColors?.colors).map(item => ({
          ...item,
          saturation: item.saturation * 10,
          value: item.value * 10,
        }));
      }

      const nextColorList = getArray(colorList).length > 0 ? getArray(colorList) : [];
      // ------------------------

      const musicData = {
        mode: mode as any,
        hue: data?.hue,
        saturation: data?.saturation,
        value: data?.value,
        brightness: data?.bright,
        temperature: data?.temperature,
      };

      const { db, dB } = data?.extra || {};
      const _db = dB || db || 0;

      if (nextColorList.length > 0) {
        // Randomly get one from colorList
        // ------------------------
        const randomColor = nextColorList[Math.floor(Math.random() * nextColorList.length)];

        musicData.hue = randomColor.hue;
        musicData.saturation = randomColor.saturation;
        let brightness = randomColor.value;

        const dBRange = [40, 80];
        const [minDB, maxDB] = dBRange;
        const [minBright, maxBright] = [0, 1000];

        if (_db <= minDB) {
          brightness = minBright;
        } else if (_db >= maxDB) {
          brightness = maxBright;
        } else {
          brightness = Math.round(calcPosition(_db, minDB, maxDB, minBright, maxBright));
        }
        musicData.value = brightness;
      }

      const SupportUtils = devices.common.model.abilities.support;
      if (SupportUtils.isSupportColour()) {
        // When color light is supported, remove white light
        musicData.brightness = 0;
        musicData.temperature = 0;
      }

      structuredActions.music_data.set(musicData, {
        throttle: 300,
      });
    },
    {
      mode: 1,
      colorList: defaultAppMusicList[0].colorArea,
    }
  );
};

Audio Data Processing SDK:

// File: src/components/music/app/sdk.ts
export const onMusic2RgbChange = (
  callback: (musicData: {
    mode: number;
    hue: number;
    saturation: number;
    value: number;
    bright: number;
    temperature: number;
    extra: any;
  }) => void,
  musicOption?: TMusicOption
) => {
  // If currently listening, return directly
  if (isListening) {
    return;
  }
  if (!manager) {
    try {
      manager = (ty as any)?.media?.getRGBAudioManager?.();
    } catch (err) {
      console.warn('no getRGBAudioManager', err);
      return;
    }
  }
  if (!manager) {
    return;
  }
  const handleAudioRgbChange = throttle((data: string) => {
    const {
      R,
      G,
      B,
      C: temp,
      L: bright,
      db,
      dB,
    } = (JSON.parse(data) || {}) as {
      R: number;
      G: number;
      B: number;
      C: number;
      L: number;
      db: number;
      dB: number;
      index: number;
    };
    // Compatible with different version fields
    const _db = dB || db || 0;
    let hue = 0;
    let saturation = 1000;
    let value = 1000;
    const { mode = 1, colorList, dBRange = [40, 80] } = musicOption || {};
    const [minDB, maxDB] = dBRange;
    const [minBright, maxBright] = [0, 1000];
    if (colorList) {
      // Randomly get one from colorList
      const randomColor = colorList[Math.floor(Math.random() * colorList.length)];
      if (!randomColor) {
        console.log('onMusic2RgbChange => Failed to get random color');
        return;
      }
      hue = randomColor.hue;
      saturation = randomColor.saturation;
      let brightness = randomColor.value;
      if (_db <= minDB) {
        brightness = minBright;
      } else if (_db >= maxDB) {
        brightness = maxBright;
      } else {
        brightness = Math.round(calcPosition(_db, minDB, maxDB, minBright, maxBright));
      }
      value = brightness;
    } else {
      [hue, saturation, value] = rgb2hsb(R, G, B).map((v, i) => (i > 0 ? v * 10 : v));
    }

    const musicData = {
      mode,
      hue: Math.round(hue),
      saturation: Math.round(saturation),
      value: Math.round(value),
      bright: Math.round(bright * 10),
      temperature: Math.round(temp * 10),
      extra: {
        R,
        G,
        B,
        C: temp,
        L: bright,
        db,
        dB,
      },
    };

    callback && callback(musicData);
  }, 300);
  manager.onAudioRgbChange(({ body }) => {
    handleAudioRgbChange(body);
  });
  start();
};

Sensitivity Adjustment:

// File: src/hooks/useLocalMusicInit.ts
const onSensitivityChange = (sensitivity: number) => {
  localMusicList.sensitivity = sensitivity;
  setIdSi(localMusicList?.id, String(sensitivity));
  structuredActions.dreamlightmic_music_data.set(
    {
      ...localMusicList,
      brightness: 100,
    },
    {
      throttle: 100,
    }
  );
};

#Data Structures

Local Music DP Data Structure:

type TMusic = {
  v: number; // Version number
  power: boolean; // Whether enabled
  id: number; // Music mode number
  isLight: number; // 0: Light off when silent, 1: Light maintains 10% brightness when silent
  mode: number; // 0: Jump, 1: Gradient, 2: Breathing, 3: Flashing
  speed: number; // Speed (0-100)
  sensitivity: number; // Sensitivity (0-100)
  a: number; // Random flag (0: Preset color, 1: Random color)
  b: number; // Reserved field
  c: number; // Reserved field
  brightness: number; // Brightness (0-100)
  colors: Array<{
    hue: number; // Hue (0-360)
    saturation: number; // Saturation (0-100)
  }>; // Color list
};

Cloud Music DP Data Structure:

type MusicData = {
  mode: number; // Music mode (0: Jump, 1: Gradient)
  hue: number; // Hue (0-360)
  saturation: number; // Saturation (0-1000)
  value: number; // Brightness (0-1000)
  brightness: number; // White light brightness (0-1000)
  temperature: number; // Color temperature (0-1000)
};

Audio Data Callback Structure:

type AudioData = {
  R: number; // Red component (0-255)
  G: number; // Green component (0-255)
  B: number; // Blue component (0-255)
  C: number; // Color temperature
  L: number; // Brightness
  db: number; // Decibel value
  dB: number; // Decibel value (compatible field)
  index: number; // Index
};

Random Color Data Structure:

type RandomItem = {
  id: number; // Color combination ID
  colors: Array<{
    hue: number; // Hue
    saturation: number; // Saturation (0-100)
    value: number; // Brightness (0-100)
  }>; // Color list
};

#API Calls

Local Music DP Interface:

• DP Point: dreamlightmic_music_data
• Call Method: structuredActions.dreamlightmic_music_data.set(data, options)
• Data Format: Hexadecimal string encoded by MicMusicFormater
• Call Timing:
  • When selecting music mode
  • When adjusting sensitivity
  • When switching between random/preset colors
  • When restoring previous state during initialization

Cloud Music DP Interface:

• DP Point: music_data
• Call Method: structuredActions.music_data.set(data, options)
• Data Format: Protocol-encoded data object
• Call Timing:
  • On audio data callback (throttled 300ms)
  • When selecting music mode
  • When switching between random/preset colors

Audio Interface:

• Interface: ty.media.getRGBAudioManager()
• Methods:
  • startRGBRecord(): Start audio recording
  • stopRGBRecord(): Stop audio recording
  • onAudioRgbChange(callback): Register audio data callback
  • offAudioRgbChange(): Unregister audio data callback
• Call Timing:
  • Register when entering cloud music mode
  • Unregister when leaving cloud music mode

Cloud Storage Interface:

• Storage Keys:
  • CLOUD_KEY_LOCAL_MUSIC_ID: Local music ID
  • CLOUD_KEY_APP_MUSIC_ID: Cloud music ID
  • CLOUD_RANDOM_COLOR_ID: Random color ID
• Call Method: useCloudStorageKey(key, options)
• Data Format: String (music ID or color ID)

#Notes

#Implementation Challenges

1. Audio Data Processing Performance:

   • Challenge: High frequency of audio data callbacks, frequent calculation and sending may cause performance issues
   • Solution: Use throttle to limit processing frequency (300ms), reduce calculation and sending times

2. State Synchronization Issue:

   • Challenge: Using closures in audio callbacks, React state may not be the latest value
   • Solution: Use useRef to save latest state, read from ref in callbacks

3. Color Calculation Logic:

   • Challenge: Need to dynamically calculate brightness based on decibel values while supporting both random and preset colors
   • Solution: Use calcPosition function to calculate decibel to brightness mapping, select color strategy based on mode

4. Mode Switch Cleanup:

   • Challenge: Need to cleanup previous listeners and state when switching music modes
   • Solution: Call offMusic2RgbChange when component unmounts, cleanup audio listeners and screen-on state

5. Sensitivity Persistence:

   • Challenge: Different music modes need separate sensitivity save and restore
   • Solution: Use useStoreSensitivity Hook, store sensitivity values with music ID as key

#Common Issues

1. Issue: Cloud music mode has no effect

   • Cause: Audio interface not initialized or permission not granted
   • Solution: Check if getRGBAudioManager is available, ensure microphone permission is granted

2. Issue: Local music mode device not responding

   • Cause: DP data format error or device doesn't support the music mode
   • Solution: Check if DP data encoding is correct, confirm device supported music mode list

3. Issue: Color changes not smooth

   • Cause: Throttle time set too long or DP sending failed
   • Solution: Adjust throttle time (recommended 300ms), check network connection and device status

4. Issue: Sensitivity adjustment not working

   • Cause: Sensitivity value not saved correctly or DP sending failed
   • Solution: Check if cloud storage is working, confirm DP send success callback

5. Issue: State confusion after mode switch

   • Cause: Previous mode state and listeners not properly cleaned up
   • Solution: Call cleanup function when switching modes, reset related states

#Performance Optimization

1. Throttle Processing:

   • Audio data callback uses throttle (300ms), reduces calculation and sending frequency
   • Sensitivity adjustment uses throttle (100ms), avoids frequent updates

2. Ref Usage:

   • Use useRef to save latest state, avoid closure issues
   • Use useRef to cache audio manager instance, avoid repeated creation

3. Conditional Rendering:

   • Conditionally render corresponding components based on music mode, reduce unnecessary rendering
   • Use useMemo to cache music list data

4. Data Caching:

   • Sensitivity data cached to cloud storage, avoid repeated calculation
   • Music list data uses useMemo for caching

5. Screen-on Management:

   • Only keep screen on during audio listening, restore when exiting
   • Use setKeepScreenOn to manage screen state

#Edge Cases

1. Audio Interface Unavailable:

   • Handling: Show error prompt, disable cloud music mode
   • Code: Check getRGBAudioManager return value, provide fallback solution

2. Device Doesn't Support Music Mode:

   • Handling: Determine support status based on DP schema, hide unsupported features
   • Code: Use useSupport to check DP support status

3. Audio Data Anomaly:

   • Handling: Validate data format, don't process abnormal data
   • Code: Add data validation logic in callback

4. Decibel Value Out of Range:

   • Handling: Limit decibel value range, use boundary values when out of range
   • Code: Use Math.max and Math.min to limit range

5. Empty Color List:

   • Handling: Use default color or calculate color from audio RGB values
   • Code: Check color list length, provide default values

6. Component Unmount Cleanup:

   • Handling: Cancel listeners and restore state in useEffect cleanup function
   • Code: Return cleanup function, call offMusic2RgbChange

#Dependencies

1. Dependent Modules:

   • @ray-js/panel-sdk: Provides DP operations and structured data interface
   • @ray-js/ray: Provides screen-on management interface
   • lodash-es: Provides throttle function
   • MediaKit: Provides audio interface (TTT capability)

2. Dependent By Modules:

   • Home page module: Uses music rhythm component
   • Work mode switch: Music mode as one of work modes

3. Dependency Notes:

   • MediaKit needs to be enabled in TTT configuration
   • Audio permission needs to be granted in App
   • DP protocol parser needs to be registered after device initialization

4. Decoupling Suggestions:

   • Encapsulate audio processing logic as independent SDK for reuse and testing
   • Extract color calculation logic as utility functions for unit testing
   • Use TypeScript type definitions to ensure data structure consistency
   • Make music mode data configurable for easy extension and maintenance