AI Text-to-ImageOn-App AI
### Function Introduction
The AI Pixel Screen Image Generation Module generates pixel images locally using an **On-App AI Edge Model**, eliminating the need for cloud-based inference capabilities. Users simply select a tag to generate a pixel-style image in real-time within 1-2 seconds, supporting collection and Bluetooth delivery to the device for display.
This module is suitable for various pixel screen devices (car screens, clock screens, doorbell screens, custom pixel screens, etc.), meeting users' needs for creating personalized pixel patterns.
### Function Description
1. **Tag-Based Image Generation**
After the user clicks on a tag (such as "maple tree," "bird," "emoji," etc.), the client will call the **local AI model** to generate a pixel image of the corresponding size.
2. **Collecting Images**
Clicking the "Collect" button on the generated image saves it to the local gallery for easy reuse.
3. **Sending Images to Devices (Bluetooth Transparent Transmission)**
Clicking the "Send Preview" button on the image will send the generated pixel image to the device for real-time display via **Bluetooth Big Data Transmission**.
### Interaction Flow
```mermaid
sequenceDiagram
participant U as User
participant UI as Frontend Page
participant AI as On-Device AI Model
participant Dev as Device
U->>UI: Enter AI Drawing Page
UI->>UI: Check if model is initialized
alt Not Initialized
UI->>AI: Call pixelImageInit to initialize model
AI-->>UI: Initialization success
UI->>AI: fetchPixelImageCategoryInfo to get label data
AI-->>UI: Return label list
else Already Initialized
UI->>AI: fetchPixelImageCategoryInfo to get label data
end
U->>UI: Select a label
UI->>UI: Update message flow (user message + placeholder for generation)
UI->>AI: generationPixelImage(label)
AI-->>UI: Return generated image path
UI->>UI: Update message flow to display the image
U->>UI: Click the Favorite button
UI->>UI: Save to local gallery
U->>UI: Click the Send Preview button
UI->>Dev: Send image to device via BLE transparent transmission
Dev-->>UI: Display confirmation
```
### Dynamic datasets
```mermaid
flowchart LR
%% --- Business Definition Module ---
subgraph BD[Business Definition]
A[Define Category] --> B[Define Label]
B --> C[Reference Image]
end
%% --- T2I Generation Module ---
subgraph TG[Text-to-Image Generation]
C --> D[Image Preprocessing]
D --> E[Prompt Inversion]
E --> F[Image Generation]
end
%% --- Dataset Output Module ---
subgraph DS[Dataset]
F --> G["Image Dataset\n(Image Data + Image Labels)"]
end
```
**Custom Pixel Image Dataset Process Description:**
- Define Categories: e.g., Fruits, Emoticons.
- Define Tags: e.g., Avocado.
- Reference Image: Select a pixel-style image as a reference image.
- Image Preprocessing: Image cutout, preserving the main subject.
- Hint Word Derivation: Derive hint words from the image to obtain hint words for the text-to-image format.
- Image Generation: Batch generate images using the hint words.
- Image Dataset: Includes image data and image tags.
### Module Working Principle
The core of the module is based on the following logic:
#### 1. Local AI Model Initialization
Upon entering the page:
```ts
if (!hasModelInit) {
pixelImageInit(); // Initialize local AI model
fetchPixelImageCategoryInfo(); // Fetch label list
}
```
#### 2. Retrieving the Tag List
After the local model initialization is complete, it will return the available tags:
```ts
const labelInfo = await fetchPixelImageCategoryInfo();
dispatch(updateLabelAsync(labelInfo));
```
#### 3. Generate Pixel Image
After clicking the tag, an image is generated from the client's local model:
```ts
generationPixelImage({
deviceId: devInfo.devId,
label,
imageWidth: 462,
imageHeight: 462,
outImagePath: localPath,
});
```
#### 4. Update the Dialogue Message Flow
The module displays the generation process in a chat format:
- Add user message
- Add "Generating" placeholder message
- Replace with real image content
```ts
updateMessages([...messages, newMsg]);
```
#### 5. Add to Favorites and Send
**Add to Favorites:** Adds the image path to your local gallery.
**Send to Device:** Transmit via Bluetooth:
```ts
sendImageToDevice(path);
```