On-App AI Video Subject Detection: EfficientDet-D0 On-Device Inference and Dynamic Frame Rate Strategy

As smart IPC devices (such as security cameras, pet companion robots, baby monitors, etc.) become increasingly popular, more and more life scenarios are being recorded in real time. However, in actual use, due to improper device installation positions and overly wide-angle lenses, the recorded subject often occupies too small a portion of the frame — resulting in "full-scene clarity but blurry subject," severely impacting the viewing experience.

---

1. Tuya AI Inference Model: Automatic Subject Enhancement

Tuya launches the On-App AI Video Subject Enhancement Solution, leveraging Tuya’s app-empowered AI capabilities for real-time object detection, automatically locating and identifying subjects, then applying image algorithms for intelligent adaptive zoom to make subjects more prominent and frames more focused. Finally, the processed video is encoded and packaged to output optimized video content.

This solution is suitable for expansion across various life scenarios:

• In indoor monitoring — for baby care, pet monitoring, pet companion robots, and other devices;
• In medical imaging or research — combined with endoscopes or other medical devices to highlight key surgical field areas;
• In outdoor recording — for smart bird feeders, trail cameras, nature landscape cameras, and other devices;
• In security and industrial inspection — for patrol, maintenance, and other devices requiring inspection of local details.

a. For example, for a cat in a home scenario, the camera automatically identifies the subject and zooms in:

b. In complex outdoor scenarios, the Tuya On-App AI video subject solution can still precisely identify target objects, perform motion tracking, and zoom in on the frame.

---

2. Tuya AI Video Subject Enhancement Technology Explained

1. Overall Technical Architecture

Across all Tuya AI products/hardware solutions, we have established end-to-end AI capabilities: device-side, cloud-side, and app-side tri-end collaboration. By deploying advanced AI inference models on Tuya-empowered apps, we help brand owners and developers seamlessly integrate advanced AI technology into mobile devices — building a more flexible AI architecture, superior user experience, and more secure computing capabilities.

• In mobile model deployment, Tuya uses a lightweight technology architecture (TensorFlow Lite, etc.), which offers advantages of efficient inference, low latency, and low power consumption. It supports local model offline operation, system updates, deployment, and other on-demand loading mechanisms, improving runtime efficiency and enabling a more flexible AI architecture.
• For image processing, Tuya uses OpenGL ES for more efficient rendering and optimization, fully leveraging the GPU to accelerate image processing;
• For video codec, Tuya’s platform provides hardware decoding to improve video processing performance, reduce CPU load, and ensure smooth video playback with low power consumption.

2. Technical Highlights

2.1 More Flexible AI Architecture: Lightweight and Dynamic

2.1.1 Lightweight Detection Model

Tuya uses EfficientDet-D0, a lightweight object detection model specifically optimized for mobile. With fewer parameters, lower computation, and fast inference speed, it can accurately detect the position and category of specified subjects (such as pets, people) in video. Combined with intelligent analysis capabilities, it quickly filters out segments without subjects, effectively reducing computational burden and improving processing efficiency.

2.1.2 Dynamic Model Management

Using an on-demand loading dynamic model management mechanism that supports online model download, update, and deployment — ensuring video applications always use the optimal model version while reducing initial installation package size and improving runtime efficiency.

2.2 Superior User Experience: Real-time and High Efficiency

2.2.1 Real-time Interactive Processing

This solution supports real-time response to user interaction needs. Relying on efficient local computation, it ensures smooth, lag-free user experience without depending on network connectivity for fast response and real-time processing.

2.2.2 Anti-Shake Image Algorithm

During the video subject position detection process, the detection box may experience jitter and offset, causing the zoomed-in subject frame to also shake. To address this, Tuya adds a jitter threshold to ensure smooth and fluid video processing.

2.2.3 Fine-tuned Models for Higher Accuracy

Tuya has accumulated massive subject images across different scenarios, covering various lighting conditions including day and night. After data desensitization, Tuya carefully trains and fine-tunes these models to ensure generalization capability during model inference.

2.3 More Secure Computing: Low Cost and Privacy Protection

2.3.1 Reduced Processing Cost

Tuya supports developers in dynamically adjusting video frame inference strategies. When no video subject is detected, the system processes one frame every several frames; when a subject is detected, it automatically adjusts to process one frame every 3 frames. This fully leverages mobile hardware acceleration, significantly improving video codec speed, reducing CPU load, and enhancing overall performance.

Moreover, local processing is lower cost than cloud processing and saves cloud workload — computing power consumption is reduced by 25 TFLOPs per 10,000 operations, saving approximately 10% in costs.

2.3.2 Privacy Protection

All data processing in this solution is completed locally, significantly reducing latency, improving response speed, while avoiding data transmission externally — enhancing user privacy protection and providing a more secure and efficient computing environment for applications.

---

3. Development Tutorials and Support

1. How to Develop AI Video Subject Enhancement?

The Tuya On-App AI Video Subject Enhancement Solution is built on Tuya’s smart IPC capabilities. To develop this solution, you need to first integrate the IPC SDK. Device-side solutions can reference IPC SDK development.

Development tutorial link:

https://t.tuya.com/AY1D3VbxRO

2. Technical Principles

For more information on AI video subject enhancement technical principles and development guide:

https://developer.tuya.com/cn/miniapp/solution-ai/case

Technical Principles Overview

3. Developer Support

For any issues during development, please post your questions on the Tuya Developer Forum:

https://www.tuyaos.com/viewforum.php?f=3