Omni AI Foundation v2.6 Architecture Overview

Omni AI Foundation is Tuya’s platform for multimodal AI hardware. It combines Tuya Real-Time Communication (T-RTC), AI agent runtime, the Dynamic Orchestration Agent System (DOS), and global distributed deployment. The platform provides a full-stack solution for AI hardware, covering device access, multimodal interaction, and workflow orchestration.

Version 2.6 introduces three key architecture upgrades:

• Workflow orchestration engine based on DOS: Now fully open. You can build complex AI applications with low-code workflows.
• End-to-end multimodal interaction optimization: Average latency in complex global scenarios stays within 1.3 s.
• OmniMem individual memory system: Now commercially available, addressing temporal reasoning and dynamic updates in AI memory.

Global real-time communication architecture

Infrastructure topology

The platform delivers global real-time interaction over the T-RTC acceleration network.

Dimension	Scale
Data centers	Seven global regions
Media acceleration network	Dozens of major countries
Edge acceleration nodes	Thousands of major cities

End-to-end latency

In complex scenarios involving memory retrieval, knowledge base retrieval, and tool calls, the average end-to-end interaction latency stays within 1.3 s.

Test conditions:

• Set Voice Activity Detection (VAD) silence to 800 ms.
• The overall experience outperforms common industry setups that use 300–500 ms VAD (short VAD windows increase false triggers; tune the value for your scenario).

High availability and disaster recovery

• Dynamic load balancing: Routes major global LLMs through the nearest access points. Real-time compute scheduling is available in shared and dedicated modes.
• Millisecond-level failover: Delivers 99.95% service availability with a financial-grade service-level agreement (SLA).
• Multi-protocol adaptation: Supports Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and WebSocket, with adaptive congestion control for different application scenarios.

Protocol availability

Protocol	Scenario	Status
WebSocket	PC and browser access	Available
UDP	App SDK integration	Available
TCP	Embedded devices	Planned for a future release

DOS multi-agent orchestration engine

DOS is the core architecture upgrade in v2.6. It aims to solve the problems of complex multimodal capability integration and insufficient development efficiency in AI hardware scenarios.

Architecture

DOS adopts a unified orchestration architecture that coordinates intent understanding, multi-agent execution, and response generation through a standardized workflow.

Core design principles

Unified input → Intent understanding and classification → Parallel multi-agent processing → Unified output

Key features

• Shortest-path routing: Dynamically determines the optimal execution path for complex workflows, balancing functional complexity with real-time responsiveness.
• Parallel multi-agent execution: Runs multiple agents in parallel to reduce serial wait time.
• MCP tool isolation: Enables or disables Model Context Protocol (MCP) tools per agent. Narrow each agent’s scope to capability boundaries to simplify prompts and reduce hallucinations.

Visual workflow orchestration

You can build complex workflows through drag-and-drop without writing orchestration code. The orchestration engine handles runtime routing, concurrency control, and exception handling.

Unified device-cloud MCP integration

Version 2.6 introduces an architectural redesign of device-side MCP integration.

Design description

Common device capabilities are abstracted into standardized cloud services, allowing AI agents to invoke device capabilities through a unified, secure, and controllable framework.

Standardized capabilities include:

• Real-time image capture
• Image recognition
• Sensor data collection

Performance improvements

End-to-end visual understanding latency improves by about 50% compared with the v2.5 device-side processing approach.

Prerequisite

Device-side MCP integration requires Wukong v3.13.0 or later.

Multi-terminal access

Access method	Description
Tuya Wukong AI	Embedded AI devices
TuyaOpen	Open-source hardware ecosystem
Tuya App SDK	Mobile apps
WebSocket	PC and browser terminals (new in v2.6)
Foundation SDK	Open system integration (planned)

Core algorithm models

Version 2.6 integrates core algorithm models for Voice Activity Detection (VAD), intent classification, and Automatic Speech Recognition (ASR) to support multimodal interaction.

High-precision VAD model

VAD is central to voice interaction. It balances response speed and false detection rate.

Metric	Tuya VAD	Benefit
Silence detection	500 ms	Reduces false cuts on valid speech and avoids frequent interruptions.
Interruption detection	300 ms	Responds quickly when the user interrupts.

Recommended configuration

Set interruption detection to 300 ms (fast response) and VAD silence detection to 800 ms (balanced). Under this configuration, global average end-to-end latency stays within 1.3 s while balancing responsiveness and fewer false triggers.

Domain intent classification model

Tuya trained a dedicated intent classification model on years of AI hardware data to reduce intent hallucinations and shorten response chains as LLM capabilities expand.

Technical advantages

• Pre-intent routing: Classifies intent before LLM inference to shorten the decision path.
• Hallucination mitigation: Applies domain constraints to reduce incorrect responses outside model capabilities.
• Plug-and-play extensibility: Enables the capability directly in workflows without changing the core application logic.

Current coverage

All official Tuya skills are supported. Third-party MCP tools and custom skill classification and retrieval are planned.

Multilingual ASR

To support global deployment, the ASR model includes targeted optimization for these languages:

Language	Optimization
English	Multiple accents
Spanish	Latin American and Iberian variants
Japanese	Grammar and intonation
Southeast Asian languages	Code-mixing scenarios

Benchmarks use the CommonVoice open test set, with recognition accuracy evaluated against Whisper-large-v3 (offline model). The platform matches each region with the best ASR provider for the lowest Word Error Rate (WER) in real-time streaming recognition.

OmniMem individual memory system

OmniMem provides commercially available individual memory capabilities. Enable it with a one-step configuration.

Technical challenges

AI memory systems must address four core problems:

Challenge	Description
Temporal processing	Reasoning over temporal relationships and modeling memory decay
Information noise	Impact of noisy data on effective memory retrieval
Memory discontinuity	Maintain continuity across sessions and devices
Dynamic updates	Memory correction and overwriting as user preferences change

Solution

OmniMem balances low latency and high accuracy through architecture optimization and algorithm innovation. Enable it with one platform configuration, without implementing memory management logic.

Roadmap

• Unified multimodal memory across text, voice, and visual data.
• Enhanced cross-device memory migration.
• More human-like memory interactions.

Technical highlights

The following table summarizes the core capabilities of Omni AI Foundation v2.6.

Capability	Tuya Omni AI Foundation v2.6	Advantage
End-to-end latency	1.3 s (including memory, knowledge base, and tool calls)	Significantly better than industry average in complex scenarios.
Service availability	99.95%	Financial-grade SLA
Workflow orchestration	DOS multi-agent parallel orchestration	Low-code workflows and shortest-path routing
Vision pipeline	Unified device-cloud MCP integration with a 50% performance improvement	Lower device-side development cost
VAD	500 ms silence and 300 ms interruption	Optimal balance of fluency and accuracy
Memory system	OmniMem with one-step commercial configuration	Leading scores in open test sets
Global deployment	Seven data centers and thousands of edge nodes	Consistent global experience
Protocol support	TCP, UDP, and WebSocket	Flexible connectivity across multiple terminals

Access and development resources

• Device access protocols: Tuya Wukong AI, TuyaOpen, App SDK, and WebSocket.
• Workflow orchestration: Visual workflow builder on the Tuya AI Developer Platform.
• Device-side MCP: Requires Wukong v3.13.0 or later.
• OmniMem: Available through one-step platform configuration.