#t-agent-plugin-aistream

#Overview

t-agent-plugin-aistream is a core plugin for t-agent that provides the capability to connect to the mini-program AI agent platform. This plugin replaces the deprecated @ray-js/t-agent-plugin-assistant and offers more powerful features including streaming conversations, speech recognition, and multimodal support.

⚠️ Important Notice: @ray-js/t-agent-plugin-assistant has been deprecated, please use @ray-js/t-agent-plugin-aistream instead.

For more detailed information, please visit the official documentation.

#Core Concepts

#withAIStream

withAIStream is the main entry function of this plugin, used to create connections to the mini-program AI agent platform. It supports streaming conversations, speech recognition, and other features.

Main configuration parameters:

• agentId: Agent ID (required)
• clientType: Client type, default is APP (2)
• deviceId: Device ID, required when clientType is DEVICE (1)
• wireInput: Whether to pass input blocks to the agent, default is true
• historySize: History message size, default is 1000
• indexId: Index ID, default is 'default'
• homeId: Home ID, defaults to current home if not specified
• earlyStart: Whether to establish connection during onAgentStart phase
• tokenOptions: Parameters for getting agent token
  • api: API interface name
  • version: Interface version
  • extParams: Additional parameters

Usage example:

const agent = createChatAgent(
  withUI(),
  withAIStream({
    agentId: 'your-agent-id',
    earlyStart: true,
    historySize: 500,
  })
);

#withBuildIn

withBuildIn provides a series of built-in features including smart home control, knowledge base search, etc. Must be used after applying the withAIStream plugin.

Main features:

• Smart Home: Device control, scene management
• Knowledge Base Search: Associated document display
• Recommended Actions: Intelligent user operation recommendations
• Button Interaction: Dynamic button responses
• Workflow Guidance: Process-oriented operation guidance
• Smart Cards: Rich card displays

Usage example:

const agent = createChatAgent(
  withUI(),
  withAIStream({ agentId: 'your-agent-id' }),
  withBuildIn()
);

#Core Features

#Speech Recognition (ASR)

Built-in AsrAgent speech recognition agent that supports real-time speech-to-text conversion.

import { createAsrAgent } from '@ray-js/t-agent-plugin-aistream';

const asrAgent = createAsrAgent({
  agentId: 'your-agent-id',
  onMessage: message => {
    if (message.type === 'text') {
      console.log('Recognition result:', message.text);
    }
  },
});

#Hook Mechanism

Provides rich Hooks to customize behavior:

• onMessageParse: Triggered when parsing messages
• onSkillCompose: Triggered when receiving skill data
• onSkillsEnd: Triggered when all skills are processed
• onTTTAction: Triggered when tile uses sendAction
• onCardsReceived: Triggered when receiving card data

#Mock Mechanism

For development convenience, a mock mechanism is provided:

import { mock } from '@ray-js/t-agent-plugin-aistream';

// Mock AI Stream response
mock.hooks.hook('sendToAIStream', context => {
  if (context.options.blocks?.some(block => block.text?.includes('hello'))) {
    context.responseText = 'Hello! How can I help you?';
  }
});

// Mock ASR speech recognition
mock.hooks.hook('asrDetection', context => {
  context.responseText = 'Hello world! I am a virtual assistant.';
});

#Summary

t-agent-plugin-aistream is a core plugin for t-agent that provides more powerful features and better development experience compared to the deprecated assistant plugin. It supports streaming conversations, speech recognition, multimodal interaction, and other core features required for modern AI applications.

This plugin seamlessly integrates with the t-agent core library and t-agent-ui-ray component library, enabling developers to easily build feature-rich intelligent conversation applications. Through rich configuration options and Hook mechanisms, developers can perform deep customization according to specific requirements.

For more detailed information and API documentation, please visit the official documentation.