Emotion Perception

Overview

Emotion perception technology represents a critical challenge in the field of artificial intelligence. It aims to automatically detect and recognize human emotional states, such as happiness, sadness, and anger, through interactions with users. Accurate emotion recognition can help the system make more appropriate responses and improve user experience.

Wukong AI Hardware Development Framework supports emotion perception technology. You can obtain the emotional state of each interaction and implement some customized expressions and actions based on the emotion type. This can improve the interactive experience and implement some special features.

Functional description

The emotion skill shall be delivered as a JSON-formatted text payload during interactions with large language models, enabling contextual emotional expression in responses. To enable emotion perception, the system must register a text content callback during initialization. The registered callback function shall parse and express emotional cues from incoming text messages.

The specific JSON string format is as follows:

{"bizId":"skill-........emo","bizType":"SKILL","eof":1,"data":{"code":"emo","skillContent":{"emotion":["NEUTRAL"]}}}

Express emotions

After the emotions returned by the model are received, they can be expressed in user-defined ways, such as expressions and actions.

Example

STATIC OPERATE_RET _parse_skill(ty_cJSON *skill)
{
    ty_cJSON *node, *skill_content;
    const CHAR_T *code;

    // Parse code
    node = ty_cJSON_GetObjectItem(skill, "code");
    code = ty_cJSON_GetStringValue(node);
    if (!code)
        return OPRT_OK;

    TAL_PR_DEBUG("skill code: %s", code);
    skill_content = ty_cJSON_GetObjectItem(skill, "skillContent");
    if (s_chat_cbc.tuya_ai_chat_custom) {
        s_chat_cbc.tuya_ai_chat_custom(code, skill_content, s_chat_cbc.user_data);
    }

    if (strcmp(code, "emo") == 0) {
        // Detect emo
        // TBD...
    }

    return OPRT_OK;
}

STATIC OPERATE_RET _ai_agent_txt_cb(AI_TEXT_TYPE_E type, CHAR_T *data, INT_T len)
{
    ty_cJSON *json, *node;

    // {"bizId":"asr-1741763201372","bizType":"ASR","eof":1,"data":{"text":"This is ASR text!"}}
    // {"bizId":"nlg-1741763173046","bizType":"NLG","eof":0,"data":{"content":"This is the NLG response text!","appendMode":"append","finish":false}}
    // {"bizId":"skill-........emo","bizType":"SKILL","eof":1,"data":{"code":"emo","skillContent":{"emotion":["NEUTRAL"]}}}
    if ((json = ty_cJSON_Parse(data)) == NULL) {
        return OPRT_OK;
    }

    // Parse bizType
    node = ty_cJSON_GetObjectItem(json, "bizType");
    const CHAR_T *bizType = ty_cJSON_GetStringValue(node);
    // Parse eof
    node = ty_cJSON_GetObjectItem(json, "eof");
    BOOL_T eof = ty_cJSON_GetNumberValue(node);
    // TAL_PR_DEBUG("bizType: %s, eof: %d", bizType, eof);

    if (eof && strcmp(bizType, "ASR") == 0) {
        node = ty_cJSON_GetObjectItem(json, "data");
        _parse_asr(node);
    } else if (strcmp(bizType, "NLG") == 0) {
        node = ty_cJSON_GetObjectItem(json, "data");
        _parse_nlg(node, eof);
    } else if (eof && strcmp(bizType, "SKILL") == 0) {
        node = ty_cJSON_GetObjectItem(json, "data");
        _parse_skill(node);
    }

    ty_cJSON_Delete(json);
    return OPRT_OK;
}

STATIC OPERATE_RET __ai_demo_init(VOID)
{
    OPERATE_RET rt = OPRT_OK;
    AI_AGENT_CFG_T ai_agent_cfg = {0};
    ai_agent_cfg.biz_code = TY_BIZCODE_AI_CHAT,
    ai_agent_cfg.output.text_cb = _ai_agent_txt_cb;
    TUYA_CALL_ERR_RETURN(tuya_ai_agent_init(&ai_agent_cfg));
    return rt;
}
STATIC OPERATE_RET __ai_demo_text_input(VOID)
{
    tuya_ai_input_start();
    tuya_ai_text_input("I failed the exam again!", strlen("I failed the exam again!"));
    tuya_ai_input_stop();
}

Support and help

If you have any problems with TuyaOS development, you can post your questions in the Tuya Developer Forum.