Live Preview and Two-Way Talk

IPC Development Framework supports real-time video delivery powered by peer-to-peer (P2P) and Web Real-Time Communication (WebRTC).

• P2P

  Live preview and two-way talk work on top of Tuya’s proprietary P2P technology. A device and a mobile app can establish a secure P2P connection for real-time audio and video communication.

• WebRTC

  WebRTC can enable real-time communication of audio, video, and data for the web. With WebRTC, the IPC can establish a connection with WebRTC-compatible browsers and speakers to transfer audio and video in real time.

How it works

P2P

Devices in different local area networks (LANs) cannot communicate directly using their LAN IP addresses because of network address translation (NAT). P2P technology solves this issue.

To establish a P2P connection, two parties use the Session Traversal Utilities for NAT (STUN) protocol to request their public IP addresses from the STUN server. Then, they can talk to each other with their public IP addresses.

However, certain types of NAT do not allow hole punching, preventing a direct connection between two parties. The following table lists the support for NAT traversal.

NAT type	Full cone	Restricted cone	Port restricted cone	Symmetric
Full cone	Possible	Possible	Possible	Possible
Restricted cone	Possible	Possible	Possible	Possible
Port restricted cone	Possible	Possible	Possible	Not possible
Symmetric	Possible	Possible	Not possible	Not possible

If a direct P2P connection cannot be made, you can forward data through the cloud server.

Tuya’s proprietary P2P technology encrypts audio and video data with AES to secure data transfer.

WebRTC

On top of the existing P2P technology, WebRTC implements secure data transfer on the P2P channel.

• Stage 1: Create a data transmission channel with the P2P protocol.
• Stage 2: Two parties use Datagram Transport Layer Security (DTLS) on top of the User Datagram Protocol (UDP) to agree on keys for secure data transmission.
• Stage 3: With the agreed key and Secure Real-time Transport Protocol (SRTP), audio and video data is encrypted and transmitted between two parties.

Features

• SD and HD mode. Switch between SD and HD output on the mobile or web app based on video resolution.
• Two-way talk. The device’s microphone picks up sound and sends it to the mobile or web app and vice versa.
• Status notification. With the event callbacks, you will get notified of live preview status, such as streaming status and resolution switching.
• Data encryption.
• Mutual authentication. Two parties authenticate each other to ensure secure communication.

Related files

• tuya_ipc_media_adapter.h
• tuya_ipc_media_stream.h
• tuya_ipc_media_stream_event.h

Integration process

The audio and video service relies on basic services such as network and time. Therefore, initialize and start the audio and video service after the device connects to the MQTT service and syncs its clock with the internet time. It takes time for a low power device to connect to the MQTT service. During this period, you can start the network-independent part of audio and video service to speed up streaming.

Mains-powered device

The demo included in the development kit provides a typical startup procedure for a mains-powered device, as shown below.

Low power device

For a low power device, you can create a separate thread to start the audio and video service, as shown below.

Implementation design

Streaming to the mobile app is divided into two parts:

1. Establish a connection

   Create a direct network link between the device and the mobile app for data exchange.

2. Protocol communication and audio and video transmission

   The device and mobile app can exchange signaling messages, such as to start preview and switch resolution, as well as transmit audio and video data.

The following sections use a live preview request from the mobile app as an example to describe the design ideas and troubleshooting steps.

Establish a connection

In this stage, the mobile app shows a message saying Establishing an encrypted channel.

The process of the mobile app establishing a connection with the device.

Request resources

A connection between the mobile app and the device uses the memory and connection channel. Below is the resource usage on a Linux system:

Connectivity	Memory	Connection channel
P2P	1 MB	1
WebRTC	1.5 MB	1

When you initialize the audio and video service, you can set the maximum simultaneous channels (three to five is recommended) and memory size. If the limit is reached, the new connection request will be forcibly closed.

• Set the maximum connection channels

  When tuya_ipc_media_stream_init is running, set the parameter MEDIA_STREAM_VAR_T:max_client_num.

• Set the maximum memory size

  When tuya_ipc_media_adapter_init is running, set the parameter TUYA_IPC_MEDIA_ADAPTER_VAR_T:available_media_memory.

Perform authentication

To protect privacy, authentication is performed after the mobile app and device are connected to ensure that the user has permission to access the streaming video.

Two key pieces of information for authentication are managed by Tuya’s SDK.

• P2P password
• Device activation information

Authentication failure of either piece of information will close the session.

Transmit audio and video data

In this stage, the mobile app shows a message saying Retrieving video stream.

After the connection is made, the device and mobile app can exchange signaling messages as well as transmit data.

• Exchange signaling messages, such as setting the resolution and turning on/off live preview and audio.

• The device sends the audio and video data to the mobile app as instructed by the signaling messages.

• The mobile app sends the audio for two-way talk to the device.

Things to note:

• For previewing on the mobile app, it is recommended to pre-feed the audio and video data into the ring buffer to avoid slow streaming.
• For two-way talk, do not block audio playing to avoid audio anomalies on the mobile app.

Close a connection

Typically, when the user exits the preview, the app will proactively request to close the session. If anomalies occur, a session will be forced to close.

• When no data is transmitted within 20 minutes in a link, a botnet detection is triggered and the session is closed.
• When no heartbeat data is transmitted within about 30 seconds in a P2P link, network disconnection is triggered and the session is closed.

Typical performance metrics and anomalies

• Streaming speed

  Based on big data analytics, it takes about 2.5 seconds for a mains-powered device to begin streaming.

• Latency

  Latency depends on the network condition. The laboratory data analytics show that the latency ranges from 250 to 400 milliseconds.

• Jank

  Both the device SDK and the mobile app use multiple buffers to cope with network jitter for smooth playback. In case of high network jitter or poor network condition, jank might occur.

• Frame skipping

  If the device’s bitrate is higher than the bandwidth between the device and the mobile app, it can cause excessive message accumulation and therefore frame skipping. When data transmission is low due to poor network conditions, frame skipping can also occur.

• Screen glitches

  The SDK uses various algorithms to prevent packet loss and ensure reliable transmission. When audio and video data transmission works fine, screen flickering will not occur in previewing on the mobile app. In case of poor network conditions and severe packet loss, screen glitches might occur in previewing on the WebRTC-based web app.

Requirements for audio and video

• Video

  To avoid decoding failures on low-performance mobile phones, it is recommended to align the resolution and encoding level (profile) with the chosen format, such as H.264 or H.265.

• Audio

  The size of the audio data fed each time must not exceed 800 bytes.

• Presentation timestamp (PTS)

  If PTS is provided before data is put into the ring buffer, it must be specified in microseconds.

Troubleshooting

Connection failed

Determine connection failure:

• The mobile app shows a message saying Failed to Establish Encrypted Channel.
• No listen ok or __p2p_deal_with_listen appears in the device logs (debug level).

Before troubleshooting, learn about how to read the MQTT message.

• Determine if the device has received MQTT messages.

  Rev MQTT: appears in the device logs (debug level). The content after the colon is the payload.

• Determine if the MQTT message from the mobile app is for establishing a connection.

  Check if "protocol":302 for streaming protocol appears in the MQTT message.

• Determine if the device successfully sends the MQTT message for establishing a connection.

  Check if Send MQTT Msg.P:302 appears in the device logs (debug level).

See the steps to establish a connection and try the following troubleshooting steps:

1. Check the MQTT connection.

   If the mobile app or device fails to connect to the MQTT service, all message exchanges over MQTT are not possible.

2. Check if the mobile app has gotten the connection configuration information (step 3).

   To verify this point, review the mobile app logs or check if the mobile app has sent the connection establishment request (step 5).

3. Check if the device has received the connection establishment request. That is, to verify steps 5 and 6.

   The MQTT message for the connection establishment request might be too long to be printed entirely. You can look for the 302 MQTT message that contains fields such as tcp_token, token, and sdp.

4. Check if the device has responded to the connection establishment request. That is, to verify step 7.

   Check if the device has sent the 302 MQTT message that contains fields such as "type":"answer", "tcp_token", and sdp.

5. Check if the device has received the IP address of the mobile app. That is, to verify steps 11 and 12.

   Check if the device has received the 302 MQTT message that contains "type":"candidate".

6. Check if the device has sent its IP address. That is, to verify step 13.

   Check if the device has sent the 302 MQTT message that contains "type":"candidate".

7. If the problem persists, submit a service ticket to request technical support.

Session full

Determine if sessions are full by logs.

• Check if incoming connection refused, alloc resource failed, or no value able session please check process appears in the logs.

• Mobile app prompts device busy.

  This is typically caused by session management and is irrelevant to your code. If it appears frequently, try the following troubleshooting steps:

  • Check if other devices are streaming and if the connection channel limit has been reached. If this is not the case, continue with the following steps.
  • If the error occurs on a different mobile app, try reproducing it using the Smart Life app.
  • If the error frequently appears on the Smart Life app, submit a service ticket to request technical help.

Authentication failed

Determine authentication failure:

• auth failed appears in the device logs.
• The error code -105 appears in the mobile app logs.

Troubleshooting steps:

1. Check the reception of the authentication information.

   If get userinfo timeout or get userinfo error appears in the device logs, it indicates the device failed to receive the authentication information from the mobile app.

2. Check if the P2P password stored locally is correct.

   In the device logs (debug level), the string after get p2p passwd = is the P2P password. This field is only used to authenticate the device and mobile app, so you do not need to worry about security risks from log exposure. If the password value does not start with ad, the password is wrong. If the P2P password is wrong, check the device’s DB file.

3. Try again after clearing the cache on the mobile app.

   The mobile app might use the cache P2P password or device activation information for authentication. After clearing the cache, the mobile app will pull the information from the cloud for authentication. If this works, you need to review the cache policy to see if it fits the current scenario.

Data transmission failed

If the mobile app is stuck on Retrieving video stream and finally shows Failed to retrieve video stream, an issue with signaling or data transmission occurs.

See the process of the audio and video service and try the following troubleshooting steps:

1. Check signaling exchange.

   Through event callbacks, check what commands the device has received, such as to start streaming and turn on background music. MEDIA_STREAM_EVENT_E is the command enum. You can identify a command by name.

2. Check the data sending thread.

   Check if media send proc alive periodically appears in the device logs (debug level). If this is the case, the thread works fine.

3. Check data read in the buffer ring.

   Check the device logs (debug level) for any sudden surge in messages about tuya_ring_buffer.c. If such a surge occurs, there is a problem with feeding audio and video data into the buffer ring.

4. Check the change in the number of data sending frames.

   Check if session send video cnt (the number of frames to send video) and session send audio cnt (the number of frames to send audio) periodically appear in the device logs (debug level). If the number grows, data sending works fine.

5. If the problem persists, submit a service ticket to request technical support.

FAQs

Why is streaming stuck on creating an encrypted channel when previewing on the mobile app?

• The device is unable to receive the connection request from the mobile app because the MQTT connection has been lost.
• There is an issue with the network that the device or the mobile app connects to.

Why does the mobile app display a Device Busy prompt when attempting to create an encrypted channel for preview?

The number of sessions has reached the upper limit max_client_num set by tuya_ipc_media_stream_init.

Why is streaming stuck on loading when previewing on the mobile app?

• Audio and video encoding parameters do not match the actual values. Check the parameters set by tuya_ipc_media_adapter_set_media_info.
• The audio and video stream fails to be put into the ring buffer.
• The device network is slow.

Why does streaming freeze when previewing on the mobile app?

• The device network is slow.
• The audio and video timestamps in the ring buffer are not correct. The timestamp should be in milliseconds, while pts timestamp should be in microseconds.

Why is streaming so slow when previewing on the mobile app?

• For a low power device, check if the system is started too slow.
• For a low power device, check if P2P is started too late.
• For a low power device, check if the MQTT connection is made too slow.
• Check if the video data has been fed into the ring buffer.

How can I determine when the preview starts and ends?

• When MEDIA_STREAM_EVENT_CB generates the event MEDIA_STREAM_LIVE_VIDEO_START, the preview starts.
• When MEDIA_STREAM_EVENT_CB generates the event MEDIA_STREAM_LIVE_VIDEO_STOP, the preview ends.

When the mobile app process on iOS gets killed, why doesn’t the device instantly detect the preview is terminated?

The mobile app sends a connection closing request to the device when it exits the preview. The device receives the event MEDIA_STREAM_LIVE_VIDEO_STOP indicating that the preview has ended. If the mobile app process gets killed, it does not send a connection closing request to the device. It should wait for a device heartbeat timeout of about one to two minutes.

Why can’t WebRTC play audio?

• WebRTC only supports two audio encoding formats: PCM and G.711u.
• For the sampling rate, PCM supports 8 kHz and 16 kHz, while G.711u only supports 8 kHz.
• The audio frame size is 640 bytes for PCM and 320 bytes for G.711u.

How does the device distinguish between the mobile app preview and WebRTC preview?

In the event MEDIA_STREAM_LIVE_VIDEO_START, type in the struct C2C_TRANS_CTRL_VIDEO_START is used for this purpose.

• type = TRANSFER_SOURCE_TYPE_P2P indicates mobile app preview.
• type = TRANSFER_SOURCE_TYPE_WEBRTC indicates WebRTC preview.

Why doesn’t the device receive audio during a two-way talk initiated through the web app?

Check if the microphone is turned on for the web app.