Audio and Video SDK

@ray-js/video-streams-sdk integrates methods that enable users to stream live videos, change voices in two-way talk, and play back recorded videos.

Installation

Import npm package

$ npm install @ray-js/video-streams-sdk
npm install weixin-js-sdk

// or

$ yarn add @ray-js/video-streams-sdk
yarn install weixin-js-sdk

Utilities

import { Utils } from '@ray-js/video-streams-sdk';

Decode an image URL

base64Decode

Request parameters

Parameter	Type	Description	Required
url	string	The encoded URL of the first-frame image. If the image is gotten from the cloud, the URL must be decoded.	true

Sample request

Utils.base64Decode(
  'aHR0cHM6Ly9lbmNyeXB0LXBpYy50dXlhY24uY29tL3YxL2FwaS9waWNzP3BpY0tleT1hYzdhOWUzZTM2MzI5YmVlY2YxNDI3MTA5ZT*****'
);

// return: https://**********

APIs

Call the following APIs on the server side to reduce the stream pulling time of the client.

import { API } from '@ray-js/video-streams-sdk';

Get an access token

getToken

Request parameters

Parameter	Type	Description	Required
params	IParams	The authentication information.	true
ctx	object	The login context on the server side. This parameter is not required for the client.	false

IParams

Parameter	Type	Description	Required
accessId	string	The clientId of the WeChat mini program. It can be gotten through the API of the WeChat mini program.	true
ticket	string	The identity ticket. It can be gotten through the API of the WeChat mini program.	true
appId	string	The AppId of the WeChat mini program.	true
schema	string	The Schema of the WeChat mini program.	true

Response parameters

Parameter	Type	Description
access_token	string	The identity token of the user.
refresh_token	string	The refresh token of the user.

Sample request

const { result } = await API.getToken({ accessId, ticket }, ctx);

Get the parameters for establishing a connection over the MQTT protocol

getMqttConfig

Request parameters

Parameter	Type	Description	Required
params	IParams	The identity information.	true
ctx	object	The login context on the server side. This parameter is not required for the client.	false

IParams

Parameter	Type	Description	Required
accessToken	string	The identity token of the user. This parameter is required for the server side.	true
refreshToken	string	The refresh token of the user. This parameter is required for the server side.	true
schema	string	The Schema of the WeChat mini program.	true

Sample request

const { result } = await API.getMqttConfig({ accessToken, refreshToken, schema }, ctx);

Sample response

// The result serves as the mqttConfigResponse parameter of requests over the WebRTC protocol.
{
  password: 'xxxxxxxxxx',
  topics: [],
  url: 'wss://xxxxxxx',
  username: 'xxxxxx'
}

Get the parameters for establishing a connection over the WebRTC protocol

getWebRTCConfig

Request parameters

Parameter	Type	Description	Required
params	IParams	The identity information and the device ID.	true
ctx	object	The login context on the server side. This parameter is not required for the client.	false

IParams

Parameter	Type	Description	Required
deviceId	string	The device ID.	true
accessToken	string	The identity token of the user. This parameter is required for the server side.	true
refreshToken	string	The refresh token of the user. This parameter is required for the server side.	true
schema	string	The Schema of the WeChat mini program.	true

Sample request

const { result } = await API.getWebRTCConfig({ deviceId, accessToken, refreshToken, schema }, ctx);

Sample response

// The result serves as the p2pConfigResponse parameter of requests over the WebRTC protocol.
{
  auth: 'xxxxxxxxxx',
  audio_attributes: [],
  gateway_id: '',
  id: 'xxxxxx',
  moto_id: 'xxxxx',
  p2p_config: [],
  skill: ''
}

Live video (the WebRTC protocol)

Support automatic audio and video streaming and two-way talk. However, automatic streaming is limited to H.264 and H.265 devices, excluding MJPEG devices.

WebRtcVideo

import { WebRtcVideo } from '@ray-js/video-streams-sdk';

Initialize the live video

Request parameters

Parameter	Type	Description	Required
video	HTMLVideoElement	The instance of the <video> label.	true
url	string	The media URL of the live video of the lock.	true
mqttConfigResponse	object	The parameters for establishing a connection over the MQTT protocol. It is the return value of the API.getMqttConfig method.	true
p2pConfigResponse	object	The parameters for establishing a connection over the WebRTC protocol. It is the return value of the API.getWebRTCConfig method.	true
codecType	string	The video protocol type of the device. Valid values: 4: H.265 device. 2: H.264 device. This default value is 2.	false
isMuted	boolean	Specifies whether to mute the live video. The default value is false.	false
angle	number	The angle of rotation of the device. The default value is 0.	false
isOnlySupportSpeak	boolean	Specifies whether to enable only two-way talk. The default value is false. This feature is suitable for MJPEG devices because although video streaming over the WebRTC protocol is not supported for them, two-way talk is supported.	false
isCarryAudio	boolean	Specifies whether to allow the audio to be played. The default value is true.	false
videoWidth	number	The width preset for the video player. The default value is the width of the screen.	false
videoHeight	number	The height preset for the video player. The default value is the height of the screen.	false

Sample request

// Get a video.
const video = document.getElementById('video');

const WebRtcVideoRef = new WebRtcVideo({
  url: 'https://xxxxxxxx',
  video,
  codecType: '2',
  mqttConfigResponse,
  mqttConfigResponse,
  angle: 90,
});

Listen to events

onStatusChange

Request parameters

Parameter	Type	Description	Required
callback	(message, data) => void	The callback event to listen to.	true

IMessage

• videoStyle: The width and height of the adapted video player.
• speed: The rate of the video. Unit: KB/s.
• connectStatus: The status of the API connection. Valid values:
  • unLink: The API is not connected.
  • linked: The API is connected.
  • linkFail: The connection fails.
  • busy: The server is busy.
  • fail: The video fails to be loaded or played.
• speakStatus: The status of two-way talk. Valid values:
  • start: Two-way talk is started.
  • success: Two-way talk succeeds.
  • inUse: Two-way talk is in use.
  • stop: Two-way talk is stopped.
  • nonSupport: The mobile phone does not support two-way talk.
• streamStatus: The status of video streaming. Valid values:
  • success: Video streaming succeeds.
  • nonSupport: The mobile phone does not support WebRTC.

Sample request

const handleStatusChange = (message, data) => {
  // message: the event name.
  // data: the value.
};

WebRtcVideoRef.onStatusChange(handleStatusChange);

Start two-way talk

startSpeak

Request parameters

Parameter	Type	Description	Required
type	string	The voice effect. Valid values: original: Original voice. female: Female voice. robot: Robotic voice. man: Male voice.	true

WebRtcVideoRef.startSpeak('female');

Stop two-way talk

stopSpeak

WebRtcVideoRef.stopSpeak();

Mute or unmute the live video

setVideoVoice

Request parameters

Parameter	Type	Description	Required
isMuted	boolean	Specifies whether to mute the live video.	true

WebRtcVideoRef.setVideoVoice(true);

Reconnect to the live video

reconnectVideo

WebRtcVideoRef.reconnectVideo();

Stop the live video

stopVideo

WebRtcVideoRef.stopVideo();

Live video (the WebSocket protocol)

WebSocketVideo

Support automatic audio and video streaming and two-way talk. However, two-way talk may be lossy. This is often used to stream audio and videos on MJPEG devices, in conjunction with two-way talk over the WebRTC protocol.

import { WebSocketVideo } from '@ray-js/video-streams-sdk';

Initialize the live video

Request parameters

Parameter	Type	Description	Required
video	HTMLCanvasElement	The instance of the <canvas> label.	true
url	string	The media URL of the live video of the lock. It is gotten through the getLockMediaStream method of the WeChat mini program.	true
expiration	string	The effective time. It is gotten through the getLockMediaStream method of the WeChat mini program.	true
signInfo	string	The signup information. It is gotten through the getLockMediaStream method of the WeChat mini program.	true
isMuted	boolean	Specifies whether to mute the live video. The default value is true.	false
angle	number	The angle of rotation of the device. The default value is 0.	false
isCarryAudio	boolean	Specifies whether to allow the audio to be played. The default value is true.	false
videoWidth	number	The width preset for the video player. The default value is the width of the screen.	false
videoHeight	number	The height preset for the video player. The default value is the height of the screen.	false

Sample request

// Get a canvas.
const canvas = document.getElementById('canvas');

const WebSocketVideoRef = new WebSocketVideo({
  url: 'https://xxxxxxxx',
  canvas,
  expiration: '1726020430',
  signInfo: 'xxxxxxxxx',
  angle: 90,
});

Listen to events

onStatusChange

Request parameters

Parameter	Type	Description	Required
callback	(message, data) => void	The callback event to listen to.	true

IMessage

• speed: The rate of the video. Unit: KB/s.
• connectStatus: The status of the API connection. Valid values:
  • unLink: The API is not connected.
  • linked: The API is connected.
  • linkFail: The connection fails.
• speakStatus: The status of two-way talk. Valid values:
  • start: Two-way talk is started.
  • success: Two-way talk succeeds.
  • inUse: Two-way talk is in use.
  • stop: Two-way talk is stopped.
  • nonSupport: The mobile phone does not support two-way talk.
• streamStatus: The status of video streaming. Valid values:
  • success: Video streaming succeeds.
  • fail: Video streaming fails.

Sample request

const handleStatusChange = (message, data) => {
  // message: the event name.
  // data: the value.
};

WebSocketVideoRef.onStatusChange(handleStatusChange);

Start two-way talk

startSpeak

Request parameters

Parameter	Type	Description	Required
type	string	The voice effect. Valid values: original: Original voice. female: Female voice. robot: Robotic voice. man: Male voice.	true

WebSocketVideoRef.startSpeak('female');

Stop two-way talk

stopSpeak

WebSocketVideoRef.stopSpeak();

Mute or unmute the live video

setVideoVoice

Request parameters

Parameter	Type	Description	Required
isMuted	boolean	Specifies whether to mute the live video.	true

WebSocketVideoRef.setVideoVoice(true);

Reconnect to the live video

reconnectVideo

WebSocketVideoRef.reconnectVideo();

Stop the live video

stopVideo

WebSocketVideoRef.stopVideo();

Recorded video

RecordVideo

import { RecordVideo } from '@ray-js/video-streams-sdk';

Initialize the recorded video

Request parameters

Parameter	Type	Description	Required
canvas	HTMLCanvasElement	The instance of the <canvas> label.	true
url	string	The media URL of the recorded video of the lock.	true
deviceWidth	number	The width of the lock device.	true
deviceHeight	number	The height of the lock device.	true
mediaKey	string	The media key of the recorded video of the lock.	true
angle	number	The angle of rotation of the device. The default value is 0.	false
canvasWidth	number	The width preset for the canvas. The default value is the width of the screen.	false
canvasHeight	number	The height preset for the canvas. The default value is the height of the screen.	false

Sample request

// Get a canvas.
const canvas = document.getElementById('canvas');

const RecordVideoRef = new RecordVideo({
  url: 'https://xxxxxxxx',
  canvas,
  deviceWidth: 1280,
  deviceHeight: 720,
  mediaKey: '9ged77k5sntq4fct',
  angle: 90,
});

Listen to events

onStatusChange

Request parameters

Parameter	Type	Description	Required
callback	(message, data) => void	The callback event to listen to.	true

IMessage

• canvasSize: The width and height of the adapted canvas.
• status: The playback status of the recorded video. Valid values:
  • ready: Resources are loaded.
  • playing: The video is being played.
  • pause: The video is paused.
  • end: The video is ended.
  • fail: The video fails to be loaded or played.
• durations: The total duration of the recorded video. Unit: second.
• currentDuration: The current progress of the recorded video. Unit: second.

Sample request

const handleStatusChange = (message, data) => {
  // message: the event name.
  // data: the value.
};

RecordVideoRef.onStatusChange(handleStatusChange);

Play the recorded video

startVideo

Sample request

RecordVideoRef.startVideo();

Pause the recorded video

pauseVideo

Sample request

RecordVideoRef.pauseVideo();

Stop the recorded video

stopVideo

Sample request

RecordVideoRef.stopVideo();