Audio and Video Features

The IPC SDK can provide a bunch of audio and video capabilities in addition to live video streaming and playing back footage on an SD card. For example, record videos on a mobile phone, capture video screenshots, talk with IPCs in real time, and switch between definition modes.

Local recording

During live video streaming or SD card-stored video playback, the ongoing videos can be recorded on a mobile phone.

Enable video recording

API description

int startRecordLocalMp4(String folderPath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
folderPath	The path in which videos are stored.
context	The context.
callBack	The callback.

Example

private ITuyaSmartCameraP2P mCameraP2P;

if (Constants.hasStoragePermission()) {
    String picPath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(picPath);
    if (!file.exists()) {
        file.mkdirs();
    }
    mCameraP2P.startRecordLocalMp4(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isRecording = true;
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_BEGIN);
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_FAIL);
        }
    });
    recordStatue(true);
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE, Constants.EXTERNAL_STORAGE_REQ_CODE, "open_storage");
}

Stop video recording

API description

int stopRecordLocalMp4(OperationDelegateCallBack callBack);

Parameters

Parameter	Description
callBack	The callback.

Example

mCameraP2P.stopRecordLocalMp4(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        // The success callback.
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        // Failed
    }
});

Video screenshots

Captures screenshots of live video images and stores them on the SD card of a mobile phone.

API description

int snapshot(String absoluteFilePath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
absoluteFilePath	The path in which the screenshots are stored.
context	The context.
callBack	The callback.

Example

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
    String path = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(path);
    if (!file.exists()) {
        file.mkdirs();
    }
    picPath = path;
}
mCameraP2P.snapshot(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
     // The file path is returned by `data`.
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_SUCCESS, data));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_FAIL));
    }
});

Audio settings

During live video streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

Set the audio channel

API description

Enables or disables the audio channel.

void setMute(int mute, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
mute	Specifies whether to mute videos. Valid values: 1: mutes videos. 0: unmutes videos.

Example

mCameraP2P.setMute(1, new OperationDelegateCallBack() {
  @Override
  public void onSuccess(int sessionId, int requestId, String data) {
    // The operation result is returned by `data`.
    previewMute = Integer.valueOf(data);
    mHandler.sendMessage(MessageUtil.getMessage(MSG_MUTE, ARG1_OPERATE_SUCCESS));
  }

  @Override
  public void onFailure(int sessionId, int requestId, int errCode) {
    mHandler.sendMessage(MessageUtil.getMessage(MSG_MUTE, ARG1_OPERATE_FAIL));
  }
});

Switch between speaker and earpiece

API description

Switches between the speaker and earpiece modes. This API method is not supported by P2P 1.0 devices.

void setLoudSpeakerStatus(boolean enable);

Parameters

Parameter	Description
enable	true: switches to the speaker mode. false: switches to the earpiece mode.

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Determine support for video talk

Indicates whether the device is equipped with a speaker. If so, the device supports video talk. If the device is equipped with both a speaker and a pickup, video streams are audible, so two-way talk is supported.

Start video talk

Transmits audio data from the mobile phone to the IPC.

Example

if (Constants.hasRecordPermission()) {
    mCameraP2P.startAudioTalk(new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isSpeaking = true;
            mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_BEGIN, ARG1_OPERATE_SUCCESS));
            ToastUtil.shortToast(CameraPanelActivity.this, "start talk success");
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            isSpeaking = false;
            mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_BEGIN, ARG1_OPERATE_FAIL));
            ToastUtil.shortToast(CameraPanelActivity.this, "operation fail");

        }
    });
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.RECORD_AUDIO, Constants.EXTERNAL_AUDIO_REQ_CODE, "open_recording");
}

Stop video talk

Stops transmitting audio data from the mobile phone to the IPC.

Example

mCameraP2P.stopAudioTalk(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        isSpeaking = false;
        mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_OVER, ARG1_OPERATE_SUCCESS));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        isSpeaking = false;
        mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_OVER, ARG1_OPERATE_FAIL));

    }
});

Two-way talk

During live video streaming, the audio channel can be enabled. In this case, the audible sound is the human voice and ambient sound collected by the IPC in real time. Then, two-way talk can be implemented through the audio channel between the app and the IPC.

One-way talk

The control of one-way talk is subject to your implementation. After the one-way talk is enabled, the video is muted. After the one-way talk is disabled, the video is unmuted.

Definition modes

Users can switch between definition modes during live video streaming. Currently, only high definition (HD) and standard definition (SD) modes are supported. A few IPCs only support one of the modes.

Parameters

Parameter	Description
clarity	The video definition mode. Valid values: 2: SD 4: HD

Get definition mode

Returns the definition mode of the videos sent from the IPC.

Example

mCameraP2P.getVideoClarity(new OperationDelegateCallBack() {
​    @Override
​    public void onSuccess(int sessionId, int requestId, String data) {

​    }

​    @Override
​    public void onFailure(int sessionId, int requestId, int errCode) {

​    }

});

Set video definition

Sets the definition mode of the videos sent from the IPC.

Example

mCameraP2P.setVideoClarity(2, new OperationDelegateCallBack() {

​    @Override

​    public void onSuccess(int sessionId, int requestId, String data) {
​        videoClarity = Integer.valueOf(data);
​        mHandler.sendMessage(MessageUtil.getMessage(MSG_GET_CLARITY, ARG1_OPERATE_SUCCESS));
​    }

​    @Override

​    public void onFailure(int sessionId, int requestId, int errCode) {
​        mHandler.sendMessage(MessageUtil.getMessage(MSG_GET_CLARITY, ARG1_OPERATE_FAIL));
​    }

});

Additional features

Get bitrate

API description

double getVideoBitRateKbps();

Example

double rate = mCameraP2P.getVideoBitRateKbps();

Set whether the renderer can zoom in by double-tapping

API description

Enables users to zoom in video views by double-tapping by default.

public void setCameraViewDoubleClickEnable(boolean enable);

Example

TuyaCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setCameraViewDoubleClickEnable(false);

Raw stream data

The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format.

API description

To enable the callback for video frames, you must register a listener with ITuyaSmartCameraP2P in AbsP2pCameraListener. You only need to rewrite the callback as preferred.

void registerP2PCameraListener(AbsP2pCameraListener listener);

Callback for video YUV data

API description

public void onReceiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height, int nFrameRate, int nIsKeyFrame, long timestamp, long nProgress, long nDuration, Object camera)

Parameters

Parameter	Description
Y	The luma (Y’) information of video streams.
u	The chroma (U) channel information of video streams.
v	The chroma (V) channel information of video streams.
width	The width of video images.
height	The height of video images.
nFrameRate	The frame rate of video streams.
nIsKeyFrame	Indicates whether a keyframe or an I-frame is used.
timestamp	The timestamp.
nProgress	The time progress of the video pushed by the Message Center module.
nDuration	The duration of the video pushed by the Message Center module.

Callback for P2P connection status

API description

public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus)

Parameters

Parameter	Description
sessionId	The ID of the P2P session.
sessionStatus	The status of the P2P connection. Valid values: 0: connected -3: timeout -12: connection closed by the IPC -13: connection closed due to timeout

Example

private ITuyaSmartCameraP2P mCameraP2P;

@Override
protected void onResume() {
    super.onResume();
    if (null != mCameraP2P) {
        // Registers a P2P listener.
         mCameraP2P.registerP2PCameraListener(p2pCameraListener);

    }
}

@Override
protected void onPause() {
    super.onPause();
    if (null != mCameraP2P) {
        // Unregisters a P2P listener.
        mCameraP2P.removeOnP2PCameraListener();
    }
}

private AbsP2pCameraListener p2pCameraListener = new AbsP2pCameraListener() {
    @Override
            public void onSessionStatusChanged(Object o, int i, int i1) {
        super.onSessionStatusChanged(o, i, i1);
        // The callback for connection status changes.
    }
};

Audio data

Audio data collected by the app can be further processed, such as voice changing. To implement this feature, the IPC SDK provides the callback that returns the audio data collected by the app. The collected audio data has been processed with echo cancellation. To enable the callback for audio data, you must register a listener with ITuyaSmartCameraP2P.

API description

Registers the listener.

void registerSpeakerEchoProcessor(ISpeakerEchoProcessor processor);

API description

Unregisters the listener.

void unregisterSpeakerEchoProcessor();

API description

Transmits the processed audio data to the IPC for playback.

void sendAudioTalkData(byte[] outbuf, int length);

Parameters

Parameter	Description
outbuf	The byte array of audio data.
length	Data length

Example

mCameraP2P.registerSpeakerEchoProcessor(new ISpeakerEchoProcessor() {
    @Override
    public void receiveSpeakerEchoData(ByteBuffer pcm, int sampleRate) {
        // The pulse-code modulation (PCM) data that indicates the sampling rate.
    }
});

Intelligent video analytics

The IVA feature can be enabled to support intelligent detection. If the IPC detects an object in motion during live video streaming, the object will be automatically framed in white on the view.

To achieve this purpose, the IVA features must be enabled for the IPC first. It will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 to enable this feature. For more information about device control API methods, see Device Control.

After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live video streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

API description

Enables or disables the IVA feature.

void setEnableIVA(boolean enableIVA);

Parameters

Parameter	Description
enable	Specifies whether to enable IVA.

Device capability

Get device capabilities

API description
ICameraConfigInfo is the device capability class that encapsulates multiple API methods to implement device features.

ICameraConfigInfo getCameraConfig(String devId);

Example

ITuyaIPCCore cameraInstance = TuyaIPCSdk.getCameraInstance();
if (cameraInstance != null) {
    ICameraConfigInfo cameraConfig = cameraInstance.getCameraConfig(devId);
    if (cameraConfig != null) {
        int videoNum = cameraConfig.getVideoNum();
    }
}

Query supported number of streams

API description

Indicates the number of streams supported by the device. If the value is 1, the device only supports HD or SD. Device streams are obtained with the default definition mode.

int getVideoNum();

Query default definition

API description

Indicates the default definition of the device. If the device only supports one stream channel, it also supports only one definition mode.

int getDefaultDefinition();

Query support for speaker

API description

Indicates whether the device is equipped with a speaker. If so, the device supports video talk.

boolean isSupportSpeaker();

Query support for pickup

API description

Indicates whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportPickup();

Query default video talk mode

API description

int getDefaultTalkBackMode();

Query support for switching between video talk modes

API description

Indicates whether users can switch between video talk modes. If so, the device supports both one-way and two-way video talk.

boolean isSupportChangeTalkBackMode();

Query raw P2P configurations

API description

Returns raw data of P2P configurations.

String getRawDataJsonStr();

Query supported play speed

API description

List<Integer> getSupportPlaySpeedList();

Return value

Return values	Description
0	0.5x
1	1x
3	2x