Audio and Video Features

Last Updated on : 2023-03-29 02:40:42download

The IPC SDK can provide a bunch of audio and video capabilities in addition to live video streaming and playing back footage on an SD card. For example, record videos on a mobile phone, capture video screenshots, talk with IPCs in real time, and switch between definition modes.

Local recording

During live video streaming or SD card-stored video playback, the ongoing videos can be recorded on a mobile phone.

  • During video recording, do not switch between video definition modes or modify the audio channel switch and live video talk settings. Otherwise, the recording might fail. We recommend that you disable the settings of the video definition modes, audio channel switch, and live video talk during video recording.
  • To save video footage to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions, but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.
  • The video length might be less than the recording duration in certain conditions. For example, the video keeps freezing, or video frames are saved starting from the key frame next to the start point. The recording duration is provided for reference only.

Enable video recording

Video recording requires the write permissions of the SD card.

API description

int startRecordLocalMp4(String folderPath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter Description
folderPath The path in which videos are stored.
context The context.
callBack The callback.

Example

private ITuyaSmartCameraP2P mCameraP2P;

if (Constants.hasStoragePermission()) {
    String picPath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(picPath);
    if (!file.exists()) {
        file.mkdirs();
    }
    mCameraP2P.startRecordLocalMp4(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isRecording = true;
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_BEGIN);
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_FAIL);
        }
    });
    recordStatue(true);
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE, Constants.EXTERNAL_STORAGE_REQ_CODE, "open_storage");
}

Stop video recording

API description

int stopRecordLocalMp4(OperationDelegateCallBack callBack);

Parameters

Parameter Description
callBack The callback.

Example

mCameraP2P.stopRecordLocalMp4(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        // The success callback.
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        // Failed
    }
});

Video screenshots

Captures screenshots of live video images and stores them on the SD card of a mobile phone.

To save screenshots to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.

API description

int snapshot(String absoluteFilePath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter Description
absoluteFilePath The path in which the screenshots are stored.
context The context.
callBack The callback.

Example

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
    String path = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(path);
    if (!file.exists()) {
        file.mkdirs();
    }
    picPath = path;
}
mCameraP2P.snapshot(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
     // The file path is returned by `data`.
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_SUCCESS, data));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_FAIL));
    }
});

Audio settings

During live video streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

Set the audio channel

API description

Enables or disables the audio channel.

void setMute(int mute, OperationDelegateCallBack callBack);

Parameters

Parameter Description
mute Specifies whether to mute videos. Valid values:
  • 1: mutes videos.
  • 0: unmutes videos.

Example

mCameraP2P.setMute(1, new OperationDelegateCallBack() {
  @Override
  public void onSuccess(int sessionId, int requestId, String data) {
    // The operation result is returned by `data`.
    previewMute = Integer.valueOf(data);
    mHandler.sendMessage(MessageUtil.getMessage(MSG_MUTE, ARG1_OPERATE_SUCCESS));
  }

  @Override
  public void onFailure(int sessionId, int requestId, int errCode) {
    mHandler.sendMessage(MessageUtil.getMessage(MSG_MUTE, ARG1_OPERATE_FAIL));
  }
});

Switch between speaker and earpiece

API description

Switches between the speaker and earpiece modes. This API method is not supported by P2P 1.0 devices.

void setLoudSpeakerStatus(boolean enable);

Parameters

Parameter Description
enable
  • true: switches to the speaker mode.
  • false: switches to the earpiece mode.

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Determine support for video talk

Indicates whether the device is equipped with a speaker. If so, the device supports video talk. If the device is equipped with both a speaker and a pickup, video streams are audible, so two-way talk is supported.

Start video talk

Transmits audio data from the mobile phone to the IPC.

Example

if (Constants.hasRecordPermission()) {
    mCameraP2P.startAudioTalk(new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isSpeaking = true;
            mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_BEGIN, ARG1_OPERATE_SUCCESS));
            ToastUtil.shortToast(CameraPanelActivity.this, "start talk success");
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            isSpeaking = false;
            mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_BEGIN, ARG1_OPERATE_FAIL));
            ToastUtil.shortToast(CameraPanelActivity.this, "operation fail");

        }
    });
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.RECORD_AUDIO, Constants.EXTERNAL_AUDIO_REQ_CODE, "open_recording");
}

Stop video talk

Stops transmitting audio data from the mobile phone to the IPC.

Example

mCameraP2P.stopAudioTalk(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        isSpeaking = false;
        mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_OVER, ARG1_OPERATE_SUCCESS));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        isSpeaking = false;
        mHandler.sendMessage(MessageUtil.getMessage(MSG_TALK_BACK_OVER, ARG1_OPERATE_FAIL));

    }
});

The video talk and video recording features are mutually exclusive. The former can only be enabled during video preview.

Two-way talk

During live video streaming, the audio channel can be enabled. In this case, the audible sound is the human voice and ambient sound collected by the IPC in real time. Then, two-way talk can be implemented through the audio channel between the app and the IPC.

Certain IPCs might not have speakers or pickups. Such cameras do not support two-way talk.

One-way talk

The control of one-way talk is subject to your implementation. After the one-way talk is enabled, the video is muted. After the one-way talk is disabled, the video is unmuted.

Definition modes

Users can switch between definition modes during live video streaming. Currently, only high definition (HD) and standard definition (SD) modes are supported. A few IPCs only support one of the modes.

This feature can be implemented only during live video streaming. Only one video definition mode is supported for storing footage on the SD card.

Parameters

Parameter Description
clarity The video definition mode. Valid values:
  • 2: SD
  • 4: HD

Get definition mode

Returns the definition mode of the videos sent from the IPC.

Example

mCameraP2P.getVideoClarity(new OperationDelegateCallBack() {
​    @Overridepublic void onSuccess(int sessionId, int requestId, String data) {

​    }

​    @Overridepublic void onFailure(int sessionId, int requestId, int errCode) {

​    }

});

This function is called after the preview images are generated.

Set video definition

Sets the definition mode of the videos sent from the IPC.

Example

mCameraP2P.setVideoClarity(2, new OperationDelegateCallBack() {

​    @Overridepublic void onSuccess(int sessionId, int requestId, String data) {
​        videoClarity = Integer.valueOf(data);
​        mHandler.sendMessage(MessageUtil.getMessage(MSG_GET_CLARITY, ARG1_OPERATE_SUCCESS));
​    }

​    @Overridepublic void onFailure(int sessionId, int requestId, int errCode) {
​        mHandler.sendMessage(MessageUtil.getMessage(MSG_GET_CLARITY, ARG1_OPERATE_FAIL));
​    }

});

Additional features

Get bitrate

API description

double getVideoBitRateKbps();

Example

double rate = mCameraP2P.getVideoBitRateKbps();

Set whether the renderer can zoom in by double-tapping

API description

Enables users to zoom in video views by double-tapping by default.

public void setCameraViewDoubleClickEnable(boolean enable);

Example

TuyaCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setCameraViewDoubleClickEnable(false);

Raw stream data

The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format.

API description

To enable the callback for video frames, you must register a listener with ITuyaSmartCameraP2P in AbsP2pCameraListener. You only need to rewrite the callback as preferred.

void registerP2PCameraListener(AbsP2pCameraListener listener);

Callback for video YUV data

API description

public void onReceiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height, int nFrameRate, int nIsKeyFrame, long timestamp, long nProgress, long nDuration, Object camera)

Parameters

Parameter Description
Y The luma (Y’) information of video streams.
u The chroma (U) channel information of video streams.
v The chroma (V) channel information of video streams.
width The width of video images.
height The height of video images.
nFrameRate The frame rate of video streams.
nIsKeyFrame Indicates whether a keyframe or an I-frame is used.
timestamp The timestamp.
nProgress The time progress of the video pushed by the Message Center module.
nDuration The duration of the video pushed by the Message Center module.

Callback for P2P connection status

API description

public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus)

Parameters

Parameter Description
sessionId The ID of the P2P session.
sessionStatus The status of the P2P connection. Valid values:
  • 0: connected
  • -3: timeout
  • -12: connection closed by the IPC
  • -13: connection closed due to timeout

Example

private ITuyaSmartCameraP2P mCameraP2P;

@Override
protected void onResume() {
    super.onResume();
    if (null != mCameraP2P) {
        // Registers a P2P listener.
         mCameraP2P.registerP2PCameraListener(p2pCameraListener);

    }
}

@Override
protected void onPause() {
    super.onPause();
    if (null != mCameraP2P) {
        // Unregisters a P2P listener.
        mCameraP2P.removeOnP2PCameraListener();
    }
}

private AbsP2pCameraListener p2pCameraListener = new AbsP2pCameraListener() {
    @Override
            public void onSessionStatusChanged(Object o, int i, int i1) {
        super.onSessionStatusChanged(o, i, i1);
        // The callback for connection status changes.
    }
};

Do not call other API methods in the callback thread of onSessionStatusChanged. Otherwise, a deadlock might occur.

Audio data

Audio data collected by the app can be further processed, such as voice changing. To implement this feature, the IPC SDK provides the callback that returns the audio data collected by the app. The collected audio data has been processed with echo cancellation. To enable the callback for audio data, you must register a listener with ITuyaSmartCameraP2P.

API description

Registers the listener.

void registerSpeakerEchoProcessor(ISpeakerEchoProcessor processor);

API description

Unregisters the listener.

void unregisterSpeakerEchoProcessor();

API description

Transmits the processed audio data to the IPC for playback.

void sendAudioTalkData(byte[] outbuf, int length);

Parameters

Parameter Description
outbuf The byte array of audio data.
length Data length

Example

mCameraP2P.registerSpeakerEchoProcessor(new ISpeakerEchoProcessor() {
    @Override
    public void receiveSpeakerEchoData(ByteBuffer pcm, int sampleRate) {
        // The pulse-code modulation (PCM) data that indicates the sampling rate.
    }
});

Intelligent video analytics

The IVA feature can be enabled to support intelligent detection. If the IPC detects an object in motion during live video streaming, the object will be automatically framed in white on the view.

To achieve this purpose, the IVA features must be enabled for the IPC first. It will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 to enable this feature. For more information about device control API methods, see Device Control.

After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live video streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

API description

Enables or disables the IVA feature.

void setEnableIVA(boolean enableIVA);

Parameters

Parameter Description
enable Specifies whether to enable IVA.

Device capability

Get device capabilities

API description
ICameraConfigInfo is the device capability class that encapsulates multiple API methods to implement device features.

ICameraConfigInfo getCameraConfig(String devId);

Example

ITuyaIPCCore cameraInstance = TuyaIPCSdk.getCameraInstance();
if (cameraInstance != null) {
    ICameraConfigInfo cameraConfig = cameraInstance.getCameraConfig(devId);
    if (cameraConfig != null) {
        int videoNum = cameraConfig.getVideoNum();
    }
}

Query supported number of streams

API description

Indicates the number of streams supported by the device. If the value is 1, the device only supports HD or SD. Device streams are obtained with the default definition mode.

int getVideoNum();

Query default definition

API description

Indicates the default definition of the device. If the device only supports one stream channel, it also supports only one definition mode.

int getDefaultDefinition();

Query support for speaker

API description

Indicates whether the device is equipped with a speaker. If so, the device supports video talk.

boolean isSupportSpeaker();

Query support for pickup

API description

Indicates whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportPickup();

Query default video talk mode

API description

int getDefaultTalkBackMode();

Query support for switching between video talk modes

API description

Indicates whether users can switch between video talk modes. If so, the device supports both one-way and two-way video talk.

boolean isSupportChangeTalkBackMode();

Query raw P2P configurations

API description

Returns raw data of P2P configurations.

String getRawDataJsonStr();

Query supported play speed

API description

List<Integer> getSupportPlaySpeedList();

Return value

Return values Description
0 0.5x
1 1x
3 2x