Audio and Video Features

Last Updated on : 2024-03-04 08:53:07

The IPC SDK can provide a bunch of audio and video capabilities in addition to live video streaming and playing back footage on an SD card. For example, record videos on a mobile phone, capture video screenshots, talk with IPCs in real time, and switch between definition modes.

Local recording

During live video streaming or SD card-stored video playback, the ongoing videos can be recorded on a mobile phone.

During video recording, do not switch between video definition modes or modify the audio channel switch and live video talk settings. Otherwise, recording might fail. We recommend that you disable the settings of the video definition modes, audio channel switch, and live video talk during video recording.
To save video footage to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions, but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.
The video length might be less than the recording duration in certain conditions. For example, the video keeps freezing, or video frames are saved starting from the key frame next to the start point. The recording duration is provided for reference only.

Enable video recording

Video recording requires the write permissions of the SD card.

API description

int startRecordLocalMp4(String folderPath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
folderPath	The path in which videos are stored.
context	The context.
callBack	The callback.

Example

private IThingSmartCameraP2P mCameraP2P;

if (Constants.hasStoragePermission()) {
    String picPath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(picPath);
    if (!file.exists()) {
        file.mkdirs();
    }
    mCameraP2P.startRecordLocalMp4(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isRecording = true;
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_BEGIN);
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_FAIL);
        }
    });
    recordStatue(true);
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE, Constants.EXTERNAL_STORAGE_REQ_CODE, "open_storage");
}

Stop video recording

API description

int stopRecordLocalMp4(OperationDelegateCallBack callBack);

Parameters

Parameter	Description
callBack	The callback.

Example

mCameraP2P.stopRecordLocalMp4(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        // The success callback.
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        // The failure callback.
    }
});

Capture video screenshot

Captures screenshots of live video images and stores them on the SD card of a mobile phone.

To save screenshots to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.

API description

int snapshot(String absoluteFilePath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
absoluteFilePath	The path in which the screenshots are stored.
context	The context.
callBack	The callback.

Example

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
    String path = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(path);
    if (!file.exists()) {
        file.mkdirs();
    }
    picPath = path;
}
mCameraP2P.snapshot(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
     // The file path is returned by `data`.
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_SUCCESS, data));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_FAIL));
    }
});

Audio settings

During live video streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

Set the audio channel

API description

Enables or disables the audio channel.

void setMute(int mute, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
mute	Specifies whether to mute videos. Valid values: `1`: mutes videos. `0`: unmutes videos.

Example

mCameraP2P.setMute(1, new OperationDelegateCallBack() {
  @Override
  public void onSuccess(int sessionId, int requestId, String data) {
    // The operation result is returned by `data`.
    previewMute = Integer.valueOf(data);
  }

  @Override
  public void onFailure(int sessionId, int requestId, int errCode) {
  }
});

Switch between speaker and earpiece

API description

Switches between the speaker and earpiece modes. This API method is not supported by P2P 1.0 devices.

void setLoudSpeakerStatus(boolean enable);

Parameters

Parameter	Description
enable	`true`: switches to the speaker mode. `false`: switches to the earpiece mode.

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Determine support for video talk

Indicates whether the device is equipped with a speaker. If so, the device supports one-way talk.
If the device is equipped with both a speaker and a pickup, video streams are audible, so two-way talk is supported.

Start video talk

Transmits audio data from the mobile phone to the IPC.

Example

if (Constants.hasRecordPermission()) {
    mCameraP2P.startAudioTalk(new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isSpeaking = true;
            ToastUtil.shortToast(CameraPanelActivity.this, "start talk success");
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            isSpeaking = false;
            ToastUtil.shortToast(CameraPanelActivity.this, "operation fail");

        }
    });
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.RECORD_AUDIO, Constants.EXTERNAL_AUDIO_REQ_CODE, "open_recording");
}

Stop video talk

Stops transmitting audio data from the mobile phone to the IPC.

Example

mCameraP2P.stopAudioTalk(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        isSpeaking = false;
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        isSpeaking = false;
    }
});

The video talk and video recording features are mutually exclusive. The former can only be enabled during video preview.

Two-way talk

During live video streaming, the audio channel can be enabled. In this case, the audible sound is the human voice and ambient sound collected by the IPC in real time. Then, enable the audio channel from the app to the IPC, so two-way talk can be implemented.

Certain IPCs might not have speakers or pickups. Such cameras do not support two-way talk.

One-way talk

The control of one-way talk is subject to your implementation.

After the one-way talk is enabled, the video is muted.
After the one-way talk is disabled, the video is unmuted.

Definition modes

Users can switch between definition modes during live video streaming. Currently, only high definition (HD) and standard definition (SD) modes are supported. A few IPCs only support one of the modes.

This feature can be implemented only during live video streaming. Only one video definition mode is supported for storing footage on the SD card.

Parameters

Parameter	Description
clarity	The video definition mode. Valid values: `2`: SD `4`: HD

Get definition mode

Returns the definition mode of the videos sent from the IPC.

Example

mCameraP2P.getVideoClarity(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {

    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {

    }

});

This function is called after startPreview is called to generate the preview images.

Set video definition

Sets the definition mode of the videos sent from the IPC.

Example

mCameraP2P.setVideoClarity(2, new OperationDelegateCallBack() {

    @Override

    public void onSuccess(int sessionId, int requestId, String data) {
        videoClarity = Integer.valueOf(data);
    }

    @Override

    public void onFailure(int sessionId, int requestId, int errCode) {
    }

});

Raw stream data

The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format.

Register callback for returned video frames

API description

To enable the callback for video frames, you must register a listener with IThingSmartCameraP2P in AbsP2pCameraListener. You only need to rewrite the callback as preferred.

void registerP2PCameraListener(AbsP2pCameraListener listener);

Callback for video YUV data

API description

public void onReceiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height, int nFrameRate, int nIsKeyFrame, long timestamp, long nProgress, long nDuration, Object camera)

Parameters

Parameter	Description
Y	The luma (Y’) information of video streams.
u	The chroma (U) channel information of video streams.
v	The chroma (V) channel information of video streams.
width	The width of video images.
height	The height of video images.
nFrameRate	The frame rate of video streams.
nIsKeyFrame	Indicates whether a keyframe or an I-frame is used.
timestamp	The timestamp.
nProgress	The time progress of the video pushed by the Message Center module.
nDuration	The duration of the video pushed by the Message Center module.

Callback for P2P connection status

API description

public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus)

Parameters

Parameter	Description
sessionId	The ID of the P2P connection.
sessionStatus	The status of the P2P connection. Valid values: `0`: connected `-3`: timeout `-12`: connection closed by the IPC `-13`: connection closed due to timeout

Example

private IThingSmartCameraP2P mCameraP2P;

@Override
protected void onResume() {
    super.onResume();
    if (null != mCameraP2P) {
        // Registers a P2P listener.
         mCameraP2P.registerP2PCameraListener(p2pCameraListener);

    }
}

@Override
protected void onPause() {
    super.onPause();
    if (null != mCameraP2P) {
        // Unregisters a P2P listener.
        mCameraP2P.removeOnP2PCameraListener(p2pCameraListener);
    }
}

private AbsP2pCameraListener p2pCameraListener = new AbsP2pCameraListener() {
    @Override
            public void onSessionStatusChanged(Object o, int i, int i1) {
        super.onSessionStatusChanged(o, i, i1);
        // The callback for connection status changes.
    }
};

Do not call other API methods in the callback thread of onSessionStatusChanged. Otherwise, a deadlock might occur.

Audio data

Audio data collected by the app can be further processed, such as voice changing. To implement this feature, the IPC SDK provides the callback that returns the audio data collected by the app. The collected audio data has been processed with echo cancellation. To enable the callback for audio data, you must register a listener with IThingSmartCameraP2P.

API description

Registers the listener.

void registerSpeakerEchoProcessor(ISpeakerEchoProcessor processor);

API description

Unregisters the listener.

void unregisterSpeakerEchoProcessor();

API description

Transmits the processed audio data to the IPC for playback.

void sendAudioTalkData(byte[] outbuf, int length);

Parameters

Parameter	Description
outbuf	The byte array of audio data.
length	Data length

Example

mCameraP2P.registerSpeakerEchoProcessor(new ISpeakerEchoProcessor() {
    @Override
    public void receiveSpeakerEchoData(ByteBuffer pcm, int sampleRate) {
        // The pulse-code modulation (PCM) data that indicates the sampling rate.
    }
});

Intelligent video analytics

If the IPC detects an object in motion during live video streaming, the intelligent video analytics (IVA) feature allows the object to be framed automatically in white on the view.

To achieve this purpose, the IVA feature must be enabled for the IPC first. It will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 (ipc_object_outline) to enable this feature. For more information about device control API methods, see Device Control.

After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live video streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

Enable IVA

API description

Enables or disables the IVA feature. This API method is usually called after startPreview for video previewing or setVideoClarity for definition settings.

void setEnableIVA(boolean enableIVA);

Parameters

Parameter	Description
enable	Specifies whether to enable IVA.

Configure IVA properties

You can configure the properties of IVA to control its style, such as the frame color, brush width, and flash frequency.

API description

Defines the properties of IVA in the specified format of JSON strings based on the Supplemental Enhancement Information (SEI) reported by the device.

void setSmartRectFeatures(String rectFeatures);

Parameters

The parameter rectFeatures is a JSON string in the following format:

{
    "SmartRectFeature":[
        {
            "type":0,
            "index":0,
            "brushWidth":1,
            "flashFps":{
                "drawKeepFrames":2,
                "stopKeepFrames":2
            },
            "rgb":0xFF0000,
            "shape":0
        },
        {
            "type":0,
            "index":1,
            "brushWidth":2,
            "flashFps":{
                "drawKeepFrames":3,
                "stopKeepFrames":2
            },
            "rgb":0x00FF00,
            "shape":1
        }
    ]
}

Parameter	Type	Description
SmartRectFeature	Array	(Required) The identifier in the fixed format of arrays to represent multiple frame settings.
type	Int	The type of frame. Valid values: `0`: smart bounding box (default) `1`: Line crossing detection
index	Int	The index of the frame, corresponding to each ID in `od` of SEI.
shape	Int	The shape of the rectangular frame. Valid values: `0`: closed rectangle `1`: four corners only
rgb	Int	The color of the rectangular frame, represented by the red-green-blue (RGB) color model. Value range: `0x000000` to `0xFFFFFF`. Default value: `0xFC4747`.
brushWidth	Int	The brush stroke of the rectangular frame. Valid values: `0`: thin `1`: medium `2`: thick
flashFps	String	The flash frequency of the rectangular frame. Valid values: `drawKeepFrames`: the number of frames with drawing on. `stopKeepFrames`: the number of frames with drawing off.

Define SEI protocol

The protocol that governs communication with the device. The IPC SDK parses the data over SEI and implements IVA at the positions with the specified properties.

{
    "AGTX":{
        "time":6885,
        "chn":0,
        "key":1,
        "iva":{
            "od":[
                {
                    "obj":{
                        "id":0,
                        "type":1,
                        "twinkle":1,
                        "rect":[
                            0,0,
                            25,25,
                            50,50,
                            80, 80,
                            100,100
                        ],
                        "vel":[0,10],
                        "cat":"PEDESTRIAN"
                    }
                },
                {
                    "obj":{
                        "id":1,
                        "type":1,
                        "twinkle":1,
                        "rect":[
                            0,0,
                            100,100
                        ],
                        "vel":[0,10],
                        "cat":"PEDESTRIAN"
                    }
                }
            ]
        }
    }
}

The following table describes the parsed parameters in iva reported by the device.

Parameter	Description
id	The index of each frame.
type	The type of frame. Valid values: `0`: closed rectangle `1`: four corners only
twinkle	The flash setting. Valid values: `0`: The frame is displayed, but flash is disabled. `1`: The frame flashes at the frequency passed in.
rect	Each pair of numbers represents the coordinates of a point. All points are arranged in clockwise order. The SDK draws these points into a closed polygon. Two points are used to represent a rectangular box.

The coordinates of the rect points in the SEI protocol are typically passed from the app to the device in the format of the device control data point (DP). Coordinate anchor data follows these rules:

An even-numbered position represents the numerator of a percentage value on the horizontal (x) axis.
An odd-numbered position represents the numerator of a percentage value on the vertical (y) axis.

The maximum value is 100, and even and odd positions appear in pairs.

Video rendering

After video decoding, rendered images can be further processed by settings. For example, the following features are supported: stretching or scaling, horizontal or vertical mirroring, and rotation by 90, 180, or 270 degrees.

API description

void setDeviceFeatures(String renderFeatures);

Parameters

The parameter renderFeatures is a JSON string in the following format:

{
    "DecPostProcess":{
        "video":[
            {
                "restype":"4",
                "oldres":"944*1080",
                "newres":"1920*1080"
            },
            {
                "restype":"2",
                "oldres":"944*1080",
                "newres":"1920*1080"
            }
        ],
        "mirror":0,
        "rotation":2
    }
}

Parameter	Description
DecPostProcess	(Required) The identifier in the fixed format.
video	The array of video resolution settings. The following settings are supported: `restype`: the type of video resolution. `oldres`: the size of the original video reported by the device. `newres`: the expected size of the video after stretching or scaling.
mirror	The mirroring setting. Valid values: `0`: unchanged. `1`: horizontal mirroring. `2`: vertical mirroring.
rotation	The angle of rotation. Valid values: `0`: no rotation. `1`: rotate 90 degrees. `2`: rotate 180 degrees. `3`: rotate 270 degrees.

Additional features

Get bitrate

API description

double getVideoBitRateKbps();

Example

double rate = mCameraP2P.getVideoBitRateKbps();

Set whether the renderer can zoom in by double-tapping

API description

Enables users to zoom in video views by double-tapping by default. You can also disable this feature.

public void setCameraViewDoubleClickEnable(boolean enable);

Example

ThingCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setCameraViewDoubleClickEnable(false);

Device capabilities

Get device capabilities

API description

ICameraConfigInfo is the device capability class that encapsulates multiple API methods to implement device features.

ICameraConfigInfo getCameraConfig(String devId);

Example

IThingIPCCore cameraInstance = ThingIPCSdk.getCameraInstance();
if (cameraInstance != null) {
    ICameraConfigInfo cameraConfig = cameraInstance.getCameraConfig(devId);
    if (cameraConfig != null) {
        int videoNum = cameraConfig.getVideoNum();
    }
}

Query supported number of streams

API description

Indicates the number of streams supported by the device. If the value is 1, the device only supports HD or SD. Device streams are obtained with the default definition mode.

int getVideoNum();

Query default definition

API description

Indicates the default definition of the device. If the device only supports one stream channel, it also supports only one definition mode.

int getDefaultDefinition();

Query support for speaker

API description

Indicates whether the device is equipped with a speaker. If so, the device supports video talk.

boolean isSupportSpeaker();

Query support for pickup

API description

Indicates whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportPickup();

Query default video talk mode

API description

int getDefaultTalkBackMode();

Query support for switching between video talk modes

API description

Indicates whether users can switch between video talk modes. If so, the device supports both one-way and two-way video talk.

boolean isSupportChangeTalkBackMode();

Query raw P2P configurations

API description

Returns raw data of P2P configurations.

String getRawDataJsonStr();

Query supported play speed

API description

List<Integer> getSupportPlaySpeedList();

Return value

The following table lists the return values that are mapped to constants of ThingIPCConstant.

Constant of `ThingIPCConstant`	Return value	Description
ThingIPCConstant.THING_SPEED_05TIMES	0	0.5x
ThingIPCConstant.THING_SPEED_10TIMES	1	1x
ThingIPCConstant.THING_SPEED_20TIMES	3	2x
ThingIPCConstant.THING_SPEED_40TIMES	7	4x
ThingIPCConstant.THING_SPEED_80TIMES	8	8x
ThingIPCConstant.THING_SPEED_160TIMES	9	16x
ThingIPCConstant.THING_SPEED_320TIMES	10	32x

Prev DocSD Card Management

Next DocCloud Storage