Audio and Video Features

The IPC SDK can provide a bunch of audio and video capabilities in addition to live streaming and playing back footage on an SD card. For example, record videos on a mobile phone, capture video screenshots, talk with IPCs in real time, and switch between definition modes.

Local recording

During live streaming or record playback, the ongoing videos can be recorded on a mobile phone.

Start video recording

API description

int startRecordLocalMp4(String folderPath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
folderPath	The path in which videos are stored.
context	The context.
callBack	The callback.

Example

private IThingSmartCameraP2P mCameraP2P;

if (Constants.hasStoragePermission()) {
    String picPath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(picPath);
    if (!file.exists()) {
        file.mkdirs();
    }
    mCameraP2P.startRecordLocalMp4(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isRecording = true;
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_BEGIN);
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            mHandler.sendEmptyMessage(MSG_VIDEO_RECORD_FAIL);
        }
    });
    recordStatue(true);
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.WRITE_EXTERNAL_STORAGE, Constants.EXTERNAL_STORAGE_REQ_CODE, "open_storage");
}

Stop video recording

API description

int stopRecordLocalMp4(OperationDelegateCallBack callBack);

Parameters

Parameter	Description
callBack	The callback.

Example

mCameraP2P.stopRecordLocalMp4(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        // The success callback.
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        // The failure callback.
    }
});

Take video screenshots

Capture screenshots of live video images and store them on the SD card of a mobile phone.

API description

int snapshot(String absoluteFilePath, Context context, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
absoluteFilePath	The path in which the screenshots are stored.
context	The context.
callBack	The callback.

Example

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
    String path = Environment.getExternalStorageDirectory().getAbsolutePath() + "/Camera/";
    File file = new File(path);
    if (!file.exists()) {
        file.mkdirs();
    }
    picPath = path;
}
mCameraP2P.snapshot(picPath, CameraPanelActivity.this, new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
     // The file path is returned by `data`.
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_SUCCESS, data));
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        mHandler.sendMessage(MessageUtil.getMessage(MSG_SCREENSHOT, ARG1_OPERATE_FAIL));
    }
});

Video sound

During live streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

Mute or unmute video

API description

Mute or unmute the video sound.

void setMute(int mute, OperationDelegateCallBack callBack);

Parameters

Parameter	Description
mute	Audio mode 1: Mute 0: Unmute

Example

mCameraP2P.setMute(1, new OperationDelegateCallBack() {
  @Override
  public void onSuccess(int sessionId, int requestId, String data) {
    // The operation result is returned by `data`.
    previewMute = Integer.valueOf(data);
  }

  @Override
  public void onFailure(int sessionId, int requestId, int errCode) {
  }
});

Switch between speaker and earpiece

API description

Switch between the speaker and earpiece modes. This API method is not supported by P2P 1.0 devices.

void setLoudSpeakerStatus(boolean enable);

Parameters

Parameter	Description
enable	true: Switch to the speaker mode. false: Switch to the earpiece mode.

Custom audio effect

API description

Set up a custom audio effect.

void setAudioEffect(@AudioEffect int type);

Parameters

Parameter	Description
type	The audio effect type.

AudioEffect

Enum value	Description
NONE	The original voice.
BOY	The voice of a boy.
UNCLE	The voice of an uncle.
ROBOT	The voice of a robot.
GIRL	The voice of a girl.

Live talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Check support for live talk

• Check whether the device is only equipped with a speaker. If so, the device supports one-way talk.
• If the device is equipped with both a speaker and a pickup, video streams are audible, so the two-way talk is supported.

Start video talk

Transmit audio data from the mobile phone to the IPC.

Example

if (Constants.hasRecordPermission()) {
    mCameraP2P.startAudioTalk(new OperationDelegateCallBack() {
        @Override
        public void onSuccess(int sessionId, int requestId, String data) {
            isSpeaking = true;
            ToastUtil.shortToast(CameraPanelActivity.this, "start talk success");
        }

        @Override
        public void onFailure(int sessionId, int requestId, int errCode) {
            isSpeaking = false;
            ToastUtil.shortToast(CameraPanelActivity.this, "operation fail");

        }
    });
} else {
    Constants.requestPermission(CameraPanelActivity.this, Manifest.permission.RECORD_AUDIO, Constants.EXTERNAL_AUDIO_REQ_CODE, "open_recording");
}

Stop video talk

Stop transmitting audio data from the mobile phone to the IPC.

Example

mCameraP2P.stopAudioTalk(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {
        isSpeaking = false;
    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {
        isSpeaking = false;
    }
});

Two-way talk

During live streaming, the audio from the video is the human voices and ambient sounds collected by the IPC in real time. Turning on the audio channel between the app and the IPC enables a two-way talk.

One-way talk

The control of one-way talk is subject to your implementation.

• Mute the video when one-way talk starts.
• Unmute it when the talk ends.

Definition modes

Users can switch between definitions during live streaming. Currently, only high definition (HD) and standard definition (SD) modes are supported. A few IPCs only support one of the modes.

Parameters

Parameter	Description
clarity	The video definition mode. 2: SD 4: HD 8: Ultra HD 0xFFFF: Audio-only mode

Get definition mode

Request the definition mode of the videos sent from the IPC.

Example

mCameraP2P.getVideoClarity(new OperationDelegateCallBack() {
    @Override
    public void onSuccess(int sessionId, int requestId, String data) {

    }

    @Override
    public void onFailure(int sessionId, int requestId, int errCode) {

    }

});

Set video definition

Set the definition mode of the videos sent from the IPC.

Example

mCameraP2P.setVideoClarity(2, new OperationDelegateCallBack() {

    @Override

    public void onSuccess(int sessionId, int requestId, String data) {
        videoClarity = Integer.valueOf(data);
    }

    @Override

    public void onFailure(int sessionId, int requestId, int errCode) {
    }

});

Raw stream data

The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format.

Register callback for receiving video frames

API description

To enable the callback for video frames, you must register a listener with IThingSmartCameraP2P in AbsP2pCameraListener. You only need to rewrite the callback as preferred.

void registerP2PCameraListener(AbsP2pCameraListener listener);

Callback for video YUV data

API description

public void onReceiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height, int nFrameRate, int nIsKeyFrame, long timestamp, long nProgress, long nDuration, Object camera)

Parameters

Parameter	Description
Y	The luma (Y′) information of video streams.
u	The chroma (U) channel information of video streams.
v	The chroma (V) channel information of video streams.
width	The width of video images.
height	The height of video images.
nFrameRate	The frame rate of the video.
nIsKeyFrame	Indicates whether a keyframe or an I-frame is used.
timestamp	The timestamp.
nProgress	The video’s time progress displayed in the Message Center module.
nDuration	The video duration displayed in the Message Center module.

Callback for P2P connection status

API description

public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus)

Parameters

Parameter	Description
sessionId	The ID of the P2P connection.
sessionStatus	The status of the P2P connection. 0: The connection is created. -3: The connection timed out. -12: The connection is closed by the IPC. -13: The connection is closed due to timeout.

Example

private IThingSmartCameraP2P mCameraP2P;

@Override
protected void onResume() {
    super.onResume();
    if (null != mCameraP2P) {
        // Registers a P2P listener.
         mCameraP2P.registerP2PCameraListener(p2pCameraListener);

    }
}

@Override
protected void onPause() {
    super.onPause();
    if (null != mCameraP2P) {
        // Unregisters a P2P listener.
        mCameraP2P.removeOnP2PCameraListener(p2pCameraListener);
    }
}

private AbsP2pCameraListener p2pCameraListener = new AbsP2pCameraListener() {
    @Override
            public void onSessionStatusChanged(Object o, int i, int i1) {
        super.onSessionStatusChanged(o, i, i1);
        // The callback for connection status changes.
    }
};

Audio data

Audio data collected by the app can be further processed, such as voice changing. To implement this feature, the IPC SDK provides the callback that returns the audio data collected by the app. The collected audio data has been processed with echo cancellation. To enable the callback for audio data, you must register a listener with IThingSmartCameraP2P.

API description

Register the listener.

void registerSpeakerEchoProcessor(ISpeakerEchoProcessor processor);

API description

Destroy the listener.

void unregisterSpeakerEchoProcessor();

API description

Transmit the processed audio data to the IPC for playback.

void sendAudioTalkData(byte[] outbuf, int length);

Parameters

Parameter	Description
outbuf	The byte array of audio data.
length	The data length.

Example

mCameraP2P.registerSpeakerEchoProcessor(new ISpeakerEchoProcessor() {
    @Override
    public void receiveSpeakerEchoData(ByteBuffer pcm, int sampleRate) {
        // The pulse-code modulation (PCM) data that indicates the sampling rate.
    }
});

Intelligent video analytics

If the IPC detects an object in motion during live streaming, the intelligent video analytics (IVA) feature allows the object to be framed automatically in white on the view.

To achieve this purpose, the IVA feature must be enabled for the IPC first. The device will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 (ipc_object_outline) to enable this feature. For more information about device control API methods, see Device Control.

After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

Enable IVA

API description

Enable or disable the IVA feature. This API method is usually called after startPreview for video previewing or setVideoClarity for definition settings.

void setEnableIVA(boolean enableIVA);

Parameters

Parameter	Description
enable	Specifies whether to enable IVA.

Configure IVA properties

You can configure the properties of IVA to control its style, such as the frame color, brush width, and flash frequency.

API description

Define the properties of IVA in the specified format of JSON strings based on the Supplemental Enhancement Information (SEI) reported by the device.

void setSmartRectFeatures(String rectFeatures);

Parameters

rectFeatures is a JSON string in the following format:

{
    "SmartRectFeature":[
        {
            "type":0,
            "index":0,
            "brushWidth":1,
            "flashFps":{
                "drawKeepFrames":2,
                "stopKeepFrames":2
            },
            "rgb":0xFF0000,
            "shape":0
        },
        {
            "type":0,
            "index":1,
            "brushWidth":2,
            "flashFps":{
                "drawKeepFrames":3,
                "stopKeepFrames":2
            },
            "rgb":0x00FF00,
            "shape":1
        }
    ]
}

Parameter	Type	Description
SmartRectFeature	Array	(Required) The identifier in the fixed format of arrays to represent multiple frame settings.
type	Int	The type of frame. Valid values: 0: Smart bounding box (default) 1: Line crossing detection
index	Int	The index of the frame, corresponding to each ID in od of SEI.
shape	Int	The shape of the rectangular frame. Valid values: 0: Closed rectangle 1: Four corners only
rgb	Int	The color of the rectangular frame, represented by the red-green-blue (RGB) color model. Value range: 0x000000 to 0xFFFFFF. Default value: 0xFC4747.
brushWidth	Int	The brush stroke of the rectangular frame. Valid values: 0: Thin 1: Medium 2: Thick
flashFps	String	The flash frequency of the rectangular frame. Valid values: drawKeepFrames: The number of frames with drawing on. stopKeepFrames: The number of frames with drawing off.

Define SEI protocol

The protocol that governs communication with the device. The IPC SDK parses the data over SEI and implements IVA at the positions with the specified properties.

{
    "AGTX":{
        "time":6885,
        "chn":0,
        "key":1,
        "iva":{
            "od":[
                {
                    "obj":{
                        "id":0,
                        "type":1,
                        "twinkle":1,
                        "rect":[
                            0,0,
                            25,25,
                            50,50,
                            80, 80,
                            100,100
                        ],
                        "vel":[0,10],
                        "cat":"PEDESTRIAN"
                    }
                },
                {
                    "obj":{
                        "id":1,
                        "type":1,
                        "twinkle":1,
                        "rect":[
                            0,0,
                            100,100
                        ],
                        "vel":[0,10],
                        "cat":"PEDESTRIAN"
                    }
                }
            ]
        }
    }
}

The following table describes the parsed parameters in iva reported by the device.

Parameter	Description
id	The index of each frame.
type	The type of frame. Valid values: 0: Closed rectangle 1: Four corners only
twinkle	The flash setting. Valid values: 0: The frame is displayed, but flash is disabled. 1: The frame flashes at the frequency passed in.
rect	Each pair of numbers represents the coordinates of a point. All points are arranged in clockwise order. The SDK draws these points into a closed polygon. Two points are used to represent a rectangular box.

Video rendering

After video decoding, rendered images can be further processed by settings. For example, the following features are supported: stretching or scaling, horizontal or vertical mirroring, and rotation by 90, 180, or 270 degrees.

API description

void setDeviceFeatures(String renderFeatures);

Parameters

renderFeatures is a JSON string in the following format:

{
    "DecPostProcess":{
        "video":[
            {
                "restype":"4",
                "oldres":"944*1080",
                "newres":"1920*1080"
            },
            {
                "restype":"2",
                "oldres":"944*1080",
                "newres":"1920*1080"
            }
        ],
        "mirror":0,
        "rotation":2
    }
}

Parameter	Description
DecPostProcess	(Required) The fixed identifier.
video	The array of video resolution settings. restype: The type of video resolution. oldres: The size of the original video reported by the device. newres: The expected size of the video after stretching or scaling.
mirror	The mirroring setting. Valid values: 0: Unchanged. 1: Horizontal mirroring. 2: Vertical mirroring.
rotation	The angle of rotation. Valid values: 0: No rotation. 1: Rotate 90 degrees. 2: Rotate 180 degrees. 3: Rotate 270 degrees.

Local video

Use the phone’s camera features to perform tasks. For example, start/stop data collection, start/stop/pause/resume sending device data, and switch between the front and rear cameras.

Turn on camera

Initialize the phone’s camera and start capturing video.

API description

Turn on the phone’s camera and return a Boolean value to indicate the operation result. Invoke a callback through IRegistorIOTCListener#receiveLocalVideoFrame(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height).

   @MainThread
   @RequiresPermission(Manifest.permission.CAMERA)
   boolean startVideoCapture();

   @MainThread
   @RequiresPermission(Manifest.permission.CAMERA)
   boolean startVideoCapture(int width, int height, int frameRate);

Parameters

Parameter	Description
width	The width of the video.
height	The height of the video.
frameRate	The frame rate of the video.

Turn off camera

API description

Turn off the phone’s camera.

   void stopVideoCapture();

Switch between cameras

Switch to the front or rear camera on the phone.

API description

Switch between the front and rear cameras on the phone and return a Boolean value to indicate the operation result.

boolean switchCamera();

Local video stream

Based on Live Streaming, create a player, implement the method IRegistorIOTCListener#receiveLocalVideoFrame(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height), and bind with the P2P object to render local videos.

Example

The local camera player inherits the YUV player.

public class LocalCameraMonitor extends YUVMonitorTextureView implements IRegistorIOTCListener {

    public LocalCameraMonitor(Context context) {
        this(context, null);
    }

    public LocalCameraMonitor(Context context, AttributeSet attrs) {
        super(context, attrs);
    }

    @Override
    public void receiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, ThingVideoFrameInfo videoFrameInfo, Object camera) {
    }

    @Override
    public void receiveLocalVideoFrame(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height) {
        updateFrameYUVData(y, u, v, width, height);
    }

    @Override
    public void receivePCMData(int sessionId, ByteBuffer pcm, ThingAudioFrameInfo audioFrameInfo, Object camera) {

    }

    @Override
    public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus) {

    }
}

Start video talk

After a P2P connection is created, start a video talk to send the local video to the device.

API description

 void startVideoTalk(OperationDelegateCallBack callBack);

Stop video talk

Stop a video call to disconnect the channel for transmitting local video to the device. This action does not terminate the P2P connection.

API description

 void stopVideoTalk(OperationDelegateCallBack callBack);

Pause video talk

Pause a video talk.

API description

 void pauseVideoTalk(OperationDelegateCallBack callback);

Resume video talk

Resume a video talk.

API description

void resumeVideoTalk(OperationDelegateCallBack callback);

Additional features

Get bitrate

API description

double getVideoBitRateKbps();

Example

double rate = mCameraP2P.getVideoBitRateKbps();

Set double-tap zoom

API description

Enable users to zoom in video views by double-tapping by default. You can also disable this feature.

public void setCameraViewDoubleClickEnable(boolean enable);

Example

ThingCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setCameraViewDoubleClickEnable(false);

Device capabilities

Get device capabilities

API description

ICameraConfigInfo is the device capability class that encapsulates multiple API methods to implement device features.

ICameraConfigInfo getCameraConfig(String devId);

Example

IThingIPCCore cameraInstance = ThingIPCSdk.getCameraInstance();
if (cameraInstance != null) {
    ICameraConfigInfo cameraConfig = cameraInstance.getCameraConfig(devId);
    if (cameraConfig != null) {
        int videoNum = cameraConfig.getVideoNum();
    }
}

Get supported streams

API description

Request the streams supported by the device. If the value is 1, the device only supports HD or SD. Device streams are obtained with the default definition mode.

int getVideoNum();

Get default definition

API description

Request the default definition of the device. If the device only supports one stream channel, it also supports only one definition mode.

int getDefaultDefinition();

Check support for speaker

API description

Check whether the device is equipped with a speaker. If so, the device supports video talk.

boolean isSupportSpeaker();

Check support for pickup

API description

Check whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportPickup();

Get default video talk mode

API description

int getDefaultTalkBackMode();

Check support for switching between video talk modes

API description

Check whether users can switch between video talk modes. If so, the device supports both one-way and two-way video talk.

boolean isSupportChangeTalkBackMode();

Get raw P2P configurations

API description

Request the raw data of P2P configurations.

String getRawDataJsonStr();

Get play speed options

API description

List<Integer> getSupportPlaySpeedList();

Return value

The following table lists the return values that are mapped to constants of ThingIPCConstant.

Constant of ThingIPCConstant	Return value	Description
ThingIPCConstant.THING_SPEED_05TIMES	0	0.5x
ThingIPCConstant.THING_SPEED_10TIMES	1	1x
ThingIPCConstant.THING_SPEED_20TIMES	3	2x
ThingIPCConstant.THING_SPEED_40TIMES	7	4x
ThingIPCConstant.THING_SPEED_80TIMES	8	8x
ThingIPCConstant.THING_SPEED_160TIMES	9	16x
ThingIPCConstant.THING_SPEED_320TIMES	10	32x