Wi-Fi Smart Lock Audio and Video Capabilities

This topic describes the API methods respecting audio and video capabilities, log management, member management, and temporary passwords.

Audio and video capabilities

The following API methods can be called to implement audio and video capabilities of Wi-Fi smart locks. For example, smart video locks and camera locks support live video streaming and video talks.

Initialize and access audio and video capabilities

Class name	Description
ThingOptimusSdk	Provides access to the SDK initialization feature and returns the lock management class.
IThingLockManager	The lock management class that is used to get different types of lock classes.
IVideoLockManager	Provides access to audio and video capabilities of a Wi-Fi smart lock.

Register a listener

Listens for different types of data point (DP) data reported by the smart lock, for example, live video requests, remote locking and unlocking, and alerts.

void registerLockReportListener(ILockReportListener iLockReportListener);

void unRegisterLockReportListener();

Description

ILockReportListener:

	 /**
     * `dpCode:initiative_message`: initiates push notifications, including the path in which captured images are stored.
     * @param devId The device ID.
     * @param dpCode
     * @param fileBucketInfoBean  The image information in cloud storage.
     */
    void onLockMessageArrived(String devId, String dpCode, FileBucketInfoBean fileBucketInfoBean);
    /**
     * `dpCode:video_request_realtime`: requests live videos.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void onVideoRequestRealtime(String devId, String dpCode,String dpValue);

    /**
     * `dpCode:unlock_request`: the countdown for remote unlocking.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void unlockRequestCountdown(String devId,String dpCode,Object dpValue);

    /**
     * `dpCode:alarm_request`: the countdown for confirming a capturing alert.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void alarmRequestCountdown(String devId, String dpCode,Object dpValue);

    /**
     * `dpCode:remote_no_dp_key`: reports remote unlocking.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void onRemoteUnlockReport(String devId, String dpCode,Object dpValue);

    /**
     * `dpCode:reverse_lock`: reports forced double locking.
     * @param devId
     * @param dpCode
     */
    void onForceLockUpReport(String devId, String dpCode,Object dpValue);

    /**
     * Reports other DP data.
     * @param devId
     * @param dpCode
     */
    void onLockDpUpdate(String devId, Map<String,Object> dpCode);

Example

// Initialize the SDK for only once.
ThingOptimusSdk.init(getApplicationContext());
// Get the `ThingLockManager` class.
val iThingLockManager = ThingOptimusSdk.getManager(IThingLockManager::class.java)
// Get the `IVideoLockManager` class.
iThingVideoLockManager = iThingLockManager.newVideoLockManagerInstance(mDevId)
// Register a listener.
iThingVideoLockManager?.registerLockReportListener(object:ILockReportListener{
            override fun onLockMessageArrived(
                devId: String?,
                dpCode: String?,
                fileBucketInfoBean: FileBucketInfoBean?
            ) {
                TODO("Not yet implemented")
            }

            override fun onVideoRequestRealtime(devId: String?, dpCode: String?, dpValue: String?) {
                TODO("Not yet implemented")
            }

            override fun unlockRequestCountdown(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun alarmRequestCountdown(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onRemoteUnlockReport(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onForceLockUpReport(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onLockDpUpdate(devId: String?, dpCode: MutableMap<String, Any>?) {
                TODO("Not yet implemented")
            }

        })

Remote unlocking

API description

Implements remote operations, including unlocking, locking, unlocking rejection, and locking rejection.

void remoteLock(boolean isOpen, boolean confirm, IResultCallback callback);

Parameters

Parameter	Optional	Description
isOpen	No	Specifies whether to unlock. Valid values: true: unlock false: lock
confirm	No	Specifies whether to accept unlocking. Valid values: true: accept false: reject
Callback	Yes	The callback.

Example

fun remoteUnLock(isOpen: Boolean, confirm: Boolean) {
        iThingVideoLockManager?.remoteLock(isOpen, confirm, object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                Toast.makeText(mContext, error, Toast.LENGTH_SHORT).show()
            }

            override fun onSuccess() {
                Toast.makeText(mContext, "publish success", Toast.LENGTH_SHORT).show()
            }

        })
    }

Capture image again

API description

Captures an image again. Then, the device transfers the captured image to cloud storage. onLockMessageArrived is called to report the image path. Due to device limits, this operation can only be implemented once within 15 seconds.

void reTakePhoto(boolean isTakePhotoAgain, IResultCallback callback);

Parameters

Parameter	Optional	Description
isTakePhotoAgain	No	Specifies whether to capture an image again.
callback	Yes	The callback.

Example

iThingVideoLockManager?.reTakePhoto(true,object :IResultCallback{
            override fun onError(code: String?, error: String?) {
                L.e(TAG,"RetakePhoto error:$error")
            }

            override fun onSuccess() {
                L.i(TAG,"Retake photo publish success")
            }

        })

Forced double locking

API description

Implements forced double locking. The success callback onForceLockUpReport is executed.

void forceLock(boolean isForceLock, IResultCallback callback);

Parameters

Parameter	Optional	Description
isForceLock	No	Specifies whether to implement forced double locking.
callback	Yes	The callback.

Example

iThingVideoLockManager?.forceLock(true,object :IResultCallback{
            override fun onError(code: String?, error: String?) {
                L.e(TAG,"code $code error:$error")
                ToastUtil.shortToast(mContext,error)
            }

            override fun onSuccess() {
                L.i(TAG,"force anti-lock success")
                ToastUtil.shortToast(mContext,"force anti-lock success")
            }

        })

Get the latest image URL

API description

Returns the URL of the latest image from cloud storage. This request is made along with getPictureAndVideo.

void getLatestPicture(FileTypeEnum fileTypeEnum, IThingResultCallback<FileBucketInfoBean> callback);

Parameters

Parameter	Optional	Description
fileTypeEnum	No	The type of image. Valid values: image captured due to remote unlocking image captured due to an alert
callback	Yes	The callback.

FileBucketInfoBean: the image details in cloud storage.

Field	Description
filePath	The path of the image in cloud storage.
fileKey	The key used to decrypt the image.
bucket	The bucket in which the image is stored.

Example

iThingVideoLockManager?.getLatestPicture(FileTypeEnum.REMOTE_UNLOCK,
            object : IThingResultCallback<FileBucketInfoBean> {
                override fun onSuccess(result: FileBucketInfoBean?) {
                    L.i(TAG, "result filePath:${result?.toString()}")
                    result?.let {
                        requestPictureAndVideo(it.bucket, it.filePath, it.fileKey)
                    }

                }

                override fun onError(errorCode: String?, errorMessage: String?) {
                    Toast.makeText(mContext, errorMessage, Toast.LENGTH_SHORT).show()
                }

            })

Get URL of captured image

API description

Returns the URL of the image captured and uploaded to cloud storage. This request is made along with onLockMessageArrived or getLatestPicture.

void getPictureAndVideo( String bucket, String filePath, IThingResultCallback<RemoteMediaBean> callback);

Parameters

Parameter	Optional	Description
bucket	No	The bucket in which the image is stored.
filePath	No	The encrypted URL of the image in cloud storage.
callback	Yes	The callback.

RemoteMediaBean

Field	Description
fileUrl	The path of the image in cloud storage.
mediaUrl	The path of the video in cloud storage.
angle	The rotation angle configured on the Tuya IoT Development Platform. Valid values: 0, 90, 180, and 270.

Example

override fun onLockMessageArrived(
        devId: String?,
        dpCode: String?,
        fileBucketInfoBean: FileBucketInfoBean?
    ) {
    	fileBucketInfoBean?.let {
    		requestPictureAndVideo(it.bucket,it.filePath,it.fileKey)
    	}
    }
...
fun requestPictureAndVideo(
        bucket: String?,
        filePath: String?,
        fileKey: String?
    ) {
        iThingVideoLockManager?.getPictureAndVideo(
            bucket,
            filePath,
            object : IThingResultCallback<RemoteMediaBean> {
                override fun onSuccess(result: RemoteMediaBean?) {
                    L.i(TAG, result?.toString())
                    if (fileKey != null) {
                        refreshImage(result?.fileUrl, fileKey)
                    }

                }

                override fun onError(errorCode: String?, errorMessage: String?) {
                    Toast.makeText(mContext, errorMessage, Toast.LENGTH_SHORT).show()
                }

            })
    }
    
    ...
    fun refreshImage(fileUrl:String?,fileKey:String){
      fileUrl?.let{
        val imageView:DecryptImageView = findViewById(R.id.div_picture)
        imageView.setImageURI(it,fileKey.toByteArray())
      }
    }

Get video object

API description

IIPCManager getIPCManager();

Multiple features are implemented, including (peer-to-peer) P2P connection, video rendering, and video talks, as shown in the following process.

Import view rendering container to page layout file

ThingCameraView is the video rendering view provided by the SDK.

<com.thingclips.smart.camera.middleware.widget.ThingCameraView
  android:id="@+id/camera_video_view"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  />

Set callback for view rendering container

API description

public void setViewCallback(AbsVideoViewCallback callback);

Parameters

Parameter	Description
callback	The callback.

Example

ThingCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setViewCallback(new AbsVideoViewCallback() {
	@Override
	public void onCreated(Object view) {
		super.onCreated(view);
        // The callback for the view rendering container.
	}
});

AbsVideoViewCallback

The abstract class of the callback for the view rendering.
You only need to focus on rewriting the callback as needed. In most cases, only the onCreated method needs to be rewritten.

Build a rendered view

API description

public void createVideoView(String devId);

Parameters

Parameter	Description
devId	The device ID.

Example

ThingCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.createVideoView(devId);

Bind a rendered view

API description

void generateCameraView(T view);

Example

iIPCManager?.generateCameraView(mVideoView.createdView())

Register a P2P listener

Registers a listener. Otherwise, video images cannot be displayed as expected.
For more information, see Raw stream data.

API description

void registerP2PCameraListener(AbsP2pCameraListener listener);

Example

// 1. Get `IIPCManager`.
iIPCManager = iThingVideoLockManager?.ipcManager
ThingCameraView cameraView = findViewById(R.id.camera_video_view)
// 2. Set the callback for the view rendering container.
cameraView.setViewCallback(object : AbsVideoViewCallback() {
            override fun onCreated(p0: Any?) {
                super.onCreated(p0)
                iIpcManager?.generateCameraView(p0)
            }
        })
// 3. Build the rendered view.
cameraView.createVideoView(devId)
// 4. Register a P2P listener.
if (null != iIPCManager){
    iIPCManager.registerP2PCameraListener(object: AbsP2pCameraListener() {
        override fun onReceiveAudioBufferData(
                nSampleRate: Int,
                nChannelNum: Int,
                nBitWidth: Int,
                nTimeStamp: Long,
                progress: Long,
                duration: Long
            ) {
                super.onReceiveAudioBufferData(
                    nSampleRate,
                    nChannelNum,
                    nBitWidth,
                    nTimeStamp,
                    progress,
                    duration
                )
            }

            override fun onReceiveFrameYUVData(
                sessionId: Int,
                y: ByteBuffer?,
                u: ByteBuffer?,
                v: ByteBuffer?,
                videoFrameInfo: ThingVideoFrameInfo?,
                camera: Any?
            ) {
                super.onReceiveFrameYUVData(sessionId, y, u, v, videoFrameInfo, camera)
            }

            override fun onReceiveFrameYUVData(
                sessionId: Int,
                y: ByteBuffer?,
                u: ByteBuffer?,
                v: ByteBuffer?,
                width: Int,
                height: Int,
                nFrameRate: Int,
                nIsKeyFrame: Int,
                timestamp: Long,
                nProgress: Long,
                nDuration: Long,
                camera: Any?
            ) {
                super.onReceiveFrameYUVData(
                    sessionId,
                    y,
                    u,
                    v,
                    width,
                    height,
                    nFrameRate,
                    nIsKeyFrame,
                    timestamp,
                    nProgress,
                    nDuration,
                    camera
                )
            }

            override fun onReceiveSpeakerEchoData(pcm: ByteBuffer?, sampleRate: Int) {
                super.onReceiveSpeakerEchoData(pcm, sampleRate)
            }

            override fun onSessionStatusChanged(camera: Any?, sessionId: Int, sessionStatus: Int) {
                super.onSessionStatusChanged(camera, sessionId, sessionStatus)
            }


        })
    });   
}

onReceiveFrameYUVData raw data

Parameter	Description
sessionId	The session ID.
Y	The luma (Y’) information of video streams.
u	The chroma (U) channel information of video streams.
v	The chroma (V) channel information of video streams.
width	The width of video images.
height	The height of video images.
nFrameRate	The frame rate of video streams.
nIsKeyFrame	Indicates whether a keyframe or an I-frame is used.
timestamp	The timestamp.
nProgress	The time progress of the video pushed by the Message Center module.
nDuration	The duration of the video pushed by the Message Center module.

onSessionStatusChanged P2P connection status callback

Parameter	Description
sessionId	The session ID.
sessionStatus	The session status.

Session status code	Description
0	The connection is created.
-3	The connection timed out.
-12	The connection is closed by the device.
-13	The connection is closed due to timeout after no response is returned.

P2P connections

Before live video streaming, a P2P connection must be created. The P2P connection status is subject to your maintenance. The SDK can only send commands to and receive responses from the device.

API description

Create a P2P connection

void connect(IResultCallback callBack);

Close a P2P connection

void disconnect(IResultCallback callBack);

Parameters

Parameter	Description
callBack	The callback.

Whether P2P connection is built

boolean isConnected();

Whether P2P connection in progress

boolean isConnecting();

Play live video

After a P2P connection is created, live video streaming can be started.

API description

Start live video streaming

void startPreview(int clarity, IResultCallback callBack);

Stop live video streaming

int stopPreview(IResultCallback callBack);

Parameters

Parameter	Description
clarity	The video definition.
callBack	The callback.

Video definition

Mode	Value
Standard definition	2
High definition	4

Example

iIPCManager.startPreview(new IResultCallback() {
    @Override
    public void onSuccess() {
        // Live video streaming is started.
    }

    @Override
    public void onError(String code, String error) {
        // Failed to start live video streaming.
    }
});

Local recording

During live video streaming or record playback, the ongoing videos can be recorded on a mobile phone.

Enable video recording

API description

void startRecord(String folderPath, Context context, RotateModeEnum rotateMode, IResultCallback callback);

Parameters

Parameter	Description
folderPath	The path in which videos are stored.
context	The context.
rotateMode	The rotation mode.
callBack	The callback.

Stop video recording

API description

void stopRecord(IThingResultCallback<Map<String, String>> callback);

Callback parameter description

In the callback, the video path is returned as the value of the key video, and the cover image is returned as the value of the key coverImage.

Whether recording in progress

API description

boolean isRecording();

Example

if(iIpcManager?.isRecording == true){
	stopRecorder()
}else{
	startRecord()
}

...
fun startRecord(){
	iIpcManager?.startRecord(
                path,
                mContext,
                rotateMode ?: RotateModeEnum.NORMAL,
                object : IResultCallback {
                    override fun onError(code: String?, error: String?) {
                        
                    }

                    override fun onSuccess() {
                        L.i(TAG, "recorder success")
                    }

                })
}
fun stopRecord(){
	iIpcManager?.stopRecord(object : IThingResultCallback<Map<String, String>> {
            override fun onSuccess(result: Map<String, String>?) {
                result?.let {
                    val videoPath = it.get("video")
                    val coverImagePath = it.get("coverImage")                    
                }
            }

            override fun onError(errorCode: String?, errorMessage: String?) {
                
            }

        })
}

Video screenshots

Captures screenshots of live video images and stores them on the SD card of a mobile phone.

API description

void snapshot(String folderPath, Context context, ICameraP2P.PLAYMODE playmode, RotateModeEnum rotateMode, IThingResultCallback<String> callback);

Parameters

Parameter	Description
folderPath	The path in which the screenshots are stored.
context	The context.
playmode	The video playback mode. Currently, only PLAYMODE.LIVE is supported.
rotateMode	The rotation angle. Valid values: 0, 90, 180, and 270.
callBack	The callback. onSuccess returns the complete path of the screenshot.

Example

iIpcManager?.snapshot(
                path,
                mContext,
                ICameraP2P.PLAYMODE.LIVE,
                rotateMode ?: RotateModeEnum.NORMAL,
                object : IThingResultCallback<String> {
                    override fun onSuccess(result: String?) {
                        L.i(TAG, "snapshot success :$result")
                    }

                    override fun onError(errorCode: String?, errorMessage: String?) {
                        
                    }

                })

Audio settings

During live video streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

API description

Enables or disables the audio channel.

void enableMute(boolean isMute,IResultCallback callback);

Audio mode	Value
Mute	true
Unmuted	false

Whether audio enabled

boolean isMuting();

Example

iIpcManager?.enableMute(mute, object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                
            }

            override fun onSuccess() {
                L.i(TAG, "set mute $mute success ")
            }

        })

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Enable or disable video talks

Enables or disables transmission of audio data from the mobile phone to the IPC.

void startTalk(IResultCallback callback);
void stopTalk(IResultCallback callback);

Video talk to be supported or already enabled

boolean isSupportTalk();
boolean isTalkBacking();

Example

iIpcManager?.startTalk(object : IResultCallback {
                override fun onError(code: String?, error: String?) {
                    
                }

                override fun onSuccess() {
                    L.i(TAG, "startTalk success")
                }

            })
iIpcManager?.stopTalk(object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                
            }

            override fun onSuccess() {

            }

        })

Others

API description

Indicates whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportSound();

API description

Indicates the default video talk mode.

int supportAudioMode();

Destroy

Destroy IIPCManager

If the camera features are not required, the P2P listener must be removed and the P2P object must be destroyed.

void onDestroy();

Destroy IVideoLockManger

If the features of the visual lock and camera lock are not required, the IVideoLockManger and IIPCManager objects must be destroyed.

void onDestroy();