English
English
简体中文
Contact Us
Register
Log In
layoutIndex

Wi-Fi Smart Lock Audio and Video Capabilities

Last Updated on : 2022-08-31 14:28:45download

This topic describes the API methods respecting audio and video capabilities, log management, member management, and temporary passwords.

Audio and video capabilities

The following API methods can be called to implement audio and video capabilities of Wi-Fi smart locks. For example, smart video locks and camera locks support live video streaming and video talks.

Initialize and access audio and video capabilities

Class name Description
TuyaOptimusSdk Provides access to the SDK initialization feature and returns the lock management class.
ITuyaLockManager The lock management class that is used to get different types of lock classes.
IVideoLockManager Provides access to audio and video capabilities of a Wi-Fi smart lock.

Register a listener

Listens for different types of data point (DP) data reported by the smart lock, for example, live video requests, remote locking and unlocking, and alerts.

void registerLockReportListener(ILockReportListener iLockReportListener);

void unRegisterLockReportListener();

Description

ILockReportListener:

	 /**
     * `dpCode:initiative_message`: initiates push notifications, including the path in which captured images are stored.
     * @param devId The device ID.
     * @param dpCode
     * @param fileBucketInfoBean  The image information in cloud storage.
     */
    void onLockMessageArrived(String devId, String dpCode, FileBucketInfoBean fileBucketInfoBean);
    /**
     * `dpCode:video_request_realtime`: requests live videos.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void onVideoRequestRealtime(String devId, String dpCode,String dpValue);

    /**
     * `dpCode:unlock_request`: the countdown for remote unlocking.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void unlockRequestCountdown(String devId,String dpCode,Object dpValue);

    /**
     * `dpCode:alarm_request`: the countdown for confirming a capturing alert.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void alarmRequestCountdown(String devId, String dpCode,Object dpValue);

    /**
     * `dpCode:remote_no_dp_key`: reports remote unlocking.
     * @param devId The device ID.
     * @param dpCode
     * @param dpValue
     */
    void onRemoteUnlockReport(String devId, String dpCode,Object dpValue);

    /**
     * `dpCode:reverse_lock`: reports forced double locking.
     * @param devId
     * @param dpCode
     */
    void onForceLockUpReport(String devId, String dpCode,Object dpValue);

    /**
     * Reports other DP data.
     * @param devId
     * @param dpCode
     */
    void onLockDpUpdate(String devId, Map<String,Object> dpCode);

Example

// Initialize the SDK for only once.
TuyaOptimusSdk.init(getApplicationContext());
// Get the `TuyaLockManager` class.
val iTuyaLockManager = TuyaOptimusSdk.getManager(ITuyaLockManager::class.java)
// Get the `IVideoLockManager` class.
iTuyaVideoLockManager = iTuyaLockManager.newVideoLockManagerInstance(mDevId)
// Register a listener.
iTuyaVideoLockManager?.registerLockReportListener(object:ILockReportListener{
            override fun onLockMessageArrived(
                devId: String?,
                dpCode: String?,
                fileBucketInfoBean: FileBucketInfoBean?
            ) {
                TODO("Not yet implemented")
            }

            override fun onVideoRequestRealtime(devId: String?, dpCode: String?, dpValue: String?) {
                TODO("Not yet implemented")
            }

            override fun unlockRequestCountdown(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun alarmRequestCountdown(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onRemoteUnlockReport(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onForceLockUpReport(devId: String?, dpCode: String?, dpValue: Any?) {
                TODO("Not yet implemented")
            }

            override fun onLockDpUpdate(devId: String?, dpCode: MutableMap<String, Any>?) {
                TODO("Not yet implemented")
            }

        })

Remote unlocking

API description

Implements remote operations, including unlocking, locking, unlocking rejection, and locking rejection.

void remoteLock(boolean isOpen, boolean confirm, IResultCallback callback);

Parameters

Parameter Optional Description
isOpen No Specifies whether to unlock. Valid values:
  • true: unlock
  • false: lock
confirm No Specifies whether to accept unlocking. Valid values:
  • true: accept
  • false: reject
Callback Yes The callback.

If onSuccess is returned by Callback, it only means a success callback, but cannot indicate the result of the command executed by the lock.

  • confirm = true: The device status changes to run the unlocking or locking command. The command result depends on the listener for the result of the method onRemoteUnlockReport.
  • confirm=false: The device status remains unchanged and no DP data is reported. onSuccess: The unlocking or locking request is rejected.

Example

fun remoteUnLock(isOpen: Boolean, confirm: Boolean) {
        iTuyaVideoLockManager?.remoteLock(isOpen, confirm, object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                Toast.makeText(mContext, error, Toast.LENGTH_SHORT).show()
            }

            override fun onSuccess() {
                Toast.makeText(mContext, "publish success", Toast.LENGTH_SHORT).show()
            }

        })
    }

Capture image again

API description

Captures an image again. Then, the device transfers the captured image to cloud storage. onLockMessageArrived is called to report the image path. Due to device limits, this operation can only be implemented once within 15 seconds.

void reTakePhoto(boolean isTakePhotoAgain, IResultCallback callback);

Parameters

Parameter Optional Description
isTakePhotoAgain No Specifies whether to capture an image again.
callback Yes The callback.

Example

iTuyaVideoLockManager?.reTakePhoto(true,object :IResultCallback{
            override fun onError(code: String?, error: String?) {
                L.e(TAG,"RetakePhoto error:$error")
            }

            override fun onSuccess() {
                L.i(TAG,"Retake photo publish success")
            }

        })

Forced double locking

API description

Implements forced double locking. The success callback onForceLockUpReport is executed.

void forceLock(boolean isForceLock, IResultCallback callback);

Parameters

Parameter Optional Description
isForceLock No Specifies whether to implement forced double locking.
callback Yes The callback.

Example

iTuyaVideoLockManager?.forceLock(true,object :IResultCallback{
            override fun onError(code: String?, error: String?) {
                L.e(TAG,"code $code error:$error")
                ToastUtil.shortToast(mContext,error)
            }

            override fun onSuccess() {
                L.i(TAG,"force anti-lock success")
                ToastUtil.shortToast(mContext,"force anti-lock success")
            }

        })

Get the latest image URL

API description

Returns the URL of the latest image from cloud storage. This request is made along with getPictureAndVideo.

void getLatestPicture(FileTypeEnum fileTypeEnum, ITuyaResultCallback<FileBucketInfoBean> callback);

Parameters

Parameter Optional Description
fileTypeEnum No The type of image. Valid values:
  • image captured due to remote unlocking
  • image captured due to an alert
callback Yes The callback.

FileBucketInfoBean: the image details in cloud storage.

Field Description
filePath The path of the image in cloud storage.
fileKey The key used to decrypt the image.
bucket The bucket in which the image is stored.

Example

iTuyaVideoLockManager?.getLatestPicture(FileTypeEnum.REMOTE_UNLOCK,
            object : ITuyaResultCallback<FileBucketInfoBean> {
                override fun onSuccess(result: FileBucketInfoBean?) {
                    L.i(TAG, "result filePath:${result?.toString()}")
                    result?.let {
                        requestPictureAndVideo(it.bucket, it.filePath, it.fileKey)
                    }

                }

                override fun onError(errorCode: String?, errorMessage: String?) {
                    Toast.makeText(mContext, errorMessage, Toast.LENGTH_SHORT).show()
                }

            })

Get URL of captured image

API description

Returns the URL of the image captured and uploaded to cloud storage. This request is made along with onLockMessageArrived or getLatestPicture.

void getPictureAndVideo( String bucket, String filePath, ITuyaResultCallback<RemoteMediaBean> callback);

Parameters

Parameter Optional Description
bucket No The bucket in which the image is stored.
filePath No The encrypted URL of the image in cloud storage.
callback Yes The callback.

RemoteMediaBean

Field Description
fileUrl The path of the image in cloud storage.
mediaUrl The path of the video in cloud storage.
angle The rotation angle configured on the Tuya IoT Development Platform. Valid values: 0, 90, 180, and 270.

fileUrl is still an encrypted URL. onLockMessageArrived or getLatestPicture must be called to get the value of fileKey to decrypt the URL. Tuya provides the control DecryptImageView to decrypt and display the image.

Example

override fun onLockMessageArrived(
        devId: String?,
        dpCode: String?,
        fileBucketInfoBean: FileBucketInfoBean?
    ) {
    	fileBucketInfoBean?.let {
    		requestPictureAndVideo(it.bucket,it.filePath,it.fileKey)
    	}
    }
...
fun requestPictureAndVideo(
        bucket: String?,
        filePath: String?,
        fileKey: String?
    ) {
        iTuyaVideoLockManager?.getPictureAndVideo(
            bucket,
            filePath,
            object : ITuyaResultCallback<RemoteMediaBean> {
                override fun onSuccess(result: RemoteMediaBean?) {
                    L.i(TAG, result?.toString())
                    if (fileKey != null) {
                        refreshImage(result?.fileUrl, fileKey)
                    }

                }

                override fun onError(errorCode: String?, errorMessage: String?) {
                    Toast.makeText(mContext, errorMessage, Toast.LENGTH_SHORT).show()
                }

            })
    }
    
    ...
    fun refreshImage(fileUrl:String?,fileKey:String){
      fileUrl?.let{
        val imageView:DecryptImageView = findViewById(R.id.div_picture)
        imageView.setImageURI(it,fileKey.toByteArray())
      }
    }

Get video object

API description

IIPCManager getIPCManager();

Multiple features are implemented, including (peer-to-peer) P2P connection, video rendering, and video talks, as shown in the following process.

Wi-Fi Smart Lock Audio and Video Capabilities

Import view rendering container to page layout file

TuyaCameraView is the video rendering view provided by the SDK.

<com.tuya.smart.camera.middleware.widget.TuyaCameraView
  android:id="@+id/camera_video_view"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  />

Set callback for view rendering container

API description

public void setViewCallback(AbsVideoViewCallback callback);

Parameters

Parameter Description
callback The callback.

Example

TuyaCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.setViewCallback(new AbsVideoViewCallback() {
	@Override
	public void onCreated(Object view) {
		super.onCreated(view);
        // The callback for the view rendering container.
	}
});

AbsVideoViewCallback

The abstract class of the callback for the view rendering.
You only need to focus on rewriting the callback as needed. In most cases, only the onCreated method needs to be rewritten.

Build a rendered view

API description

public void createVideoView(String devId);

Parameters

Parameter Description
devId The device ID.

Example

TuyaCameraView mVideoView = findViewById(R.id.camera_video_view);
mVideoView.createVideoView(devId);

Bind a rendered view

API description

void generateCameraView(T view);

Example

iIPCManager?.generateCameraView(mVideoView.createdView())

Register a P2P listener

Registers a listener. Otherwise, video images cannot be displayed as expected.
For more information, see Raw stream data.

API description

void registerP2PCameraListener(AbsP2pCameraListener listener);

Example

// 1. Get `IIPCManager`.
iIPCManager = iTuyaVideoLockManager?.ipcManager
TuyaCameraView cameraView = findViewById(R.id.camera_video_view)
// 2. Set the callback for the view rendering container.
cameraView.setViewCallback(object : AbsVideoViewCallback() {
            override fun onCreated(p0: Any?) {
                super.onCreated(p0)
                iIpcManager?.generateCameraView(p0)
            }
        })
// 3. Build the rendered view.
cameraView.createVideoView(devId)
// 4. Register a P2P listener.
if (null != iIPCManager){
    iIPCManager.registerP2PCameraListener(object: AbsP2pCameraListener() {
        override fun onReceiveAudioBufferData(
                nSampleRate: Int,
                nChannelNum: Int,
                nBitWidth: Int,
                nTimeStamp: Long,
                progress: Long,
                duration: Long
            ) {
                super.onReceiveAudioBufferData(
                    nSampleRate,
                    nChannelNum,
                    nBitWidth,
                    nTimeStamp,
                    progress,
                    duration
                )
            }

            override fun onReceiveFrameYUVData(
                sessionId: Int,
                y: ByteBuffer?,
                u: ByteBuffer?,
                v: ByteBuffer?,
                videoFrameInfo: TuyaVideoFrameInfo?,
                camera: Any?
            ) {
                super.onReceiveFrameYUVData(sessionId, y, u, v, videoFrameInfo, camera)
            }

            override fun onReceiveFrameYUVData(
                sessionId: Int,
                y: ByteBuffer?,
                u: ByteBuffer?,
                v: ByteBuffer?,
                width: Int,
                height: Int,
                nFrameRate: Int,
                nIsKeyFrame: Int,
                timestamp: Long,
                nProgress: Long,
                nDuration: Long,
                camera: Any?
            ) {
                super.onReceiveFrameYUVData(
                    sessionId,
                    y,
                    u,
                    v,
                    width,
                    height,
                    nFrameRate,
                    nIsKeyFrame,
                    timestamp,
                    nProgress,
                    nDuration,
                    camera
                )
            }

            override fun onReceiveSpeakerEchoData(pcm: ByteBuffer?, sampleRate: Int) {
                super.onReceiveSpeakerEchoData(pcm, sampleRate)
            }

            override fun onSessionStatusChanged(camera: Any?, sessionId: Int, sessionStatus: Int) {
                super.onSessionStatusChanged(camera, sessionId, sessionStatus)
            }


        })
    });   
}

P2P connections

Before live video streaming, a P2P connection must be created. The P2P connection status is subject to your maintenance. The SDK can only send commands to and receive responses from the device.

API description

Create a P2P connection

void connect(IResultCallback callBack);

Close a P2P connection

void disconnect(IResultCallback callBack);

Parameters

Parameter Description
callBack The callback.

Whether P2P connection is built

boolean isConnected();

Whether P2P connection in progress

boolean isConnecting();

Play live video

After a P2P connection is created, live video streaming can be started.

API description

Start live video streaming

void startPreview(int clarity, IResultCallback callBack);

Stop live video streaming

int stopPreview(IResultCallback callBack);

Parameters

Parameter Description
clarity The video definition.
callBack The callback.

Video definition

Mode Value
Standard definition 2
High definition 4

Example

iIPCManager.startPreview(new IResultCallback() {
    @Override
    public void onSuccess() {
        // Live video streaming is started.
    }

    @Override
    public void onError(String code, String error) {
        // Failed to start live video streaming.
    }
});

After startPreview returns the success callback, the callback onReceiveFrameYUVData starts receiving video data and transmitting it to the business layer.

Local recording

During live video streaming or record playback, the ongoing videos can be recorded on a mobile phone.

During video recording, do not switch between video definition modes or modify the audio channel switch and live video talk settings.

To save video footage to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.

Enable video recording

API description

void startRecord(String folderPath, Context context, RotateModeEnum rotateMode, IResultCallback callback);

Video recording requires the write permissions of the SD card.

Parameters

Parameter Description
folderPath The path in which videos are stored.
context The context.
rotateMode The rotation mode.
callBack The callback.

Stop video recording

API description

void stopRecord(ITuyaResultCallback<Map<String, String>> callback);

Callback parameter description

In the callback, the video path is returned as the value of the key video, and the cover image is returned as the value of the key coverImage.

Whether recording in progress

API description

boolean isRecording();

Example

if(iIpcManager?.isRecording == true){
	stopRecorder()
}else{
	startRecord()
}

...
fun startRecord(){
	iIpcManager?.startRecord(
                path,
                mContext,
                rotateMode ?: RotateModeEnum.NORMAL,
                object : IResultCallback {
                    override fun onError(code: String?, error: String?) {
                        
                    }

                    override fun onSuccess() {
                        L.i(TAG, "recorder success")
                    }

                })
}
fun stopRecord(){
	iIpcManager?.stopRecord(object : ITuyaResultCallback<Map<String, String>> {
            override fun onSuccess(result: Map<String, String>?) {
                result?.let {
                    val videoPath = it.get("video")
                    val coverImagePath = it.get("coverImage")                    
                }
            }

            override fun onError(errorCode: String?, errorMessage: String?) {
                
            }

        })
}

Video screenshots

Captures screenshots of live video images and stores them on the SD card of a mobile phone.

To save screenshots to a system album, you must implement this feature on your own. Starting from Android 10, scoped storage is applicable. This feature can be disabled for most of these versions but is required by Android 11. To store media files to a system album, the MediaStore API methods must be used.

API description

void snapshot(String folderPath, Context context, ICameraP2P.PLAYMODE playmode, RotateModeEnum rotateMode, ITuyaResultCallback<String> callback);

Parameters

Parameter Description
folderPath The path in which the screenshots are stored.
context The context.
playmode The video playback mode. Currently, only PLAYMODE.LIVE is supported.
rotateMode The rotation angle. Valid values: 0, 90, 180, and 270.
callBack The callback. onSuccess returns the complete path of the screenshot.

Example

iIpcManager?.snapshot(
                path,
                mContext,
                ICameraP2P.PLAYMODE.LIVE,
                rotateMode ?: RotateModeEnum.NORMAL,
                object : ITuyaResultCallback<String> {
                    override fun onSuccess(result: String?) {
                        L.i(TAG, "snapshot success :$result")
                    }

                    override fun onError(errorCode: String?, errorMessage: String?) {
                        
                    }

                })

Audio settings

During live video streaming or video playback, the audio channel can be enabled or disabled. By default, it is disabled.

API description

Enables or disables the audio channel.

void enableMute(boolean isMute,IResultCallback callback);
Audio mode Value
Mute true
Unmuted false

Whether audio enabled

boolean isMuting();

Example

iIpcManager?.enableMute(mute, object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                
            }

            override fun onSuccess() {
                L.i(TAG, "set mute $mute success ")
            }

        })

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Enable or disable video talks

Enables or disables transmission of audio data from the mobile phone to the IPC.

void startTalk(IResultCallback callback);
void stopTalk(IResultCallback callback);

Video talk to be supported or already enabled

boolean isSupportTalk();
boolean isTalkBacking();

Example

iIpcManager?.startTalk(object : IResultCallback {
                override fun onError(code: String?, error: String?) {
                    
                }

                override fun onSuccess() {
                    L.i(TAG, "startTalk success")
                }

            })
iIpcManager?.stopTalk(object : IResultCallback {
            override fun onError(code: String?, error: String?) {
                
            }

            override fun onSuccess() {

            }

        })

Others

API description

Indicates whether the device is equipped with a pickup. If so, videos from the device are audible.

boolean isSupportSound();

API description

Indicates the default video talk mode.

int supportAudioMode();

Raw stream data

The SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format.

API description

To enable the callback of video frames, you must register a listener. You only need to rewrite the callback as preferred.

void registerP2PCameraListener(AbsP2pCameraListener listener);

Main methods of AbsP2pCameraListener:

API description

Executes the callback for video YUV data.

public void onReceiveFrameYUVData(int sessionId, ByteBuffer y, ByteBuffer u, ByteBuffer v, int width, int height, int nFrameRate, int nIsKeyFrame, long timestamp, long nProgress, long nDuration, Object camera)

Parameters

Parameter Description
sessionId The session ID.
Y The luma (Y’) information of video streams.
u The chroma (U) channel information of video streams.
v The chroma (V) channel information of video streams.
width The width of video images.
height The height of video images.
nFrameRate The frame rate of video streams.
nIsKeyFrame Indicates whether a keyframe or an I-frame is used.
timestamp The timestamp.
nProgress The time progress of the video pushed by the Message Center module.
nDuration The duration of the video pushed by the Message Center module.

API description

Executes the callback of P2P connection status.

public void onSessionStatusChanged(Object camera, int sessionId, int sessionStatus)

Parameters

Parameter Description
sessionId The session ID.
sessionStatus The session status.
Session status code Description
0 The connection is created.
-3 The connection timed out.
-12 The connection is closed by the device.
-13 The connection is closed due to timeout after no response is returned.

Destroy

Destroy IIPCManager

If the camera features are not required, the P2P listener must be removed and the P2P object must be destroyed.

void onDestroy();

Destroy IVideoLockManger

If the features of the visual lock and camera lock are not required, the IVideoLockManger and IIPCManager objects must be destroyed.

void onDestroy();