English
English
简体中文
Contact Us
Register
Log In
layoutIndex

Audio and Video Features

Last Updated on : 2022-10-31 12:19:42download

The IPC SDK can provide a bunch of audio and video capabilities in addition to live video streaming and playing back footage on an SD card.

Local recording

During live video streaming or record playback, the ongoing videos can be recorded on a mobile phone.

API description

Records videos and saves them to the album of a mobile phone.

- (void)startRecord;

API description

Saves video records to a specified path. This API method is not supported by P2P 1.0 devices.

- (void)startRecordWithFilePath:(NSString *)filePath;

Parameters

Parameter Description
filePath The file path to save video records as MP4 files that are suffixed with .mp4.

API description

Stops recording and saves the records.

- (void)stopRecord;

API description

The delegate callback that is executed after video recording is started.

- (void)cameraDidStartRecord:(id<TuyaSmartCameraType>)camera;

API description

The delegate callback that is executed after video recording is stopped and records are saved.

- (void)cameraDidStopRecord:(id<TuyaSmartCameraType>)camera;

The call to start or stop recording might fail. In this case, call the delegate method -(void)camera:didOccurredErrorAtStep: specificErrorCode:; to return specific error messages.

Example

ObjC

- (void)startRecord {
	if (self.isRecording) {
		return;
	}
	// Video recording can be enabled only during video streaming.
	if (self.previewing || self.playbacking) {
		[self.camera startRecord];
		self.recording = YES;
	}
}

- (void)stopRecord {
	if (self.isRecording) {
		[self.camera stopRecord];
		self.recording = NO;
	}
}

- (void)cameraDidStartRecord:(id<TuyaSmartCameraType>)camera {
		// Starts video recording and updates the UI.
}

- (void)cameraDidStopRecord:(id<TuyaSmartCameraType>)camera {
		// Stops video recording and saves records.
}

- (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
	// Failed to start or stop video recording.
	if (errStepCode == TY_ERROR_RECORD_FAILED) {
		self.recording = NO;
	}
}

Swift

func startRecord() {
	if self.isRecording {
		return
	}
	guard self.isPreviewing || self.isPlaybacking else {
		return;
	}
	self.camera.startRecord()
	self.isRecording = true
}

func stopRecord() {
	guard self.isRecording else {
		return
	}
	self.camera.stopRecord()
}

func cameraDidStartRecord(_ camera: TuyaSmartCameraType!) {
	// Starts video recording and updates the UI.
}

func cameraDidStopRecord(_ camera: TuyaSmartCameraType!) {
	// Stops video recording and saves records.
}

func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
	// Failed to start or stop video recording.
	if errStepCode == TY_ERROR_RECORD_FAILED {
		self.isRecording = false
	}
}

During video recording, do not switch between video definition modes or modify the audio channel switch and live video talk settings.

Video screenshots

During live video streaming or record playback, users can capture screenshots of video views. The IPC SDK supports three screenshot capturing methods. Two of the following API methods are provided by the TuyaSmartCameraType object.

API description

Captures a video screenshot and saves it to the album of a mobile phone.

- (UIImage *)snapShoot;

Return value

Type Description
UIImage The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

API description

Captures a video screenshot and saves it to a specified path. This API method is not supported by P2P 1.0 devices.

- (UIImage *)snapShootSavedAtPath:(NSString *)filePath thumbnilPath:(NSString *)thumbnilPath;

Parameters

Parameter Description
filePath The path in which the screenshot is stored.
thumbnilPath The path to save the thumbnail. Set the value to nil if this parameter is not required.

Return value

Type Description
UIImage The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

API description

Calls the screenshot capturing API method of TuyaSmartVideoType that processes video rendering views. Only the UIImage object is returned, and the screenshots are not automatically saved.

- (UIImage *)screenshot;

Return value

Type Description
UIImage The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

Example

ObjC

- (void)snapShoot {
	// Screenshots can be captured only during video playing.
	if (self.previewing || self.playbacking) {
		if ([self.camera snapShoot]) {
			// Saves screenshots to the album of a mobile phone.
		}
	}
}

Swift

func snapShoot() {
	guard self.isPreviewing || self.isPlaybacking else {
		return;
	}
	if let _ = self.camera.snapShoot() {
		// Saves screenshots to the album of a mobile phone.
	}
}

To enable video recording and screenshot capturing, the app must be granted access to the album of the mobile phone. Otherwise, the app might crash.

Audio settings

During live video streaming or record playback, the audio channel can be enabled or disabled. By default, it is disabled.

API description

Enables or disables the audio channel.

- (void)enableMute:(BOOL)mute forPlayMode:(TuyaSmartCameraPlayMode)playMode;

Parameters

Parameter Description
mute Specifies whether to mute videos. Valid values:
  • YES: mutes videos.
  • NO: unmutes videos.
playMode The current playing mode.

API description

The delegate callback of the audio status.

- (void)camera:(id<TuyaSmartCameraType>)camera didReceiveMuteState:(BOOL)isMute playMode:(TuyaSmartCameraPlayMode)playMode;

Parameters

Parameter Description
camera The Camera object that enables or disables the audio channel.
isMute Indicates whether videos are muted.
playMode The current playing mode.

API description

Switches between the speaker and earpiece modes. If the return value is not 0, the switching operation failed. This API method is not supported by P2P 1.0.

- (int)enableSpeaker:(BOOL)enabled;

Parameters

Parameter Description
enabled Valid values:
  • YES: switches to the speaker mode.
  • NO: switches to the earpiece mode.

API description

Returns the current audio playing mode. This API method is not supported by P2P 1.0. Valid values:

  • YES: the speaker mode.
  • NO: the earpiece mode.

- (BOOL)speakerEnabled;

After the system switches between live video streaming and video playback, the IPC SDK does not memorize the audio status of the previous playing mode. For example, the audio channel is enabled for live video streaming. After the playing mode is switched to video playback, videos are still audible. Then, disable the audio channel and switch back to the live video streaming mode. Videos are thus inaudible. Therefore, after switching between the playing modes, the audio status must be adjusted as expected.

Video bitrate

Live video bitrate can be obtained after videos are streamed.

API description

Returns the live video bitrate.

- (double)getVideoBitRateKBPS;

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Determine support for video talk

Indicates whether the device is equipped with a speaker. If so, the device supports video talk. If the device is equipped with both a speaker and a pickup, video streams are audible, so two-way talk is supported. You can query device capabilities to check whether the device supports a speaker or pickup.

Two-way talk

During live video streaming, the audio channel can be enabled. In this case, the audible sound is the human voice and ambient sound collected by the IPC in real time. Then, enable the audio channel from the app to the IPC, so two-way talk can be implemented.

Certain IPCs might not have speakers or pickups. Such cameras do not support two-way talk.

One-way talk

The control of one-way talk is subject to your implementation. After the one-way talk is enabled, the video is muted. After the one-way talk is disabled, the video is unmuted.

API description

Enables the audio channel from the app to the IPC.

- (void)startTalk;

API description

Disables the audio channel from the app to the IPC.

- (void)stopTalk;

API description

The delegate callback that is executed after the audio channel from the app to the IPC is enabled.

- (void)cameraDidBeginTalk:(id<TuyaSmartCameraType>)camera;

API description

The delegate callback that is executed after the audio channel from the app to the IPC is disabled.

- (void)cameraDidStopTalk:(id<TuyaSmartCameraType>)camera;

Example

The following example of one-way talk shows the API calls to use the audio switch and live video talk.

ObjC

- (void)startTalk {
	[self.camera startTalk];
	// Disables the audio channel if videos are not muted.
	if (!self.isMuted) {
		[self.camera enableMute:YES forPlayMode:TuyaSmartCameraPlayModePreview];
	}
}

- (void)stopTalk {
	[self.camera stopTalk];
}

- (void)cameraDidBeginTalk:(id<TuyaSmartCameraType>)camera {
		// Starts live video talk.
}

- (void)cameraDidStopTalk:(id<TuyaSmartCameraType>)camera {
		// Stops live video talk.
		// Enables the audio channel if videos are muted.
	if (self.isMuted) {
		[self.camera enableMute:NO forPlayMode:TuyaSmartCameraPlayModePreview];
	}
}

- (void)camera:(id<TuyaSmartCameraType>)camera didReceiveMuteState:(BOOL)isMute playMode:(TuyaSmartCameraPlayMode)playMode {
		// Receives the mute status changes and updates the UI.
		self.isMuted = isMute;
}

- (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
	if (errStepCode == TY_ERROR_START_TALK_FAILED) {
			// Failed to start live video talk and enables the audio channel again.
				if (self.isMuted) {
				[self.camera enableMute:NO forPlayMode:TuyaSmartCameraPlayModePreview];
			}
	}
	else if (errStepCode == TY_ERROR_ENABLE_MUTE_FAILED) {
				// Failed to enable the muted state.
	}
}

Swift

// The live video streaming mode.
func startTalk() {
	self.camera.startTalk()
	guard self.isMuted else {
		self.camera.enableMute(true, for: .preview)
		return
	}
}

func stopTalk() {
	self.camera.stopTalk()
}

func cameraDidBeginTalk(_ camera: TuyaSmartCameraType!) {
	// Starts live video talk.
}

func cameraDidStopTalk(_ camera: TuyaSmartCameraType!) {
	// Stops live video talk.
	if self.isMuted {
		self.camera.enableMute(false, for: .preview)
	}
}

func camera(_ camera: TuyaSmartCameraType!, didReceiveMuteState isMute: Bool, playMode: TuyaSmartCameraPlayMode) {
	self.isMuted = isMute
	// Receives the audio status changes and updates the UI.
}

func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
	if errStepCode == TY_ERROR_START_TALK_FAILED {
		// Failed to start video talk and enables the audio channel again.
		self.camera.enableMute(false, for: .preview)
	}else if errStepCode == TY_ERROR_ENABLE_MUTE_FAILED {
		// Failed to enable the muted state.
	}
}

Definition modes

During live video streaming, users can switch between video definition modes for most mobile phones. Only a few mobile phone models support only one video definition mode. Currently, only high definition (HD) and standard definition (SD) modes are available, and they are supported only when live videos are streamed. Only one video definition mode is supported for storing footage on the SD card.

API description

Returns the definition mode of current video images by using a delegate method.

- (void)getHD;

API description

Switches between the video definition modes. Valid values:

  • YES: switches to the HD mode.
  • NO: switches to the SD mode.

    - (void)enableHD:(BOOL)hd;
    

    Parameters

    Parameter Description
    hd Specifies whether to enable the HD mode. Valid values:
    • YES: enables the HD mode.
    • NO: enables the SD mode.

    API description

    The delegate callback that is executed after the video definition mode is changed.

    - (void)camera:(id<TuyaSmartCameraType>)camera didReceiveDefinitionState:(BOOL)isHd;
    

    Parameters

    Parameter Description
    camera The Camera object that switches between the definition modes.
    isHd Specifies whether the HD mode is enabled. Valid values:
    • YES: The HD mode is enabled.
    • NO: The SD mode is enabled.

    The preceding API methods are supported by P2P 1.0 devices. However, in v3.20.0, these API methods are deprecated and replaced with the following three API methods that are not supported by P2P 1.0 devices.

    API description

    Returns video definition.

    - (void)getDefinition;
    

    API description

    Sets video definition.

    - (void)setDefinition:(TuyaSmartCameraDefinition)definition;
    

    API description

    The delegate callback that is executed after the video definition mode is changed.

    - (void)camera:(id<TuyaSmartCameraType>)camera definitionChanged:(TuyaSmartCameraDefinition)definition;
    

    Enum values of TuyaSmartCameraDefinition

    Value Description
    TuyaSmartCameraDefinitionProflow Bandwidth efficient streaming
    TuyaSmartCameraDefinitionStandard SD
    TuyaSmartCameraDefinitionHigh HD
    TuyaSmartCameraDefinitionSuper Ultra HD
    TuyaSmartCameraDefinitionSSuper Super Ultra HD

    The supported definition modes vary depending on different devices. Currently, generic devices only support SD and HD.

    Example

    ObjC

    - (void)changeHD {
    		TuyaSmartCameraDefinition definition = self.HD ? TuyaSmartCameraDefinitionStandard : TuyaSmartCameraDefinitionHigh;
    		[self.camera setDefinition:definition];
    }
    
    // The delegate callback that is executed after the video resolution is changed. It is also executed when live video streaming or video playback is started.
    - (void)camera:(id<TuyaSmartCameraType>)camera resolutionDidChangeWidth:(NSInteger)width height:(NSInteger)height {
    		// Returns the current definition mode.
    		[self.camera getDefinition];
    }
    
    // The delegate callback that is executed when the video definition mode is changed.
    - (void)camera:(id<TuyaSmartCameraType>)camera definitionChanged:(TuyaSmartCameraDefinition)definition {
    	self.HD = definition >= TuyaSmartCameraDefinitionHigh;
    }
    
    - (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
    	if (errStepCode == TY_ERROR_ENABLE_HD_FAILED) {
    				// Failed to switch between the video definition modes.
    	}
    }
    

    Swift

    func changeHD() {
    	let definition = self.isHD ? TuyaSmartCameraDefinition.standard : TuyaSmartCameraDefinition.high
    	self.camera.setDefinition(definition)
    }
    
    // The delegate callback that is executed after the video resolution is changed. It is also executed when live video streaming or video playback is started.
    func camera(_ camera: TuyaSmartCameraType!, resolutionDidChangeWidth width: Int, height: Int) {
    	// Returns the current definition mode.
    	self.camera.getDefinition()
    }
    
    func camera(_ camera: TuyaSmartCameraType!, definitionChanged definition: TuyaSmartCameraDefinition) {
    		self.isHD = definition.rawValue >= TuyaSmartCameraDefinition.high.rawValue
    }
    
    func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
    	if errStepCode == TY_ERROR_ENABLE_HD_FAILED {
    		// Failed to switch between the video definition modes.
    	}
    }
    

    Raw stream data

    The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format. This format is implemented by kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange for iOS.

    API description

    The delegate callback that is executed to return raw stream data of video frames.

    - (void)camera:(id<TuyaSmartCameraType>)camera ty_didReceiveVideoFrame:(CMSampleBufferRef)sampleBuffer frameInfo:(TuyaSmartVideoFrameInfo)frameInfo;
    

    Parameters

    Parameter Description
    camera The Camera object that receives video data.
    sampleBuffer The YUV data of video frames.
    frameInfo The information about video frames.

    TuyaSmartVideoFrameInfo structure

    Field Type Description
    nWidth int The width of video images.
    nHeight int The height of video images.
    nFrameRate int The video frame rate.
    nTimeStamp unsigned long long The timestamp of a video frame.
    nDuration unsigned long long The total duration of the video attached to an alert. Unit: milliseconds.
    nProgress unsigned long long The time point of a video frame in the video attached to an alert. Unit: milliseconds.

    You can render video images with your own method, or further process video images. For this purpose, set the autoRender property of the TuyaSmartCameraType object to NO, and implement the delegate method. This way, the IPC SDK does not automatically render video images.

    You can force sampleBuffer into CVPixelBufferRef. To asynchronously process video frame data, remember to retain the pixel buffer. Otherwise, the video frame data will be released after the delegate method is executed. This will cause a wild pointer exception during asynchronous processing.

    Example

    ObjC

    - (void)camera:(id<TuyaSmartCameraType>)camera ty_didReceiveVideoFrame:(CMSampleBufferRef)sampleBuffer frameInfo:(TuyaSmartVideoFrameInfo)frameInfo {
    	CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)sampleBuffer;
    	// Retains `pixelBuffer` to avoid unexpected release.
    	CVPixelBufferRetain(pixelBuffer);
    	dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    		// Processes and renders `pixelBuffer`.
    		// ...
    		// Releases `pixelBuffer`.
    		CVPixelBufferRelease(pixelBuffer);
    	});
    }
    

    Swift

    func camera(_ camera: TuyaSmartCameraType!, ty_didReceiveVideoFrame sampleBuffer: CMSampleBuffer!, frameInfo: TuyaSmartVideoFrameInfo) {
    		// Processes and renders video data.
    }
    

    IVA

    The IVA feature can be enabled to support intelligent detection. If the IPC detects an object in motion during live video streaming, the object will be automatically framed in white on the view.

    To achieve this purpose, the IVA features must be enabled for the IPC first. It will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 to enable this feature.

    if ([self.dpManager isSupportDP:@"198"]) {
    	[self.dpManager setValue:@(YES) forDP:@"198" success:nil failure:nil];
    }
    

    After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live video streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

    API description

    Enables or disables the IVA feature.

    - (void)setOutLineEnable:(BOOL)enable;
    

    Parameters

    Parameter Description
    enable Specifies whether to enable IVA.

    Device capabilities

    The TuyaSmartCameraAbility class can be used to parse device configurations and get basic device capabilities.

    Property Description
    defaultDefinition Indicates the default definition of live video streaming.
    videoNum Indicates the number of streams supported by the device. If the value is 1, the device only supports one definition mode. Switching between definition modes is not supported. The defaultDefinition property indicates the only definition mode. If the value is 2, the device supports SD and HD.
    isSupportSpeaker Indicates whether the device is equipped with a speaker. If so, the device supports video talk.
    isSupportPickup Indicates whether the device is equipped with a pickup. If so, the audio channel can be enabled when video streams are previewed on the app.
    rowData Returns raw data of P2P configurations.

    API description

    Creates the device capabilities class object based on a device data model. This API method is called after a P2P connection is created. After the initial P2P connection, the raw data of P2P configurations is cached in a local sandbox.

    + (instancetype)cameraAbilityWithDeviceModel:(TuyaSmartDeviceModel *)deviceModel;