Audio and Video Features

The IPC SDK can provide a bunch of audio and video capabilities in addition to live video streaming and playing back footage on an SD card.

Local recording

During live video streaming or record playback, the ongoing videos can be recorded on a mobile phone.

API description

Records videos and saves them to the album of a mobile phone.

- (void)startRecord;

API description

Saves video records to a specified path. This API method is not supported by P2P 1.0 devices.

- (void)startRecordWithFilePath:(NSString *)filePath;

Parameters

Parameter	Description
filePath	The file path to save video records as MP4 files that are suffixed with .mp4.

API description

Stops recording and saves the records.

- (void)stopRecord;

API description

The delegate callback that is executed after video recording is started.

- (void)cameraDidStartRecord:(id<TuyaSmartCameraType>)camera;

API description

The delegate callback that is executed after video recording is stopped and records are saved.

- (void)cameraDidStopRecord:(id<TuyaSmartCameraType>)camera;

Example

ObjC

- (void)startRecord {
	if (self.isRecording) {
		return;
	}
	// Video recording can be enabled only during video streaming.
	if (self.previewing || self.playbacking) {
		[self.camera startRecord];
		self.recording = YES;
	}
}

- (void)stopRecord {
	if (self.isRecording) {
		[self.camera stopRecord];
		self.recording = NO;
	}
}

- (void)cameraDidStartRecord:(id<TuyaSmartCameraType>)camera {
		// Starts video recording and updates the UI.
}

- (void)cameraDidStopRecord:(id<TuyaSmartCameraType>)camera {
		// Stops video recording and saves records.
}

- (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
	// Failed to start or stop video recording.
	if (errStepCode == TY_ERROR_RECORD_FAILED) {
		self.recording = NO;
	}
}

Swift

func startRecord() {
	if self.isRecording {
		return
	}
	guard self.isPreviewing || self.isPlaybacking else {
		return;
	}
	self.camera.startRecord()
	self.isRecording = true
}

func stopRecord() {
	guard self.isRecording else {
		return
	}
	self.camera.stopRecord()
}

func cameraDidStartRecord(_ camera: TuyaSmartCameraType!) {
	// Starts video recording and updates the UI.
}

func cameraDidStopRecord(_ camera: TuyaSmartCameraType!) {
	// Stops video recording and saves records.
}

func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
	// Failed to start or stop video recording.
	if errStepCode == TY_ERROR_RECORD_FAILED {
		self.isRecording = false
	}
}

Video screenshots

During live video streaming or record playback, users can capture screenshots of video views. The IPC SDK supports three screenshot capturing methods. Two of the following API methods are provided by the TuyaSmartCameraType object.

API description

Captures a video screenshot and saves it to the album of a mobile phone.

- (UIImage *)snapShoot;

Return value

Type	Description
UIImage	The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

API description

Captures a video screenshot and saves it to a specified path. This API method is not supported by P2P 1.0 devices.

- (UIImage *)snapShootSavedAtPath:(NSString *)filePath thumbnilPath:(NSString *)thumbnilPath;

Parameters

Parameter	Description
filePath	The path in which the screenshot is stored.
thumbnilPath	The path to save the thumbnail. Set the value to nil if this parameter is not required.

Return value

Type	Description
UIImage	The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

API description

Calls the screenshot capturing API method of TuyaSmartVideoType that processes video rendering views. Only the UIImage object is returned, and the screenshots are not automatically saved.

- (UIImage *)screenshot;

Return value

Type	Description
UIImage	The UIImage object of a video screenshot. If nil is returned, the system failed to save the screenshot.

Example

ObjC

- (void)snapShoot {
	// Screenshots can be captured only during video playing.
	if (self.previewing || self.playbacking) {
		if ([self.camera snapShoot]) {
			// Saves screenshots to the album of a mobile phone.
		}
	}
}

Swift

func snapShoot() {
	guard self.isPreviewing || self.isPlaybacking else {
		return;
	}
	if let _ = self.camera.snapShoot() {
		// Saves screenshots to the album of a mobile phone.
	}
}

Audio settings

During live video streaming or record playback, the audio channel can be enabled or disabled. By default, it is disabled.

API description

Enables or disables the audio channel.

- (void)enableMute:(BOOL)mute forPlayMode:(TuyaSmartCameraPlayMode)playMode;

Parameters

Parameter	Description
mute	Specifies whether to mute videos. Valid values: YES: mutes videos. NO: unmutes videos.
playMode	The current playing mode.

API description

The delegate callback of the audio status.

- (void)camera:(id<TuyaSmartCameraType>)camera didReceiveMuteState:(BOOL)isMute playMode:(TuyaSmartCameraPlayMode)playMode;

Parameters

Parameter	Description
camera	The Camera object that enables or disables the audio channel.
isMute	Indicates whether videos are muted.
playMode	The current playing mode.

API description

Switches between the speaker and earpiece modes. If the return value is not 0, the switching operation failed. This API method is not supported by P2P 1.0.

- (int)enableSpeaker:(BOOL)enabled;

Parameters

Parameter	Description
enabled	Valid values: YES: switches to the speaker mode. NO: switches to the earpiece mode.

API description

Returns the current audio playing mode. This API method is not supported by P2P 1.0. Valid values:

• YES: the speaker mode.
• NO: the earpiece mode.

- (BOOL)speakerEnabled;

Video bitrate

Live video bitrate can be obtained after videos are streamed.

API description

Returns the live video bitrate.

- (double)getVideoBitRateKBPS;

Live video talk

After a P2P connection is created, the live video talk feature can be enabled to talk to an IP camera (IPC). Before the talk, the app must be granted access to the microphone of the mobile phone.

Determine support for video talk

Indicates whether the device is equipped with a speaker. If so, the device supports video talk. If the device is equipped with both a speaker and a pickup, video streams are audible, so two-way talk is supported. You can query device capabilities to check whether the device supports a speaker or pickup.

Two-way talk

During live video streaming, the audio channel can be enabled. In this case, the audible sound is the human voice and ambient sound collected by the IPC in real time. Then, enable the audio channel from the app to the IPC, so two-way talk can be implemented.

One-way talk

The control of one-way talk is subject to your implementation. After the one-way talk is enabled, the video is muted. After the one-way talk is disabled, the video is unmuted.

API description

Enables the audio channel from the app to the IPC.

- (void)startTalk;

API description

Disables the audio channel from the app to the IPC.

- (void)stopTalk;

API description

The delegate callback that is executed after the audio channel from the app to the IPC is enabled.

- (void)cameraDidBeginTalk:(id<TuyaSmartCameraType>)camera;

API description

The delegate callback that is executed after the audio channel from the app to the IPC is disabled.

- (void)cameraDidStopTalk:(id<TuyaSmartCameraType>)camera;

Example

The following example of one-way talk shows the API calls to use the audio switch and live video talk.

ObjC

- (void)startTalk {
	[self.camera startTalk];
	// Disables the audio channel if videos are not muted.
	if (!self.isMuted) {
		[self.camera enableMute:YES forPlayMode:TuyaSmartCameraPlayModePreview];
	}
}

- (void)stopTalk {
	[self.camera stopTalk];
}

- (void)cameraDidBeginTalk:(id<TuyaSmartCameraType>)camera {
		// Starts live video talk.
}

- (void)cameraDidStopTalk:(id<TuyaSmartCameraType>)camera {
		// Stops live video talk.
		// Enables the audio channel if videos are muted.
	if (self.isMuted) {
		[self.camera enableMute:NO forPlayMode:TuyaSmartCameraPlayModePreview];
	}
}

- (void)camera:(id<TuyaSmartCameraType>)camera didReceiveMuteState:(BOOL)isMute playMode:(TuyaSmartCameraPlayMode)playMode {
		// Receives the mute status changes and updates the UI.
		self.isMuted = isMute;
}

- (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
	if (errStepCode == TY_ERROR_START_TALK_FAILED) {
			// Failed to start live video talk and enables the audio channel again.
				if (self.isMuted) {
				[self.camera enableMute:NO forPlayMode:TuyaSmartCameraPlayModePreview];
			}
	}
	else if (errStepCode == TY_ERROR_ENABLE_MUTE_FAILED) {
				// Failed to enable the muted state.
	}
}

Swift

// The live video streaming mode.
func startTalk() {
	self.camera.startTalk()
	guard self.isMuted else {
		self.camera.enableMute(true, for: .preview)
		return
	}
}

func stopTalk() {
	self.camera.stopTalk()
}

func cameraDidBeginTalk(_ camera: TuyaSmartCameraType!) {
	// Starts live video talk.
}

func cameraDidStopTalk(_ camera: TuyaSmartCameraType!) {
	// Stops live video talk.
	if self.isMuted {
		self.camera.enableMute(false, for: .preview)
	}
}

func camera(_ camera: TuyaSmartCameraType!, didReceiveMuteState isMute: Bool, playMode: TuyaSmartCameraPlayMode) {
	self.isMuted = isMute
	// Receives the audio status changes and updates the UI.
}

func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
	if errStepCode == TY_ERROR_START_TALK_FAILED {
		// Failed to start video talk and enables the audio channel again.
		self.camera.enableMute(false, for: .preview)
	}else if errStepCode == TY_ERROR_ENABLE_MUTE_FAILED {
		// Failed to enable the muted state.
	}
}

Definition modes

During live video streaming, users can switch between video definition modes for most mobile phones. Only a few mobile phone models support only one video definition mode. Currently, only high definition (HD) and standard definition (SD) modes are available, and they are supported only when live videos are streamed. Only one video definition mode is supported for storing footage on the SD card.

API description

Returns the definition mode of current video images by using a delegate method.

- (void)getHD;

API description

Switches between the video definition modes. Valid values:

• YES: switches to the HD mode.
• NO: switches to the SD mode.

  - (void)enableHD:(BOOL)hd;
  

  Parameters

  Parameter	Description
  hd	Specifies whether to enable the HD mode. Valid values: YES: enables the HD mode. NO: enables the SD mode.

  API description

  The delegate callback that is executed after the video definition mode is changed.

  - (void)camera:(id<TuyaSmartCameraType>)camera didReceiveDefinitionState:(BOOL)isHd;
  

  Parameters

  Parameter	Description
  camera	The Camera object that switches between the definition modes.
  isHd	Specifies whether the HD mode is enabled. Valid values: YES: The HD mode is enabled. NO: The SD mode is enabled.

  The preceding API methods are supported by P2P 1.0 devices. However, in v3.20.0, these API methods are deprecated and replaced with the following three API methods that are not supported by P2P 1.0 devices.

  API description

  Returns video definition.

  - (void)getDefinition;
  

  API description

  Sets video definition.

  - (void)setDefinition:(TuyaSmartCameraDefinition)definition;
  

  API description

  The delegate callback that is executed after the video definition mode is changed.

  - (void)camera:(id<TuyaSmartCameraType>)camera definitionChanged:(TuyaSmartCameraDefinition)definition;
  

  Enum values of TuyaSmartCameraDefinition

  Value	Description
  TuyaSmartCameraDefinitionProflow	Bandwidth efficient streaming
  TuyaSmartCameraDefinitionStandard	SD
  TuyaSmartCameraDefinitionHigh	HD
  TuyaSmartCameraDefinitionSuper	Ultra HD
  TuyaSmartCameraDefinitionSSuper	Super Ultra HD

  The supported definition modes vary depending on different devices. Currently, generic devices only support SD and HD.

  Example

  ObjC

  - (void)changeHD {
  		TuyaSmartCameraDefinition definition = self.HD ? TuyaSmartCameraDefinitionStandard : TuyaSmartCameraDefinitionHigh;
  		[self.camera setDefinition:definition];
  }
  
  // The delegate callback that is executed after the video resolution is changed. It is also executed when live video streaming or video playback is started.
  - (void)camera:(id<TuyaSmartCameraType>)camera resolutionDidChangeWidth:(NSInteger)width height:(NSInteger)height {
  		// Returns the current definition mode.
  		[self.camera getDefinition];
  }
  
  // The delegate callback that is executed when the video definition mode is changed.
  - (void)camera:(id<TuyaSmartCameraType>)camera definitionChanged:(TuyaSmartCameraDefinition)definition {
  	self.HD = definition >= TuyaSmartCameraDefinitionHigh;
  }
  
  - (void)camera:(id<TuyaSmartCameraType>)camera didOccurredErrorAtStep:(TYCameraErrorCode)errStepCode specificErrorCode:(NSInteger)errorCode {
  	if (errStepCode == TY_ERROR_ENABLE_HD_FAILED) {
  				// Failed to switch between the video definition modes.
  	}
  }
  

  Swift

  func changeHD() {
  	let definition = self.isHD ? TuyaSmartCameraDefinition.standard : TuyaSmartCameraDefinition.high
  	self.camera.setDefinition(definition)
  }
  
  // The delegate callback that is executed after the video resolution is changed. It is also executed when live video streaming or video playback is started.
  func camera(_ camera: TuyaSmartCameraType!, resolutionDidChangeWidth width: Int, height: Int) {
  	// Returns the current definition mode.
  	self.camera.getDefinition()
  }
  
  func camera(_ camera: TuyaSmartCameraType!, definitionChanged definition: TuyaSmartCameraDefinition) {
  		self.isHD = definition.rawValue >= TuyaSmartCameraDefinition.high.rawValue
  }
  
  func camera(_ camera: TuyaSmartCameraType!, didOccurredErrorAtStep errStepCode: TYCameraErrorCode, specificErrorCode errorCode: Int) {
  	if errStepCode == TY_ERROR_ENABLE_HD_FAILED {
  		// Failed to switch between the video definition modes.
  	}
  }
  

  Raw stream data

  The IPC SDK provides the callback that returns raw stream data, including the YUV data of video frames. YUV 420SP is used as the color encoding format. This format is implemented by kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange for iOS.

  API description

  The delegate callback that is executed to return raw stream data of video frames.

  - (void)camera:(id<TuyaSmartCameraType>)camera ty_didReceiveVideoFrame:(CMSampleBufferRef)sampleBuffer frameInfo:(TuyaSmartVideoFrameInfo)frameInfo;
  

  Parameters

  Parameter	Description
  camera	The Camera object that receives video data.
  sampleBuffer	The YUV data of video frames.
  frameInfo	The information about video frames.

  TuyaSmartVideoFrameInfo structure

  Field	Type	Description
  nWidth	int	The width of video images.
  nHeight	int	The height of video images.
  nFrameRate	int	The video frame rate.
  nTimeStamp	unsigned long long	The timestamp of a video frame.
  nDuration	unsigned long long	The total duration of the video attached to an alert. Unit: milliseconds.
  nProgress	unsigned long long	The time point of a video frame in the video attached to an alert. Unit: milliseconds.

  You can render video images with your own method, or further process video images. For this purpose, set the autoRender property of the TuyaSmartCameraType object to NO, and implement the delegate method. This way, the IPC SDK does not automatically render video images.

  You can force sampleBuffer into CVPixelBufferRef. To asynchronously process video frame data, remember to retain the pixel buffer. Otherwise, the video frame data will be released after the delegate method is executed. This will cause a wild pointer exception during asynchronous processing.

  Example

  ObjC

  - (void)camera:(id<TuyaSmartCameraType>)camera ty_didReceiveVideoFrame:(CMSampleBufferRef)sampleBuffer frameInfo:(TuyaSmartVideoFrameInfo)frameInfo {
  	CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)sampleBuffer;
  	// Retains `pixelBuffer` to avoid unexpected release.
  	CVPixelBufferRetain(pixelBuffer);
  	dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
  		// Processes and renders `pixelBuffer`.
  		// ...
  		// Releases `pixelBuffer`.
  		CVPixelBufferRelease(pixelBuffer);
  	});
  }
  

  Swift

  func camera(_ camera: TuyaSmartCameraType!, ty_didReceiveVideoFrame sampleBuffer: CMSampleBuffer!, frameInfo: TuyaSmartVideoFrameInfo) {
  		// Processes and renders video data.
  }
  

  IVA

  The IVA feature can be enabled to support intelligent detection. If the IPC detects an object in motion during live video streaming, the object will be automatically framed in white on the view.

  To achieve this purpose, the IVA features must be enabled for the IPC first. It will then report the coordinates of the object along with the video frames. You can use the data point (DP) ID 198 to enable this feature.

  if ([self.dpManager isSupportDP:@"198"]) {
  	[self.dpManager setValue:@(YES) forDP:@"198" success:nil failure:nil];
  }
  

  After IVA is enabled for the IPC, this feature must also be enabled for the IPC SDK during live video streaming. This allows the SDK to frame the object in white on the view based on the received coordinates of the object.

  API description

  Enables or disables the IVA feature.

  - (void)setOutLineEnable:(BOOL)enable;
  

  Parameters

  Parameter	Description
  enable	Specifies whether to enable IVA.

  Device capabilities

  The TuyaSmartCameraAbility class can be used to parse device configurations and get basic device capabilities.

  Property	Description
  defaultDefinition	Indicates the default definition of live video streaming.
  videoNum	Indicates the number of streams supported by the device. If the value is 1, the device only supports one definition mode. Switching between definition modes is not supported. The defaultDefinition property indicates the only definition mode. If the value is 2, the device supports SD and HD.
  isSupportSpeaker	Indicates whether the device is equipped with a speaker. If so, the device supports video talk.
  isSupportPickup	Indicates whether the device is equipped with a pickup. If so, the audio channel can be enabled when video streams are previewed on the app.
  rowData	Returns raw data of P2P configurations.

  API description

  Creates the device capabilities class object based on a device data model. This API method is called after a P2P connection is created. After the initial P2P connection, the raw data of P2P configurations is cached in a local sandbox.

  + (instancetype)cameraAbilityWithDeviceModel:(TuyaSmartDeviceModel *)deviceModel;