Hello,
I have implemented Low-Latency Frame Interpolation using the VTFrameProcessor framework, based on the sample code from https://developer.apple.com/kr/videos/play/wwdc2025/300. It is currently working well for both LIVE and VOD streams.
However, I have a few questions regarding the lifecycle management and synchronization of this feature:
1. Common Questions (Applicable to both Frame Interpolation & Super Resolution)
1.1 Dynamic Toggling
- Do you recommend enabling/disabling these features dynamically during playback?
- Or is it better practice to configure them only during the initial setup/preparation phase?
- If dynamic toggling is supported, are there any recommended patterns for managing VTFrameProcessor session lifecycle (e.g., startSession / endSession timing)?
1.2 Synchronization Method
- I am currently using CADisplayLink to fetch frames from AVPlayerItemVideoOutput and perform processing.
- Is CADisplayLink the recommended approach for real-time frame acquisition with VTFrameProcessor?
- If the feature needs to be toggled on/off during active playback, are there any concerns or alternative approaches you would recommend?
1.3 Supported Resolution/Quality Range
- What are the minimum and maximum video resolutions supported for each feature?
- Are there any aspect ratio restrictions (e.g., does it support 1:1 square videos)?
- Is there a recommended resolution range for optimal performance and quality?
2. Frame Interpolation Specific Questions
2.1 LIVE Stream Support
- Is Low-Latency Frame Interpolation suitable for LIVE streaming scenarios where latency is critical?
- Are there any special considerations for LIVE vs VOD?
3. Super Resolution Specific Questions
3.1 Adaptive Bitrate (ABR) Stream Support
- In ABR (HLS/DASH) streams, the video resolution can change dynamically during playback.
- Is VTLowLatencySuperResolutionScaler compatible with ABR streams where resolution changes mid-playback?
- If resolution changes occur, should I recreate the VTLowLatencySuperResolutionScalerConfiguration and restart the session, or does the API handle this automatically?
3.2 Small/Square Resolution Issue
- I observed that 144x144 (1:1 square) videos fail with error:
"VTFrameProcessorErrorDomain Code=-19730: processWithSourceFrame within VCPFrameSuperResolutionProcessor failed"
- However, 480x270 (16:9) videos work correctly.
- minimumDimensions reports 96x96, but 144x144 still fails. Is there an undocumented restriction on aspect ratio or a practical minimum resolution?
3.3 Scale Factor Selection
- supportedScaleFactors returns [2.0, 4.0] for most resolutions.
- Is there a recommended scale factor for balancing quality and performance?
- Are there scenarios where 4.0x should be avoided?
The documentation on this specific topic seems limited, so I would appreciate any insights or advice.
Thank you.