Hello, I am getting the following errors when building a Mac Camera Extension with web sockets. I am using URLSessionWebsocketTask as my web socket library. I built a test program for my code and in there I can see my web sockets are working properly, but when I run it from the System Extension I get the following errors. The socket opens for two - three messages then crashes. I couldnt find any documentation online for the following errors CMIOExtensionProvider.m:1975:-[CMIOExtensionProvider removeProviderContext:]_block_invoke Unregistered provider context <CMIOExtensionProviderContext: ->, don't be surprised if things go badly CMIOExtensionProviderContext.m:64:-[CMIOExtensionProviderContext initWithConnection:]_block_invoke [391] received Connection invalid``

I struggle to add custom properties to my streams as described in the WWDC22 video https://developer.apple.com/videos/play/wwdc2022/10022/ minute 28:17 The speaker describes using this technique in his CIFilterCam demo (would the source code be available please?) to let the app control which filter the extension should apply. Presumably, there's thus a way to: 1 - define a custom property in the camera extension's stream/device/provider? 2 - be able to use CoreMediaIO (from Swift?) in the app in order to set values of that custom property. This is not documented anywhere I could find. Help and sample code would be greatly appreciated. Thank you. Laurent

My goal is to implement a moving background in a virtual camera, implemented as a Camera Extension, on macOS 13 and later. The moving background is available to the extension as a H.264 file in its bundle. I thought i could create an AVAsset from the movie's URL, make an AVPlayerItem from the asset, attach an AVQueuePlayer to the item, then attach an AVPlayerLooper to the queue player. I make an AVPlayerVideoOutput and add it to each of the looper's items, and set a delegate on the video output. This works in a normal app, which I use as a convenient environment to debug my extension code. In my camera video rendering loop, I check self.videoOutput.hasNewPixelBuffer , it returns true at regular intervals, I can fetch video frames with the video output's copyPixelBuffer and composite those frames with the camera frames. However, it doesn't work in an extension - hasNewPixelBuffer is never true. The looping player returns 'failed', with an error which simply says "the operation could not be completed". I've tried simplifying things by removing the AVPlayerLooper and using an AVPlayer instead of an AVQueuePlayer, so the movie would only play once through. But still, I never get any frames in the extension. Could this be a sandbox thing, because an AVPlayer usually renders to a user interface, and camera extensions don't have UIs? My fallback solution is to use an AVAssetImageGenerator which I attempt to drive by firing off a Task for each frame each time I want to render one, I ask for another frame to keep the pipeline full. Unfortunately the Tasks don't finish in the same order they are started so I have to build frame-reordering logic into the frame buffer (something which a player would fix for me). I'm also not sure whether the AVAssetImageGenerator is taking advantage of any hardware acceleration, and it seems inefficient because each Task is for one frame only, and cannot maintain any state from previous frames. Perhaps there's a much simpler way to do this and I'm just missing it? Anyone?

Background: Our CMIOExtensionDevice may change its supported formats in run time when video source changes. (i.e. set of resolutions change). A custom callback is triggered once this happens. According to Apple doc CMIOExtensionStreamSource … “formats should not change during the life cycle of the associated stream” . So we can’t update formats of existing CMIOExtensionStreamSource once it has been created. Initially, we thought stopping the stream -> removing the stream-> adding new stream -> starting the new stream would accomplish this. CMIOExtensionStreamSource::stopStreamAndReturnError CMIOExtensionDevice::removeStream CMIOExtensionDevice::addStream CMIOExtensionStreamSource::startStreamAndReturnError but we ran into strange CMIO behaviors. In the os_log we could see error messages like CMIOExtensionProvider.m:3085:-[CMIOExtensionProvider stopStreamForClientID:streamID:reply:]_block_invoke Invalid streamID CMIOExtensionProviderContext.m:887:-[CMIOExtensionProviderContext stopStream:message:]_block_invoke stopStream failed Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list" UserInfo={NSLocalizedDescription=Invalid streamID} CMIOExtensionSession.m:400:-[CMIOExtensionSessionStream stopStream:]_block_invoke stop stream index error Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list" CMIO_DAL_CMIOExtension_Stream.mm:1981:Stop_block_invoke 37 unexpected Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list" CMIO_DAL_CMIOExtension_Device.mm:1565:StopStream 36 could not find stream with ID 37 It seemed like CMIO or AVFoundation was not made aware of such add/remove change. On a different approach, it would work if in the CMIOExtensionProvider level we remove the old device and a create new device + stream with updated list of formats. However, such action will cause the client app to move to next available AVCaptureDevice (i.e FaceTime Camera) to preview, leading to a strange user experience. DTS engineer recommendations: I see, I agree that this would create a strange user experience, that is, reporting support for a format but not actually having hardware connected that supports that format. Unfortunately there is no other option, you can provide the array of all possible formats, and then maintain a dynamic sub-set of these formats that you actually support at a given time. Then, if a format index is set (by whatever client app is using the device) that isn’t supported at a particular time, you could throw an error from setStreamProperties. Or you can tear everything down and create a new device + stream with the currently supported formats. Beyond those options, I recommend that you file an enhancement request using Feedback Assistant, to explain to the engineering team your need for a way to update the formats array during the life cycle of the associated stream. We filed an enhancement request a while ago in Feedback Assistant. Thought I might share this in the forum.

I built a Camera Extension that augments the Facetime hd camera stream and I have noticed that it works just fine with Facetime and Photobooth but when I choose my camera from the camera picker of a non Apple application the camera becomes extremely laggy. Is this a known issue because the non Apple apps have yet to be optimized for mac os 13?

I have a macOS app, which detects if a camera is being used via the kCMIOHardwarePropertyDevices : typedef void (^DeviceIterator)(CMIOObjectID cameraObject, BOOL *stop); static void iterateThroughAllInputDevices(DeviceIterator deviceIterator) { // Check the number of devices. CMIOObjectPropertyAddress address = makeGlobalPropertyAddress(kCMIOHardwarePropertyDevices); UInt32 devicesDataSize; auto status = CMIOObjectGetPropertyDataSize(kCMIOObjectSystemObject, &address, 0, nil, &devicesDataSize); if (isError(status)) { return; } // Get the devices. UInt32 devicesDataUsed; int count = devicesDataSize / sizeof(CMIOObjectID); if (!count) { LOG_INFO("video device list is empty"); return; } I am using virtual machines for my automated end-to-end tests (which tests are written in Python). These Virtual Machines don't have a Camera device available, so I quickly wrote a macOS Camera Extension (based on the "Create camera extensions with Core Media IO" WWDC session ) in hope that it could act like a real camera. Unfortunately this Virtual Camera is not being detected by the code above. I am getting the video device list is empty message. How could I create a virtual (software) camera that would be listed by the API above?

I have solved most of the obvious challenges with CMIO Extensions but am finding some strange edge cases that are proving difficult to debug. I have a CMIO Extension which is essentially the straight boilerplate code from the Xcode template. I always return here when I have strange behaviour just in case it's my strange behaviour and not the system. The problem appears on Ventura 13.1 (Apple Silicon - I'll also check Monterey when I'm back at that machine later tonight). The CMIO Extension is installed and recognized. If I launch QuickTime Player and start a recording, everything looks good. If I also launch Photo Booth while QuickTime is running, the camera frames are also streaming. However, if I start quitting and launching QuickTime and Photo Booth to force disconnection/connection events in the extension eventually there are some messages spewed into the os_log by CMIO, the frames stop running and QuickTime hangs when I try to quit.

CMIO Extension - mostly working; I'm trying to narrow down some edge cases which appear to be related to the buffering of CMSampleBuffers arriving via a sink stream and being subsequently vended to clients via a source stream. The sink stream may not be an issue, I cross posted about the boilerplate CMIO extension having problems with multiple clients. In particular I have a case where two clients are running and the flow of sample buffers is smooth until I connect a third client (based loosely on RosyWriter ported to Mac). I think this third client has a much different cadence of consuming the CMSampleBuffers because it's using AVAssetWriter to asynchronously write the buffers to disk. It seems under this load the CVPixelBufferPool inside the extension runs out of CVPixelBuffers. For reference, my CFPlugIn DAL sample driver is functioning correctly (obviously not on Ventura) with this RosyWriter-esque client so I suspect its the buffering between the client and the extension. Hoping for some hints about what kind of buffering is between the client and the extension. To that end, is there any clarification on: What values might be appropriate for CMIOExtensionPropertyStreamSinkBufferQueueSize? I have it randomly set to 3 in my complex extension. Also, if the sink client disconnects, it is necessary to set CMIOExtensionPropertyStreamSinkEndOfData? Does the CMIO system read or interpret CMIOExtensionPropertyStreamSinkBufferUnderrunCount? Is it even necessary to expose any of these properties to the CMIO subsystem? Or can I remove them and let CMIO infer default values? Some os_log messages that seem to be related to presentation timestamps: CMIO_Unit_Synchronizer_Video.cpp:1435:SyncUsingIntegralFrameTiming observing getting frames too quickly by a lot: 284681993 : 100000 < 284682999 : 100000 (-0.0101), prev time / dur: 284682180 : 100000 / 8189 : 100000 : dropping this frame CMIO_Unit_Synchronizer_Video.cpp:1470:SyncUsingIntegralFrameTiming here's the normal bounding info: 284681993 : 100000 < 284686275 : 100000 (-0.0428), prev time / dur: 284682180 : 100000 / 8189 : 100000 : dropping this frame

Hello. I'm working on a Core Media IO camera extension, including a source and sink stream. My application sends sample buffers to the sink stream, which are passed to the source stream. The sample buffers' pixel format is 420v. This format is determined in the application (not the camera extension) by a VTDecompressionSession, which has the image buffer attribute requested: kCVPixelBufferIOSurfaceCoreAnimationCompatibilityKey; and a decoder specification that requests the key kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder. However, no client can connect to the extension's source stream. Photo Booth shows a blank screen. I noticed the second argument to the function CMVideoFormatDescriptionCreate, which has type CMVideoCodecType, doesn't explicitly support 420v. In fact there is no 4:2:0 format in the list at all. Comparing the headers, the only format in common between CoreVideo and CoreMedia is 422YpCbCr8... % sed -n -e 's/.*\(kCMVideoCodecType_[a-zA-Z0-9_]*\).*/\1/p' < /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreMedia.framework/Versions/A/Headers/CMFormatDescription.h | sort | uniq > kCMVideoCodecType.txt % sed -n -e 's/.*\(kCVPixelFormatType_[a-zA-Z0-9_]*\).*/\1/p' < /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreVideo.framework/Versions/A/Headers/CVPixelBuffer.h | sort | uniq > kCVPixelFormatType.txt % comm <(sed -e 's/kCMVideoCodecType_//' < kCMVideoCodecType.txt) <(sed -e 's/kCVPixelFormatType_//' < kCVPixelFormatType.txt) What is the guidance for pixel formats in camera extensions? The sample camera extension code produced by Xcode uses kCVPixelFormatType_32BGRA. Should camera extensions all be using this format? That's an extra step of YUV→RGB conversion that I'd rather avoid if possible.

Is there a way to tell using device / stream properties if a camera extension is being actively used by a client?

_streamSinkIn = [[CMIOExtensionStream alloc] initWithLocalizedName:localizedName streamID:streamInputID direction:CMIOExtensionStreamDirectionSink clockType:CMIOExtensionStreamClockTypeHostTime source:self]; Attempting to publish a CMIOExtensionStream with the 'sink' direction (i.e. print-to-tape) as alluded to in Brad Ford's presentation. Any attempt to create such a stream yields and invalid argument exception and if you examine the header files all the init methods are described as returning stream instances that source data (ie camera publishers). *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: 'Invalid argument'

I am working on an camera extension for macOS 12.3 and later. This extension acts as a webcam for a WebRTC stream from a different device, you can imagine it like a IP webcam. To the issue at hand. I use the native WebRTC library from google and on macOS 13 it works like a charm. But on macOS 12.3 it gives me the following console error while gathering the ICE candidates: Sandbox: bundle id(PID) deny(1) network-bind*:0 I only found this related issue here but I already included the entitlements for client and server from the beginning. And as mentioned it works on macOS 13. Some additional information is that the local ICE candidates show only TCP offers on macOS 12.3 but also include the for me needed UDP offers on macOS 13. Does anyone know how to fix this issue? Or can tell me why UDP is not an option from a camera extension?

i am trying to send data from the container app to its system extension (specifically, a camera extension. i am not entirely sure how to go about this. i read that i could utilize XPC, and i have tried this but for some reason, the system extension returns nothing when trying to connect to the XPC Service. below is an example code snipper let connectionToService = NSXPCConnection(serviceName: "example.VirtualCameraXPC")     connectionToService.remoteObjectInterface = NSXPCInterface(with:VirtualCameraXPCProtocol.self)     connectionToService.resume() //checking processes available shows that the XPC Server process is never activated also. if let proxy = connectionToService.remoteObjectProxy as? VirtualCameraXPCProtocol {       proxy.getNewString() { aString in //this is never logged.         NSLog("Second result string was: \(aString)")       }     } when i run the above code from the container app, it works properly but from the system extension, i run into the problems commented above in the code block. i would appreciate any help that will help me successfully send data from my container app to my camera extension.

Hi, I have been following the steps presented in https://developer.apple.com/documentation/coremediaio/creating_a_camera_extension_with_core_media_i_o to create a core media I/O camera. However when calling let activationRequest = OSSystemExtensionRequest.activationRequest( forExtensionWithIdentifier: identifier, queue: .main ) activationRequest.delegate = installDelegate OSSystemExtensionManager.shared.submitRequest(activationRequest) The delegate's request(_ request: OSSystemExtensionRequest, didFailWithError error: Error) method is being called with error code 9 (OSSystemExtensionError.Code.validationFailed) and the system dialog does not appear.

I was following the WWDC demo and I was able to get the virtual camera running pretty easily. Now I am trying to convert this into a hardware camera and I am not finding any examples. In my ExtensionProvider.Swift file I am attempting the following      let session = AVCaptureSession()     session.sessionPreset = .hd1920x1080     session.startRunning()     guard let captureDevice = AVCaptureDevice.default(for: .video) else {      return     }     guard let deviceInput = try? AVCaptureDeviceInput(device: captureDevice) else {      return     }     session.addInput(deviceInput)           let deviceID = UUID() // replace this with your device UUID I have my AVFoundation stuff but am I will changing my device ID (UUID) actually change the stream to originate from the hardware camera?

Hello, According to Create camera extensions with Core Media IO from WWDC2002, one can use driverkit-extension (dext) to communicate with hardware. Does it mean there will be a .dext and a .systemextension in the SystemExtensions folder and they both need to be activated via user prompt separately? We looked into Communicating Between a DriverKit Extension and a Client App and USBApp as our starting point. With NewUserClient implemented, we could open our dext via IOServiceOpen in a user space program. We have been unable to apply similar approach in a cameraextension code sample with the sandbox violation: Violation: deny(1) iokit-open-user-client IOUserUserClient In the plist we did add com.apple.security.temporary-exception.iokit-user-client-class entitlement array with IOUserUserClient like in our other test app. Our goal is to let CMIOExtension exchange data / information with our usb device via function calls like IOConnectCallAsyncStructMethod upon successful opening our userClient Has anyone run into similar problem or is this not the right way to do it? Thanks for your time.

I can't see it under the video or at the sample code website.

While trying to re-create the CIFilterCam demo shown in the WWDC session, I hit a roadblock when trying to access a hardware camera from inside my extension. Can I simply use an AVCaptureSession + AVCaptureDeviceInput + AVCaptureVideoDataOutput to get frames from an actual hardware camera and pass them to the extension's stream? If yes, when should I ask for camera access permissions? It seems the extension code is run as soon as I install the extension, but I never get prompted for access permission. Do I need to set up the capture session lazily? What's the best practice for this use case?

It’d be nice if the extension could inherit the permissions given to the main app by the user, in my case screen capture. Alternatively, is there a way to send data to the extension from the app, or vice versa? Especially image or video data. thanks! laurent