Hi,
since RealityKit 4 now supports Blend Shapes I was wondering if there are any workflow or tooling recommendations to bake/export them into a USDZ.
Are Blender or Cinema4D capable to do that out of the box? Should we look into NVIDIA omniverse (https://docs.omniverse.nvidia.com/connect/latest/blender/manual.htm)
So far this topic seems very sparsely documented and I would appreciate any hints. Thank you!
Discuss spatial computing on Apple platforms and how to design and build an entirely new universe of apps and games for Apple Vision Pro.
Selecting any option will automatically load the page
Post
Replies
Boosts
Views
Activity
I'm trying to implement a prototype to render virtual objects in a mixed immersive space on the camer frames captured by CameraFrameProvider.
Here are what I have done:
Get camera's instrinsics from frame.primarySample.parameters.intrinsics
Get camera's extrinsics from frame.primarySample.parameters.extrinsics
Get the device anchor by worldTrackingProvider.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())
Setup a RealityKit.RealityRenderer to render virtual objects on the captured camera frames
let realityRenderer = try RealityKit.RealityRenderer()
realityRenderer.cameraSettings.colorBackground = .outputTexture()
let cameraEntity = PerspectiveCamera()
// see https://developer.apple.com/forums/thread/770235
let cameraTransform = deviceAnchor.originFromAnchorTransform * extrinsics.inverse
cameraEntity.setTransformMatrix(cameraTransform, relativeTo: nil)
cameraEntity.camera.near = 0.01
cameraEntity.camera.far = 100
cameraEntity.camera.fieldOfViewOrientation = .horizontal
// manually calculated based on camera intrinsics
cameraEntity.camera.fieldOfViewInDegrees = 105
realityRenderer.entities.append(cameraEntity)
realityRenderer.activeCamera = cameraEntity
Virtual objects, which should be seen in the camera frames, are clipped out by the camera transform.
If I use deviceAnchor.originFromAnchorTransform as the camera transform, virtual objects can be rendered on camera frames at wrong positions (I think it is because the camera extrinsics isn't used to adjust the camera to the correct position).
My question is how to use the camera extrinsic matrix for this purpose?
Does the camera extrinsics point to a similar orientation of the device anchor with some minor rotation and postion change? Here is an extrinsics from a camera frame. It seems that the direction of Y-axis and Z-axis are flipped by the extrinsics. So the camera is point to a wrong direction.
simd_float4x4([[0.9914258, 0.012555369, -0.13006608, 0.0], // X-axis
[-0.0009778949, -0.9946325, -0.10346654, 0.0], // Y-axis
[-0.13066702, 0.10270659, -0.98609203, 0.0], // Z-axis
[0.024519, -0.019568002, -0.058280986, 1.0]]) // translation
Hi All,
We're a studio building an app and as part of a scene we have a 3D asset with a smoke particle emitter and a curved mesh that plays video. I notice that when the video alone is played or the particle effect alone is done then the scene works fine but the frame rate drops drastically when both are turned on.
How do I solve this because this is an important storytelling feature.
Hi everyone,
We’re developing a Unity project for Apple Vision Pro that connects PSVR2 Sense controllers for advanced interaction and input.
We’ve encountered a major limitation:
when the controller is not held close to the designated hand (e.g., resting on a table or held by the non designated hand), the Sense controller enters a low-power or reduced-update mode. This results in noticeably reduced tracking update frequency and responsiveness until the controller is held again.
For certain use cases, this behavior is undesirable. In our case, it prevents continuous real-time tracking of the controller even when it’s stationary or being tracked externally.
Request:
Please consider exposing an API flag or developer option in ARKit to disable and optionally delay the low-power mode when the app requires full-rate updates regardless of proximity or hand pose detection.
Hi,
I am creating an ECS. With this ECS I will need to register several DragGesture.
Question: Is it possible to define DragGestures in ECS? If yes, how do we do that? If not, what is the best way to do that?
Question: Is there a "gesture" method that takes an array of gestures as a parameter?
I am interested in any information that can help me, if possible with an example of code.
Regards
Tof
Hello,
I have downloaded and run the sample object tracking app for visionos.
Now I'm working on my own objects for tracking. I have made a model using Create ML using images of my object.
However, I cannot see how to convert the Create ML output file (xxx.mlmodel) into a reference object like the files in the sample project.
is there a tool for converting them?
TIA
Topic:
Spatial Computing
SubTopic:
ARKit
Does anyone have any idea if Apple plans to add in UDIM support for its 3D development? It is a real bummer to not have this feature and makes an otherwise clean USD pipeline kinda suck.
I’m working on an iOS app that needs to measure the area of planes or surfaces, like the length and width of objects, just like the Apple Measure app does. I’ve been exploring ARKit, but I’m curious if there are any APIs or techniques that can help automate the process of detecting and measuring planes.
Specifically, I’m looking for a way to automatically detect and measure planes (e.g., from a top-down view). For example: Measuring a box width and length. I have attached a screenshot and a video of the Apple Measure App doing it.
Does Apple provide any tools or APIs for this, or are there any best practices I should know about? I’d love to hear from anyone who’s tackled something similar.
Video: https://drive.google.com/file/d/1BxM7fIbFxsCsYwY7w8ZxIeq_4WTGkkwA/view?usp=drive_link
I have read in the apple documentation and on forums that in order to access the camera and capture images on VisionPro, both an Entitlement and an Enterprise.license are required. I already have the Entitlement, but I don’t yet have the Enterprise.license. I would like to ask: is the Enterprise.license strictly required to gain camera access for capturing images? How can I obtain this file, and does it require an Enterprise account? Currently, my developer account is a regular Developer 99$, not an Enterprise account.
Topic:
Spatial Computing
SubTopic:
ARKit
Hi, I'm developing an app for Vision Pro using Xcode, while updating the latest update, things that worked in my app suddenly didn't.
in my app flow I'm tapping spheres to get their positions, from some reason I get an offset from where I tap to where a marker on that position is showing up.
here's the part of code that does that, and a part that is responsible for an alignment that happens afterwards:
func loadMainScene(at position: SIMD3) async {
guard let content = self.content else { return }
do {
let rootEntity = try await Entity(named: "surgery 16.09", in: realityKitContentBundle)
rootEntity.scale = SIMD3<Float>(repeating: 0.5)
rootEntity.generateCollisionShapes(recursive: true)
self.modelRootEntity = rootEntity
let bounds = rootEntity.visualBounds(relativeTo: nil)
print("📏 Model bounds: center=\(bounds.center), extents=\(bounds.extents)")
let pivotEntity = Entity()
pivotEntity.addChild(rootEntity)
self.pivotEntity = pivotEntity
let modelAnchor = AnchorEntity(world: [1, 1.3, -0.8])
modelAnchor.addChild(pivotEntity)
content.add(modelAnchor)
updateModelOpacity(0.5)
self.modelAnchor = modelAnchor
rootEntity.visit { entity in
print("👀 Entity in model: \(entity.name)")
if entity.name.lowercased().hasPrefix("focus") {
entity.generateCollisionShapes(recursive: true)
entity.components.set(InputTargetComponent())
print("🎯 Made tappable: \(entity.name)")
}
}
print("✅ Model loaded with collisions")
guard let sphere = placementSphere else { return }
let sphereWorldXform = sphere.transformMatrix(relativeTo: nil)
var newXform = sphereWorldXform
newXform.columns.3.y += 0.1 // move up by 20 cm
let gridAnchor = AnchorEntity(world: newXform)
self.gridAnchor = gridAnchor
content.add(gridAnchor)
let baseScene = try await Entity(named: "Scene", in: realityKitContentBundle)
let gridSizeX = 18
let gridSizeY = 10
let gridSizeZ = 10
let spacing: Float = 0.05
let startX: Float = -Float(gridSizeX - 1) * spacing * 0.5 + 0.3
let startY: Float = -Float(gridSizeY - 1) * spacing * 0.5 - 0.1
let startZ: Float = -Float(gridSizeZ - 1) * spacing * 0.5
for i in 0..<gridSizeX {
for j in 0..<gridSizeY {
for k in 0..<gridSizeZ {
if j < 2 || j > gridSizeY - 5 { continue } // remove 2 bottom, 4 top
let cell = baseScene.clone(recursive: true)
cell.name = "Sphere"
cell.scale = .one * 0.02
cell.position = [
startX + Float(i) * spacing,
startY + Float(j) * spacing,
startZ + Float(k) * spacing
]
cell.generateCollisionShapes(recursive: true)
gridCells.append(cell)
gridAnchor.addChild(cell)
}
}
}
content.add(gridAnchor)
print("✅ Grid added")
} catch {
print("❌ Failed to load: \(error)")
}
}
private func handleModelOrGridTap(_ tappedEntity: Entity) {
guard let modelRootEntity = modelRootEntity else { return }
let localPosition = tappedEntity.position(relativeTo: modelRootEntity)
let worldPosition = tappedEntity.position(relativeTo: nil)
switch tapStep {
case 0:
modelPointA = localPosition
modelAnchor?.addChild(createMarker(at: worldPosition, color: [1, 0, 0]))
print("📍 Model point A: \(localPosition)")
tapStep += 1
case 1:
modelPointB = localPosition
modelAnchor?.addChild(createMarker(at: worldPosition, color: [1, 0.5, 0]))
print("📍 Model point B: \(localPosition)")
tapStep += 1
case 2:
targetPointA = worldPosition
targetMarkerA = createMarker(at: worldPosition,color: [0, 1, 0])
modelAnchor?.addChild(targetMarkerA!)
print("✅ Target point A: \(worldPosition)")
tapStep += 1
case 3:
targetPointB = worldPosition
targetMarkerB = createMarker(at: worldPosition,color: [0, 0, 1])
modelAnchor?.addChild(targetMarkerB!)
print("✅ Target point B: \(worldPosition)")
alignmentReady = true
tapStep += 1
default:
print("⚠️ Unexpected tap on model helper at step \(tapStep)")
}
}
func alignModel2Points() {
guard let modelPointA = modelPointA,
let modelPointB = modelPointB,
let targetPointA = targetPointA,
let targetPointB = targetPointB,
let modelRootEntity = modelRootEntity,
let pivotEntity = pivotEntity,
let modelAnchor = modelAnchor else {
print("❌ Missing data for alignment")
return
}
let modelVec = modelPointB - modelPointA
let targetVec = targetPointB - targetPointA
let modelLength = length(modelVec)
let targetLength = length(targetVec)
let scale = targetLength / modelLength
let modelDir = normalize(modelVec)
let targetDir = normalize(targetVec)
var axis = cross(modelDir, targetDir)
let axisLength = length(axis)
var rotation = simd_quatf()
if axisLength < 1e-6 {
if dot(modelDir, targetDir) > 0 {
rotation = simd_quatf(angle: 0, axis: [0,1,0])
} else {
let up: SIMD3<Float> = [0,1,0]
axis = cross(modelDir, up)
if length(axis) < 1e-6 {
axis = cross(modelDir, [1,0,0])
}
rotation = simd_quatf(angle: .pi, axis: normalize(axis))
}
} else {
let dotProduct = dot(modelDir, targetDir)
let clampedDot = max(-1.0, min(dotProduct, 1.0))
let angle = acos(clampedDot)
rotation = simd_quatf(angle: angle, axis: normalize(axis))
}
modelRootEntity.scale = .one * scale
modelRootEntity.orientation = rotation
let transformedPointA = rotation.act(modelPointA * scale)
pivotEntity.position = -transformedPointA
modelAnchor.position = targetPointA
alignedModelPosition = modelAnchor.position
print("✅ Aligned with scale \(scale), rotation \(rotation)")
Topic:
Spatial Computing
SubTopic:
General
Hi, would love for your help in that matter.
I try to get the position in space of two QR codes to make an alignment to their positions in space. The detection shows that the QR codes position is always 0,0,0 and I don't understand why. Here's my code:
import SwiftUI
import RealityKit
import RealityKitContent
struct AnchorView: View {
@ObservedObject var qrCoordinator: QRCoordinator
@ObservedObject var coordinator: ImmersiveCoordinator
let qrName: String
@Binding var startQRDetection: Bool
@State private var anchor: AnchorEntity? = nil
@State private var detectionTask: Task<Void, Never>? = nil
var body: some View {
RealityView { content in
// Add the QR anchor once (must exist before detection starts)
if anchor == nil {
let imageAnchor = AnchorEntity(.image(group: "QRs", name: qrName))
content.add(imageAnchor)
anchor = imageAnchor
print("📌 Created anchor for \(qrName)")
}
}
.onChange(of: startQRDetection) { enabled in
if enabled {
startDetection()
} else {
stopDetection()
}
}
.onDisappear {
stopDetection()
}
}
private func startDetection() {
guard detectionTask == nil, let anchor = anchor else { return }
detectionTask = Task {
var detected = false
while !Task.isCancelled && !detected {
print("🔎 Checking \(qrName)... isAnchored=\(anchor.isAnchored)")
if anchor.isAnchored {
// wait a short moment to let transform update
try? await Task.sleep(nanoseconds: 100_000_000)
let worldPos = anchor.position(relativeTo: nil)
if worldPos != .zero {
// relative to modelRootEntity if available
var posToSave = worldPos
if let modelEntity = coordinator.modelRootEntity {
posToSave = anchor.position(relativeTo: modelEntity)
print("converted to model position")
} else {
print("⚠️ modelRootEntity not available, using world position")
}
print("✅ \(qrName) detected at position: world=\(worldPos) saved=\(posToSave)")
if qrName == "reanchor1" {
qrCoordinator.qr1Position = posToSave
let marker = createMarker(color: [0,1,0])
marker.position = .zero // sits directly on QR
marker.position = SIMD3<Float>(0, 0.02, 0)
anchor.addChild(marker)
print("marker1 added")
} else if qrName == "reanchor2" {
qrCoordinator.qr2Position = posToSave
let marker = createMarker(color: [0,0,1])
marker.position = posToSave // sits directly on QR
marker.position = SIMD3<Float>(0, 0.02, 0)
anchor.addChild(marker)
print("marker2 added")
}
detected = true
} else {
print("⚠️ \(qrName) anchored but still at origin, retrying...")
}
}
try? await Task.sleep(nanoseconds: 500_000_000) // throttle loop
}
print("🛑 QR detection loop ended for \(qrName)")
detectionTask = nil
}
}
private func stopDetection() {
detectionTask?.cancel()
detectionTask = nil
}
private func createMarker(color: SIMD3<Float>) -> ModelEntity {
let sphere = MeshResource.generateSphere(radius: 0.05)
let material = SimpleMaterial(color: UIColor(
red: CGFloat(color.x),
green: CGFloat(color.y),
blue: CGFloat(color.z),
alpha: 1.0
), isMetallic: false)
let marker = ModelEntity(mesh: sphere, materials: [material])
marker.name = "marker"
return marker
}
}
Topic:
Spatial Computing
SubTopic:
General
Hi !
I'm new on this forum, so if I need to update this post to have more info, or anything else, please let me know.
I'm using the Apple Vision Pro to develop some app (with unity). To demonstrate what the user see on the headset, I would like to mirror the view on a device (an iPad in this case). I managed to do this without any issue.
My problem is that, in the Vision Pro, I have an interface that the user can interact with. But I would like to be able to manage myself the interface on the iPad. What I mean is that the user can (or can't, doesn't matter) see the interface in the headset, and the interface is controlled by myself on the iPad.
Is there any way to do this ? Is this a question I should ask on unity's forum ? (I don't think so, because it should be related to the mirroring function non ?)
Topic:
Spatial Computing
SubTopic:
General
As far as I know, Apple hasn’t opened access to the Vision Pro camera for developers yet, so I’m trying to find possible workarounds within the current capabilities. I’m wondering if there’s any way to apply a mesh to a person in the scene in Vision Pro, or if there’s an alternative approach to roughly detect a human shape in front of the user?
Here is my code in visionOS 2.3
NavigationSplitView {
List {
}
.navigationTitle("Passwords")
} detail: {
Text("Hello")
.navigationTitle("All")
}
The font size of "Passwords" and "All" are smaller than the ones in Passwords app.
As I understand it there are two ways I can track a hand, or a joint, in RealityKit:
either, create an AnchorEntity, for example AnchorEntity(.hand(.left, location: .palm))
or, set up an ARSession with a HandTrackingProvider ( a lot more code which I haven't repeated here).
Assuming this is correct, when would I want to use one over the other?
Hi Apple engineers,
I'm currently working on an app that uses the incoming microphone audio and gives visual feedback to the user about the incoming audio.
I would like to use Reality Composer Pro's Developer Capture to get a high quality recording of the app and its use cases for the App Store — but any time I have an in-progress capture, my app stops receiving the incoming audio. It almost seems as if the microphone audio is getting 'hijacked' during the screen capture, which prevents me from demonstrating the app's core features.
Could you please advise on how to proceed?
Topic:
Spatial Computing
SubTopic:
Reality Composer Pro
Goal: To render in an apple vision pro app, the solid-mechanics 3D simulation results coming form an FEA code.
Starting point: I have surface vtks with deformations on each node. Each time step has a a mesh with the nodal coordinates. This is straighforward translatable to a usd MeshSequence. Unfortunately, the results cannot be simplified to a scaling o linear transformation as you would do with other game-oriented animations.
Tools: Right now, I am using Xcode and reality composer pro (RCP) to build the scenes.
Technical limitations: I am aware that RCP can do animations with BlendMesh and skeletons and that MeshSequence is not a problem.
Progress:
Coverting to the sequence of vtk meshes to a usd MeshSequence is straighforward. This animates correctly in Preview and Blender (see screenshot).
I managed to convert from MeshSequence to multiple keys and BlendMesh. This also animates correctly in Blender and preview. Unfortunately, the BlendMesh of multiple blended meshes shows a zero animation time in RCP (see screenshot below)
Also, see below usda file scheme for the animation. Of course I am not showing full vectors such as faceVertexCounts, faceVertexIndex, normals.
Question: what is the right set up to create a BlendMesh animation that RCP will correctly import and animate, form a set of Meshes or multiple key shapes?
Blender animation
Time zero RCP "animations"
#usda 1.0
(
defaultPrim = "BlendMeshRoot"
doc = "Blender v4.5.3 LTS"
endTimeCode = 48
framesPerSecond = 24
metersPerUnit = 1
startTimeCode = 0
timeCodesPerSecond = 24
upAxis = "Z"
)
def Xform "BlendMeshRoot" (
customData = {
dictionary Blender = {
bool generated = 1
}
}
)
{
def SkelRoot "Mesh"
{
custom string userProperties:blender:object_name = "Mesh"
float3 xformOp:rotateXYZ = (89.99999, -0, 0)
float3 xformOp:scale = (0.009999999, 0.01, 0.01)
double3 xformOp:translate = (0, 0, 0)
uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateXYZ", "xformOp:scale"]
def Mesh "Mesh" (
active = true
prepend apiSchemas = ["MaterialBindingAPI", "SkelBindingAPI"]
)
{
uniform bool doubleSided = 1
float3[] extent = [(25.091871, -34.121277, -13.298501), (299.94482, 245.10088, 202.35126)]
int[] faceVertexCounts = [3, 3, ...
int[] faceVertexIndices = [0, 10293, ...
rel material:binding = </BlendMeshRoot/_materials/MeshSequence_Default>
normal3f[] normals = [(-0.3632836, -0.9102419, -0.19870725), ....
point3f[] points = [(244.41148, 155.42062, 70.454926),.....
float3[] primvars:node_displacement = [(93.54703, 110.9341, 48.37992)....
float3[] primvars:Normals = [(-0.0050530406, -0.9910114, -0.13368203),...
int[] primvars:skel:jointIndices = [0, 0, 0, 0, 0 ...
float[] primvars:skel:jointWeights = [1, 1, 1, 1, 1...
uniform token[] skel:blendShapes = ["frame_0000", "frame_0001", "frame_0002", "frame_0003", "frame_0004", "frame_0005"]
rel skel:blendShapeTargets = [
</BlendMeshRoot/Mesh/Mesh/frame_0000>,
.......
</BlendMeshRoot/Mesh/Mesh/frame_0005>,
]
prepend rel skel:skeleton = </BlendMeshRoot/Mesh/Skel>
uniform token subdivisionScheme = "none"
custom string userProperties:blender:data_name = "Mesh"
custom float userProperties:originalTime
float userProperties:originalTime.timeSamples = {
0: 0,
}
def BlendShape "frame_0000"
{
uniform vector3f[] offsets = [(0, 0, 0), (0, 0, 0),.....
uniform int[] pointIndices = [0, 1, 2, .....
}
.....
.....
#### BlendShape frame to 0005
.....
def Skeleton "Skel" (
prepend apiSchemas = ["SkelBindingAPI"]
)
{
uniform matrix4d[] bindTransforms = [( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0, 0, 0, 1) )]
uniform token[] joints = ["joint1"]
uniform matrix4d[] restTransforms = [( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0, 0, 0, 1) )]
prepend rel skel:animationSource = </BlendMeshRoot/Mesh/Skel/Anim>
def SkelAnimation "Anim"
{
uniform token[] blendShapes = ["frame_0000", "frame_0001", "frame_0002", "frame_0003", "frame_0004", "frame_0005"]
float[] blendShapeWeights.timeSamples = {
0: [1, 0, 0, 0, 0, 0],
1: [0.9697085, 0.03029152, 0, 0, 0, 0],
2: [0.88787615, 0.11212383, 0, 0, 0, 0],
.....
46: [0, 0, 0, 0, 0.11212379, 0.8878762],
47: [0, 0, 0, 0, 0.030291557, 0.96970844],
48: [0, 0, 0, 0, 0, 1],
}
}
}
}
def Scope "_materials"
{
def Material "MeshSequence_Default"
{
token outputs:surface.connect = </BlendMeshRoot/_materials/MeshSequence_Default/Principled_BSDF.outputs:surface>
custom string userProperties:blender:data_name = "MeshSequence_Default"
def Shader "Principled_BSDF"
{
uniform token info:id = "UsdPreviewSurface"
float inputs:clearcoat = 0
float inputs:clearcoatRoughness = 0.03
color3f inputs:diffuseColor = (0.8, 0.4, 0.3)
float inputs:ior = 1.5
float inputs:metallic = 0
float inputs:opacity = 1
float inputs:roughness = 0.5
float inputs:specular = 0.2
token outputs:surface
}
}
}
def Scope "AnimationClips"
{
custom rel animations = </BlendMeshRoot/Mesh/Skel/Anim>
}
def RealityKitComponent "AnimationLibrary"
{
custom rel animations = </BlendMeshRoot/Mesh/Skel/Anim>
custom token info:id = "RealityKit.AnimationLibrary"
custom double realitykit:approximateDuration = 2
custom double[] realitykit:clipDurations = [2]
custom string[] realitykit:clipNames = ["Anim"]
custom rel realitykit:clipTargets = </BlendMeshRoot/Mesh/Skel/Anim>
custom double realitykit:frameRate = 24
custom bool realitykit:isAnimationLibrary = 1
}
}
Topic:
Spatial Computing
SubTopic:
Reality Composer Pro
Tags:
Swift Packages
Developer Tools
Reality Converter
Reality Composer
Hi,
I am in the process of implementing SharePlay into our app. The shared experience opens an Immersive Space and we set systemCoordinator.configuration.supportsGroupImmersiveSpace = true
Now visionOS establishes a shared coordinate space for the immersive space.
From the docs:
To achieve consistent positioning of RealityKit entities across multiple devices in an immersive space during a SharePlay session
There are cases where we want to position content in front of the user (independent of the shared session, and for each user individually). Normally to do that we use the transform retrieved via worldTrackingProvider.queryDeviceAnchor.originFromAnchorTransform
to position content in front of the user (plus some Z Offset and smooth interpolation).
This works fine in non-SharePlay instances and the device transform is where I would expect it to be but during the FaceTime call deviceAnchor.originFromAnchorTransform seems to use the shared origin of the immersive space and then I end up with a transform that might be offset.
Here is a video of the issue in action: https://streamable.com/205r2p
The blue rect is place using AnchorEntity(.head, trackingMode: .continuous). This works regardless of the call and the entity is always placed based on the head position.
The green rect is adjusted on every frame using the transform I get from worldTrackingProvider.queryDeviceAnchor. As you can see it's offset.
Is there any way I can query query this transform locally for the user during a FaceTime call?
Also I would like to know if it's possible to disable this automatic entity transform syncing behavior?
Setting entity.synchronization = nil results in the entity not showing up at all.
https://developer.apple.com/documentation/realitykit/synchronizationcomponent
Is SynchronizationComponent only relevant for the legacy MultiPeerConnectivity approach?
Thank you!
let component = GestureComponent(DragGesture())
iOS: ☑️
visionOS: ❌
This bug from beta to public, please fix it.
The landing page for visionOS 26 mentions
The Unified Coordinate Conversion API makes moving views and entities between scenes straightforward — even between views and ARKit accessory anchors.
This WWDC session very briefly shows a single example of using this, but with no context. For example, they discuss a way to tell the distance between a Model3D and an entity in a RealityView. But they don't provide any details for how they are referencing the entity (bolts in the slide).
The session used the BOT-anist example project that we saw in visionOS 2, but the version on in the Sample Code library has not been updated with these examples.
I was able to put together a simple example where we can get the position of a window relative to the world origin. It even updates when the user recenters.
struct Lab080: View {
@State private var posX: Float = 0
@State private var posY: Float = 0
@State private var posZ: Float = 0
var body: some View {
GeometryReader3D { geometry in
VStack {
Text("Unified Coordinate Conversion")
.font(.largeTitle)
.padding(24)
VStack {
Text("X: \(posX)")
Text("Y: \(posY)")
Text("Z: \(posZ)")
}
.font(.title)
.padding(24)
}
.onGeometryChange3D(for: Point3D.self) { proxy in try! proxy
.coordinateSpace3D()
.convert(value: Point3D.zero, to: .worldReference)
} action: { old, new in
posX = Float(new.x)
posY = Float(new.y)
posZ = Float(new.z)
}
}
}
}
This is all that I've been able to figure out so far. What other features are included in this new Unified Coordinate Conversion?
Can we use this to get the position of one window relative to another? Can we use this to get the position of a view in a window relative to an entity in a RealityView, for example in a Volume or Immersive Space? What else can Unified Coordinate Conversion do?
Are there documentation pages that I'm missing? I'm not sure what to search for. Are there any Sample projects that use these features? Any additional information would be very helpful.
Topic:
Spatial Computing
SubTopic:
General