Knuckle Position Synchronization

TODO: collapasable, copyable, download, etc

I have a bug in my Swift RealityKit project, that I don't understand. My logs say

pos4 = 0.0, 0.0, 0.0

The first log is from this class ``` @MainActor class HandTrackingViewModel: ObservableObject { private let session = ARKitSession() private let handTracking = HandTrackingProvider() @Published var knucklePosition: SIMD4 = SIMD4(0, 0, 0, 1)

In processing the hand updates:

        self.knucklePosition = originFromIndex.columns.3
        
        print("Setting HandTrackingViewModel.knucklePosition \(self.knucklePosition)")```

The second log line is from this class class SyncIPhoneSystem : System { ... func update(context: SceneUpdateContext) { ... if let handModel = Self.handTrackingViewModel { let pos4 = handModel.knucklePosition print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")

Why can't I read knucklePosition? Why doesn't it get updated? Why am I always getting the original value?The issue seems to be related to how @Published properties and @MainActor work in SwiftUI and RealityKit. The @Published property knucklePosition is likely being updated correctly within your HandTrackingViewModel, but when you read it in your SyncIPhoneSystem, the update might not be visible due to the way threading and actor isolation work.

Here are a few steps to troubleshoot and potentially fix this issue:

1. Check MainActor Execution: Ensure that the update method in SyncIPhoneSystem is being called on the main thread. @Published properties updated on the main thread need to be read on the main thread as well.

2. Use Combine to Observe Changes: Instead of directly accessing knucklePosition, you can observe changes to the HandTrackingViewModel using Combine. This ensures that updates are propagated correctly to the SyncIPhoneSystem.

3. Ensure Proper Synchronization: Make sure that the HandTrackingViewModel instance is shared correctly and not being re-initialized elsewhere in your code.

Here's an updated version of your code using Combine to observe changes:

HandTrackingViewModel

@MainActor class HandTrackingViewModel: ObservableObject {
    private let session = ARKitSession()
    private let handTracking = HandTrackingProvider()
    
    @Published var knucklePosition: SIMD4<Float> = SIMD4(0, 0, 0, 1)
    
    func updateKnucklePosition(with position: SIMD4<Float>) {
        knucklePosition = position
        print("Setting HandTrackingViewModel.knucklePosition \(knucklePosition)")
    }
}

SyncIPhoneSystem

class SyncIPhoneSystem: System {
    private var cancellable: AnyCancellable?
    static var handTrackingViewModel: HandTrackingViewModel?
    
    init() {
        if let handTrackingViewModel = Self.handTrackingViewModel {
            cancellable = handTrackingViewModel.$knucklePosition.sink { [weak self] newPosition in
                self?.handleKnucklePositionUpdate(newPosition)
            }
        }
    }
    
    func handleKnucklePositionUpdate(_ position: SIMD4<Float>) {
        let pos4 = position
        print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")
    }
    
    func update(context: SceneUpdateContext) {
        // Other update logic
    }
}

Usage

Ensure that you initialize and set the handTrackingViewModel correctly and that it is shared across the relevant parts of your code.

let handTrackingViewModel = HandTrackingViewModel()
SyncIPhoneSystem.handTrackingViewModel = handTrackingViewModel

// Update knuckle position somewhere in your code
handTrackingViewModel.updateKnucklePosition(with: SIMD4(-0.24767806, 0.9377898, -0.4052379, 0.99999994))

Key Points:

1. Combine: Use Combine to observe changes to @Published properties, ensuring that updates are propagated correctly.
2. Threading: Ensure that update and property changes are performed on the main thread if necessary.
3. Singleton/Shared Instance: Ensure that handTrackingViewModel is shared correctly and not re-initialized.

By observing the knucklePosition changes using Combine, you ensure that updates are handled correctly across different parts of your app.I'm happy to remove @Published, I just want a way to get those values into my System classIf you want to remove @Published and still ensure that your HandTrackingViewModel values are accessible within your System class, you can use a more direct approach to update and read the values. Here’s how you can achieve this:

HandTrackingViewModel

@MainActor class HandTrackingViewModel {
    private let session = ARKitSession()
    private let handTracking = HandTrackingProvider()
    
    private(set) var knucklePosition: SIMD4<Float> = SIMD4(0, 0, 0, 1)
    
    func updateKnucklePosition(with position: SIMD4<Float>) {
        knucklePosition = position
        print("Setting HandTrackingViewModel.knucklePosition \(knucklePosition)")
    }
}

SyncIPhoneSystem

Ensure that SyncIPhoneSystem can access the HandTrackingViewModel instance directly. You can pass the HandTrackingViewModel instance to SyncIPhoneSystem during its initialization or via a setter method.

class SyncIPhoneSystem: System {
    private var handTrackingViewModel: HandTrackingViewModel?

    init(handTrackingViewModel: HandTrackingViewModel) {
        self.handTrackingViewModel = handTrackingViewModel
    }

    func update(context: SceneUpdateContext) {
        if let handModel = handTrackingViewModel {
            let pos4 = handModel.knucklePosition
            print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")
        }
    }
}

Usage

Initialize and set the HandTrackingViewModel and SyncIPhoneSystem correctly:

let handTrackingViewModel = HandTrackingViewModel()
let syncSystem = SyncIPhoneSystem(handTrackingViewModel: handTrackingViewModel)

// Update knuckle position somewhere in your code
handTrackingViewModel.updateKnucklePosition(with: SIMD4(-0.24767806, 0.9377898, -0.4052379, 0.99999994))

Key Points

1. Remove @Published: By removing @Published, you handle updates manually, which gives you more control over when and how values are propagated.
2. Direct Access: Pass the HandTrackingViewModel instance to the SyncIPhoneSystem directly, ensuring that both classes can access and update the same data.
3. Initialization: Ensure that the HandTrackingViewModel is properly initialized and passed to the SyncIPhoneSystem.

This approach ensures that your SyncIPhoneSystem always has the latest knucklePosition from HandTrackingViewModel without relying on @Published and Combine.Stepping back. KNowing I have HandTrackingViewModel and a System SyncIPhoneSystem. How would you coorindate getting the SIMD4 from one to the other?To coordinate getting the SIMD4<Float> value from HandTrackingViewModel to the SyncIPhoneSystem, you can use a shared instance of HandTrackingViewModel that the SyncIPhoneSystem can access directly. Here’s how you can achieve this:

Step-by-Step Solution

1. Define the HandTrackingViewModel: This class will manage the knucklePosition and provide a method to update it.

@MainActor
class HandTrackingViewModel {
    private let session = ARKitSession()
    private let handTracking = HandTrackingProvider()
    
    private(set) var knucklePosition: SIMD4<Float> = SIMD4(0, 0, 0, 1)
    
    func updateKnucklePosition(with position: SIMD4<Float>) {
        knucklePosition = position
        print("Setting HandTrackingViewModel.knucklePosition \(knucklePosition)")
    }
}

2. Define the SyncIPhoneSystem: This class will read the knucklePosition from the HandTrackingViewModel.

class SyncIPhoneSystem: System {
    private var handTrackingViewModel: HandTrackingViewModel?

    init(handTrackingViewModel: HandTrackingViewModel) {
        self.handTrackingViewModel = handTrackingViewModel
    }

    func update(context: SceneUpdateContext) {
        if let handModel = handTrackingViewModel {
            let pos4 = handModel.knucklePosition
            print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")
        }
    }
}

3. Initialize and Use the Classes: Ensure that you initialize and set the HandTrackingViewModel instance and pass it to the SyncIPhoneSystem.

// Initialize HandTrackingViewModel
let handTrackingViewModel = HandTrackingViewModel()

// Initialize SyncIPhoneSystem with HandTrackingViewModel
let syncSystem = SyncIPhoneSystem(handTrackingViewModel: handTrackingViewModel)

// Simulate updating the knuckle position
handTrackingViewModel.updateKnucklePosition(with: SIMD4(-0.24767806, 0.9377898, -0.4052379, 0.99999994))

// Simulate a scene update
syncSystem.update(context: SceneUpdateContext())

Key Points

1. Shared Instance: Ensure that handTrackingViewModel is a shared instance that both the view model and the system class can access.

2. Initialization: Pass the handTrackingViewModel instance to the SyncIPhoneSystem during initialization to ensure that the system has access to the latest knucklePosition.

3. Manual Update: Update the knucklePosition through a method in the HandTrackingViewModel and ensure the SyncIPhoneSystem reads the value during its update cycle.

By following this approach, you maintain a clear separation of concerns while ensuring that the SyncIPhoneSystem can access the latest knucklePosition from the HandTrackingViewModel.Given your expertise in building VisionOS apps and handling 3D worlds, let's focus on a clean way to coordinate data transfer between HandTrackingViewModel and SyncIPhoneSystem to ensure smooth updates.

Approach

1. Shared Instance: Use a shared instance of HandTrackingViewModel that both classes can access.
2. Direct Method Calls: Call a method on SyncIPhoneSystem to update the knuckle position whenever it changes in HandTrackingViewModel.

Implementation

HandTrackingViewModel

First, set up the HandTrackingViewModel with a way to notify SyncIPhoneSystem of updates:

@MainActor class HandTrackingViewModel {
    private let session = ARKitSession()
    private let handTracking = HandTrackingProvider()
    
    private var syncSystem: SyncIPhoneSystem?
    
    private(set) var knucklePosition: SIMD4<Float> = SIMD4(0, 0, 0, 1)
    
    func updateKnucklePosition(with position: SIMD4<Float>) {
        knucklePosition = position
        print("Setting HandTrackingViewModel.knucklePosition \(knucklePosition)")
        syncSystem?.updateKnucklePosition(with: knucklePosition)
    }
    
    func setSyncSystem(_ system: SyncIPhoneSystem) {
        syncSystem = system
    }
}

SyncIPhoneSystem

Next, set up SyncIPhoneSystem to receive updates:

class SyncIPhoneSystem: System {
    private var handTrackingViewModel: HandTrackingViewModel?

    init(handTrackingViewModel: HandTrackingViewModel) {
        self.handTrackingViewModel = handTrackingViewModel
        handTrackingViewModel.setSyncSystem(self)
    }

    func updateKnucklePosition(with position: SIMD4<Float>) {
        let pos4 = position
        print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")
    }

    func update(context: SceneUpdateContext) {
        // Other update logic
    }
}

Usage

Initialize and set up the HandTrackingViewModel and SyncIPhoneSystem correctly:

let handTrackingViewModel = HandTrackingViewModel()
let syncSystem = SyncIPhoneSystem(handTrackingViewModel: handTrackingViewModel)

// Update knuckle position somewhere in your code
handTrackingViewModel.updateKnucklePosition(with: SIMD4(-0.24767806, 0.9377898, -0.4052379, 0.99999994))

Key Points

• Direct Notification: By having HandTrackingViewModel directly notify SyncIPhoneSystem, you ensure that updates are immediate and consistent.
• Centralized Initialization: Make sure the HandTrackingViewModel is properly initialized and shared with SyncIPhoneSystem.
• Scalability: This approach scales well with your other interests, such as handling complex 3D interactions and maintaining low latency in your UI updates.

This method ensures a streamlined and efficient way to coordinate data transfer between HandTrackingViewModel and SyncIPhoneSystem, aligning with your focus on precise and responsive updates in your VisionOS app.Currently I never instantiate SyncIPhoneSystem.If SyncIPhoneSystem is never instantiated directly by you, it is likely being instantiated by RealityKit or another framework component that manages the system lifecycle. In this case, you can use a singleton pattern for HandTrackingViewModel to ensure that it is accessible globally.

Implementing a Singleton for HandTrackingViewModel

1. Define the Singleton for HandTrackingViewModel: This ensures that there is a single instance of HandTrackingViewModel accessible throughout your app.

@MainActor
class HandTrackingViewModel {
    static let shared = HandTrackingViewModel()

    private let session = ARKitSession()
    private let handTracking = HandTrackingProvider()
    
    private(set) var knucklePosition: SIMD4<Float> = SIMD4(0, 0, 0, 1)
    
    private init() {}
    
    func updateKnucklePosition(with position: SIMD4<Float>) {
        knucklePosition = position
        print("Setting HandTrackingViewModel.knucklePosition \(knucklePosition)")
    }
}

2. Modify SyncIPhoneSystem to Use the Singleton: This ensures that SyncIPhoneSystem accesses the singleton instance of HandTrackingViewModel.

class SyncIPhoneSystem: System {
    func update(context: SceneUpdateContext) {
        let handTrackingViewModel = HandTrackingViewModel.shared
        let pos4 = handTrackingViewModel.knucklePosition
        print("pos4 = \(pos4.x), \(pos4.y), \(pos4.z)")
    }
}

3. Use the Singleton in Your Application: Ensure that you use the HandTrackingViewModel.shared instance to update the knucklePosition.

// Update knuckle position somewhere in your code
HandTrackingViewModel.shared.updateKnucklePosition(with: SIMD4(-0.24767806, 0.9377898, -0.4052379, 0.99999994))

Key Points

1. Singleton Pattern: Use a singleton pattern for HandTrackingViewModel to ensure a single, globally accessible instance.

2. Access Singleton in System: Modify SyncIPhoneSystem to access the singleton instance of HandTrackingViewModel.

3. Update Singleton Instance: Update the knucklePosition using the singleton instance, ensuring that all parts of the application access the same data.

This approach ensures that SyncIPhoneSystem can always access the latest knucklePosition from the HandTrackingViewModel without needing to explicitly instantiate or pass instances around.

Knuckle Position Synchronization created at Tue Jan 20 1970 21:06:52 GMT+0000 (Coordinated Universal Time)