Frame Photos with SwiftUI

TODO: collapasable, copyable, download, etc

I want to write an iOS app that lets you frame your photos. I am trying to use @Observable but I don't really understand what I'm doing. Here is my code

FrameMakerApp - SwiftUI App which has a ContentView and a Frame ContentView - Has a PhotosPicker, a @State var selected: PhotosPickerItem? and a @State private var picture: UIImage? . When a photo is chosen, we load it into picture. If the picture is present, we use a RealityView to render the picture onto a Plane.

I was trying to extract the ReailtyView into a new Frame class and to have the photo be changable. I got stuck on @ It sounds like you're working on an iOS app in SwiftUI that uses ARKit (since you mentioned RealityView) for framing photos. Extracting your RealityView into a separate Frame class and managing a changeable photo involves a few steps in SwiftUI. Here’s a general approach you might take to refactor your code and use the @ObservableObject and @Published property wrappers to manage state across your components.

1. Define an ObservableObject

You can create an ObservableObject that holds the state for the selected photo. This object will update any views that depend on it when the photo changes.

import SwiftUI
import Combine

class PhotoModel: ObservableObject {
    @Published var picture: UIImage?
    
    // Add any other photo-related properties and methods here
}

2. Update ContentView

In ContentView, you can use an instance of PhotoModel. When a photo is picked, update the picture property of this model.

struct ContentView: View {
    @StateObject private var photoModel = PhotoModel()
    @State private var isPickerPresented = false
    
    var body: some View {
        VStack {
            if let picture = photoModel.picture {
                Frame(photoModel: photoModel)
            } else {
                Button("Pick Photo") {
                    isPickerPresented = true
                }
            }
        }
        .sheet(isPresented: $isPickerPresented) {
            PhotosPicker(
                selection: $photoModel.picture,
                matching: .images,
                photoLibrary: .shared()) {
                    // Handle completion
                }
        }
    }
}

3. Define the Frame View

Create a new SwiftUI view called Frame that takes PhotoModel as a parameter and observes it. This view will re-render whenever the picture in PhotoModel changes.

struct Frame: View {
    @ObservedObject var photoModel: PhotoModel

    var body: some View {
        if let picture = photoModel.picture {
            RealityView(picture: picture)
                .frame(width: 300, height: 300)
        } else {
            Text("No picture selected")
        }
    }
}

4. Define RealityView

Make sure RealityView can take a UIImage and display it appropriately. Assuming you're using RealityKit, you might need to update a TextureResource or similar.

struct RealityView: View {
    var picture: UIImage

    var body: some View {
        // Example RealityKit view
        ARViewContainer(image: picture).edgesIgnoringSafeArea(.all)
    }
}

struct ARViewContainer: UIViewRepresentable {
    var image: UIImage

    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        // Setup ARView with the image
        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {
        // Update the ARView if needed
    }
}

Final Notes

• Make sure you have the necessary permissions to access the photo library and the camera if you're integrating AR features.
• Test the flow from picking a photo to displaying it in the AR view to ensure all data passes correctly between your views.

This setup should help you structure your app for better state management and separation of concerns. If you have specific errors or parts of the code you're stuck on, feel free to ask!I never see my RealityView code run more than once. I would expect it to run each time I choose a different photo: ```// // ContentView.swift // FrameMaker // // Created by Austin King on 3/9/24. //

import SwiftUI import RealityKit import RealityKitContent import PhotosUI

struct ContentView: View { // Observable object that can be changed @State private var userPhotoModel = UserPhoto()

// Temporary machinery for PhotosPicker
@State var selected: PhotosPickerItem?
// @State private var picture: UIImage?

@Environment(\.openWindow) private var openWindow

var body: some View {
    VStack {
        // if let picture {
        if let aPicture = userPhotoModel.selectedPhoto {
            Image(uiImage: aPicture)
                .resizable()
                .scaledToFit()
            Spacer()
        }
            
        if let aPicture = userPhotoModel.selectedPhoto {
            
        RealityView { content in
            
            
            let width: Float = Float(aPicture.size.width) / 10000.0
            let height: Float = Float(aPicture.size.height) / 10000.0
            
            print("generating \(width) x \(height)")
            
            let planeMesh = MeshResource.generatePlane(width: width, depth: height)
            
           //  let planeMaterial = // UnlitMaterial.Color(patternImage: picture)
            // AOK start copy/pate
            let texture = try! TextureResource.generate(from: aPicture.cgImage!, options: .init(semantic: .color))
            print("Running after image changed \(aPicture)")

            var material = UnlitMaterial()

                    // note: color will be different if anything other than .white is specified
                    material.color = .init(tint: .white, texture: .init(texture))

                    // note: Using 1.0 was too dark, so 2.0 is used
                    material.blending = .transparent(opacity: PhysicallyBasedMaterial.Opacity(floatLiteral: 1.0))
            // AOK end copy / paste
            
            let plane = ModelEntity(mesh: planeMesh)
            plane.model?.materials = [material]
            // plane.transform.rotation = simd_quatf(angle: Float.pi, axis: SIMD3(x: 1, y: 0, z: 0))
            // plane.transform.rotation = simd_quatf(angle: Float.pi, axis: SIMD3(x: 0, y: 0, z: 1))
            // plane.transform.rotation = simd_quatf(angle: Float.pi, axis: SIMD3(x: 0, y: 1, z: 0))
            
            // https://stackoverflow.com/questions/59294602/how-do-i-rotate-an-object-around-only-one-axis-in-realitykit
            plane.orientation = simd_quatf(angle: .pi/2,     /* 45 Degrees   */
                                           axis: [1,0,0])   /* About Z axis */
            
            plane.transform.translation.x -= 0.1
            plane.transform.translation.y -= 0.2
            
            plane.transform.translation.z += 0.2
            
            
            // plane.position = World
            // TODO: fuck around with anchros
            // let anchor = AnchorEntity(world: .zero)
            
            content.entities.removeAll()
            content.add(plane)
            
        }
            
        }
        
        /* Model3D(named: "Scene", bundle: realityKitContentBundle)
            .padding(.bottom, 50)*/
        
        

        Text("Make a Frame")
        /* Button(action: pickImage, label: {
            Text("Choose a Photo")
        })*/
        Button(action: pickImage, label: {
            Text("Show Frame")
        })
        PhotosPicker(
            selection: $selected,
            matching: .images
        ) {
            Text("Select a Photo")
        }
    }
    .padding()
    .onChange(of: selected) {
        Task(priority: .background) {
            if let data = try? await selected?.loadTransferable(type: Data.self) {
                print("AOK photo picker onChange fired")
                userPhotoModel.selectedPhoto = UIImage(data: data)
            }
        }
    }
}
    
    func pickImage() {
        // https://developer.apple.com/videos/play/wwdc2023/10107/
        openWindow(id: "frame")
    }

}

/* #Preview(windowStyle: .automatic) {

ContentView(currentUserPhoto)

} */ ```It looks like the issue where your RealityView code is not re-running might be tied to how SwiftUI views react to data changes. If RealityView is a custom component that internally processes significant changes (like changing the photo to be rendered), it needs to be made reactive to changes in its input data — in this case, the aPicture passed to it.

Here's how you can adjust your setup to ensure the view updates reactively:

1. Ensure RealityView is a Proper SwiftUI View

First, ensure that RealityView is a proper SwiftUI view that reacts to changes in its properties. If RealityView is a custom SwiftUI view that wraps a ARView, it should take the image as a @Binding or at least as a property that SwiftUI can track.

If you haven't already, modify RealityView to accept the image directly as a property:

struct RealityView: View {
    var picture: UIImage

    var body: some View {
        // Your AR rendering logic here, which should use 'picture'
        ARViewContainer(image: picture).edgesIgnoringSafeArea(.all)
    }
}

2. Use @ObservedObject or @StateObject

If RealityView relies heavily on userPhotoModel, ensure this model is provided as an @ObservedObject (if passed from a parent view) or @StateObject (if owned by the view) so that changes in userPhotoModel.selectedPhoto trigger view updates:

struct ContentView: View {
    @StateObject private var userPhotoModel = UserPhoto()  // This should probably be @StateObject if it's initialized here

    var body: some View {
        VStack {
            if let aPicture = userPhotoModel.selectedPhoto {
                RealityView(picture: aPicture)
            }

            Button(action: pickImage) {
                Text("Show Frame")
            }
            PhotosPicker(selection: $userPhotoModel.selected) {
                Text("Select a Photo")
            }
        }
        .onChange(of: userPhotoModel.selected) {
            // Load the new image
            // This might not be necessary if UserPhoto handles it
        }
    }
}

3. Debugging

To ensure that changes are captured, add more debugging print statements within theRealityView is part of the iOS frameworkAh, if you're referring to a RealityKit view or something similar, that does change things a bit. Since RealityKit isn't inherently designed to be reactive like SwiftUI components, you need to explicitly handle updates to make sure the AR content reflects the new image each time it changes.

You can manage updates in SwiftUI by ensuring that whenever the image changes, the whole AR scene or the relevant parts get properly refreshed. Here’s how you might approach it:

Ensuring Updates in RealityKit

1. Explicitly Remove and Re-add Content: Since AR content doesn't automatically update when a state changes, you need to make sure you're properly refreshing your content. This might involve removing all existing entities and re-adding them when the image changes.

2. Use @State or @ObservableObject for Image Updates: You are currently using an @ObservableObject which is good. However, make sure every part of your SwiftUI body that depends on this object correctly reacts to changes. It seems like you're doing this, but you may need to explicitly reset the AR scene when the image changes.

Here’s a refined example, assuming RealityView is a placeholder for where you use RealityKit views:

struct ContentView: View {
    @StateObject private var userPhotoModel = UserPhoto()
    
    var body: some View {
        VStack {
            if let aPicture = userPhotoModel.selectedPhoto {
                ARViewContainer(image: aPicture)
            }
            Button(action: pickImage) {
                Text("Show Frame")
            }
            PhotosPicker(selection: $userPhotoModel.selected) {
                Text("Select a Photo")
            }
        }
        .onChange(of: userPhotoModel.selected) { _ in
            loadNewImage()
        }
    }
    
    func loadNewImage() {
        Task(priority: .background) {
            if let data = try? await userPhotoModel.selected?.loadTransferable(type: Data.self) {
                userPhotoModel.selectedPhoto = UIImage(data: data)
            }
        }
    }
    
    func pickImage() {
        // Implementation depends on how you want to trigger photo selection
    }
}

struct ARViewContainer: View {
    let image: UIImage

    var body: some View {
        CustomARView(image: image)
            .edgesIgnoringSafeArea(.all)
    }
}

struct CustomARView: UIViewRepresentable {
    let image: UIImage

    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        setupARContent(arView, with: image)
        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {
        // This gets called if SwiftUI sees a change in the `image`
        setupARContent(uiView, with: image)
    }

    private func setupARContent(_ arView: ARView, with image: UIImage) {
        arView.scene.anchors.removeAll()
        
        let texture = try! TextureResource.generate(from: image.cgImage!, options: .init(semantic: .color))
        var material = UnlitMaterial()
        material.color = .init(tint: .white, texture: .init(texture))
        
        let width: Float = Float(image.size.width) / 10000.0
        let height: Float = Float(image.size.height) / 10000.0
        let planeMesh = MeshResource.generatePlane(width: width, depth: height)
        
        let plane = ModelEntity(mesh: planeMesh, materials: [material])
        let anchor = AnchorEntity(world: .zero)
        anchor.addChild(plane)
        arView.scene.addAnchor(anchor)
    }
}

Key Points:

• React to Changes: Use .onChange(of:) to react to changes in the photo picker selection and then update the AR content.
• Update AR Content Dynamically: When updating AR content, make sure to clear the previous content and set up new content with the updated image. This ensures your AR scene reflects the new image each time it changes.

This should help make sure your AR content updates whenever you choose a new photo, making it more interactive and responsive.

Frame Photos with SwiftUI created at Tue Jan 20 1970 20:22:04 GMT+0000 (Coordinated Universal Time)