The Road to RealityKit 🥽

[tikimcfee]

Current state

• Working Metal implementation with picking / tracing / highlighting / glyph instancing.
• Old and still working commits and branches with existing SceneKit / ARKit implementations that may help with RealityKit port.
• MetalLink is essentially a fork of a ; I've only bolted on functionality whilst learning and building.
• Tracing is currently MIA and needs some touch up to work either within BitHandling or as a separate module. There might also be some #macro magic that can work to replace it.. for my use cases, at least.

Early things to do

• The ReadMe needs a major update, haven't really touched it since the first Metal implementation.
• It may be time to rework from building a scene-graph to just doing mesh and texture rendering.
• Test whether anything is salvageable from the old SceneKit / ARKit implementations. The goal is to use the Immersive or Volumetric space.

Looking for help

As always, I'd love input! Feel free to chat here, post issues, or anything in between.

Let's do this thing.

[tikimcfee]

After a a great technical conversation, I'm going to start looking in to some of the mesh / entity primitives to see if any of the direct instancing parts can be reused or salvaged. It seems doing direct-Metal rendering is currently a VR-Only experience (need a source), and one most use the RealityKit/Reality view pairing to go full immersive. So, the experiments will begin!

[lqj01]

I have found that this project may be helpful for the support work of visionOS. By transferring the visionOS simulator screen to other VR devices, it may be possible to more intuitively view the working status in visionOS. https://github.com/zhuowei/VisionOSStereoScreenshots/tree/alvr

[tikimcfee]

Hey, many apologies, I never saw this notification, or forgot, that's on me. This looks cool, I'll need to dig into it to see how it works. It would be great if it was more a less a drop in to be able to manually feed the rendering flow that vOS is expecting when using that system.

[MostafaOkasha]

I know someone will full-heartedly build an amazing code editor for VR - but with Github Copilot, and GPT integrations like Diagrams:Show Me... there is an idea for creating a super comprehensive AI-powered VR code editor. I mean... the only problem is it might be too soon to go all out on a platform when their device is selling for such a high price point... The idea is 100%, the timing feels like 50-70%. Anyway, sorry for blabbering, I'm really excited about this space and you two seemed to be the only ones interested lol

[MostafaOkasha]

On a separate note... Not sure if you saw this but was pretty cool. https://developer.apple.com/videos/play/wwdc2023/10073/

[MostafaOkasha]

@tikimcfee ok... I just saw your video presenting this and I'm in love! I love how your brain is thinking about this. I'd love to work with you on this. @lqj01 did you see it?

[tikimcfee]

@MostafaOkasha Well hello there! I'm super excited to see ya here, and moreso that you're digging the project!

In reverse order:

I'm really glad you found that video but I'm curious... what video did you find? 😂 I remember I was on a bit of a sharing spree a while back and sent some things around. Whatever version it is, it's a really cool feeling to know it connected with ya.

I agree with your point about this being inevitable. It's been done, in fact! The nearest functioning version I've seen of this is Primitive, and while it looks really awesome, I've never been able to get a working copy on my headset - or at least, I haven't tried as hard in the same way as others. It's unfortunate, but I think the folks over there are probably hard at work making awesome stuff 'cause they're hard to track down. And don't worry - you're not blabbering. You're excited, and I am too - it's a great capital-z-feelZ to share that, haha.

And the best part in my books: I would love for you to help with this. I've been scouring the corners looking for folks that would have a few minutes here and there to brainstorm, maybe even throw some code at the wall to experiment. To that end, I humbly offer up any time you'd like to get a little walkthrough (flythrough? ;) ) of the stuff that makes this work, and see what you have to think and say. Sincerely - just drop a line to my DMs, or even fire something into my calendly (I'm NOT busy at all, lol, it's just an easy way to share up links and times) https://calendly.com/thelugos/talk-with-ivan.

I'm at your disposal, and I just got a little spark of motivation to try a couple more things out =) I'm sure having another pair of eyes and brainial lobes on it can help narrow what to work on too.

Thanks again for reaching out @MostafaOkasha - really looking forward to seeing what we do!

[MostafaOkasha]

Hello Hello 😄 Excited to have run into this honestly.

In reverse reverse order:

I tried looking for the video link but I go through 1000+ tabs per day D: You were in your backyard and you used your phone to demo the spacial placement! Def. connected with me though. I've been thinking about how to slowly start building a fully immersive coding experience that can shatter the disconnect between coders and code. VR has always been the answer but the technology was never there yet. I can only begin to imagine how crazy it would be on the Vision Pro... 🤤 🤤

I've seen Primitive too but I don't necessarily like what the main dev is doing with it. It's pretty cool ngl, and they've put in a lot of work I know. But I think there is a group of right answers when it comes to asking what components are required to build the best user experience in this AR/VR coding world, and I don't see it with them (yet). I think you're on the right path though! The path to visualization will be different, and I think there will be a lot of people building in this space soon, each with their unique visualization methods. Either way, I can imagine how time-consuming it would be to get it into a headset, something to def. talk about.

I'd love to help as well, I'm fairly new to Swift but I'm currently learning! Trying to build a push-to-talk Whisper Menubar app for Mac. Shouldn't take me long to get to a state where I can contribute and help coding too. Wait, are you the same dude I connected on Linkedin with? Pretty sure it's you. Shoot me a message and we can jump into a proper place to talk! I put us up for a meeting on Thursday either way for a code flythrough 🛸

Glad to have motivated you :D

[tikimcfee]

@MostafaOkasha amigo, I'm sorry I missed our meeting. I messed up timing with work, and my little margin was chewed up by another "impromptu chat". Let's try again, and I'll happily jump on whatever schedule you have of course - only fair. And again, whatever makes it easier.

Thanks again for taking your time, sir.

[JaLnYn]

I've recently been looking into building a VR ide. I imagine the ide would still connect to a laptop or some cloud computer so programs are runnable. I've set up a backend for it here. I've recently moved on to the frontend. I'm currently planning a ui that could make use of VR. This looks like a really interesting way to go!

Let me know if you want to discuss some ideas!

[tikimcfee]

@JaLnYn I would absolutely love to chat with you about your work. That backend looks like exactly the kind of tool that this would fit on top of. The rendering system is just generic Metal draw calls and a little compute, so strapping a web socket in to the current flow of data and UI would be easy, and even fun. Did you want to schedule up a call, or throw some note in this discussion? I'm game for either or both =)

[JaLnYn]

Yea let's do both :). Some notes would be good. I'll read up before our chat. Let's set up a time.

If you want we can connect, this is my linkedin.