This is an implementation of generalized Perspective Projection.

Enter your physical Screen-Size into 'Screen Values' in the sidebar to make it work for your screen.

Starting point was this article: https://edom18.medium.com/implementation-of-generalized-perspective-projection-on-the-unity-c9472a94f083

which follows this paper: http://160592857366.free.fr/joe/ebooks/ShareData/Generalized%20Perspective%20Projection.pdf

I did this for a research project into "Virtual Windows".

The idea was to give a camera operator the option to have a "virtual window" even on a small set.

For position tracking, I would usually use the zigSimPro app on the iPhone.

Because I wanted to avoid the need for an OSC server, I also worked in unity to create a websocket server on the phone. That way I could send data directly for ARkit to the browser (you only need to turn off the security settings). But the phone usually got to hot running the server, throttling the position-tracking in the process...

The downfall for this setup was the latency between devices. If you want a filmable virtual window, you need close to no latency between the motion of the camera and the simulated motion of the screen. At the moment, this is not really possible without extra hardware.