First off, here's a visual representation of my objective:
https://i.stack.imgur.com/5Uo1h.png
(Credit for the photo: )
The concise question
How can I use HTML5 video & canvas to execute a 4-point perspective transform in order to display only the "TV screen" portion of the frame on the canvas? Additionally, why does my current implementation not show the correct area?
Insight into my goal
I am in the process of constructing a webpage with the following functionalities:
- The user aims their webcam at a TV which may be positioned at any angle within the frame.
- The webcam footage is captured and previewed on the webpage using HTML5 video & canvas.
- The user defines the positions of the 4 corners of the TV screen by clicking on the preview (creating 4 pairs of x/y coordinates).
- ** The challenge lies in warping the video via some type of perspective transform so that the canvas exclusively displays the content within the actual TV screen, not the complete webcam view. **
- Following the transformation, image processing tasks are carried out such as identifying predominant colors. These processes do not fall within the scope of this query, except that I will eventually need access to the content/pixels of an HTML5 canvas.
My hurdle lies in step 4. To ensure that only the relevant part of each video frame is processed, it's crucial to "warp" the image to solely exhibit the "TV screen" region instead of the entire webcam feed.
Upon research, I've come to understand that:
- This necessitates a form of perspective transform, given the non-parallel alignment due to various angles the webcam could be set at; thus, a 3-dimensional transform is imperative over a 2D transform. A 2D transform wouldn't address the issue of converging sides effectively.
- HTML5 canvas is limited to two-dimensional transformations, rendering a 3D transform unfeasible. Since a solution compatible with
canvasis sought, leveraging a 3D CSS transform like https://developer.mozilla.org/en-US/docs/Web/CSS/transform-function/matrix3d is impractical. Perhaps delving into WebGL is necessary to accommodate the 3D aspect.
My Approach So Far
To tackle this, I pursued the subsequent method:
a) Capture webcam footage using a video tag.
b) Utilize three.js to construct a 3D scene rendered onto a canvas element allowing subsequent image processing.
c) The three.js scene involves:
- a flat mesh displaying the video on one side through a VideoTexture.
- an initial perspective camera configuration showcasing the entire webcam image.
d) Enable users to designate the four corner points indicating the TV location, compute respective x/y coordinates, and save them.
e) Compute a perspective transform stretching the image to fill the viewport based on the clicked "TV corner" points; I've been relying on this library: https://github.com/jlouthan/perspective-transform.
f) By applying the applicable transformation to the video-containing mesh while maintaining a fixed camera stance, I anticipate obtaining the desired image on the output canvas from a 2D viewpoint.
Link to My Current (Faulty) Implementation
Here's a link to my ongoing attempt outlined above. It permits video display and corner-clicking functionality. While successful around the center origin, issues arise when selecting other areas within the image.
https://bitbucket.org/mattwilson1024/perspective-transform/src/master/
In Conclusion
Your insights on why this isn't functioning as intended or suggestions for alternative/easier approaches to accomplish my requirements would be greatly appreciated.