I recently had to prototype an app that would use webcam videofeed across multiple devices. It was a challenging problem to tackle, because even if your code worked on one device, there was no guarantee that it would work on the other devices. It was mainly due to the differences between how Apple develops their browser compared to Google. Painful.
Here are a couple of issues that I encountered. Here are some gotchas.
Gotcha 0: Https - A coworker of mine was working on the initial implementation before I came in. He was accessing the website with http, and couldn’t figure out why the video wasn’t loading. Well, it turns out that if you try to access any webcam feed through http, it will not pop-up asking for your permission to use the camera. Camera access must be done through secure protocol. A simple fix would be to always redirect http to https.
Gotcha 1: iOS devices require an inline tag in order to render the video on the browser. So your video tag should look like this:
Your video will not play if you don’t have the playinline tag.
Gotcha 2: This was for my application. The requirement was that we needed to draw objects on top of the video. For instnace, you want to draw an oval on top of a video.
The first step is you get a canvas, and then draw an oval like this.
First, you can add a canvas and draw your elements.
Link to Demo - enable video permissions.
OK, so you have both html elements. Now how are you going to put the oval on top of the video? There’s two approaches. But my fellow developer friend took the first link on Google and did this:
First, here’s the modified the code to takes frames from the video and draw to the top canvas.
function processVideo() {
let canvas = document.getElementById("canvas");
let canvasCtx = canvas.getContext('2d');
canvasCtx.drawImage(video, 0, 0, video.videoWidth, video.videoHeight); //this line was added here
canvasCtx.beginPath();
canvasCtx.strokeStyle = 'red';
canvasCtx.ellipse(video.videoWidth / 2, video.videoHeight / 2, video.videoWidth*0.3, video.videoHeight * 0.3, 0, 0, 2 * Math.PI);
canvasCtx.stroke();
well, now, we can’t have two elements showing our face now, right? So we’re going to hack our way and hide the video element by doing this:
let streaming = false;
video.addEventListener("canplay", function (ev) {
if (!streaming) {
let canvas = document.getElementById("canvas");
let canvasCtx = canvas.getContext('2d');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
//Chrome, Android will play this, but iOS and Safari will not!
video.width = 0 + "px";
video.height = 0 + "px";
video.style.position = "absolute";
video.style.left = 0 + "px";
video.style.top = 900 + "px";
streaming = true;
}
processVideo();
}, false);
OK. Yay, we fixed the issue, right? The video is 0px by 0px, but it will still be on the page! And it will write to the canvas!
Here’s a demo of that here…
And the full code here.
<canvas id="canvas"></canvas>
<video id="video" playsinline autoplay muted>Your browser does not support the video tag.</video>
<script>
var constraints = {
video: {
width: {min: 640, ideal: 1920 },
height: {min: 480, ideal: 1080 }
}
};
var video = document.getElementById('video');
if(navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
// Not adding `{ audio: true }` since we only want video now
navigator.mediaDevices.getUserMedia({ video: true }).then(function(stream) {
//video.src = window.URL.createObjectURL(stream);
video.srcObject = stream;
video.play();
});
}
let streaming = false;
video.addEventListener("canplay", function (ev) {
if (!streaming) {
let canvas = document.getElementById("canvas");
let canvasCtx = canvas.getContext('2d');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
//Chrome, Android will play this, but iOS and Safari will not!
video.width = 0;
video.height = 0;
video.style.position = "absolute";
video.style.left = 0;
video.style.top = 0;
streaming = true;
}
processVideo();
}, false);
function processVideo() {
let canvas = document.getElementById("canvas");
let canvasCtx = canvas.getContext('2d');
canvasCtx.drawImage(video, 0, 0, video.videoWidth, video.videoHeight);
canvasCtx.beginPath();
canvasCtx.strokeStyle = 'red';
canvasCtx.ellipse(video.videoWidth / 2, video.videoHeight / 2, video.videoWidth*0.3, video.videoHeight * 0.3, 0, 0, 2 * Math.PI);
canvasCtx.stroke();
try {
// drawOval(canvasOutputCtx, faces, 'red', size, eyes, eyesize);
var req = requestAnimationFrame(processVideo);
} catch (err) {
console.log(err);
}
}
</script>
Now, try this on iOS on an iPhone. It doesn’t work…
It’s interesting because on iOS Safari, it won’t render the 0 pixel by 0 pixel element. It just won’t. Similarly, if you set the pixel y coordinate to be off the screen, Apple’s Safari on Mobile Deice they will not render the video. For example, let’s say the viewing window is 900x1600, and your video element falls on position (900, 1700).
So who’s in the right here? Should the video element is 0 pixels by 0 pixels, should it actually function like it does in Chrome? Or should the browser not render the actual video at all like it in does in Safari? At this point, I’m not sure about which approach is correct. Chrome’s approach makes sense to have the video element function since there’s a downstream dependency on it. Safari’s approach isn’t worth either - why have the underlying element function if it’s not being rendered on screen?
Gotcha 3: Shape Flickering - If you draw something like a circle on the Canvas, but then delete the Canvas and draw the same one each frame, then the object will flicker on Apple devices. It will not on Android. Similarly, I found that canvas implementations seemed to be less robust on iOS. It’s not as smooth on Android when you have the same shape on canvas multiple times per frame. RangeError: Range consisting of offset and length are out of bounds set. You must copy the exact number of pixel values to an buffer object in IOS compared to Android.
Other Interesting Points: Part of the challenge was implementing some sort of facial detection for the project. If you Google this, the top results all point to using openCV’s landmark face detection. Unfortunately it currently doesn’t support the api call in JavaScript. So here I was, thinking of getting the source code and implementing that portion of the code myself.
However, what I later learned was that the weight files for that function requires 100MBs for a dependency! 100MB! You can’t load wait for that kind of data to download on the browser, so if I had gone through the approach it would have been a lot of time wasted since it would lead to a deadend. It made me reflect on the engineering process in general. What you conceptualize inside your head you’d work beautifully, but when it comes to practice there’s things you don’t know that you don’t know.
A good example of this is nuclear weapons. Just because you know the fundamental concepts that entail making a nuclear bomb, doesn’t mean that you can. The details and unexpected challenges in the physical world are much more nuanced and challenging - you can’t predict some of the obstacles.
DeviceOrientation and Motion:These set of events are triggered for accelerometer and gyroscope for handheld devices. Android enables permissions on devices by default. iOS requests permissions on your first time accessing the device. However, if you deny permissions, it will never ask you twice. Even if you reload the webpage or revisit you. You have to clear the cache.
Takeaway
It’s interesting because some minor things you can get away on Android, and then you debug on iOS it doesn’t work. I guess the big takeaway picture is something like this: Suppose you have two teams of engineers and let them implement X with the specs and let them work for a year without communication. Sure, it’ll support all the major functional requirements, but the underlying implementation could be very different.
The other thing it makes me wonder is about all the dominant products and services out in the market. Are they truly the best that society could offer that have been won through thick and thin of competitive forces? Are the best versions that are offered? Or are most of them the first implementations of an emerging idea that lucked out with the first mover advantage and shut out better ideas from existence?
What you see is a very fast simplifcation of everything that’s going on under the hood…
Resources used
https://www.html5rocks.com/en/tutorials/getusermedia/intro/ - This site doesn’t work on iOS
https://stackoverflow.com/questions/27420581/get-maximum-video-resolution-with-getusermedia
https://hibbard.eu/how-to-center-an-html-element-using-javascript/