Using VideoTexture with Camera in AIR

These days I am trying to write simple iOS app that takes live camera stream and applies custom shader to it, as my pet project. And – surprise – I am doing this with AIR. That’s right, after I have finally come to terms with the fact that flash is dead and, considering how much GLSL is superior to AGAL as a shader language, I am still using AIR to build this app. Why? Because it works :) Unlike HTML5 aka flash killer. WebRTC is still not supported on iOS, and community polyfills simply are not enough to plug the gap. So, the only other option would be to use ObjC or Swift, but I know neither yet, while with flash I am at home (and, potentially, can easily port the app to Android later).

VideoTexture ?

Ok, so we have simple access to live camera feed, but how to bring it to Stage3D (flash version of WebGL) to apply the shader? Traditionally, we had to draw the display list video object into bitmap and re-upload it to GPU every frame, which was costly. But quite recently Adobe gave us more straightforward option – VideoTexture. Whole new runtime feature with amusing history, however, flash is still dead, so its documentation is crap. Just following NetStream-based snippets and changing attachNetStream to attachCamera like this

var videoTexture:VideoTexture = stage3D.context3D.createVideoTexture ();
videoTexture.attachCamera (camera);
videoTexture.addEventListener (Event.TEXTURE_READY, renderFrame);

did not work.

The missing piece ?

There is RENDER_STATE event that can tell you if video decoding is GPU accelerated (in the case of live camera stream it is not, btw). Turns out, you have to subscribe to this event even if you don’t care, or else the magic does not happen – camera stream data will never reach the texture and, correspondingly, TEXTURE_READY events will never fire. You’d think this is stupid and can’t possibly be true, but then adding event listeners is exactly how you get webcam activityLevel property to work, for example, so I am not surprised. With this, the final code to make it work is

var videoTexture:VideoTexture = stage3D.context3D.createVideoTexture ();
videoTexture.attachCamera (camera);
videoTexture.addEventListener (VideoTextureEvent.RENDER_STATE, function (e:*):void {
	// yes, we want TEXTURE_READY events
});
videoTexture.addEventListener (Event.TEXTURE_READY, renderFrame);