Using VideoTexture with Camera in AIR

These days I am trying to write simple iOS app that takes live camera stream and applies custom shader to it, as my pet project. And – surprise – I am doing this with AIR. That’s right, after I have finally come to terms with the fact that flash is dead and, considering how much GLSL is superior to AGAL as a shader language, I am still using AIR to build this app. Why? Because it works :) Unlike HTML5 aka flash killer. WebRTC is still not supported on iOS, and community polyfills simply are not enough to plug the gap. So, the only other option would be to use ObjC or Swift, but I know neither yet, while with flash I am at home (and, potentially, can easily port the app to Android later).

VideoTexture ?

Ok, so we have simple access to live camera feed, but how to bring it to Stage3D (flash version of WebGL) to apply the shader? Traditionally, we had to draw the display list video object into bitmap and re-upload it to GPU every frame, which was costly. But quite recently Adobe gave us more straightforward option – VideoTexture. Whole new runtime feature with amusing history, however, flash is still dead, so its documentation is crap. Just following NetStream-based snippets and changing attachNetStream to attachCamera like this

var videoTexture:VideoTexture = stage3D.context3D.createVideoTexture ();
videoTexture.attachCamera (camera);
videoTexture.addEventListener (Event.TEXTURE_READY, renderFrame);

did not work.

The missing piece ?

There is RENDER_STATE event that can tell you if video decoding is GPU accelerated (in the case of live camera stream it is not, btw). Turns out, you have to subscribe to this event even if you don’t care, or else the magic does not happen – camera stream data will never reach the texture and, correspondingly, TEXTURE_READY events will never fire. You’d think this is stupid and can’t possibly be true, but then adding event listeners is exactly how you get webcam activityLevel property to work, for example, so I am not surprised. With this, the final code to make it work is

var videoTexture:VideoTexture = stage3D.context3D.createVideoTexture ();
videoTexture.attachCamera (camera);
videoTexture.addEventListener (VideoTextureEvent.RENDER_STATE, function (e:*):void {
	// yes, we want TEXTURE_READY events
});
videoTexture.addEventListener (Event.TEXTURE_READY, renderFrame);

One year of COLT

A year ago we have released first public version of COLT (short for Code Orchestra Livecoding Tool). It’s about time for some kind of reflection post.

Like other livecoding tools, COLT is watching your source files for changes that you make in any IDE or text editor. Unlike other livecoding tools, COLT does not restart your application but updates it with new code, and it keeps running as if nothing happened. This is cool and unprecedented feature, but of course we pay the price – we have to transform original code for this to work, and this transformation occasionally stands in the way of, or even breaks some things in your code – which looks like yet another bug in COLT to the user.

COLT was initially released for flash platform (if you was not around back then, check this demo or presentation to get a picture). We have received a lot of positive feedback and decided to expand the technology into other languages, such as javascript.

While javascript normally does not have any “compile time” and you can refresh the browser to see your changes instantly, there is the same common problem – you want to tweak the application logic in specific state, but you have not added easy way to save and restore that state. Then, there is crappy android emulator that is times slower than actual device to work with, when you are developing phonegap apps. Finally, there is node that you have to restart for source changes to take effect when you work on nodejs server application. We thought COLT would help you with all of the above (see this nodejs video, for example).

However, even the most enthusiastic users would perceive COLT as some weekend toy, opting for time-tested workflow for their actual work. We had to find another way to make COLT useful, and in the twisted turn of evolution COLT has developed special powers, when it comes to extendable text editors such as Sublime or Webstorm. Leveraging COLT’s connection to your application, our plugins for those editors can improve their limited code completion with runtime information

or provide new useful features such as variables inspection, navigating to function declaration or rendering pretty documentation. People seem to appreciate it already, e.g. check out slides 77 to 84 of this EmpireJS talk by Kirill Cherkashin. Funny thing about these features is that we could do them with minimal non-invasive code transformations, not having to deal with source maps or aforementioned “bugs”, but then COLT had to be renamed to COT.

For those who were still not convinced to try COLT we have added angularjs support. I mean, our live html editing support was already really good, but for angular fans we made it even better. Since I have probably exhausted average reader attention span by now, I will just briefly mention few other things here, such as new runtime environment support, chromeless COLT version, or proxy mode – but you can learn more in COLT 2.0 blog post if you feel like. And if you want to learn even more, consult our docs page, or just ask someone in the forum.

How to check if URL is not MP3 ?

Since flash player can’t send HEAD requests, you have to resort to dirty tricks if you want to check URL content type. In this example, we are going to try and figure out if some URL points to playable sound. Since we’re dealing with unknown URL, chances are that its domain does not permit us to access raw data, so we can’t really use URLLoader – we need to actually load whatever URL points to into Sound object. However, if you thought that you could just catch i/o error 2124, like you would do with image Loader, for example, you’re in for big surprise: it does not fire. Instead, Sound object fires progress and complete events, and is even happy to play (although you will not hear any noise, or anything at all).

So, how do we detect that our Sound object is actually broken? First thing you’ll notice is that even after complete event it is still buffering. Unusual, but the same thing happens with really short sounds, so this check is not reliable. The other thing is that extract() method obviously fails to return any samples. This in itself is again not enough, since it does happen with partially loaded sounds too. However, the combination of the two seems to hit the nail:

private function onComplete(e:Event):void {
   var s:Sound = e.target as Sound;
   if (s.isBuffering && (s.extract(new ByteArray(), 1, 0) < 1)) {
      // url does not point to playable sound
   }
}

See this method in action at wonderfl.net :)

COLT was updated to 1.2

By the way, codeorchestra livecoding tool (aka COLT) was updated to version 1.2 just now. Ironically, the changes are far from minor, since half of the COLT was refactored to support multiple languages (at the moment ActionScript3 and JavaScript) and use JavaFX instead of Eclipse RCP for UI. Let me quickly go over these news in detail:

JavaScript

JavaScript has eval() so at first it seems too easy to support (remember this ?) but it is not. We had to solve the problem of identifying multiple anonymous and / or nested functions that we could ignore in ActionScript. We had to change code transformation logic over and over because of sneaky corner cases that did not work. We had to deal with multiple runtimes, some of which do not support “basic” things like web sockets. On the bright side, surprising number of things has worked out of the box, or with minor modifications. Thanks to prior AS3 experience, JS support comes with all its blows and whistles such as infinite loops or recursion checks. For another example, being written with browsers and mobile apps in mind, COLT never the less is perfectly able to work with server side Node.JS code:

This does not look like a big deal to PHP programmers, but Node.JS guys actually have to restart the server to see their code changes. Check this post for more information about this use case.

ActionScript3

Unfortunately our work on AS3 was mostly frozen (much like the whole AS3 world itself). But we still love AS3. Few days ago, after many months of watching the grass grow, Adobe (actually their GPM, Bill “Machete” Howard) was finally able to grant us redistribution rights to AIR SDK. So now we can get rid of long-standing AS3 workflow issues. Hopefully.

Photoshop assets generator

Ten days ago Adobe started making noise around their Photoshop “autosave” feature called assets generator. Awesome feature, I totally loved it, and surely it works out of the box with COLT:

However, there is some room for improvement here. E.g. we could do what loom guys did and give you access to layers positioning information, if there is any interest. Let me know.

Spread the news!

If you like COLT and want to help us spread the news, download COLT, try it with your project and write the review on your blog. Contact me for more information, if interested.

About instance metadata in FP (continued)

Today Buraks released their final version of ASV update that supports PlaceObject4. And, I had to say, they did better job with their tool than Adobe did with their Flash CC. This is how metadata inspector dialog looks in ASV:

This is how it looks in Flash CC:

That’s not the first time the team of two does better job at tooling, so why should it be surprising? Any way, this inspired me to play with PlaceObject4 tag again.

What else is new?

Last time we learned that PlaceObject4 can carry complex nested objects, and “Integer, Double, or String” limitation is that of JSFL, not flash player runtime. It turns out the runtime does have limitations of its own – for example, you can’t have plain data there. If you place a string at the end of PO4 tag, flash player will not complain (you don’t get VerifyError) but corresponding metaData property will be null.

Btw, metaData is not really a property, but a reference. Documentation says it has a setter:

public function set metaData(value:Object):void

but IRL any attempt to use it results in TypeError #2004. However, if you change the value of the subfield, it sticks. So, metaData object is created once and the reference is returned from there on.

Another thing to know about instance metadata is that you can make it strictly typed. If you have the type registered at the time of publishing SWF, you will be able to specify what type will the metaData property be of when you access it. Make sure that corresponding class has default constructor, or else accessing metaData property will throw ArgumentError #1063.

But we still need tools to write that metadata :(

Right. Unfortunately there are none, and you will have to write your own. But don’t be sad – I can give you head start:

You’re welcome :) I originally wrote this with wonderfl in mind (hence 465×465) but they do not compile version 19 SWFs yet. Not tested thoroughly, feel free to report/fix bugs.