Archive for the '3D' Category

Transparent video texture in three.js

Some weeks ago I was asked if it is possible to have transparent video texture in THREE.js and, although I was sure it’s totally possible, I could not find any examples of it. Today I am sharing one such example with you, so you don’t have to code it yourself in case someone will ask you the same question :)


This demo is using good old “green screen” trick to create transparency in otherwise opaque texture. You can find similar examples out there, but they are neither THREE.js- nor GPU- based.

So, where do you start? I started here – you can easily tell how much code was borrowed from Lee Stemkoski. Basically, I only had to replace MeshBasicMaterial with ShaderMaterial. Creating custom material sounded scary at first, but turned out to be really easy in the end thanks to helpful tutorials (such as this one). Finally, the shader – I want it to be as simple as possible, so no fancy color conversion stuff in there – it operates directly in RGB using 1st method from my earlier post (err, no, it’s different method). The result is basically two lines of code:

vec3 tColor = texture2D(texture, vUv).rgb;
gl_FragColor = vec4(tColor, (0.9 - dot(color, tColor) / (length(tColor) + 0.05)) * 3.0);

Let’s break 2nd line down. It starts at the dot product of colors – the dot product approaches 1 when texture color approaches the color we want to remove. In theory, it should have been dot (normalize(color), normalize(tColor)), but normalize(tColor) part fails on pure black pixels (such as girl hair), so I had to use tColor / (length(tColor) + small number) instead. This handles black colors better, but now the dot is always less than 1 – hence you see 0.9 in there. You can also see that I do not normalize “color” parameter – there is no good reason to do it inside the shader for every pixel. Finally, 3.0 is randomly chosen magic number that determines how fast texture colors become transparent as they approach the color to remove – check out this graph to see its effect.

Update: I have just made the shader even simpler using length(tColor – color). Initially I thought absolute color values will not work well, but the actual test says they do:

Angle-based shader on the left, distance-based shader on the right.

Angle-based shader on the left, distance-based shader on the right.

Using D3.js with THREE.js, part II

So, our little D3 + 3D experiment continues. This time we will be trying to replicate bar chart tutorial 2 in 3D.


Two things need to be done before D3 transitions could work. First, D3 must be able to interpolate “attribute” values, so we need to implement both setAttribute and getAttribute methods in THREE.js:

// this one is to use D3's .attr() on THREE's objects
THREE.Object3D.prototype.setAttribute =
  function (name, value) {
    var chain = name.split('.');
    var object = this;
    for (var i = 0; i < chain.length - 1; i++) {
        object = object[chain[i]];
    object[chain[chain.length - 1]] = value;

// and this one is to make'em work with D3's .transition()-s
THREE.Object3D.prototype.getAttribute =
  function (name) {
    var chain = name.split('.');
    var object = this;
    for (var i = 0; i < chain.length - 1; i++) {
        object = object[chain[i]];
    return object[chain[chain.length - 1]];

Second, D3’s selectAll() now needs to actually select something to build updating selection. We can make it select various descendants of Object3D type, for example:

// now selectAll must actually do something
THREE.Object3D.prototype.querySelectorAll =
  function (selector) {
    var matches = [];
    var type = eval(selector);
    for (var i = 0; i < this.children.length; i++) {
        var child = this.children[i];
        if (child instanceof type) {
    return matches;

These changes are already enough to replicate their simple transition example in THREE.js:

var t = 1297110663,
    v = 30,
    data = d3.range(9).map(next);

function next() {
  return {
    time: ++t,
    value: v = ~~Math.max(10,
      Math.min(90, v + 10 * (Math.random() - .5))

setInterval(function() {
}, 1500);

function redraw() { chart3d )
    .attr("position.y", function(d, i) { return d.value; })
    .attr("scale.y", function(d, i) { return d.value / 10; })

But to replicate their final keyed data join example, we need to take care of one last thing…

Exit selections

We want D3’s remove() to actually remove 3D objects. Now, you might be thinking, this is easy one, we’ll just add removeChild() to Object3D prototype

THREE.Object3D.prototype.removeChild =
  function (c) {

and it should work, right? Wrong, it does not! To solve this one, I had to check D3 source. Turns out D3 does not call removeChild on selection parentNode but on node’s parentNode itself, so we need to modify our appendChild as follows:

THREE.Object3D.prototype.appendChild = function (c) {
    // create parentNode property
    c.parentNode = this;
    return c;

Final result

With these changes, it is now possible to do this:

function redraw() {

	var bars = chart3d )
		.data(data, function(d) { return d.time; });

	// move existing bars to their new place
			function(d, i) { return 30 * i; })

	// add new bar and make it grow
	bars.enter().append( newBar )
			function(d, i) { return 30 * (i + 1); })
		.attr("position.y", 0)
		.attr("scale.y", 1e-3)
			function(d, i) { return 30 * i; })
			function(d, i) { return d.value; })
			function(d, i) { return d.value / 10; })

	// remove the bar that is no longer in data set
			function(d, i) { return 30 * (i - 1); })
		.attr("position.y", 0)
		.attr("scale.y", 1e-3)

Final demo

This is still not complete THREE.js “plugin” but, as you can see, it’s a nice start.

Using D3.js with THREE.js

This blog name has “3D” in it, but there was no 3D here for a long time now. So I am fixing this tonight.

Once upon a time we did simple AS3 port of D3.js lib here at CodeOrchestra, codenamed “D6” by Potapenko. The name D6 came from D3 + 3D, since our port was able to work with 3D engines to enable 3D data visualization. That port never went into production, but the idea behind it is still valid today.

I mean, can you do 3D with D3? There is no 3D examples in their gallery, so the obvious answer seems to be “no”. The reason is that D3 was built with browser DOM in mind, and it is not straightforward to extend it to support arbitrary object models such as THREE.js, de facto standard for 3D in javascript world. Nevertheless, as you can guess, it could be done.

Let’s take a look at D3 simplest example (bar chart):".chart")
    .style("width", function(d) { return d * 10 + "px"; })
    .text(function(d) { return d; });

Here, a number of methods accept browser-specific magic strings such as “.chart” selector or “div” tag name. While convenient in 2D browser environment, this is totally useless for our purposes. But have no fear, for D3 comes with alternative signatures for these methods! These signatures were designed for boring things like reusing existing selections, but we can actually bend them to swallow non-DOM objects. For example, we can rewrite the bar chart above like this:

function createDiv() {
	return document.createElement("div");

var chart = {
	appendChild: function (child) {
		// this is called by append() after createDiv()
		return document.getElementById("chartId")
	querySelectorAll: function () {
		// this is called by selectAll()
		return [];
} chart )
  .enter().append( createDiv )
    .style("width", function(d) { return d * 10 + "px"; })
    .text(function(d) { return d; });

Here, we 1) explicitly instruct D3 how to create “div” tags, and 2) trick D3 into thinking that “chart” is normal DOM object, while the end result of the script is exactly the same as before. Now when we learned how to work around D3 helpful APIs, we can easily do the same with THREE.js:

THREE.Object3D.prototype.appendChild = function (c) {
	this.add(c); return c;

THREE.Object3D.prototype.querySelectorAll = function () {
	return [];

chart3d = new THREE.Object3D(); chart3d )
  function (d, i) {
	var bar = new THREE.Mesh( geometry, material );
	bar.position.x = 30 * i;
	bar.position.y = d;
	bar.scale.y = d / 10;
	return bar;

In this demo, I actually went ahead and made D3’s attr() calls work for why not.

Well, this is it for today :) I hope to continue this post some day with part 2, where I’d get D3’s transitions to work, but right now I need to sleep. Zzzz..

Gyroscope adventure

In case you can’t fully understand yet what Adobe means by focusing on (desktop) gaming and video and not general purpose apps, allow me to demonstrate. Suppose you are making mobile app and for whatever reason you want to track device orientation. How hard can that be? Check it out:

Apple way

If you’re native iOS app developer, you are dealing with ecosystem that does support general purpose apps. You have simple CMDeviceMotion class that performs sensor data fusion, filtering, bias removal, etc, and gives you orientation data in any form you want (roll + pitch + yaw, or matrix, or quaternion). Case closed. The same class conveniently splits accelerometer data into gravity and user applied acceleration, btw. In a word, it’s perfect.

Adobe way

If you’re AIR app developer, prepare yourself – you’re going on the adventure. You can start by looking at flash.sensors package – for some mysterious reason, it has accelerometer API for desktop, but no gyroscope API for mobile. Why? Because Adobe does not feel obliged to include every piece of code you might think of in their runtime. But don’t give up your hope yet – Adobe has the solution for this kind of problems – AIR native extensions. And, it turns out that Adobe made one for you to access gyroscope data. Despite of being abandoned in 2011, apparently just before Adobe’s focus shift kicked in, this ANE still works with current AIR 3.x.

Okay, this is the part where you’re happy that you don’t have to write ANE yourself, and it seems that getting your hands on device orientation data is now only a matter of few additional build settings. Well…

This extension API, designed to be Accelerometer class twin, has two major problems in the way. Number one is that it only exposes raw gyroscope data, i.e. rotation rate. Since this is not something that flash developers do every day, you probably have no idea how to go from radians per second to orientation quaternion, or transformation matrix. So to use this ANE, you will have to learn some new tricks, such as finding approximate differential equation solutions, Taylor expansion of matrix functions, series summation and evaluating indeterminate forms. If you are still with me, good guy Oliver J. Woodman will guide you through this hell (pages 21 to 23).

Ok, now you’re done reading that and ready for action, right? Not so fast – there is problem number two. Let me just quote this comment from ANE source code:

// The singleton ExtensionContext object listens for StatusEvent events that
// the native implementation dispatches. These events contain the device’s
// gyroscope x,y,z data.
// However, each Gyroscope instance has its own interval timer. When the timer
// expires, the Gyroscope instance dispatches a GyroscopeEvent that contains
// the current x,y,z data.

Thank you, Adobe engineer, for this brilliant idea. Because there just can’t be enough lag or noise in my data. Sigh… so what do you do about this? You bypass ANE AS3 wrapper, and access its internal methods over low level interface. Of course, this means that you have to study both wrapper and native code beforehand :(

And here, my friend, we come to the end of our wonderful journey. I will go back to work on my AIR app, and you will maybe go and write your own ANE, with blackjack and hookers. Or you may choose to continue where I left off – maybe add Kalman filter or something. And don’t forget to write back, if you do :)

Grab my balls

So, I have started cross-engine 3D primitives project at github. Just like as3dmod, it plugs seamlessly into your favorite engine to help you out whenever you feel limited by it. Initial commit supports Alternativa3D 8.12, Away3D 4 Alpha, Minko 1 and contains two spheric primitives: UniformSphere and Globe.

UniformSphere is a mesh built from the spiral of vertices uniformly distributed across the sphere. You have two options here: Rakhmanov formula (default, the spiral joins sphere poles) or Bauer formula (the spiral does not contain poles). This primitive was ported from my 3D metaballs code.

Globe is a mesh that minimizes texture distortions for classic plate carré projection. It does so by placing equilateral triangles along the equator and gradually compressing triangles towards each pole (where high triangle density is actually good property in terms of texture distortions). This primitive was ported from my globe component for FP9, which is now also hosted at github.

I plan to add few more primitives over time, both useful and not really, and maybe more engine proxies if there will be any interest.

Still messing with qtrack

Ok, boring things first. I uploaded another qtrack version with some minor fixes and improvements, and also added broomstick example there. Nothing groundbreaking so far.

Now, to fun things. I tried to make non-square marker qtrack mod based on this method. You can see in this video how far I could push it:

but, overall, this was a failure. Unlike that guy, I had only one shape to detect, so at 1st I thought I could use max ICF value between blob and marker contours (ideally 1) and compare it to confidence threshold. However, this picked way too much false positives. So in 2nd iteration, that you see in the video, I had to calculate max ICF values for few hundred blobs 1st, and then pick the blob with max of those values. This worked somewhat better, but in the long run, it is still unreliable. Here is a distribution of max max ICF values thresholded at 0.7:

As you see, even shape as simple as flash logo produces really poor values most of the time. Another equally significant problem is that BitmapData’s threshold() method produces poor contours when threshold is not optimal: over-illuminated patches of the marker punch random “holes” in its binarized version and contaminate detected contour. To sum up, you will have to stick with quad markers for some time, sorry.

Update on qtrack

For those of you who are still interested in my free marker-based flash AR tracker, I updated qtrack to v1.0. There were actually only few minor changes, however. 1st, I changed detector code to avoid v0.9 marching squares bug; interesting side effect of this change is that now literally anything that remotely resebles a quad can be used for the marker (like those examples on the right). 2nd, I added requested option to track the marker in NetStream-ed video. 3rd, there is new makePlaneTransform method that should help you with molehill setup (but no molehill demo included for now). 4th, some minor stuff you wouldn’t care about.

On a related note, back in December I promised to make Alternativa3D AR video tutorial for russian xmas, that would include qtrack and other trackers available through FLARManager. I did not finish that video in time, so for this post I uploaded qtrack part only (and chances are that FLARManager part will be only available in text version of tutorial). The video is in russian, but you can try english subtitles. Enjoy:

Old stuff

January 2017
« Jan    

Oh, btw…