This site is now just an archive over the rise and fall of Flash. The domain flashmagazine.com is available for sale
Login | Register
Using Pixel Bender to calculate information

Using Pixel Bender to calculate information

Pixel Bender can be used for calculations of many specific cases. Using Pixel Bender can increase your application's performance as Pixel Bender runs on a faster and separate thread than the Flash Player. In this article I cover how to use pixel bender for calculations as well as show you tutorials of building a real-life application.

This is how it works. The syntax is based on GLSL. The Pixel Bender graph is an XML language for combining individual pixel-processing operations called kernels, which are in PBK file format. We can create the PBK XML file using the Adobe Pixel Bender toolkit.

After our kernel is ready we can then export it as byte-code in a .PBJ file. PBJ can then be used in the Flash 10 player. Additionally, there is another file format called Pixel Bender graphs (PBG), which is supported directly in the Photoshop/After Effects CS4 Pixel Bender extension. The use of these are outside the scope of this article.

So what can you do with Pixel Bender? You can use Pixel Bender for the following graphics:

  • Filters: You can create a Pixel Bender shader (using the ShaderFilter method) and use the Pixel Bender kernel as a filter on any component based on DisplayObject, just as you use other filters
  • Fills: The beginShaderFill method allows you to create custom fills to be used as backgrounds and add to the existing fills built into Flex (i.e., solid, gradient, and bitmap)
  • Blend modes: By assigning the blendShader property, you can control how DisplayObject is composited and what’s beneath it
  • Calculations: You can use the Pixel Bender kernel for mathematical calculations

A lot was written regarding using Pixel Bender for filters, fills and Blend modes. In this article we are going to talk about using Pixel Bender for calculations.

Why use Pixel Bender for calculations?

The Flash Player runs as a single-thread on all machines and when creating large applications that uses a lot of resources such as memory and CPU, we should use everything we can to increase performance and remove large calculations from the player. Pixel Bender can be leveraged for calculations of data, which may increase our application performance if used correctly.

Most Pixel Bender instructions execute faster than similar instructions do in the Flash Player. Since Pixel Bender kernels and the Flash Player will run as separate threads, most modern processors with multiple cores can run the same code more than twice as fast or even faster. On an older machine with a single core CPU, Pixel Bender kernels will be executed on the same core as the Flash Player resulting in little or no speed gain.

Pixel Bender roundtrip workflow:

  • Create .pbk in the Pixel Bender Toolkit.
  • Export – Once the pbk file is created you can run the Pixel Bender Toolkit and convert the XML into .pbj binary.
  • Embed or dynamically load – use the .pbj file in your application.

The Pixel Bender Toolkit supports the following math functions that we can use:

  1. sin(x) - Trigonometric sine function
  2. cos(x) - Cosine function
  3. tan(x) – Tangent function
  4. asin(x) - Arcsine (inverse sine) function
  5. acos(x) - inverse cosine (arccosine) function
  6. atan(x) - Arctangent (inverse tangent) function
  7. atan(x, y) - Arctangent (inverse tangent) function
  8. exp(x) - exponential function
  9. log(x) – Logarithm function
  10. pow(x, y) - power of function
  11. reciprocal(x) - multiplicative inverse function.
  12. sqrt(x) - square root function.

Here are some examples of real-life applications use sine, cosine, tan etc:

  • Compression algorithms
  • 3D calculations – such as Papervision3D and Matrix3D
  • Surveying, navigation and astronomy position calculations
  • Audio is composed of waves
  • Ballistic trajectories
  • Space flight and satellite positioning
  • GPS and cellphones rely on triangulation and formulas involving sin/cos
  • Signal transmission, such as TV or radio broadcasting, involves waves described with sin/cos waves

Flash Player single thread limitation

How useful is Pixel Bender for these calculations? The Flash Player being single threaded means that while the Player is processing information it cannot do other things. We often find the player get “stuck” when doing heavy lifting. Pixel Bender can help in certain cases. I created an example and an API that uses Pixel Bender to calculate information. In the implementation example you can run calculations while the video is playing and compare performance with the Flash Player doing the these two tasks. In the application we will built we will be using the cos function.

To get started download and install the Pixel Bender toolkit. Open the Pixel Bender toolkit, and paste this script to calculate a cos function, see figure below.

<languageVersion : 1.0;>
kernel SinCalculator
<
    namespace : "pixelBender";
    vendor : "Elad Elrom";
    version : 1;
    description : "Cos Calculator";
>
{
    input image1 src;
    output pixel3 result;

    void evaluatePixel()
    {
        pixel1 value = pixel1(cos(sample(src, outCoord())));
        result = pixel3(value, 0.0, 0.0);
    }
}

 

image

Adobe Pixel Bender ToolKit running filter.

The class set pixel1 to do the sine function. Then, we set the result to pixel3, which will output the result. You will need to load an image as a placeholder, since Pixel Bender toolkit was not intended to do these type of calculations. Run the code, and you can export the code for Flash player.

To export the code for Flash player, select File -> Export for Flash Player, you can then save the file as a PBJ and use it in our application. There is a limit on the number of calculations that Pixel Bender can perform, but we can create a helper class that can handle the communication with Pixel Bender. As an example download the helper class called: PixelBenderCalculator.

I am not going to walk you through the entire code, but the code is pretty heavily documented and easy to understand. We first define all the variables we will be using:

        //  We need to define the PBJ class we will be 
        // using; we'll assign a class in our implementation.
        
        public var KernalClass:Class;
        
        // We also need to define an array, numberCollection, 
        // which will hold the collection of numbers we need to do 
        // a calculation for, as well as shader and shaderJob
        
        public var numberCollection:Array;
        private var shader:Shader;
        private var shaderJob:ShaderJob;
        private var input:ByteArray;
        private var output:ByteArray;
        private var destinationCollection:Array;
        private var requestsCounter:Number;
        private var numberOfRequest:Number;

createShaderJob method creates a shader class based on the kernel to pass the numbers and start the calculations. This method makes the request and is very similar to any loader we would use in Flash. Once we set the ShaderJob, we can define a listener and start the job.

        private function createShaderJob():void
        {
            var width:int = input.length >> 2;
            var height:int = 1;
            
            shader = new Shader(new KernalClass());
            shader.data.src.width = width;
            shader.data.src.height = height;
            shader.data.src.input = input;                
            
            shaderJob = new ShaderJob(shader, output, width, height);
            shaderJob.addEventListener(Event.COMPLETE, shaderJobCompleteHandler);
            shaderJob.start();                
        }

The addByteArrayToCollection method takes the byte array we received from Pixel Bender kernel, converts the information to float number, and adds the information to the array we will return to the user. We are checking the data to ensure we are passing three variables; if(i % 3 == 0), since we had to add three extra pieces of data to pass to pixel bender kernel: pixel3(value, 0.0, 0.0);

        private function addByteArrayToCollection(byteArray:ByteArray):void
        {
            var length:int = byteArray.length;
            var number:Number;
            
            var i:int;
            for(i = 0; i<length; i+=4)
            {
                number = byteArray.readFloat();
                if(i % 3 == 0)
                {
                    destinationCollection.push(number);
                }
            }
        }

Using this API, we can create a class to implement and check if we achieved any performance gains. We will create two methods: one will call the Pixel Bender kernel, and the other will use the Flash Player for calculations. Take a look at Figure below. We will run a video in the background, while we calculate result so we can observe performance. Complete implementation code can be found here.

 

image

Screen shot showing implementation of Pixel Bender calculation

Let’s take a look at the code to create an implementation. We define our Pixel Bender class using the following expression:

            [Embed(source="assets/pbj/CosCalculator.pbj", mimeType="application/octet-stream")]
            private var kernalClass:Class;
            private var pixelBenderCalc:PixelBenderCalculator;

We instruct our method to start the calculations in Pixel Bender, we will create a number collection in the size of 5 Million then start Pixel Bender shader. An event will listen to the complete event we created in the API.

            protected function startCalculatorPixelBender():void
            {
                // create a number collection
                var numberCollection:Array = new Array();
                
                for (var i:int=0; i<5000000; i++)
                {
                    numberCollection.push( i );
                }
                
                // calculate
                pixelBenderCalc = new PixelBenderCalculator(numberCollection, kernalClass);
                pixelBenderCalc.addEventListener( PixelBenderCompleteEvent.COMPLETED_EVENT, onComplete );
                
                cursorManager.setBusyCursor();
                pixelBenderCalc.start();
            }

This method does exactly what Pixel Bender did but, this time, uses the Flash Player, so everything is done on the same thread. This method will be used as a way to test if we gain in terms of performance.

            private function startCalculatorFlashPlayer():void
            {
                // create a number collection
                var numberCollection:Array = new Array();
                
                cursorManager.setBusyCursor();
                
                for (var i:int=0; i<5000000; i++)
                {
                    numberCollection.push( Math.cos(i) );
                }
                
                console0.text = "Flash Player calculations completed";
                list0.dataProvider = new ArrayCollection(numberCollection);
                
                cursorManager.removeBusyCursor();
            }

Our components include a list to display the results, video player, and buttons. You can test the performance by starting the video, calling Flash Player, and then comparing the time it took to calculate as well as performance to the Pixel Bender shader.

    <s:List id="list"  width="200" height="531.5"/>
    <s:List id="list0" width="200" height="531.5" x="212" y="0"/>
    
    <s:Button label="Calculate with Pixel Bender" width="200" height="20" y="545" click="startCalculatorPixelBender()"/>
    <s:Button label="Calculate with Flash Player" width="200" height="20" y="545" click="startCalculatorFlashPlayer()" x="212"/>
    
    <s:TextArea id="console"  x="0"   y="585" width="200" height="125"/>
    <s:TextArea id="console0" x="209" y="585" width="200" height="125"/>
    
    <s:VideoPlayer id="vid" width="355" height="290" 
                   source="http://www.safeexpress.se/video/flv/radiobil-remastered.flv" 
                   autoPlay="true" x="440" y="10"/>
    
    <mx:UIComponent id="component" x="436" y="343" width="70" height="100"/>
 
</s:WindowedApplication>

The results are astonishing. Using Flash Player, running the code took 6 to 8 seconds (on an iMac) and the video paused. Using Pixel Bender, the video had a slight glitch when pixel bender starts the job and when the list is populated, and we received the results back once the task was completed. The see the result run the example. When the example is running click to play the video and use the Flash player button to calculate results you will notice that the video completely stalled while the Flash player is calculating the results since we are using the same thread. The second time run the video and use the pixel bender to calculate, you will notice that you had a tiny glitch in the video, but the video keep playing while the pixel bender calculate the number.

Calculating mixing of audio files

Let’s look at a more practical example. We will create a track mixer, which will take two tracks and mix them into one track using pixel bender.

The Pixel Bender Toolkit comes with some limitations, when you try to add a third src you get the following message: “This version of Adobe Pixel Bender Toolkit does not support kernel with more than 2 inputs”. However many calculations will need to use more than two sources. The solution is to compile assembler code instead of using Pixel Bender Toolkit.

Assembler code

By using C++ code you can compile assembler code instead of using the Pixel Bender Toolkit. Tinic Uro from Adobe released C++ code to compile assembler files into pbj files. I may post a related article showing calculations with assembler code so sign up to my blog feeds to be notified.

The class that will use pixel bender to mix the two audio files can be downloaded from here.

The API is relatively simple, we define the variables we will be using:

        // pixel bender class and shader
        private var KernalClass:Class;
        private var shader:Shader;
        private var shaderJob:ShaderJob;
        
        // num of tracks
        private var numOfTracks:Number = 0;
        
        // counter
        private var trackDownloadCounter:int = 0;
        
        // buffer & sound objects
        private var buffer:Vector.<ByteArray> = new Vector.<ByteArray>;
        private var sound:Vector.<Sound> = new Vector.<Sound>;

Once we are ready to get started the start method will load the tracks we will be mixing.

        public function start(urls:Array):void 
        {
            if (urls.length >2)
            {
                this.dispatchErrorEvent( "API only supports two tracks at this point." );
            }
            
            numOfTracks = urls.length;
            var i:int;
            for (i = 0; i< numOfTracks; ++i)
            {
                sound[ i ] = new Sound(new URLRequest(urls[ i ]));
                sound[ i ].addEventListener(Event.COMPLETE, onSoundLoaded);
                sound[ i ].addEventListener(IOErrorEvent.IO_ERROR, onError);
            }
        }

Once we completed loading the tracks we set the shader job and parameters and start the job.

        private function onSampleDataHandler(event:SampleDataEvent):void 
        {
            var width:int = 1;
            var height:Vector.<int> = new Vector.<int>(numOfTracks);
            
            var i:int;
            for (i = 0; i < numOfTracks; ++i)
            {
                buffer[ i ] = new ByteArray();
                sound[ i ].extract(buffer[ i ],BUFFER_SIZE * 4);
                height[ i ] = buffer[ i ].length >> 4;
                buffer[ i ].position = 0;
                
                shader.data["src"+i]["input"] = buffer[ i ];
                shader.data["src"+i]["width"] = width;
                shader.data["src"+i]["height"] = height[ i ];
            }
            
            shader.data.distort.value = [this.balance];
            shaderJob = new ShaderJob(shader, event.data,width, height[0]);
            shaderJob.start(true);
        }

Next, we will create a class that implement the API, see screen shot of implementation, in figure below.

 

image

Track mixer screen shot

Implementation passes the pbj and tracks we will be using and allow changing the balance of the track, see code below:

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
               xmlns:s="library://ns.adobe.com/flex/spark" 
               xmlns:mx="library://ns.adobe.com/flex/halo"
               xmlns:local="*"
               minWidth="1024" minHeight="768" creationComplete="creationCompleteHandler(event)">
    
    <fx:Script>
        <![CDATA[
            import com.elad.framework.sound.events.TrackMixerErrorEvent;
            import com.elad.framework.sound.TrackMixer;
            import mx.events.FlexEvent;
            
            [Embed("assets/pbj/TwoTracksMixer.pbj", mimeType="application/octet-stream")]
            private var Kernal:Class;
            
            private var trackMixer:TrackMixer;

            protected function creationCompleteHandler(event:FlexEvent):void
            {
                trackMixer = new TrackMixer(Kernal);
                trackMixer.addEventListener(TrackMixerErrorEvent.TRACK_MIXER_ERROR, function(e:*):void { trace(e.message); } );
                trackMixer.start( ["assets/tracks/FeelinGood.mp3", "assets/tracks/Sunshine.mp3"] );
            }

            protected function balanceSliderChangeHandler(event:Event):void
            {
                trackMixer.balance = event.currentTarget.value ;
            }

        ]]>
    </fx:Script>
    
    <mx:Slider x="28" y="171" id="balanceSlider" 
              labels='["Track1","Mix","Track2"]'
              minimum="0" maximum="1" 
              liveDragging="true" value="0.5"
              change="balanceSliderChangeHandler(event)" />
    
    <local:Visualization type="wave" bars="32" width="300" height="137" x="28" y="14"/>
    
</s:Application>

 

To learn more about Pixel Bender visit Adobe Pixel Bender official site. Happy pixel-bending :o)

 

About Elad Elrom

Elad Elrom is a technical writer, technical lead, and senior Flash engineer. As a technical writer, Elad wrote books covering Flash technologies. He maintains an active blog and has spoken at several conferences regarding the Flash platform. He has helped companies follow the XP and Scrum methodologies to implement popular frameworks, optimize and automate built processors and code review, and follow best practices. Elad has consulted a variety of clients in different fields and sizes, from large corporations such as Viacom, NBC Universal, and Weight Watchers to startups such as MotionBox.com and KickApps.com.

Get new stories first

Click to follow us on Twitter!

 

Comments


Posted by jlward4th on 08/13 at 09:38 PM

Great article!  I recently created an AS3 library which allows you to create Pixel Bender filters at runtime.  Check it out:
http://www.jamesward.com/blog/2009/04/29/announcing-pbjas-an-actionscript-3-pixel-bender-shader-library/

Let me know what you think.

-James


Posted by Elad Elrom on 08/16 at 12:31 AM

Hi James, great article.  I played around with haXe to assembler and disassembler pbj files based on Tinic Uro C++ code.  Nice job creating the API and publishing for all to use.


Posted by Vadim on 01/26 at 12:55 PM

Interesting way. Right now I trying to use Pixel Bender to make something like this: http://blog.bellinsky.com/2011/01/07/pixel-bender-filter-slide-wring-for-flash/

Submit a comment

Only registered members can comment. Click here to login or here to register