Another scream on Flash, Alchemy Memory and compilers.

Before I start, I want to explain the timeline which led to this blog post. This was supposed to become a post of being able to get much faster access to bytearray’s through ActionScript, by (ab)using the Alchemy memory opcodes. I spent quite some time trying to build a swc library which would allow you to do just that. Only to learn that you can’t build a generalized API for this in ActionScript. It just can’t work out with the current ActionScript compiler technology; here’s the story of why.

Leveling the field between Alchemy, Haxe and Actionscript

The reason I set out to do this, is because I found it somewhat strange that the only language in the row above that is not able to access the faster memory, is the platform’s native language, ActionScript. I could use many strong terms for this, but it is what it is. So why not change this myself ? Surely adding a piece of custom built (byte) code, should be able to allow access to this hidden gem ? I had discussed this option with Joa before and it seemed to be a viable path. So why not do it ? A good thing to note in the mention above is that Haxe has already implemented this. Read Nicolas his post about haxe flash.Memory.

Compiling inline ABC with Alchemy’s asc.jar, a bit different.

As I posted earlier about Alchemy, one of it’s advantages is the fact that Flash Player 10 has an implementation for actionscript byte code opcodes that are currently not supported within the ActionScript API. These opcodes allow for faster access to 1 bytearray inside the player, which you can set by accessing ApplicationDomain.currentDomain.domainMemory.

Strangely enough, that is actually opened up to AS API, presumably to allow for “regular” ActionScript access to the Alchemy memory. The actual opcodes have no ActionScript equivalent though.

Since Alchemy boils C code down to ActionScript, how does it obtain access to these opcodes ? The answer lies in the compiler which comes with Alchemy. The asc.jar in the /bin package allows for so called “inline asm”.  This type of code looks something like the following.

__asm(op(0xD2),op(0xD1),op(0x3A));

This code can be put anywhere, and the opcodes will be added to the abc bytestream. Great huh ? So, ideally, we would just switch out our asc.jar in our Flex install directory, and we could just start doing this. Well. NO.
Compc, which is used for actionscript project compilation isn’t compatible with this special asc.jar. Not too worry, we can get around this, more on that later.

Discovering special memory opcodes
Although at the first release of Alchemy, these special memory opcodes where somewhat ellusive, by now there’s loads of information on that. Most specific info comes from Adobe Alchemy’s creator, Scott Petersen, in his presentation FlashCCompiler (pdf).

Now, armed with these special opcodes, writting an API around it shouldn’t be to hard, so let’s go at it.

public function setFastMem(byteArray : ByteArray):void
{
 currentDomain.domainMemory = byteArray;
}

public function writeByte(position:uint, value:int):void
{
__asm(op(0xD2),op(0xD1),op(0x3A));
}

Ha, yes, combined with the Alchemy asc.jar, this code works, and translates out into working bytecode! So, now by calling writeByte, we are using one of the opcodes to write an opcode into the FastMem. Great! When I had all the opcodes to work, I forgot to do one thing, which I should have known and could have saved me a lot of time. Benchmark it. Instead, I first wanted to figure out a way to distribute this, so developers could easily do this.

The long path from .abc to .swc
Now, when using the asc.jar, we can not compile a swc directly, since the asc.jar is incompatible with the compc compiler.
It’s only able to output a swf, or the abc bytecode for that class.
What I went and did was examine the gcc script in the alchemy/achacks folder. Interestingly, the gcc script is a replacement for the normal gnu gcc compiler, written in c. It handles the steps for alchemy to go from c to llvm to actionscript to swf/swc distibutable. Even more interesting is HOW alchemy creates a swc, if you use the -swc compiler option. You’d think there would be some magic going on there, but instead it is not so magic at all. When compiling to a swc file, the compiler extracts the abc data from the previously compiled swf using a script called GetABC2.pl, and then replaces the code in an existing .swf file which is in the achacks directory, using PutABC2.pl.

A thing to note about that .swf file it uses, is that the file is an uncompressed .swf, marked in the header by FWS. These are getting more and more rare, since most swf files are compressed these days, and start with CWS instead. This is mostly common knowledge. Now, the problem here is that I wanted to replace that template.swf by my own….only to find out there’s no documented way of compiling a swf file without compression. To make matters worse, the perl helper scripts seem to have a bug, where they do not work with compressed files. Although these scripts can be patched, that’s not what I wanted to spend my time on, so I went my own route.

Injecting bytecode
Instead, I wrote a little air app, which scanned a swf for the DoABC tag, (id 82), and replaced it with my own ABC bytecode. This involved changing the size tags of both the swf and the tag, and saving it out again. It took little time, but in the end it worked like a charm, and I could now generate my own swf’s with custom inject ABC bytecode, great! Now I needed to get this swf into a .swc file.

SWC and zip, a little tip
As most of you would know, as SWC file is nothing more then a .zip file, normally containing a library.swf and a catalog file. In my naive thought I would just unzip the swc, replace the library with my abc injected library and things would work. Not so. I’ll save you the hours of pondering on this, but as it turns out, when using the Mac OS X finder, and unzipping it through that, leaves you with something crippled; what I don’t know. So here’s a good tip, if you want to unzip and rezip a SWC, do so using the terminal / unzip & zip. Really, I wish someone had blogged about this, because it drove me crazy, ending up with Flex Builder saying “can’t load swc”, even if I didn’t change anything in the .swc, just uncompressed and compressed it again. I caught on to that quite fast, but the fact it was the uncompressing the swc which was the problem, took me a little while to realize.

It works! Now let’s benchmark
As I said before, with some logical reason and benchmarking before I went on a spent a lot of time on bytecode injectors, build scripts and swc compilation, I could have saved myself a lot of time. The reason why I wanted to have this distributable was the entire reason to do so, so I didn’t want to go on, without knowing how to get from a special __asm .as file to a working .swc. In the end it was all in vain. When I started benchmarking this, the result was disappointing.

The FastMem bytearray, and the special functions I had built for it, where nearly 2x SLOWER then just using ByteArray.writeByte(). And then I figured it out. I have forget to think about the overhead of calling a function in ActionScript. Since a built in function is much faster then an actionscript function, the overhead of calling that function far outweighed the faster access speed to the data.

So it ended up in a failed experiment. That’s ok. Failing is a way of learning, and during this process I learned a lot of new things. But in this case, the biggest lesson wasn’t how to inject bytecode, use the asc.jar, write your own swc compilation pipeline. It was that with the current compiler for ActionScript, it’s just not possible to use the FastMem opcodes in a easy and generalized way.

The difference between the compilers

Now, the saddening conclusion is that I wasn’t able to build a FastMemory.swc that all Flash developers could use. The more alarming conclusion I take from this, that the problem doesn’t just resides in the compiler not having any way to do so, but the fact that it’s due to missing features other compilers already have, leaving ActionScript developers behind.

Haxe is able to pull the exact same thing off. With Nicolas his work on the flash.Memory api, you can select a bytearray, and get very fast read and write access. So why isn’t this possible with ActionScript ? The Haxe compiler supports inlining. For those unfamiliar with this, this allows you to keep portable and reusable functions, while the compiler inserts these functions directly in the place they are called. So you’re not actually calling function, you are essentially inserting a piece of code inside your code. This eliminates the overhead of the function call during the compilation phase.

The reason this does work within Alchemy is that it not only supports inlining (less important in this case), but the Alchemy code actually is processed before hand by the llvm compiler with ActionScript target. Even further, within Alchemy, your code lives inside the reserved fast memory, and access to for instance malloc’ed pieces of memory are inside this memory.

Conclusion and final shout out to Adobe

This leaves ActionScript behind. Not because of the language, just because of the compilers. ActionScript doesn’t support inlining, nor does the compiler do any optimizing steps. We could start modifying our own asc.jar and doing this, but with the Flex SDK being opensource, it’s still not legal to modify the asc.jar and release, essentially making that step useless to do.

All in all, ActionScript isn’t able to target it’s own platform as well as both Haxe and Alchemy. Sure, you can insert optimized hand written bytecode in performance critical places, but at that point, why shouldn’t I just switch to using Haxe or Alchemy ?  In my opinion, both have their downsides.

Alchemy is great, but as it’s currently in beta, it’s not even an industry standard. Sure, C as a language is, but Alchemy just isn’t. Also, when using Alchemy, even the smallest piece of code results in a huge .swc because of all the standard C library code. All in all, it’s amazing Alchemy runs this fast considering all the steps the code takes before it actually runs on your machine.

Haxe is a work of genius. It’s a great compiler, I like the language and I specifically love the .abc format it contains, allowing me to write bytecode inside the Flash runtime and executing it. In my book, that’s awesome. As of today, Haxe can even target the iPhone ! The language is great and has many features ActionScript lacks. It baffles me how one man army nicolas can single handedly write a language and a multi target compiler, and in many respects stay on top of the ActionScript compiler.

My first problem with Haxe is just that. The one man army. Should nicolas decide he doesn’t feel like it anymore, the platform is dead. Sure, there’s more people working on it, but in it’s core it seems to be that nicolas is the driving force. My second problem is that although there is a huge pool of ActionScript developers, which can work together within the industry standard called ActionScript. There’s not even near that many people who’ve even touched haxe, let alone the number of developers who are able to call themselves “Haxe Professionals”.

Instead, I’ll stick with ActionScript for now, and hand roll my optimized code using pure abc, haxe or Alchemy, which ever suits the purpose best. But the necessity to do so leaves me wondering what the state of ActionScript and it’s compilers is. Should we start building our own ActionScript compiler ? Add our own features to the language ?

[EDIT]

What I failed to mention is that due to the overhead the asc.jar added to the bytecode (some return’s and at every function a getlocal0 + pushscope I didn’t need) I eventually switched from using that to moving to Haxe with it’s Haxe format library. One of the supported formats is ABC, and I ended up using that to create my own .abc bytecode, injecting that instead of the asc.jar output. Again, I’m not saying either Haxe or Alchemy are bad tools, I’m just saying currently ActionScript is lagging behind.

This entry was posted in Uncategorized. Bookmark the permalink.

31 Responses to Another scream on Flash, Alchemy Memory and compilers.

  1. Matt says:

    Great Post!

    Hey, I think your issue with rezipping the SWC file is actually a result of the way Mac OS zips files by adding extra hidden meta data when it compresses files (e.g. __MACOSX directory). I believe Adobe’s compilers don’t like these extra hidden files and folders within their precious swc files.

  2. UnitZeroOne says:

    @Matt Actually, this was the surprising thing; it wasn’t the compression, but the decompression for some reason. When I decompressed the swc using the Finder it wouldn’t work after repacking it. I found it very odd too; I already went down the path of packing it using the terminal / zip, instead of Finder, but that didn’t work. Only when I also unpacked from the terminal it worked.

  3. Matthew Fabb says:

    At FITC Toronto, Jim Corbett, one of the Flash Player engineers, mentioned in his presentation on the “Flash Player Internals v2″, that making the faster ByteArray available in pure AS3, was one of the things Adobe was currently working on. There was no mention of when we might see this, this as just one of the things he mentioned at the end of his presentation on what Adobe were working on next for Flash.

  4. Joa Ebert says:

    Hey Ralph,

    a great post and a long journey. I can definitly understand your problems with the SWF/ABC format and how to put everything together. When I wrote AS3C there was even no documentation available.

    Actually what frustrates me currently the most is a part of the Flex SDK which says, that you are not allowed to redistribute “Build Tools” which include for instance the asc.jar and you are also not allowed to modify the source-code of those tools. When I want to add some more functionality into ASC — I can not without breaking the license agreement.

    This is a really sad story. Extending the ASC to allow method inlining is not really hard. There are also tons of other features which could make it into the ASC but there is no way to extend and distribute it.

    Actually, there are only two options. Adobe should really begin fixing those problems. If they do not want to or do not have the resources: remove that damn license terms and let the community fix it. I had a long talk with Nicolas in Germany at the FFK09 and my main problem with haXe is the missing IDE — but we can fix that in no time. The compiler has a lot of features which make IDE development very easy.

    When writing our own compiler, custom features are a little bit complicated. Generics/Templates for instance. How would generics work with reflection since they are not part of the runtime? They don’t. But I guess the best option to do inlining is using the ASC. Hopefully Adobe will change their license terms.

  5. Great Post Ralph!

    Nicolas is a genius, that’s for sure!

    The fact that most of your troubles are caused by odd licence terms is actually one more argument for me for moving to haXe.

    As soon as there is a decent haXe IDE I’m switching. (heck I’m already a believer).

  6. Subb says:

    “Adobe, make some compiler” campaign perhaps?

  7. Josh Tynjala says:

    I wasn’t aware that you couldn’t modify and distribute asc.jar. I can’t seem to find any document that defines this restriction. Will someone post a link so that I can take a look? Thanks.

  8. UnitZeroOne says:

    @Josh joa pointed this out to me. Apparentky it is in the 3rd party license,

  9. Hristo says:

    What about writing a compiler from scratch? A pure Actionscript compiler, without funny flex stuff in it. Having the sdk sources should do for a quick starter, right?

  10. Great post Ralph, very interesting and very valuable.

    Totally reminds me of the old __bytecode__ trick for AS2. Example – pop this in some AS in Flash and publish to AS2:
    __bytecode__(“88240002006900486F6C7920736869742C2062797465636F646520696E2074686520666C612100960B0008000600000000000000003C96020008001C960500070A00000048129D02001500960200080126960400080008001C501D990200D6FF00″);

    Definitely interested in this area. I was also going to mention that Jim Corbett and others including Scott Peterson during his MAX session last year on Alchemy, all have mentioned that getting these opcodes, and other optimizations into the compiler is something they are working on, but no timeline for doing so was discussed.

    I also have to believe that now that Adobe isn’t bound by the limitations of trying to adhere to ECMAscript for ActionScript the language (reference: http://blogs.adobe.com/open/2008/08/blog_entry_dated_81408_715_pm.html) , that this will also open the door to adjustments, enhancements in the language itself to make it easier to support advanced inline compiler directives and outright new features and optimizations. I am all for standards, but I think that’s one thing that has actually held them back to a certain degree, and certainly an advantage Nicholas (the brainiac) has in developing Haxe – he has nearly no limitations imposed on him other than the ones he makes for himself. Every few months or when something new comes out I find myself drawn to haxe to investigate, and end up playing around and am always impressed, but I have the same concerns you mentioned.

    Again great post and keep up the awesome work!

  11. Forgot to mention- looking forward to your session at Flash On Tap next week!

  12. Chris Allen says:

    Great article Ralph! As far as I know, haXe supports converting AS3 code to the haXe language, which you could then convert to ABC using the haXe compiler. I suppose that there would be no way of triggering this inlining feature from the AS3 language though, without some reworking of this feature from haXe. Either, way it should be possible to use haXe as an intermediary for compiling AS3 code. Also, I know this is not something that our man Nicolas is really into, but we could get the haXe compiler to interpret AS3 directly too. No matter what though, it would not be standard AS3, and would sort of be branching the language that Adobe has created. Thus you would not be able to compile the code from the Flex SDK.

    Anyway, it’s definitely an interesting topic, and I look forward to talking to you and Joa about it at Flash on Tap. See you next week!

  13. Nicolas says:

    Hi Ralf, long time no see !

    Thanks for all the good words about haXe ;)

    However, I have to answer that your main reason for not using haXe is no longer true for more than one year : several people are contributing to the compiler – Franco Ponticelli did the PHP backend and Hugh Sanderson did the C++ backend. Both are skilled enough to take over the whole compiler if for example I would get an accident.

    It’s pretty much the same as saying that I can’t be using Papervision3D because it looks from me like a 2-man army (you and Carlos, with separated roles). The main difference is that haXe is open source, with a lot of skilled developers contributing in different ways to it. And who better than a developer can understand other developers needs ?

    As for getting people into haXe, everybody coming from AS3 say they were able to get used to haXe in just a few days. Nothing hard here ;)

    And with all the things that are left to do in terms of optimizations, I’m pretty confident that haXe will stay ahead for a long time. So you better start using it asap, or prepare yourself for more long posts such as this one :D

  14. Weird looks like that __bytecode__ line got truncated – it should have been:

    __bytecode__(“88240002006900486F6C7920736869742C2062797465636F646520696E2074686520666C612100960B0008000600000000000000003C96020008001C960500070A00000048129D02001500960200080126960400080008001C501D990200D6FF00″);

    Rob

  15. liuhuan says:

    thanks a lot, it’s very helpful.

  16. dim says:

    great post !

    Nikolas and Ralf – i’m yours fan ! damn ! ! !

  17. UnitZeroOne says:

    @Subb Yes, or we could try and see how far we can push that license ?

    @Robert M. Hall Yes, in AS2 there where more options. I actually kind of fond of the AS3 DoABC tags, they are very well structured, although a bit hard to get too at first. Nicolas is definitely a master on that one. Looking forward to see you man!

    @Chris Allen Yes, I’d love to see an AS3 target for Haxe; but it would defy the purpose of being able to cater to a better compiler, with more optimized output. What Alchemy and Haxe can do are testaments what can be achieved, and I’m getting more and more convinced that it’s the combination of the language and compiler not getting any real big upgrades. Maybe FP11 would show us a new path. Also, I’m really curious where Adobe will be taking AS4.

    @Nicolas good to hear the team is that strong. It was in no means intended in a bad way, but it is an honest observation. It’s a stark contrast to have a big corporate team working on this, and at the same time seeing the awesomeness of Haxe / Alchemy which are manned by much smaller teams, but pushing the platform further. The corporate backing is what makes it more “firm” imho, but when things get more standardized, we might see things change ?

    @liuhuan @dim thank you both.

  18. Joa Ebert says:

    One important thing: The stuff I said about the license is wrong. Adobe confirmed that the license ending up in the root of the SDK is not the open source SDK license.

  19. PB says:

    At FITC Toronto, Jim Corbett, one of the Flash Player engineers, mentioned in his presentation on the “Flash Player Internals v2″, that making the faster ByteArray available in pure AS3, was one of the things Adobe was currently working on. There was no mention of when we might see this, this as just one of the things he mentioned at the end of his presentation on what Adobe were working on next for Flash.

  20. Rackdoll says:

    Very nice post Ralph!
    Looks like i gotta look into HaXe :)

    Also thnx for sharing your project findings with us, will help us save the crunch time!

  21. yonatan says:

    About the CWS/FWS thing; you might want to try flasm (“flasm -x comperessed-file.swf”) the next time you need to decompress a swf.

    Cheers

  22. Hi, great reading. I also like Haxe as a language. I didn’t know that it generates better abc.

    At one point, you say “it’s still not legal to modify the asc.jar and release”
    Well, that’s just what I did with this project
    http://www.victordramba.com/?p=31

    I wonder, will they come after me? :D

  23. ryan says:

    i think i speak for everybody when i say that the biggest drawback to haxe is the lack of developer tools and ide to work with… come on guys… lets change this…

  24. anonymous says:

    I disagree with you on “platform’s native language” – I don’t think AVM2 has any native language, mostly because the gap between the supposed language, ActionScript 3, and the platform, AVM2 is too large – the platform deals with stack, locals, etc, while the language deals with more refined and human-readable logic.

    haXe would be just as “native” to AVM2, as ActionScript 3. Just because Adobe decided to use ECMAScript flavour to compile into AVM2, does not make it their “native” language or make it AVM2s “native” language.

    My $.02, of course.

    • UnitZeroOne says:

      I do agree to some extent. Problem is that Adobe controls the platform and the languages which target it. Features of the Flash Player, and the API around it are being scaped with ActionScript in mind. Also, with a consectutive release of a new player, the API for AS is in place. Haxe and other languages / compilers targeting the player can only follow up. The way these features are being build is based around ActionScript, not Haxe, not Alchemy.

      So in terms of native, I do think ActionScript is the first up. Even though other compilers and languages can follow up. That is, until Adobe decides on what to do with what the player actually is targeted by. If they take that wider, then that would change things. But for now, I feel assured that they are taking the path of ActionScript. Gumbo and the effort on the language (MXML could be called a language in this context) don’t point to anything else.

  25. anonymous says:

    You have a good point time-wise, politics decides the future of AVM, even though today the relationship seems severed (which is good). And politics is what is actually a bit sad – I mean they hold a leash, however long, to their platform, ready to pull the plug anytime the wind stops blowing into their sail. This is why it took so long for SWF-related tools to pop up – I mean we waited years for a usable SWF specification, and now it is the same story with DoABC tag internals, which is what basically runs the whole platform these days. With Adobe scouting the horizon like the Eye of Sauron, they sure can scare one off even trying, because then one thinks “ah, what the hell, in a year they invent some new flashy thing that they will say is the best thing since sliced bread, and push it on us like sardines on a fish market”. Which I am afraid is true too. Which partly mirrors the status quo as you yourself have written about in the above post. Changes come too fast for a scattered community to follow, while the benemoths can actually more than keep up. Not bad as a method of control. You are absolutely right in that sense.

  26. devu says:

    So true and sad sometimes… but I am wondering about one thing.
    (On the beginning I am sorry for my English and I am not a French ;) )

    One man – Nicolas could develop such a big platform or solution able to deal with swf format more eficient way.
    Now we see clearly the point the swf and AVM2 with all of limitation is the key. Have you thing about what culd happend to the AS3 developers or even designers if any other competitor (SL JavaFX) on this market could really provide better solution? GPU acceleration, clear and easy Drawing API with GPU as well, multi threading and efficiency close to the cpp and finally designer/developer-friendly IDE… it would be disaster for Flash platform and huge leak of talented guys.

    Because Macromedia/Adobe itself learned us from 10 years how to do better and better stuff.
    A lot of guys become developers even if they started from designer background only.
    Thanks to this technology and fact that because designer and developer worlds could met in one place.
    Is anyone here thinking that designers are going to work so close with pure cpp development as with Flash could do?
    Whatever it is, game, website, application. However we still need more because as we can follow the Flash features flash could follow the technology trends around. But in the same time keep it simple and intuitive for us.

    Fortunately or not, nor of competitors could change our way … yet. But as we know SL did big storm on this sea few years ago. Think about different scenario. Don’t you thing if someone like Nicolas could develop brand new solution as a platform with this whole stuff above in mind and target to Flash Developers/ Designers? Make it more elegant, efficient and just have a chance avoid all of these bad experience and disappointments in the middle? Is it so difficult? We need another 10 years to promote brand new file extension? Personally I don’t think so. And guy like Nicolas just inspired me to thing this way.
    10 years ago was few guys around to play with flash 5. Now we are talking about potentially more than 1milion active development community. If they’ll got something more robust to deal with it will take few months to do revolution and news about this will spread the world as quick as client will be able to pay for new version of Quake on-line with GPU support. And as easy as you can do any stuff in AS3 or haXe. My conclusion is technology and standards are belong to us, developers who can follow or not the trends. If you think is not like that just go backwards 10 years and think what you though about many things, how many technologies died because no one picked it up. Only big companies job is to make you believe is only one way and cover-up the core of the invention. Sometimes is worth to reinvent the wheel.

  27. dbam says:

    i was wondering…
    When You say :
    “…Compc, which is used for actionscript project compilation isn’t compatible with this special asc.jar. Not too worry, we can get around this, more on that later…”
    i have the feeling that You never get to that point.
    Or do i miss something? I’m reading the post over and over again, but can’t connect the spots.
    Could You pin-point that step?
    Is it that AIR application of Yours (parsing output swf byte-by-byte and changing the proper values?)
    What language is that pseudo, AS3-lookin’, __asm-containing API?

    I’m working on a project, heavy memory reading-writing involved. I use Flex CL compiler to compile my .SWFs, but would love to get the most out of AVM2 ( even if this means to get hands dirty with alchemy ).

    Looks like a looong road ahead…
    Cheers,
    the-bam

  28. Pingback: Adobe Alchemy 资料整理 « 国士无双

  29. Hi!

    Great Post! Very informative.

    What about a post compiler that parses the swf file and replaces the call of the writeByte wrapper with the bytecode? Additionally it could do some optimizations like inlineing math functions.

    michael

  30. Pingback: Alchemy 와 속도 | Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Notify me of followup comments via e-mail. You can also subscribe without commenting.