If Mike Pall happens to read this, I feel like he's really selling the world short by not writing a book about dynamic language implementation. It could influence generations of programmers! Right now you have to piece together some of his mailing list posts and try to fill in the blanks, but a more thorough explanation from him would be invaluable.
Does anyone know where he disappeared off to? He seems to post every one and a while. I've never been part of the LuaJIT community but his online presence seems to have dropped off over the past few years. Was it because the new garbage collector didn't get funding?
My favorite blog post of all time is Haberman's "Hello, JIT World: The Joy of Simple JITs" which covers using DynASM. [1]
I wish there was better documentation for DynASM. I wonder what cool programming projects would be created if DynASM was better known and better documented.
I've used this. If the comparison is with writing an interpreter for whatever you want done, here's how I remember the tradeoffs.
Using DynASM itself was smooth sailing. It is officially undocumented, but like the site says, the code is sufficiently well commented.
Generating simple-minded machine code is easy enough (in my opinion) and the results run decently fast.
I didn't integrate run-time generated functions with exceptions and debugging (host language was C++). These areas seemed very sparsely documented (exceedingly sparsely, IIRC, for 64bit Windows) and looked like they would eat more work than everything else up to that point.
A while ago, I spent a fair amount of time evaluating JIT frameworks including DynASM. The one I ended up using is something called Xbyak (https://github.com/herumi/xbyak) which I think is a bit of a hidden gem. Unlike DynASM, it doesn't require a preprocessing pass, is better documented and IMHO easier to use.
So if you're in the market for a lightweight JIT engine and target C++/X86, I'd give Xbyak a whirl too and see which one you like better.
One should really note the major differences dynasm took from the more abstract and logical jit libraries, which try to abstract platform oddities from the user. Like LLVM, libjit, asmjit, v8, libgccjit, lightning or nanojit.
It doesn't care how you call a function, it doesn't even care about 2 or 3 address op syntax. All it does is adding simple template logic to let you handle the platform-specific abstractions by yourself.
And in the end it's easier to add your own prologue or prepcall1 macro, than relying on a library.
people always looked at the "most portable jit library". dynasm offers no platform support per se, as all the others do. you have to do by yourself. it's the least portable of all. but in the end it was much more powerful, and offered the most ports. you get the asm anyway by dissambling the target function. with the libraries (such as LLVM) you'd need to translate a high-level overview to the library API.
with dynasm you just need to adjust your types, prologue, struct field accesses and epilogue. and dynasm is perfect and practical in doing this, whilst the other libraries try do to much more, with much more handholding and are much bigger and less effective.
I don't agree with the genius part though. lua itself was pretty much perfect already, and lua took most of the good parts from earlier dynamic languages, lisp and self, and esp. Inferno's "Dis VM". http://doc.cat-v.org/inferno/4th_edition/dis_VM_specificatio... which was Lucent's faster and smaller competitor of Sun's stack-based and GC-heavy JVM.
it just missed besides a jit a good OO, and luajit didn't add that neither. e.g. _why the lucky stiff added a nice OO layer on top of lua, plus a nice simple jit. much simplier than luajit.
luajit just objected to some questionable decisions in lua regarding performance, and then took over. esp. nan tagging makes a lot of sense, but the compiler optimizations are also pretty cool. esp. when compared to the massive LLVM overhead.
luajit is using the slower and bigger 3-address SSA and the optimizations based on that. but the CPU and the lua bytecode is just using the fast and natural two address op syntax, where it is not easy to add SSA. This is the genius part. http://wiki.luajit.org/SSA-IR-2.0
Yes, potion. It's a better rewrite of lua, as luajit.
Same op and data layout as lua, just with a simple method jit (luajit does tracing and optimizations), plus a msg send based OO.
As an electrical engineer I deal with assembly programming all too frequently. So clearly I am missing something important about the concepts presented here. Why on earth would I go to the trouble of writing assembly code to put in the middle of my C and then use another tool to strip the assembly and convert it to C? Because it turns out that I'm even better at writing C than I am at assembly. And although the power of assembly is fantastic, I vastly prefer C.
Thanks in advance for any explanation. I'm clearly missing something that should be obvious.
Sometimes there are existing binaries you want to instrument with code. Maybe to profile or perform dynamic analysis for something like bug detection or security enhancement.
Valgrind operates like this. Instrument the allocation routines and with tracking of pointer variables you can detect dangling pointers and overflows and leaks and so on.
I'm a jack of all trades design engineer in the R&D space. Assembly is appropriate for the very low end of the 8-bit microcontroller space. Especially for timing critical code. Like a bit banged RS-232 port. There are still micros with only a few kb of flash. Can be hard to fit much of anything in there. You can do some amazing things in assembly if you're motivated, which tends to happen when you have a fifty cent budget for a micro, two weeks left for delivery of prototypes, and code left to write.
I'm not the person you're asking but I suspect many embedded systems engineers are at least familiar with the basics of assembly.
Also, it isn't that hard to pick up the basics of assembly, the main reasons it's not common anymore is tedium of writing code in assembly, and not being as portable as higher level languages.
I don't know where Mike Pall works, but one of the best things about working at Google is that you get to collaborate with a lot of folks of this caliber. It can be very stressful for some people to be in an environment like this, but if you approach it with the right mindset - it's wonderful.
I used DynASM in my little weekend project https://github.com/jsn/icfp06 , see README for my experience. In short: I like the idea a lot, but the implementation is seriously buggy. My experience with GNU Lightning (during the same weekend project) was much more smooth.
LuaJIT (of which DynASM is a component) is a JIT compiler for Lua (a dynamic language), which despite being basically a one-man project was beating contemporary Javascript compilers on performance and even the Dalvik Java compiler on Android (making it the fastest JIT compiler in existence for a dynamic language). Also, it targets multiple platforms (x86, x64, ARM, PPC, MIPS). It's been in maintenance mode for a while I think.
Luajit a wonderfully clever piece of code that only one person understands and is able to support. Before everyone chimes in that I'm mistaken ask yourself why Luajit has not evolved to be compatible with the latest Lua 5.3 release.
Lua was pretty much perfect at version 5.1 (the version LuaJIT supports). Lua is meant to be a simple language, not one of those languages which accumulate more and more features and thus complexity over time (I am looking at you C#, Java, etc.).
So unlike those languages Lua can really be considered "done". Version 5.2 was a lot like C99, maybe people considered one or two features "nice to have" but most Lua programmers saw no need to upgrade. Others like Mike Pall even considered 5.2 inferior to 5.1.
Now we have 5.3 which is really all about one special-purpose feature: 64 bit integers. That is a feature most Lua users do not need and it comes with a massive complexity and performance price tag.
Unless you really need 64 bit integer math in your scripts 5.3 is the inferior language.
I really enjoy that in classic Lua a number is a number, that is such a relief compared to the hell that is C's countless numeric types and all the potential bugs resulting from (silent) conversions between them. Lua 5.3 brought that problem to Lua e.g.
a = 9223372036854775000
b = 0.5
c = a + b
c = c - b
print(c) // 9.2233720368548e+18
The third line silently converts a 64-bit integer variable to a 64-bit floating point variable, corrupting its value.
And of course Lua 5.3 is slower and needs more memory because of the additional complexity which must be handled.
Thanks but no thanks.
I mean Lua 5.3 still makes sense if you really need 64-bit integers in your Lua scripts. But otherwise it is a downgrade.
I have little experience in lua myself, and I know only that lua is meant to be a simple language. That said, I can't see from your example what's wrong with lua5.3. I ran the same code with lua5.1, lua5.2, and lua5.3 – and the output was identical across all:
In Lua < 5.3
a = 9223372036854775000
is not even valid code. Because only 32-bit integer values are supported.
In Lua < 5.3 the value is immediately corrupted (check the value of 'a' after the initial assignment) and static analysis can detect that line as a bug.
In contrast in Lua 5.3 it is perfectly fine code and the value of 'a' will be 9223372036854775000 as one would expect. However the moment you - maybe accidentally - mix that int64 variable with a float64 variable in an expression you end up with a (silent) bug.
Such bugs will happen. They happen all the time in C and in Lua's case things are even worse because function parameters do not have types. The possibilities of hard to find, nasty bugs are endless here. E.g. simply forgetting the second '/' in '//' in one part of the program can cause bugs in a completely different part, maybe even in a third party module, and only under certain circumstances.
>That's not true at all, Lua < 5.3 uses doubles for all numbers
I know that of course, but using doubles for integer math is only safe if you limit yourself to a certain integer range.
>which on most computers is IEEE 754, they can represent integers perfectly up to 2^53
False. You get rounding errors after 10^14.
>The value is not corrupted, depending on the rounding mode is which number you're going to get out of that (I get 9223372036854774784)
The above being an example of such a rounding error.
If I assign the value 9223372036854775000 to a variable I expect said variable to afterwards have the value 9223372036854775000 not 9223372036854774784. Your example is exactly what I meant with "corrupted" i.e. the value is changed by the conversion.
As a long-term Lua fan, I completely agree with you, even though its not a popular opinion. Too many times I've been debugging some Lua code, tearing my hair out, only to discover "oh yeah, we're using Lua5.3 for this - is that a problem?" .. but at least we're catching up with Python in this regard.
It stayed at v5.1 (with some v5.2 features that don't break 5.1 code) because Mike Pall disagreed with some v5.2 features: swapping `setfenv` for `_ENV`, and some details of the bundled bitlib.
He also criticized 128 bits interpreter values mandated by the support of 64 bits integer in v5.3. These take twice as much cache space and damage performance.
Except luajit has boxed 64bit integers: there's no reason you couldn't have them act just like lua 5.3 64 bit ints.
In the majority of real use cases you can narrow down to the 32bit ints already supported by luajit.
Tarantool[1], a general-purpose database and app server based on LuaJIT, has Lua/C API to work with 64-bit numbers and even custom cdata objects [2]. I can extract this code into a separate library for you.
Currently in luajit you're just "not meant to" work with ffi cdata from the C api. You can... even if it's not pretty.
However, I was more saying that there isn't a technical reason that lua 5.3 integer behaviour couldn't reuse the existing (boxed) 64bit integer support in luajit. Which would mean that the slim tagged NaN tagged value approach can be kept.
I think the main problem here is that there are several semantic differences between 5.3's and LuaJIT's integers which would make this a bit dodgy; eg. rounding behavior and mixed float-integer arithmetic are different.
IMO 5.3 should have done what LuaJIT did and require a suffix for 64-bit integer literals (say `L` to avoid conflict with LuaJIT's `LL`). Given how rarely Lua code needs true integers this large, that probably would have worked better. It also would have avoided the gotchas of porting code to 5.3 where the integral numbers you never add `.0` to are suddenly real integers instead of floats.
I'd love for LuaJIT to get 5.3's bitwise ops but the semantic differences between the integer implementations would make this interesting to say the least.
Not sure what you talk about.. lua5.3 stores 64bit integers in the same tagged union it stored doubles so the value size didn't grow at all. 64bit ints only need a new tag value (which was available before).
Lua 5.2 beta implements the "NaN trick": it packs all Lua values into a
single double value by representing non-numeric values as signaled NaNs,
which are (usually) not produced by the system.
This trick should work on most 32-bit machines that follow IEEE
754-2008 for double representation. However, currently it is being
enabled only by predefined macro __i386__ (and similars). We would
like to know what other platforms could benefit from this trick
(and what macros should we check in luaconf.h). You can force the
trick by compiling Lua with -DLUA_NANTRICKLE (for little endian
systems) or -DLUA_NANTRICKBE (for big endian).
(Of course, we would also like to know whether there are problems
with this implementation.)
-- Roberto
----
I also think I remember people (maybe Mike Pall) reporting perf regressions between 5.2 and 5.3 because of the cache bloat introduced by mandatory 128bits values (on 64bits systems).
From what I've read, Lua has apparently done a poor job at maintaining backwards compatibility and Mike Pall is unwilling to support those kinds of changes.
I don't have enough context to know if breaking backwards compatibility is justified or not. I generally think it's a poor decision that's hostile to developers to break backwards compatibility in a programming language without providing a clear migration path. Issues between 5.1, 5.2, and 5.3 definitely seem to have splintered the community.
The backward compatibility issue Mike is ranting about in this post is not in the language, it is in the source code of Lua itself (actually a define in its configuration header file).
There was a very good reason for this change, which is that Lua introduced an integer type instead of using floating point numbers for everything. This means that LUA_NUMBER_DOUBLE and LUA_NUMBER_FLOAT do not really make sense anymore (when you check those you may want to check the type of the integer numbers or the type of the floating-point numbers).
In any case, porting luabitop doesn't really matter that much: if LuaJIT supported Lua 5.3 people would use the native 64 bit bitwise operators it introduced instead.
I feel that the language itself has remained remarkably compatible through releases. The difficult change, I believe, has been the changes to the API for native extensions (if I'm remembering my history right).
Hi, Tarantool[1] team maintains production-ready fork of LuaJIT [2]. Commercial support contract for Tarantool covers problems with LuaJIT as the integral part of the product.
It looks like the main change in 5.3 is an overhaul of how numbers work to add integers and integer division. This seems like a bad call and I'd rather not deal with it.
