Play With Lua!

So you have made a program in Lua. You want to give it to people to run. You run into two problems right away: one, you don’t want to require people to install Lua to run your program, and two, you may not want them to have your source code (if it’s a commercial program). So, what to do?

If we were in a language like C, we could precompile our program into a binary and distribute that. So, since Lua can very closely integrate with C, can we do that with Lua? The answer turns out to be “yes”.

There are three steps to this process: first, we need to turn our Lua files into objects that a linker can put into a C program. Then, we need to tell the C program how to load the embedded objects. Finally, we need a way for Lua to know how to get them out of C, so that require will work.

Turning Lua into object files

When you compile a C program, there are two general steps: the compiler (gcc, say) turns your source code into an object file. This is a binary file that defines a bunch of objects, like variable declarations and functions. Then, a linker comes along and turns the object files into a program, by pulling out objects from each file (and shared files you reference, like shared libraries) and embedding them into a binary. So, if we can turn a Lua script into an object file, we can use the linker to embed it into the program.

One way to do this is to just make a C source file with a big string that has all your code in it. This would work but it’s a little stupid; you’d have to care about escaping quotes and things, it gets real hard real fast. It turns out that there’s an easier way:

Let’s say we have the following Lua code, in hello.lua:

print "Hello, world!"

We can do this to make an object file out of it:

ld -r -b binary -o hello.o hello.lua

(Unfortunately, this is only an option with GNU ld, not Apple’s, so if you have a Mac, you’ll need something different. It’s not too hard to write a program is Lua to turn an arbitrary file to a C source file)

So, now you have an object file that you can link in with the rest of your program. This object file will define three file-scoped variables we can refer to:

extern char binary_hello_lua_start[];
extern char binary_hello_lua_end[];
extern char binary_hello_lua_size[];

We only care about the start and size variables, end isn’t really useful.

Loading embedded resources in C

In our main program file, we need to declare the variables as extern references, just like above. Then, we can use luaL_loadbuffer to compile and load the code:

lua_State *lua = lua_open();
luaL_openlibs(lua);
luaL_loadbuffer(lua, binary_hello_lua_start, (int)&binary_hello_lua_size, "hello.lua");

This compiles our code (loaded from the embedded resource) and pushes it, as a function, on to the stack. We can call it with lua_pcall:

lua_pcall(lua, 0, LUA_MULTRET, 0);

So now if we run this, it ought to say “Hello, World!”. But this is sort of pointless if we can only embed one file; most of the time we’ll have multiple modules or libraries or whatever. It would be nice if we could load things the same way in embedded mode as normal mode, too.

Letting Lua decide what to load

When you require a module in Lua, it uses the very flexible package loading system to find your module. There’s an array of searcher functions that take a module name and either return the module compiled as a function, or nil. Conveniently, they return exactly what luaL_loadbuffer does above. So, if we had a (C) function that would take a module name and either call luaL_loadbuffer or push nil, then we could register that function in Lua, add it to package.loaders, and then use require normally.

The fundamental problem here is that we need to do reflection in C: we need a way to translate between the string “hello” and the variable binary_hello_lua_start. Unfortunately there’s no easy way to do that; we have to build a lookup table, so I’ll leave that as an exercise (since the focus here is on the actual embedding). If we just have our function assume that you’re always going to load “hello”, we can write it like this:

int get_module(lua_State *lua){
    const char *modname = luaL_checkstring(lua, 1);
 
    /* This is just an example, in a real app we'd
       turn modname into a real resource pointer
       instead of just using hello.lua */
    if(1 == 1){
        luaL_loadbuffer(lua, binary_hello_lua_start, (int)&binary_hello_lua_size, modname);
    } else {
        lua_pushnil(lua);
    }
 
    return 1;
}

Now we just need to register it and put it in package.loaders:

lua_register(lua, "get_module", get_module);
luaL_dostring(lua, "table.insert(package.loaders, get_module)");

Finally we just load the module, and require will take care of the rest:

luaL_dostring(lua, "require 'hello'");

Wrapping up

So, now we can take any arbitrary Lua code, link it into a C binary, and distribute a Lua-powered application without giving up our source code. The C library can statically link Lua, and can export other functions (to tie in other libraries), so that we have an application that’s the best of both worlds: high-level garbage-collected language for all our logic, low-level language for talking to libraries, and a simple binary distribution to make it easy for users to run it.

The code for this entry is here.