Embedding Wolang
Wolang is a library before it is a tool. One header, a C ABI, and a libwolang_embed to link: a program written in something else — an engine, a server, an…
Wolang is a library before it is a tool. One header, a C ABI, and a
libwolang_embed to link: a program written in something else — an
engine, a server, an editor — gets Wolang as the language its users
write.
#include "wolang/embed/wolang.h"
WolangVm* vm = wolang_vm_create();
wolang_vm_bind(vm, "host_log", (void*)&host_log); // before loading
wolang_vm_load(vm, "scripts/enemy"); // a package or a file
WolangValue delta = wolang_float(1.0 / 60.0);
WolangValue out;
wolang_vm_call(vm, "Update", &delta, 1, &out);Everything below is the same API with the ceremony removed, for C++:
#include "wolang/embed/wolang.hpp"
wolang::Vm vm;
vm.Bind("host_log", &host_log);
if (!vm.LoadSource("enemy.wo", source)) {
std::fputs(vm.errors().c_str(), stderr);
}
vm.SetGlobal("speed", 3.0);
const wolang::Function update = vm.Lookup("Update");
for (;;) {
if (const auto answer = update(1.0f / 60.0f)) {
Apply(answer->number());
}
}Getting it
cmake --preset release
cmake --build --preset release -j
cmake --install build/release --prefix /usr/localThat lays out exactly what a host needs:
<prefix>/include/wolang/embed/wolang.h the C API
<prefix>/include/wolang/embed/wolang.hpp the C++ one, header only
<prefix>/lib/libwolang_embed.a static
<prefix>/lib/libwolang_embed.dylib|so shared
<prefix>/lib/std, http, game, core the standard library's sources
<prefix>/lib/cmake/Wolang/ find_package(Wolang CONFIG)
<prefix>/lib/pkgconfig/wolang-embed.pc for everything else
<prefix>/bin/wolang the compiler, for `wolang new`With CMake:
find_package(Wolang CONFIG REQUIRED)
target_link_libraries(my_engine PRIVATE Wolang::embed)
# ${WOLANG_STD_DIR} is where the standard library's sources are.Without CMake:
cc host.c $(pkg-config --cflags --libs wolang-embed) -o hostA release archive holds the same layout, so unpacking it beside a project
and pointing CMAKE_PREFIX_PATH at it is enough.
What a host can do
| Load | wolang_vm_load takes a package directory or a .wo file; wolang_vm_load_source takes text under a name of your choosing. A second call adds to what is loaded. A lone file or a piece of source sits in a package named host that can need the standard library, found through WOLANG_STD or a lib/std above the file (D259). |
| Call | wolang_vm_call by name, or wolang_vm_function once and wolang_call per frame. The compiled tier runs what it can and the interpreter runs the rest, which is why calls are fast without asking. |
| Ask | wolang_vm_function_count/_name/_module list what the package offered and the module that offered it, so two packages that both offer Tick are told apart as sample.mover.Tick and sample.camera.Tick; a bare Tick that two modules offer is refused (D259); wolang_function_arity, _parameter, _result and _doc say what one takes, answers and is for. |
| Configure | wolang_vm_get_global/_set_global read and write the program's module-level vars. A script's var gravity: f32 = -9.8 is a setting the host owns. |
| Extend | wolang_vm_bind makes a host function reachable: the script declares extern "C" fun host_log(message: *u8, length: u64) and calls it. |
| Capture | wolang_vm_set_writer sends the program's output to the host's console instead of standard output; wolang_vm_set_reporter does the same for diagnostics. std.io.Print and Eprint reach them too, in every tier (D259). |
| Contain | wolang_vm_set_budget stops a call after so many ticks in either tier, for scripts the host did not write (D265). wolang_vm_set_step_limit counts interpreter instructions and keeps calls on the interpreter. |
| Debug | wolang_vm_debug_listen opens a port an editor attaches to with the Debug Adapter Protocol; breakpoints stop the host's call where it is. See debugging. A trap that ends a call leaves wolang_vm_errors (and the reporter) the kind, the source line and the script's call stack, one in Function (file:line:column) per frame, innermost first; the compiled tier gives function names without lines (D269). |
| Reload | wolang_vm_reload loads everything again and keeps the program's state; wolang_vm_file_count/_name say which files to watch. |
| Tick | wolang_vm_tick gives the program's tasks a step and dt seconds of the host's clock, so wait Wait(2.0) means two seconds of the game's time (D267). |
What crosses the boundary
Integers, floats, booleans and text. A string is a pointer and a length in both directions — the host's bytes are borrowed for the length of a call, and a string the program answered is copied out, because the program's own copy may live in an arena that is about to be given back.
Objects cross too, in both directions — see below. Anything else richer — a list, a struct, an enum with a payload — stays inside the language. A function at the boundary takes and answers what C can carry, which is a constraint worth designing the script's API around rather than working past.
The host's objects in a script
An engine's actor is the engine's: it makes it, owns it, and frees it.
A script names the type with extern class and gets methods and nothing
else:
extern class Actor {
fun Position() -> Vec3
fun Hurt(amount: s32)
}
offer fun Tick(self_actor: Actor) uses unsafe {
self_actor.Hurt(5)
}The host binds each method as Class_Method, taking the object first,
and says how its objects are counted:
wolang_vm_register_class(vm, "Actor", ActorRetain, ActorRelease, ActorAlive, engine);
wolang_vm_bind(vm, "Actor_Hurt", (void*)&Actor_Hurt); // void Actor_Hurt(void* actor, int32_t amount)
WolangValue actor = wolang_handle(engine_actor);
wolang_vm_call(vm, "Tick", &actor, 1, NULL);A script that keeps an Actor in a field calls retain, and release
when it lets go — the host decides what that means, a reference count of
its own or nothing at all. A weak var costs nothing and asks alive
when it is read, which is how a script holds an actor the level may
destroy: it reads null afterwards instead of a dangling pointer. An
object passed into a call is lent for the length of it. A script cannot
construct an extern class or give it fields (E0139).
alive alone cannot tell an object from a new one built at the same address.
A host that reuses addresses registers its class with a token, through the
size-prefixed WolangHostClass (D254), before it loads the program:
WolangHostClass actor = {0};
actor.size = sizeof(actor);
actor.name = "Actor";
actor.retain = ActorRetain;
actor.release = ActorRelease;
actor.alive = ActorAlive;
actor.token = ActorToken; // uint64_t ActorToken(void* object, void* user)
actor.user = engine;
wolang_vm_register_class_ex(vm, &actor);A weak var then remembers the token the object had when it was stored, and
reads null when alive says yes but token answers something else. The
token is asked only of an object alive says is there.
The ABI keeps its shape
wolang.h defines WOLANG_ABI_VERSION (major in the high 16 bits, minor in
the low) and the library answers wolang_abi_version(). A host compares the
two once at load, because the library it links is not always the one it was
compiled against: a different major is a different ABI, a larger minor only
adds (D241). Minor 1 adds wolang_vm_function_module (D259). Minor 2 adds
wolang_vm_object_new, wolang_vm_object_set and the wolang_vm_class_*
reflection calls (D258).
if ((wolang_abi_version() >> 16) != (WOLANG_ABI_VERSION >> 16)) {
return 1; /* built against another ABI */
}
WolangVmOptions options = {sizeof(WolangVmOptions)};
WolangVm* vm = wolang_vm_create_with(&options);- A
WolangValueis 32 bytes:kind(auint32_t), a reserved word, one union ofinteger,number,booleanorhandle, thentextandlength. The member names are unchanged, so existing host code compiles. The size does not depend on the C compiler's choice of enum size. - A library built with
wolang build --library actorsexportsactors_abi_version()the same way, answering the same number. - Kinds are only ever added; a host accepts a kind it does not know.
- A struct that will grow starts with its size.
WolangVmOptionsis the first;wolang_vm_create_withanswers null for a size it does not know.wolang_vm_create()still works and means the defaults. - Fallible calls answer a
WolangStatus:kWolangOk(0),kWolangError(the call could not be made;wolang_vm_errorssays why) orkWolangTrap(the program trapped). Code is only ever added, so treat an unknown one as failure.!= 0still means "failed".
What a call costs
benchmarks/actors is a host ticking actors the way an engine does: a
Tick(actor, delta) per actor per frame that reads the actor's position
and velocity through its bindings, moves it, keeps it in bounds and
writes it back. The same logic in C++ goes through virtual calls. A
release build on Windows x64, a thousand actors:
| a frame | against C++ | |
|---|---|---|
| C++ | 5.8 µs | 1× |
| Wolang, compiled ahead of time | 9.6 µs | 1.7× |
| Wolang, the dev tier | 63 µs | 11× |
| Wolang, the interpreter | 8.1 ms | 1400× |
Every column ends in the same state as the C++ one. Ahead of time is what
a shipped game runs: wolang build --library actors writes an object with
no main, an actors_start() to call once, and actors_Tick to call
like any C function. The dev tier is what an editor runs, with hot
reloading; a call through a WolangFunction handle goes straight into
compiled code when every argument is a number, a bool or an object. The
interpreter is what runs under a debugger. wolang_actor_bench is built
with everything else, and prints the table (D219), and a second one with
calls per millisecond and objects allocated per call. The measured numbers,
on a named machine and an optimized build, are in
benchmarks.md (D273).
A game that ships a library built ahead of time does not need a C compiler on
the machine that builds it: on Linux wolang build links with a bundled
ld.lld and a musl sysroot when there is no c++ on PATH, and the object it
writes is the same every time (D272, building.md).
Bindings from an engine's reflection
An engine with reflection already knows its classes, so nobody should
write extern class declarations by hand. wolang bind --reflection
reads the JSON a header tool writes (MowoEngine's *.reflection.json)
and writes both halves:
wolang bind --reflection --effect World Intermediate/Build/Generated/Debug/MowoEngine.*/Reflection/*.json -o scripts/mowo/src/mowo.wo --glue Source/Scripting/MowoWolangBindings.cppmowo.wo— anoffer extern classper reflected class, with its base, its methods, and aGet/Setpair per property. A method that takes or answers aVec3, aQuat, aRotatoror a string has a raw form the host answers and a method in anextendblock that scripts call:actor.GetActorLocation()answers aVec3. A string comes back as an owned copy of what the engine held, and a reflected struct whose fields are all numbers is answered the way a vector is. A property of an object type (X*orTObjectPtr<X>) or ofstd::stringhas itsGet/Setpair like any other.- Statics — an engine
staticfunction is a freeoffer funin the same package, named after the function; two classes that both have aCountgetActorCountandWorldCount, the class name without its one-letter engine prefix first (D262). - The glue — an
extern "C"function per method and accessor, andMowoWolang_Bind_mowo(vm), which registers every class and binds every function. A null object, or oneMowoWolangAlivesays the engine has destroyed, answers zero rather than crashing;--no-alive-checkleaves the second test out (D262). A property is reached by its offset, asked for once from the engine's own reflection throughMowoWolangPropertyOffset(class, key).
The engine supplies five functions the glue declares: that offset, and
MowoWolangRetain/Release/Alive/Token for its objects' lifetimes.
MowoWolangToken(object, user) answers a number that tells two objects
apart that lived at the same address one after the other, such as an index
and a serial; the glue registers each class with it, so a weak var does
not mistake a new object for the one that died (D254). A host with no serial
to give passes --no-token, and the glue registers without one, as before. With
--effect World every engine call carries uses World, so a mod has to
be granted World to touch the world at all.
Which compiler the bindings fit (D271). The first thing MowoWolang_Bind_*
does is call wolang_vm_expect_abi(vm, WOLANG_ABI_VERSION) with the header the
glue was compiled against, and then wolang_vm_expect_struct once for each
struct it passes by pointer, with that struct's size, alignment and field
offsets from sizeof, alignof and offsetof. The rule is the one the ABI
already has: the major version must be equal and the bindings' minor must not
be greater than the library's. A struct is checked when the program loads
(or at once, if it is loaded already): the compiler's own layout of the
struct of that name must be the same numbers. If either fails, the load fails
and wolang_vm_errors says what each side expected, for example the bindings expect Vec3 to be size 16, align 8, offsets 0, 8, 12, but this compiler lays it out as size 12, align 4, offsets 0, 4, 8. A change to how the compiler lays
a struct out is therefore an ABI change: it bumps the major version, and
bindings made before it say so instead of reading the wrong bytes.
What is left out is printed, one line each: components (which belong on the ECS side), properties and functions of types with no binding, structs with a field that is not a number, a struct taken as a parameter, and overloads (D218, D262).
The program's objects in the host
A function that answers a class instance hands the host a
kWolangObject: handle is the object, text is its type. The host
holds a reference, and lets go of it with wolang_vm_object_release —
the last one runs the class's deinit and frees it, the same as inside
the language.
WolangValue enemy;
wolang_vm_call(vm, "Spawn", &health, 1, &enemy);
wolang_vm_object_call(vm, enemy, "Hurt", &amount, 1, &left);
wolang_vm_object_get(vm, enemy, "health", &left);
wolang_vm_object_release(vm, enemy);wolang_vm_object_retain takes another reference, for a host that keeps
the object in two places.
A host can also make an object and fill it (D258):
WolangValue spot;
wolang_vm_object_new(vm, "Spot", &spot);
wolang_vm_object_set(vm, spot, "x", wolang_int(7));
wolang_vm_object_release(vm, spot);Every field starts as zero. Only a numeric, boolean or optional field can be
left that way safely, so a class with a text, class or array field, or with
virtual methods, is refused: make it in a script function and call that.
wolang_vm_class_count/_name, wolang_vm_class_field_count/_name/_kind
and wolang_vm_class_attribute_count/_name list the program's classes, the
stored fields of each and the attributes written on it (Replicated for
@Replicated), so a host need not scan the source.
What the program says should be replicated is reported the same way (D268).
wolang_vm_class_field_replication(vm, "Pawn", i, &flags) answers
kWolangReplicated for a field marked @Replicated, with
WOLANG_REPLICATION_OWNER in flags for @Replicated(owner); the methods are
listed by wolang_vm_class_method_count/_name, and
wolang_vm_class_method_replication answers kWolangServer, kWolangClient or
kWolangMulticast for an RPC, with WOLANG_REPLICATION_UNRELIABLE for
(unreliable). wolang_vm_class_method_param_count/_param_kind give the
parameters, which are always kinds a host can read. Wolang sends nothing: the
host maps these to its own replication and RPC policy.
Saving a held object (ABI 1.6, D292) is two calls. The bytes are the ones
std.save writes, so a save written by a script loads here and the other way
round:
const uint8_t* bytes = NULL;
uint64_t length = 0;
if (wolang_vm_save(vm, mover, &bytes, &length) == kWolangOk) {
WriteFile("slot1.sav", bytes, length); // valid until the next save or reload
}
WolangValue back = wolang_void();
wolang_vm_load_save(vm, "Mover", data, size, &back); // you own one referenceThe fields marked @Save of the object and of every object they reach are
written; a shared reference stays shared and a cycle comes back a cycle. Bytes
that are not a save of the class give kWolangError and allocate nothing. A
class with a virtual method cannot be saved or loaded, and the reason is in
wolang_vm_errors.
Each vm has its own bindings, host classes and writers (D257). Two vms in one process may bind the same symbol to different functions and print to different places; a lookup asks the vm whose call it is, then the process.
Mods
A script the host wrote is trusted: it can declare extern "C" and reach
anything the process can, the same way the standard library does. A mod
is loaded differently:
wolang_vm_load(vm, "scripts/engine_api"); // the host's own, trusted
wolang_vm_load_mod(vm, "mods/better_turrets", "World, Audio");The mod is granted those capabilities and nothing else, whatever its own
manifest says. unsafe is never grantable: writing it in the list is
ignored (D270), and since the standard library's network and process calls
are unsafe too, a mod granted io and net still cannot open a socket or
start a program. It may not use unsafe or declare extern (E0705), so
the only doors out are the ones the trusted packages offer — and each of
those states the capability it needs, down to the standard library's
std.fs.Open, which is io. A mod that reaches past its grant does not
load: E0703 names the function and the capability (D217).
The grant bounds reach. wolang_vm_set_budget bounds time, for a mod
that loops forever (D265): a call that spends its ticks stops with
kWolangTrap and the message step limit, and the next call starts afresh.
A tick is one entry into a script function or one jump back to an earlier
block; it is counted in the interpreter and in the compiled tier alike, so a
budget does not cost the speed of compiled code, only a subtract and a branch
at those points. It is a guard against a script that never ends, not a meter:
the two tiers draw the line between "back" and "forward" a block or so apart.
Together they are what a mod sandbox is. Memory a
mod allocates is not bounded, and a host that runs mods from strangers
should still give them a vm of their own.
Packages held as data
A host that ships its bindings and the standard library inside the program has
no directory to point wolang_vm_load at. wolang_vm_load_package_source (ABI
1.7, D293) loads a package from memory under the name its scripts need:
const char* paths[] = {"src/main.wo", "src/ui/button.wo"};
const char* sources[] = {kMowoMain, kMowoButton};
wolang_vm_load_package_source(vm, "mowo",
"[package]\nname = \"mowo\"\nversion = \"1.0.0\"\n", paths, sources, 2);
const char* script_paths[] = {"src/main.wo"};
const char* script_sources[] = {"need mowo.ui.{Press}\nfun main() { Press() }\n"};
wolang_vm_load_package_source(vm, "app",
"[package]\nname = \"app\"\n\n[dependencies]\nmowo = { path = \".\" }\n",
script_paths, script_sources, 1);A path is below the package root and src/ui/button.wo is in the module
mowo.ui, as on disk. The manifest is the host's and is trusted like a package
on disk (unsafe and [link] allowed); a mod is still sandboxed only through
wolang_vm_load_mod. A dependency is found by the name of a package already
loaded, so load std first; the path it needs to parse is a placeholder.
Pass a null manifest and the package depends on everything loaded so far, as
source from wolang_vm_load_source does, and the standard library is looked
for the way it is for a lone file, but only when a file has a line that begins
with need (a comment that says "need" does not count), so that search can
depend on WOLANG_STD and the working directory. host, a name already
loaded, a manifest that names another package, a dependency that is not loaded,
a path outside src/ and a path with an empty, . or .. segment are refused
with E0060; a manifest that does not parse, or a dependency version that does
not fit, is refused too and leaves the name free for a corrected retry. A reload loads the package again from the
data the vm kept.
Tasks on the host's clock
A script writes code that waits as a task, and the host decides what time is:
task fun Wait(seconds: f64)
class Door {
var open: bool = false
}
var door: Door = Door()
task fun OpenLater(target: Door) {
wait Wait(2.0)
target.open = true
}
offer fun Start() {
const _handle = spawn OpenLater(door)
}wolang_vm_call(vm, "Start", NULL, 0, &ignored); // starts the task
for (;;) {
wolang_vm_tick(vm, frame_seconds, &steps); // one step each, clock moved on
}wolang_vm_tick (ABI 1.5, D267) moves the vm's clock by dt and gives every
task that was ready one step. A task that is not finished comes back on the next
tick, and one that was started during a tick starts on the next. Not calling it
is a pause and a smaller dt is slow motion, so the engine needs no hook for its
clock. wolang_vm_tasks_pending says how many are still going. Each vm has its
own tasks and its own clock. A host starts a task by calling a script function
that spawns one, and the tasks of a vm must all be declared in one of the
host's files.
Threads
Any thread may call into a WolangVm, and a vm takes one call at a time:
a second thread waits for the first to come back. A call that reaches a
host function which calls into the same vm again is the same thread and
goes straight through. That is the whole contract (D214), and it gives
an engine two ways to run scripts from a job system:
- One vm per worker. Each worker loads the program into its own vm,
and calls run in parallel. A module-level
varbelongs to its vm, so state the workers share lives on the host's side — in the engine's components, which is where it was going to be anyway. - One vm, many workers. Every job calls the same vm and the calls take turns. Simple, and right for scripts that are cheap next to what the engine does around them.
The counts on the program's objects are atomic, so an object one vm
handed to the host can be retained and released from any thread. The
callbacks an extern class registers are called on whichever thread is
running the script at the time; an engine whose own counting is not
thread-safe should keep those calls on the thread it counts on.
Hot reloading
if (ChangedOnDisk(vm)) {
wolang_vm_reload(vm); // `update` still works, and runs the new code
Log(wolang_vm_notes(vm)); // what had to start again, if anything
}A reload keeps what the program had. Every module-level var the new
program still declares with the same type keeps its value, and that
includes objects: a var player: Player is the same player, with the
health it had. When Player gained a field, the player keeps the fields
that are still there, and the new one starts from what the initializer
gave it. A var or a field whose type changed starts again from its
initializer, and wolang_vm_notes has a line saying so.
A reload that cannot work changes nothing (D255). The new program is
loaded beside the old one first, without starting anything. If it does not
compile, wolang_vm_reload answers an error, wolang_vm_errors says why, and
the program that was running keeps running. The same goes for a program that
would change the class of an object the host holds, unless the host said where
to hear about it:
static void Moved(WolangVm* vm, WolangValue was, WolangValue now, void* user) {
Swap(user, was.handle, now); // from here on, use `now`
}
wolang_vm_set_remap(vm, Moved, engine);
wolang_vm_reload(vm); // a class gained `speed`; the mover is carried overWith a function registered through wolang_vm_set_remap (ABI 1.4, D266) the
objects the host holds are carried to the new layout: each one, and every object
it reaches through the fields it stores, is written down under the old layout
and read back under the new one. A field that is still there keeps its value, a
new field is zero, a removed one is dropped, and a field whose type changed starts
from zero. The function is called once per object the host held, with the old
object and the new one; the vm has moved the host's references over, so each
wolang_vm_object_release of the old object is now one of the new. A
module-level var that pointed at the object points at the new one, so the host
and the program still share one object. The old object stays with the retired
program and is not to be used again.
A class the new program no longer has, or a field a reload cannot carry (an array, a closure, a weak reference, a host object, an enum that carries values), still refuses the reload with a line naming it; so does a changed class when no function is registered. Nothing changes until the host has let go of the object or fixed the field.
Function handles from wolang_vm_function survive a reload and call the
new code. The program as it was is kept until the vm is destroyed —
values carried over may still point into it — so a session that reloads
a thousand times uses a thousand programs' worth of memory, which is the
trade for never having a dangling string.
wolang_vm_file_count/_name give the files the program was built from,
including the standard library's, so watching them is a host's own
business and needs nothing from here.
Two files, no build system
A host that wants none of CMake, LLVM or a package manager can take the whole compiler as one source file:
python3 tools/package/make_amalgamation.py # build/amalgamationwolang.h the C interface
wolang.hpp the C++ one, over the same calls
wolang_amalgamation.cc everything else, about two megabytesclang++ -std=c++20 -fno-exceptions -fno-rtti -O2 \
-c wolang_amalgamation.cc -o wolang.o
clang++ host.cc wolang.o -o hostThat is the interpreter and the just-in-time tier — everything embedding uses. The ahead-of-time backend is not in it, because it needs LLVM and embedding does not.
What it cost to make this possible: half a dozen file-local names in the
compiler were too generic to share one translation unit — four different
Entry structs, two States, two Checkers — and they now say which
file they are in. CI builds the amalgamation and runs a host against it
on every commit, so it cannot quietly stop working (D186).