Welcome everyone, to my talk about Lua.
My name is Andreas.
You might have seen me on
the nickname of ComicSansMS
which is a really silly nickname,
so it's usually not taken,
except on Twitter, because
Twitter's full of crazy people,
so I had to use an even
sillier nickname there.
I'm one of the organizers of
the Munich C++ User Group,
so yeah, if you ever happen
to be in our around Munich
and want to get in touch
with the local community,
give us a call, we'd be happy
to set up a meeting with you.
And my day job is incredibly
working as a software architect
for BMW on the autonomous
driving software platform.
So today I'm going to talk about Lua.
So can I maybe get a quick show of hands,
how many people have used Lua
in like a serious project,
like not a toy project but like,
something production,
yeah, couple of hands.
Okay.
For those of you who have not used it,
hopefully I can convince
you that it's a language
that is worth investigating
even for serious projects.
So, what is it that is
distinctive about Lua
that sets it apart from other languages.
I'd say there is two
features that are kind
of unique to it, the first
one is that it's actually
not designed to be a standalone language.
It's an embeddable
language, how they call it.
So usually you have like some application,
like the most famous use
case for Lua is being games,
that is written in some other
programming language, often C++.
And then you have Lua
in there to customize
like some small parts.
But, it's also being used
for programming microcontrollers,
embedded microcontrollers,
or also like serious,
serious desktop applications,
like Adobe Lightroom.
The other thing that is
kind of unique about Lua
is that it's small.
It's really small.
So you're not supposed to be able
to read this thing on the left,
but that is actually the full EBNF syntax
of the language, which
fits exactly on one A4
sheet of paper, the compiled
binary is really small.
These 180 KB is actually
the full Lua package
including the standard library
if you strip out the libraries
and the parser if you only
need to run compiled BiCode,
you can make this significantly smaller.
The complete reference
manual is only 82 pages
is really the complete
documentation including
the full CAPI, that is
required for integrating it
with a enclosing program,
it only has eight basic data types.
As you can imagine, from these numbers,
there's not a whole lot
of features in there.
Sort of if you take like the Python
approach of batteries included,
Lua is sort of the opposite
of that conceptually.
But, the cool thing about Lua is that,
for them is actually not a limitation.
Is actually a virtue.
And not only because this
allows them to run on devices
where other languages
wouldn't fit, it's also
that from a language design perspective,
they really embrace this
property of being small,
and turn that into a
very elegant language.
So, I always like to
say, the whole language
fits into your head at the same time.
Which is something that
I cannot say of most
other programming languages.
I certainly cannot say that of C++,
unless maybe you're one of those guys,
but like for me, if I use
C++, I have to constantly
swap in and out of the
different features of my head,
and with Lua I don't need to do that.
Which is really nice.
So, hopefully I will be
able to convince you,
in the progress of this
talk, that although Lua
is small it is quite a powerful
and interesting language.
However, I have to say,
the problem with this talk,
is like, um, I cannot assume
that there are any Lua
experts in the audience,
so on the one hand it's going
to be an introductory talk
to Lua, but on the other
hand I also want to show
interesting examples, right.
And the problem with that
is that these interesting
examples, they're sometimes
a little bit contrived,
so and a little bit too
clever, so, what you see here
might not be representative
of what you would write in every day Lua.
Like, typically it might be a lot simpler.
In particular what I want to say is,
if at any point during the
talk you have the impression
that oh no, this is like,
really unnecessarily complex,
and I don't get it and I don't really
want to use this stuff, please blame that
on the talk, and not on the language.
That out of the way, this is Hello World.
Notice that although my syntax highlighter
likes to make the print
blue, it's actually
not a keyword, it's just a function
from the standard library,
which in Lua's case means
it's just an ordinary function it does not
get any special treatment whatsoever.
So this is how we declare a function.
It's a dynamic language, so we
don't have to give any types,
we just say that we have three arguments
and give them names, the
interesting thing about Lua,
this is the first thing
where we actually see it,
it being small and lightweight,
is that functions are
actually not like this, this
special kind of thing that they are in C,
but all functions are actually lambdas,
all functions are anonymous,
and this declaration
that we see up there,
is just syntactic sugar
for what we see down there,
which is the construction
of an anonymous function,
which is then being assigned
to a variable of type f.
Right?
So, f is just a variable like any other,
and you just assign to it
a value of type function.
And what that really
means is like functions
are true first-class values.
Function values are not
any different from strings
or numbers or anything else.
Okay?
So that way we can reuse
all of the properties
that we have for the other
values also for functions.
That keeps it small, but that also enables
some interesting things.
So, for example, let's say I'm not happy
with the print function here.
I want to replace it by my own.
I can just do that.
It's just a value.
I just assigned to it a
different function to it.
No problem.
Let's say I want to count
the number of print calls.
That could be a use case for this, right?
I just replace the vanilla print function
by the new function, and
there's actually valid syntax,
this is a very adept
function, which just increases
the counter, and then calls
the original print function within.
Now, you notice that there's actually not
any variable declarations in this code.
I just assigned values
directly to variables
that were not previously declared.
This is something that
most scripting language
actually will allow you to
do, so if you just assign
to a new variable, you
don't need to declare it,
it just automatically
creates a new global variable
and puts the value into it.
Which is not a behavior
that is always useful,
but we will get to that later how we can.
Maybe work around that
if we don't like it.
What is really not nice about this code
is that we now polluted
our global name space
with a variable named count.
So, if we only ever
count the print function,
don't do much else in the
program, that might be okay,
but really it doesn't look that nice.
So what we would like
to do instead is keep
this count function at a smaller scope,
so, for example, have a
function enable counting,
which like performs this
hooking of the print function,
and have the count function
and the old print function
as local variables in there.
So here notice if I prefix
the variable with the local
keyword that declares that
variable as a local variable,
it works pretty much exactly the same
as local variables in C++.
Now the interesting thing is
if I now write the same code
that I wrote before,
what will happen is that
the local variables from
the end closing scope
will be implicitly captured
in that new local function.
So like in C++ if you wanted to do this,
you would have to write
in the area brackets
of lambda, the captures explicitly.
So like the names of count and old print,
you would have to specify
them, write them out.
In Lua you don't need to do this.
Just as lexical scoping, so it knows
that it can pull in the
stuff from the outer scope.
The interesting question is now.
I just made the count
variable a local variable,
so how can I access my count now.
Like if I want to find out
of often print was called.
I can just return a
function that gives me back
the value of the counter.
And this is interesting now, right.
Because the count is an
integral value, so it has
value semantics, and it
also has value semantics
in Lua, just as it has in C++.
So, now have captured the same value
in two different lambdas,
in two different closures.
And it still works.
So this is actually where the full
in full lexical scoping comes from.
That allows you to do something like this.
This is actually really
difficult to pull off
in the implementation,
and Lua couldn't do this
from the beginning, but
since version 5.0, I think,
they support it, and it's
a really powerful feature.
Notice that of course
since we capture stuff
that means that our function
objects are actual objects,
so whenever you read
function, interpret that
as construction of a closure.
So there's actually memory
potentially being allocated there
and this needs to be garbage collected,
potentially, at some point.
Okay.
Let's talk about tables.
Tables are the most important
data structure in Lua.
In fact it's the only complex
data structure in Lua.
And, I actually don't
know of any other language
that can get by with
just one data structure.
Which is quite fascinating.
So what is a table, like,
this is the simplest table,
it's just constructing an
empty table with nothing in it.
But a table, imagine it as
being an associative array.
So, you can use it just as
you would use a C array,
but you can also use it as a dictionary.
Using a, using it to map keys to values.
And the interesting thing about Lua tables
is that they actually allow
you to map any type of value,
to any other value, so for
example, you can use functions
as keys, you can use other
tables as keys, if you want.
This is again, very
much, the Lua philosophy.
Right, like you only
provide very few features,
but you make those as
flexible as possible.
Notice that tables, unlike the
numbers that you saw before,
have reference semantics.
So, for example, what we do
here, is we basically built
assembly link list, we
construct like two note tables,
each of them holding a value,
and then we link them together,
through the next field of the first note.
You don't usually need to
build link lists like this,
because like if you just use plain tables,
they usually powerful enough
for what you need to do,
but it's useful to know
that you can do it.
As with functions before, whenever you,
read the curly braces, think
of it as table construction,
so this means that you will
potentially allocate memory.
Since tables are only complex data type,
if we want to build like a
struct, or like a record type,
you also have to do that with tables,
which means like we simply create entries
in our dictionary, in our table,
for the different fields of the struct.
And since the syntax that we
already know for accessing
the tables is a bit
cumbersome for this use case,
Lua provides syntactic sugar with the,
for accessing the fields with a dot.
So the last line is just syntactic sugar
for the line buff there,
completely equivalent otherwise.
So we want to add member
functions to our record now,
we can just assign a function
to one of the fields,
and then call it as member function.
So notice Lua's tables
have reference semantics,
so this function actually mutates,
the value inside the table,
and the call down there,
this is, think of it as
a member function calling
in C whether to give
this pointer explicitly.
Seems we don't want to pass it explicitly
because it's kind of silly,
we also get syntax sugar
for that, with the colon syntax.
So, the thing about tables is that,
we don't really have types right,
so all of our complex numbers
have the same type table.
So if we want to build
multiple values of this table,
we usually have to supply
a constructive function
like we see here.
So let's say now we have
these complex numbers
and we want to add to some.
For this we would actually need
operator overloading, right?
And Lua provides this through
a feature called Metatables.
So everything in Lua is a table right?
So Metatable, the idea here is basically,
I provide a table which
I then attach to a value,
and this table tells me
how to treat this type
that is attached to, in the language.
So, there's a couple of special fields,
and they allow you to
specify how it treats
the other authentic operators,
the comparison operators,
how it treats the element access
with the outright brackets and so on.
So, I just changed my
construction function
to add this Metatable, call that,
assigns the Metatable
to my newly constructed
complex number, and then I can
just add them with the plus operator.
So, let's say now I have
more complicated data type,
where I actually have invariance
between the different,
on the different fields.
Might not want to allow the
user to access them directly,
but rather encapsulate them.
So, instead of exposing
your month and date
of the date directly, I
actually want to provide
getter and setter functions.
And the way that we do this
is in the construction,
we actually build a
local table that contains
the actual data, then what goes
in the table that we return,
is just to getter and setter functions,
which again access the actual
data through the closure.
So some of the similar trick that we saw
with the counter before,
and that allows us to give,
to implement encapsulation
without any explicit
language support for it.
Which is kind of cool.
Of course, since a table is the end
just an associative array,
we also get a certain
reflection properties for free.
Like, a table is a complex
number if it has these fields,
real and imaginary.
And of course I can also
inspect all of the fields
by just iterating over the table
which in Lua I do with this
syntax, so the pairs function
just gives me back an iterator function,
which with each invocation
returns the next field
of the table, and then I
have this generic form,
that processes iterators.
So this allows me to perform
reflection on individual
table that I have already in my hand,
but what about global variables?
Right?
I might need to know, which objects
are aligned around in my environment.
So Lua has a solution for that.
Global variables are actually
not global variables,
they are entries in a table called _G.
So, as I said, everything was a table,
and global variables
are just syntactic sugar
for accessing elements
in this global table _G.
And this has a really
interesting implication.
Cause like, as we saw
before, we don't need
to be clear variables.
So what people really
don't like about this,
if I make a silly typo
like this, where I assign
to fobar instead of foobar, the language
will just silently create
a new global variable.
Which is not what I wanted here.
But, G is just a table, right?
So this creates a new entry
fobar into this table.
And I can just use
Metatables to forbid this.
So, I can use, say, you're
not allowed to create
any new entries into this table.
And Lua actually provides
on their homepage module
that does this in a little
more complicated way,
and the idea there, is that
you can sort of restrict
where you are allowed to
create global variables.
So like only allow to
declare them in like certain
parts of your script, or
only be allowed to declare
them through declare function,
which then bypasses the Metatable.
And this is I think a pretty cool feature
and it basically comes for free
from the properties of the tables,
and the fact that everything
in Lua is a table.
So, this is all pretty cool,
but if you have ever tried
to integrate scripting language with C++,
you know that this is
usually not a lot of fun.
So, how that does work with Lua.
So as we said before, Lua
is an embedded language,
that means the main functions
still belongs to C++.
And in there, we use Lua
more or less as a library.
So, we create a Lua State object, and then
we can just run some Lua code.
So who thinks that doing
something like this
with a scripting language should
be any more complicated than this?
No one.
Okay, that's what I thought.
So, just running Lua, chunks of Lua code
in isolation is not very interesting.
We also want to communicate
with our C++ program.
So in order to export
functions from C into Lua,
so that Lua code can
call them, we actually
have to wrap them in
functions of this signature.
Like any function with this
signature can be directly
injected into the Lua VM.
And as you might notice
there, this actually
does not take any function parameters
and it also does not
provide any return values.
It just takes the Lua State object.
And that is by design.
So the Lua API is all designed around
this concept of a stack.
And the idea of the stack is that whenever
you want to get a value out of the VM
or into the VM, you have
to put it on the stack,
and then you can grab it from the stack.
And this is, this might seem
a limited design at first,
because like if you imagine
if you want to set a value
inside a table, you have to
put the table on the stack,
you have to put the key on the stack,
and you have to put
the value on the stack.
And then you have to make
the call that does the actual
assignment, which is a little bit verbose.
But there are two
fundamental advantages here.
The first advantage is that
it keeps the API very small,
because we have only
eight data types overall,
and you only need two
functions per data type,
one to put it on the stack and
one to get it from the stack.
The second implication is
a little bit more subtle
but I think it's even more important,
is that it solves the ownership problem.
Cause the problem with garbage
collector languages is,
if I now grab like a table,
from inside the Lua VM,
and expose it to the C++
code, I actually need
to make sure that the
VM's aware of the fact
that I am pointing to this table,
so that the garbage collector
doesn't pull it out from under me.
And with Lua, I cannot access the table
if it's not on the stack.
But as soon as it's on the stack,
the stack has ownership on it.
This might not seem like a big deal,
but if you're using this in everyday code,
this is actually really powerful.
So, pushing values on the stack,
I'm just gonna use number and string
as the example types here,
some of the types are a little
more complicated, but not much.
So let's say I just
have two push functions,
so two overloads for push functions,
one that pushes a number, so Lua numbers
are usually double, but if you compile
for a platform where you
don't have double available
you can actually change that
very easily to like pure interval types.
I would push a string.
And then we can very easily
write a function like this
in C++ 17, which uses the fold expression,
to just push an arbitrary number
of arguments to the stack.
Is everyone familiar with this syntax?
I've seen it quite a
lot this week already.
So if you don't have C++ 17, available,
you can also use a library to do the same,
so here's an example that uses boost hana.
And if that also doesn't work for you
there was actually a
talk, I think yesterday
by Joe Fan-Cue, where he
have like some examples
on how you can emulate this
feature with all the C++
versions, I just put the
code up for reference,
you don't have to read it,
you're certainly not
expected to understand it.
Because it's horrible.
But it's possible.
So what about the other way around.
How can I get values from the stack,
so out of the VM into C++.
So here the thing is that since
Lua is a dynamic language,
I don't know what the type is
of, of the value on the stack.
So I need to do the switch case here.
But, the thing is that,
this is actually the only,
the only point where I'm willing to stick
to this factor where
I don't know the type.
Because like, as soon as
this function returns,
I'm actually again in the C++ type system,
so I want this to be like, as
rigorous as possible again.
So, what can we use to
as a return value here?
So we could of course use a base class,
like, solve the problem
with virtual inheritance,
but that's just so 90's, right.
So, in this example, you
have to define what like
your unified interface of
all the different types is,
in my case it's just a type function.
So each value can tell
you what it's type is.
And then you could just use a
C++ 17 variant, for example.
So, like in a full
implementation you would have,
there's only eight types in Lua.
And you know them all beforehand.
So variant is actually really good fit,
and you can also implement it with
very little size overhead.
Like you might lose some
bytes when saving numbers
or bullions in there,
but for tables, functions
and the more complex types,
it's pretty much uniform
for memory consumption.
And then of course, this
is how you get stuff
out of the variant with visitation.
If you don't like variants
and you would like to use a
different kind of type erasure,
Louis Ti-Yong was giving a very nice talk
yesterday about runtime polymorphism,
where he explained all the
different options that you have
if you don't want to use a variant.
Okay, so let's call a
function now from C++.
Let's call a function.
So, we load the function on the stack,
get it by its name, put
it on the Lua stack,
and you push all of our
arguments onto the stack.
And we call the function,
and then we inspect
the stack to get the return values
that were returned by the Lua function.
And I noticed that in
Lua, a function can have
an arbitrary number of return values,
so I actually have to
check, after the Lua call,
API call returns, I have
to inspect the stack
to see like how many
values were put on there.
This is also the reason
why here I have to return
a vectoral value, because
I'm not sure how many
arguments am I going to get.
And then I can just call a Lua function,
like here I'm calling a print function,
from the Lua standard library
with a couple of arguments,
and we just print them to the console.
So of course this is the most general
signature that you can give
for such a core function.
You could constrain it,
like if you know the number
of return values and the
number of arguments beforehand,
or maybe even know their types.
Right?
So, you probably want
to go to like the most
restrictive signature
that applies for your case
to make sure you get the maximum support
from the C++ type system.
Okay.
So to wrap up.
This is actually the
description that the Lua people
give like how they
understand the language,
their elevator pitch.
It's a powerful efficient lightweight
embeddable scripting language.
And I hope through this talk
I could give you an idea
of what this attributes
in the context of Lua.
I compiled a small list of literature,
the reference manual for Lua
is actually really good,
there's a book by the authors
of Lua called Programming
in Lua, which is,
it's not a big book,
it's around 300 pages,
if you're interested in the language
I would really recommend
purchasing the book,
not only because it's a great book,
but also because it's
a great way to support
the creators of Lua financially.
Because it's created by a university,
and like they don't get a
lot of funding otherwise.
And that, I think we have
some time left for questions.
Thank you.
(clapping)
- [Moderator] If you have a question,
please go up to the microphone.
Hello.
Thanks for the talk.
Yeah.
You mentioned that you have an application
that uses several libraries,
for example, dynamic libraries,
and each of these library want to do some
of its own stuff, and as
separate Lua interpreters.
So, how do they get, do
they conflict with this
or okay for them to have several
interpreters in several application?
You mean, I think,
several Lua interpreters,
in one application?
Yes, I mention several
modules are developed
independently, and all of
them rely on their own Lua,
something, so I want
those to be independent.
Yeah, so what you can
always do, like we saw
in the simple example
that you have to create
this Lua State at one
point, and if you want
the different modules
to be truly independent,
you can just have each
module use its own Lua State.
And then they can even run concurrently
in multiple threads,
there's like zero overlap
between the different
VMs, but then of course
you have to take care that
if you do want communication
from one Lua VM to the next,
than you need to model that,
but if you're fine with
them being totally separate
then that's actually a good way.
The problem there is that
there's some memory overlap
to those Lua States, so if
you're in a very constrained
environment, you might
not have that memory.
And what you can do there,
is you can also restrict the environment.
So this _G table, that we saw, that stores
all the global variables,
you can also restrict that.
So saying that I'm now
executing a chunk of Lua code
under this environment.
And sometimes that's
enough of a separation,
but it's a little bit weaker
than having truly separate States.
Yeah great.
And the second question is,
it seems like to use Lua
functions conveniently
from C++ coding, it's
some kind of boilerplate,
but we demonstrate this, so I wonder,
is there some standard solutions.
Some what?
Standard solutions, some libraries
which contains such boilerplate.
A library which takes care
of the bindings for you.
Well that provides such functions,
functions like Kol, like that is shown.
So, there are libraries that
help with the integration,
a pretty famous one is
I think called Luabind,
you can use those, but
the interesting thing,
is that the API is so
simple, and you want to roll
your own, this is not an
unreasonable thing to do.
Like you can probably get a decent
library up and running within a weekend.
So, there are libraries
available, but you don't
have to use them, like it's easy enough
to do it by hand, if you want.
Thanks.
Sure.
[Audience Member 1] So
how do you debug Lua?
Oh yeah, that's a good point.
So the interesting thing is actually part
of the standard library of Lua is actually
a debug module, so you can actually write
your debugger for Lua inside Lua.
Which is pretty cool.
[Audience Member 1]
You can separate points
and things like that.
Yeah.
And you can inspect like local variables
from closure and stuff like that.
So.
[Audience Member 1] And
then you write your own.
Like if you want it to go to Standard Out,
and have some sort of
interactive mode, you --
The thing is that this is just an API.
This is not a ready-to-use debugger.
So you would still need
to write a user interface
in order to be able to actually debug.
But since Lua's an embedded language,
so you don't know what the
environment is going to be,
they don't take care of that for you.
So if you only use
vanilla Lua, you will have
to write it for yourself, but it gives you
all the tools that you
need to be able to this.
Thank you.
[Audience Member 2] Just
wanted to make a quick comment
if you don't mind, on
the question of something
to help generate binding?
Sure.
Sol2, that's S-O-L two, is
a really well-written modern
C++ binding generator for Lua.
Okay, cool.
Thanks for the info.
Okay?
And I think we're through.
Thank you very much.
(clapping)
