I was in a conversation recently about the power of macros, and the use of syntactic abstraction in building simpler systems.
We quickly realized though: it’s tough to convey in a conversation what’s so special about macros. What can you do with macros that you couldn’t do with functions?
In this essay, we’ll use 2 examples, alongside some imaginary javascript syntax and lisp [1] to explore that question!
Note: This tutorial assumes you have a light understanding of lisp syntax. Go through this tutorial to brush up if you haven’t gotten to explore lisp yet.
Note: I’ve been meaning to write this for weeks, but was worried that it would be confusing. I am going to apologize now if that’s how you end up feeling when you read this. There may be a better way to explain it, but I needed to get this out of the head. If you have any feedback on how I could make this simpler, please let me know 🙂
[1] The specific language is Clojure, but anything done here can be done with any lisp
With this example, let’s gain an intuition for when macros run and why that can be powerful.
In any language with nulls, there’s a nullthrows abstraction: If some value evaluates to null, throw it.
Here’s how we could implement that as a function in javascript:
function nullthrows(result) {
if (result === null || result === undefined) {
throw new Error("uh oh");
}
return result;
}So if we run it, and it evaluates to null, we’ll throw an exception
nullthrows(getUser(db, 'billy'))
// if it's null, throw ExceptionThis works great…but there’s a problem. What would our stacktrace look like?
index.html:700 Uncaught Error: uh oh
at nullthrows (index.html:700)
at someStuff (index.html:1325)
...When some value is null, the stacktrace won’t have much helpful information. It will say which line threw, but we’d have to do some digging each time to find out where the code was.
One way we can fix that, is to pass in a message argument
nullthrows(getUser(db, 'billy'), 'expected billy');
function nullthrows(result, message) {
if (result === null || result === undefined) {
throw new Error(`uh oh: ${message}`);
...This could work…buut
What if I told you: I don’t want to have to pass in a message.
Instead, when the source code that nullthrows wraps is specific enough, I’d be just as happy if the error printed the offending piece of code.
For example, with nullthrows(getUser(db, ‘billy')), nullthrows is wrapping the source code getUser(db, ‘billy’))
If the error printed out “Uh oh, this returned null: getUser(db, ‘billy’)”, it would be specific enough, and I wouldn’t need a custom error message.
Well, by the time nullthrows is run, getUser(db, ‘billy’) will be long gone: all the function will see is the evaluation of getUser(db, ‘billy’). Since the evaluation will be null, there’s not much information we can gain.
To actually capture and work on source code, we need a new kind of abstraction.
This would be some kind of function, that does two things:
Let’s say Javascript had that. instead of function nullthrows, we would have macro nullthrows. It could look something like this:
macro nullthrows(sourceCodeSnippet) {
return `
const result = ${sourceCodeSnippet};
if (result === null || result === undefined) {
throw new Error("Uh oh, this returned null:" + ${stringify(sourceCodeSnippet)});
} else {
return result;
}
`;
}Here, the input would be the actual source code.
Whenever it’s called, we would replace that piece of code, the source code snippet that this abstraction generates.
For example, during the build step nullthrows(getUser(db, ‘billy’)) would be replaced with:
const res = getUser(db, 'billy');
if (res === null || res === undefined) {
throw new Error("Uh oh, this failed:" + "getUser(db, 'billy')");
} else {
return res;
}Now, you might see some potential problems here:
Snippets are just text! It’s really had to programmatically add/remove/edit text without causing a bunch of syntax errors. Imagine if you wanted to change the source code, based on what it was — is it a functional call or a value? — there would be no way to tell with just text.
With javascript, you can work on the abstract syntax tree itself with babel transforms, but that will make the implementation quite different from what we want our code to be doing.
We really want to use some better data-structures to represent our code. Turns out, this idea isn’t new: there’s a whole family of languages — the lisp family — that wanted code to be represented with data-structures so we could read/generate code snippets more easily.
It seems like a lot of work just to make a built-in code-snippet-generator for a language, but let’s see what our challenge looks like if we use lisp’s approach:
Since all code in lisp, are just lists, our lisp macro takes in a list of code, and returns a new list of code
We would write nil-throws like this:
(nil-throws (get-user "billy"))The function variant would look like this:
(defn nil-throws [res]
(if (nil? res)
(throw "uh oh")
res))Now, I’m going to show you how the macro variant would look like: (don’t worry about a few of the symbols you’ll see, they’re all simple and I’ll explain them in just a few words below)
(defmacro nil-throws [form]
`(let [result# ~form] ;; assign the evaluation of form to result#
(if (nil? result#)
(throw
(ex-info "uh oh, we got nil!" {:form '~form})) ;; save form for inspection
result#)))Here’s how we can think about it:
Similar to how we wrote ` in javascript, the backtick here does the same thing: it says, hey, here’s the code I want to return, don’t evaluate it right away.
# is a handy way to generate some symbol, that won’t interfere with any other symbol when this code gets replaced in a different scope.
~ is like our interpolation ${} in javascript, but for lists
‘ is a way to say: hey, I want to treat something as a list, and don’t want to evaluate it
This would make it so when we write: (nil-throws (get-user db “billy”))
It would be replaced (~approximately) with:
(let [result# (get-user db "billy")]
(if (nil? result#)
(throw (ex-info "uh oh, we got nil!" {:form '(get-user db "billy")}))
result#))Wow…we just wrote code that wrote more code…that’s pretty cool
Macros take code as input, and return code as output. They run during the build step
Now, let’s explore the kind of power this can give us.
The pipe operator is quite common in a bunch of languages.
It takes code that you would normally write like this:
createBill(addToCart(cart, updatePrice(item, 100)))And let’s you invert the flow visually:
item |> updatePrice($$, 100) |> addToCart(cart, $$) |> createBillWhat if our language didn’t have this, and we wanted to implement it? Maybe we’d want our syntax to look like this:
|> [
item,
updatePrice($$, 100), // updatePrice(item, 100)
addToCart(cart, $$), // addToCart(cart, updatePrice(item, 100))
createBill, // createbill(addToCart(cart, updatePrice(item, 100)))
]Now, we could do this by implementing a pipe function:
pipe(item, (item) => updatePrice(item, 100))But we would need to introduce anonymous functions, and the code would be less concise.
The only way to do this to spec, would be to change the syntax itself.
Now, with our imaginary javascript syntax, we could write something like this:
macro |> (listOfForms) {
return listOfForms.reduce(
(lastForm, thisForm) => {
if (isFunctionalCall(thisForm)) {
return `
let $$ = ${lastForm};
${thisForm};
`;
} else {
return `${callFunction(thisForm, lastForm)}`;
};
});
}Here, would start with a list of the forms, un-evaluated:
[item, updatePrice($$, 100), addToCart(cart, $$), createBill]Reduce would start with the arguments lastForm = item, thisForm = updatePrice($$, 100)
Now, we would need a way to know: is this a form of a function call updatePrice($$, 100), or just a function: createBill
If it’s a function call, we can create new code, which defines $$ as the last form, and evaluate the function call within that scope.
Otherwise, we can create new code, which calls that function with the last form.
What we want would be something like this:
(|> item
(update-price $$ 100)
(add-to-cart cart $$)
create-bill)And our macro could look like this:
(defmacro |> [form & forms]
(reduce
(fn [last-v form]
(if (seq? form) ;; am I being called: (update-price $$ 100)
`(let [~(symbol "$$") ~last-v]
~form)
`(~form ~last-v))) ;; or am I just a function: create-bill
form
forms))Our lisp code would follow the same idea as our Javascript solution. Let’s see the code we didn’t have to write:
(create-bill
(let [$$ (let [$$ item]
(update-price $$ 100))]
(add-to-cart cart $$)))…that’s pretty cool. We get the best of both worlds: efficient, well-erroring code that’s short to write and — most importantly — clear to read.
Macros let you change the language itself. You can transform code and change the syntax.
Macros let you change your language to suit your problem. This is extremely powerful: You can build up your language so you can express your problem as clearly as possible. This makes your code more concise and simple, which in turn makes your system more malleable.
At the same time, macros…change the language itself. There are few moments where this level of abstraction is warranted, so if you use them when simpler abstractions would do, you risk adding unnecessary complexity.
Yet… when they are warranted, having them as an option can change the game.
If this got you interested, here’s some reading and practice you may enjoy:
Shoutout to Daniel Woelfel: I saw his nil-throws macro years back when we worked together, and it opened my eyes to the power of syntactic abstraction.
Thanks to Sean Grove, Daniel Woelfel, Martin Raison, Alex Reichert, Mark Shlick for a beautifully deep review of this essay.
Thanks to Paul McJones, perfunctory and tzs, for their feedback on the code examples.