What Artists Notice

Painters

Painters at work notice details we’d ignore. From those details they develop delightful insights into reality. What do I mean? Let’s look at Van Eyck’s “Arnolfini Portrait” and find out:

Arnolfini Portrait, Jan van Eyck

Attention

The most whimsical idea starts with what Van Eyck painted. Do you think anyone in the world has paid more attention to this room than Van Eyk has? Look at the prayer beads beside the mirror:

Beads

Don’t they seem uncannily real? Imagine the kind of detail he’d have to notice to get the beads this right. The more you look, the more details you uncover. Notice how rich the wife’s dress is: the intricate folds, the patterns, the material. I’m not sure anyone but the dressmaker could have scrutinized it with more care. And even the dog is gifted with attention; look at its fur!

Try it yourself ^[1]. Pick a random spot and zoom into the painting. You’ll discover a world of detail you hadn’t seen. From here, it’s safe to say that at least Van Eyck noticed details we’d ignore.

Monet

But is this true for all painters? Van Eyck was known to bring his paintings to life with layers of detail. How about someone like Monet, who didn’t follow that strategy? Let’s look at his water lilies:

You’d be surprised. Monet actually created 250 paintings of water lilies in the same pond, to study the effect of light there. You saw 1 of 250, a devotion that engrossed close to three decades of his life. How much more do you think you’d see in a pond of water lilies, if you spent 30 years looking?

Our brains ignore details and see the essence of things. We barely notice the prayer beads, and wouldn’t take more than a few glances at the reflection of water lilies. The brush, however, forced Van Eyck and Monet to stop and see more. From there is born the premise that painters notice details we’d ignore.

Insight

Now details by themselves, although interesting, aren’t general. Who cares if Van Eyck was intimately aware of prayer beads or if Monet had a deeper understanding of a pond? Let’s zoom out and see some of the larger implications. Back to the Arnolfini Portrait:

Fruit

Why is there fruit by the window? How about the clogs? Do you see all the various symmetrical arrangements: the bead and the broom, the grey clogs and the red clogs, the couple’s position? And what about the colors? Why is the husband wearing a dark purple coat, and not a red one? Look at the wife’s gesture. It’s so maidenly, but why?

Subconscious

PG originally shared this idea in a masterful essay ^[2], and I’ll adopt it here. Every painting Van Eyck made had a purpose; to affect his audience. He may have wanted to please, shock, or dazzle; the spectrum of human emotions were fully available to him. This also meant that every painting worked doubly as a test; did it achieve its purpose? The more Van Eyck painted, the more he discovered exactly what worked.

Slowly but surely details amalgamated into patterns, and Van Eyck began to map our subconscious drivers. He discovered our inborn love of symmetry for example, and used it to masterful advantage throughout the painting. He discovered the gestures that make us feel innocence, and now we can’t help but see it in the wife. ^[3]

When we look at his work, feelings and thoughts burgeon up, but we can’t put our finger on why. Well, Van Eyck could put his finger on why, and there comes the painter’s first insight: they learn what affects humans.

Already we hit an interesting philosophical plane. When you learn what affects humans, you’re bound to ask why. But it gets grander.

Reality

I left the coolest part of this painting last. Look at the mirror:

Mirror

Van Eyk painted the reflection! You can even see two folks peering back at the couple. ^[4]

Outside the effort this must have taken, imagine what Van Eyck learned. How deeply have you thought about how light reflects in convex mirrors? I’d bet a lot less than Van Eyck after this painting. In fact, some have tried to see just how accurate Van Eyck was, and compared his depiction with what a real convex mirror would have reflected ^[5]:

a is Van Eyck’s work, and d is the mathematically correct version. The similarity is remarkable.

Nature

Not only did Van Eyck have to learn what affected humans, but the brush forced him to see deeply into the real world. The mirror is just the beginning. Imagine what he learned trying to draw the shadows. What about the chandelier? Here we hit the painter’s second insight: they learn to unravel the secrets of nature.

Transcendence

As a painters master these insights, they take the canvas to heights we couldn’t have imagined before. For example, look at Botticelli’s “The Birth of Venus”

Venus

Stuff like symmetry here is obvious. But, glance over Venus’ neck:

It’s longer than a normal human neck! Or look at her hands and shoulders, a bit lower and longer than normal. Botticelli hacked our minds. He knew what pushed our buttons, so he didn’t just describe reality, he accentuated it. He showed us a view of Venus’s elegance that could only be achieved through painting. ^[6]

Redefinition

Eventually, painters can eschew reality’s constraints in ground-breaking ways, and show us what our eyes could never see:

These insights touch on what makes painting art. Painting isn’t a rote exercise, but an inquiry into reality and perception.

Programmers

Here’s a jump: I think programmers have a similar experience ^[7]. Programmers at work notice details we’d ignore. From those details they develop delightful insights into reality.

Now, programming is not as far along as painting yet; our art has just begun. Like painting in the early renaissance, our work is crude, but we’re accruing insight after insight. What are these insights?

Let’s find out by answering a question: what would a programmer notice if they made an app for painters? We’ll call it pixel.fyi, and we’ll help painters draw stuff online. Here’s our goal:

Origins

Before we get into what programmers notice, let’s learn what programmers do. Today our devices are so sophisticated that they seem intelligent, as though they “understand” what we want from them.

However, this intelligence is an illusion built upon illusions, and is perhaps the masterwork of the pioneer programmers of our time. Computers aren’t intelligent. Instead, they do simple things, like add and subtract numbers, very quickly.

Adding Machine

To a get sense, here’s the great-great-grandfather of our modern computer:

Here we have three light switches labeled “1”, “2”, and “3”. We also have three light bulbs labeled “3”, “4”, and “5”.

Now, would it be possible to wire things up so that if someone turned on our “1” and “2” light switch, the “3” light bulb would turn on?

It doesn’t seem so difficult, and we’d hardly call it intelligence, but if we did this we would have created a machine that could add 1 and 2!

Speed

That’s cool, but here’s something startling: though crude, our invention adds numbers faster than any human ever could. The answer comes at the speed of light through a wire; this means our machine can make millions of calculations by the time you finish this sentence.

Language

Interestingly, our machine also has the same advantage as modern computers, in that the answer comes quickly, and the same disadvantage, in that fundamentally there is no intelligence.

So what’s the big difference? Our machine is specific; it can only add 1 to 3. Modern computers are general; they can do any computation. Instead of wiring things up a specific way, modern computers let us use a computer language to simulate wiring things up in any way we like. It looks something like this:

if light_switch_1 is on and light_switch_2 is on:
  turn_on(bulb_3)

With that code, it’s as though we “wired up” our great-great-grandfather machine!

We come to the essence of what programmers do. They write precise instructions in a computer language, to get computers to solve problems. Precise is the key word here, because at the end of the day it all boils down to switches that turn on and off.

Attention, II

Okay, back to programmers today. The most whimsical idea starts with what they create. For example, in order for pixel.fyi to work, at the very least we need to let our users draw simple shapes, like circles or squares.

Circles

Well, how would a computer draw a circle? Switches have no concept of “circle”. Here’s one idea:

Let’s imagine the monitor is like a grid, where we can turn each pixel on or off. A circle is really all the pixels that are a specific distance away from a center point. If we could just figure out all the pixels to turn on, we’d have a circle!

Trigonometry

And how do we figure out those pixels? Let’s remember some math:

We could draw a right-angle triangle going from the center to a point on our circle at any particular degree. We can then use trigonometry to find the x and y coordinates for the point! All of a sudden, we have a plan to draw a circle. Here’s how a programmer could express it in a computer language:

radius = 20
pixels = []
for degrees in [0..360]:
  x = radius * cos(degrees)
  y = radius * sin(degrees)
  into(pixels, [x, y])
  
turn_on(pixels)

For every degree between 0 and 360, we calculate the x and y coordinates, and we have our pixel. We can then turn_on pixels, and voila, our circle is born! ^[8] Here’s how this would look:

Detail

So what’s the consequence of all of this? Look how much we learned about circles. Not only did we breath life to trigonometry, but we discovered the “essence” of a circle: it’s all the points that are equidistant from a center. And if you think about it, this is the same definition for a sphere! A sphere is all the points that are equidistant from a center in three dimensions. And now for a mind-bender: this must mean that “circle” exists in infinite dimensions; there’s a “4-D circle”, with points equidistant from a center! ^[9]

Our brains ignore details and see the essence of things. When we look at circles, the only property likely to burgeon up is that they’re round. The computer language however, forced us to see more. Sound familiar? From here is born the premise that programmers notice details we’d ignore.

Insight, II

Now details by themselves, though interesting, aren’t general. Who cares if we know more about circles than the average person? Well, let’s zoom out to see some of the larger implications. We can kick off with a roundabout journey, and take a look at algebra.

Variables

Do you remember the kind of homework you did in middle school? Stuff like:

Here, x is 3. Now, where did x come from? When we write x, we are using the concept of a variable. But variables didn’t always exist! They were only invented in the 16th century ^[10].

What did mathematicians do before? They used words. Instead of x + 2 = 5, they would write:

You can already get a sense of how annoying this could become. If all you have are sentences, even the area of a triangle can get taxing to describe. With variables, we could write:

Without variables, we’d be stuck with:

At some point, problems get so complicated that we can’t solve them with words anymore; no person’s head could fit the space required. That’s where variables come to the rescue. x isn’t just any mundane character, but a shorthand for our brain: what was intractable before can now fit into a middle schooler’s head.

That's thrilling and begs the question: what other concepts exist like this that we haven't discovered yet?

Complications

Programmers think about this question all the time. Why? Let’s look at our code for clues:

radius = 20
pixels = []
for degrees in [0..360]:
  x = radius * cos(degrees)
  y = radius * sin(degrees)
  into(pixels, [x, y])

turn_on(pixels)

We were a bit mistaken here. We assumed our center was at [0,0], and our radius was 20. What if we wanted to draw another circle, but this time with a center of [5, 5] and a radius of 15?

We could write this:

radius = 20
pixels = []
for degrees in [0..360]:
  x = radius * cos(degrees)
  y = radius * sin(degrees)
  into(pixels, [x, y])

turn_on(pixels)

radius_two = 15
center_two = [5,5]
pixels_two = []
for degrees in [0..360]:
  x = center_two.x + radius_two * cos(degrees)
  y = center_two.y + radius_two * sin(degrees)
  into(pixels_two, [x, y])

turn_on(pixels_two)

But already this is getting hard to think about. What if we wanted to draw a hundred different circles? We can’t just keep copying and pasting. We’d barely get to triangles before our program spilled out of our heads.

Snippets

We’re forced to invent a new concept. Here’s one; what if our language had a “snippet”? A way for us to say draw_circle, with different options for center and radius:

This is how it could work:

def draw_circle(center, radius):
  pixels = []
  for degrees in [0..360]:
    x = center.x + radius * cos(degrees)
    y = center.y + radius * sin(degrees)
    into(pixels, [x, y])
  pixels

turn_on(draw_circle([0, 0], 20)) # Draws one circle
turn_on(draw_circle([5, 5], 15)) # And another one!

def becomes our “snippet” making concept. Whenever we write draw_circle(…), it’s as though we run the code inside it, with center and radius set differently.

Just like variables, def gives our minds a shorthand to think with. Now drawing a hundred circles is easy peasy. Not only that, but wherever someone sees draw_circle, they no longer need to think about how it works; if we find a more efficient way to draw a circle for example, we could change our snippet and nobody would be the wiser. ^[11]

Map

def is just the beginning. Let’s keep going. Look at draw_circle again:

pixels = []
for degrees in [0..360]:
  x = center.x + radius * cos(degrees)
  y = center.y + radius * sin(degrees)
  into(pixels, [x, y])
pixels

We can write it like this, but what’s really going on here?

If you think about it, we’re transforming one list (of degrees) to another list (of pixels). If we use an analogy in the real world, it could look like an assembly line:

There’s one assembly line with degrees and a worker beside it. The worker picks up each degrees, figures out the corresponding pixel, and adds them to the pixel assembly line.

It took a sentence to explain, but it’s a valuable concept, and we can generalize it. What if we called this map ? A worker “maps” degrees to pixels. Here’s how how our program could look now:

def draw_circle(center, radius):
  map [0..360]: 
    degrees =>
    [center.x + radius * cos(degrees), center.y + radius * sin(degrees)]

Now whenever we see map, the assembly line idea pops into our head. Not only does it give us a shorthand, but it opens up a new world; what else goes on in assembly lines, and can we use those concepts to help us think better? ^[12]

Logic

Just with circles, we were forced to invent snippets and map. Imagine all that we’d discover if we kept going: what about shadows? what about layers?

The computer can only understand precise instructions, and there are only so many precise instructions we can keep in our heads. The more complicated the programs we write, the more we’re forced to invent new concepts. From there comes the programmer’s first insight: they learn what helps humans think better.

Already we hit an interesting philosophical plane. When you learn what helps humans think better, you’re bound to ask why. But it gets grander.

Reality, II

Now as we progress on with pixel.fyi, we’d want to support “undo” and “redo”. If a painter drew a smiley and didn’t like the nose, they should be able to “undo” it:

Undo / Redo

How could we support this? If we just turn_on(draw_circle(...)) all over the place, there’s no way for us to know what the last instruction would be. It’s time for us to look for a new concept.

Time

Here’s a new idea. What if we kept track of the painting over time? Something like this:

We have a list that represents time. Any change we make creates a new “version” of the painting, and adds it to the list. A painting at a point in time is just all the pixels that are turned on then.

“Current” points to the version we show our user. When they press “undo” or “redo”, we simply move “current” to another version in our list.

In our computer language, it could look something like this:

painting = [[]]
current = 0

def add_circle(center, radius):
  last_version = painting[current]
  new_version = into(last_version, draw_circle(center, radius))
  into(painting, new_version)
  current = current + 1

def undo(): 
  current = current - 1

loop: 
  turn_off(all_pixels)
  turn_on(paintings[current]))

This would support undo / redo, but it also opens up some interesting questions.

Identity

First, consider that when we think about a “painting” in real life, we think about it as the end result. But here, we think about a painting as all the versions that existed throughout time.

It’s convenient to think this way for undo and redo, but isn’t life actually like this? Unfortunately, we are stuck in 3 dimensions and can't travel through time, but if we could, we would see that identity encompasses time. A painting begun, even if it goes through change after change, is the same painting in the end. That’s the same with people; no cell in your body existed when you were 5, but you’re still you.

Change

Another interesting idea, how does change happen? When we draw, we think we irrevocably modify the canvas. But if we could travel through time, we’d see something different: the canvas would evolve. After all, if you add a nose and go back in time, you’d still see the canvas without the nose, so how could the canvas have been irrevocably modified?

As it turns out, my friend Dennis ^[13] was so intrigued by this idea that he once explored a 3D animation for it, and it was about painting to boot. Here’s his rendering of a painting that evolves over time:

So darn interesting. We’ve lived in four dimensions all along but never saw it like this.

Perception

And the insights don’t stop. Remember, this is a multiplayer drawing app. Could “Stopa” and “Joe” draw a shape at the same time?

This couldn’t happen when painting in real life; our hands don’t move fast enough. But a computer moves real fast; it’s certainly possible that Joe sees a blue shape and makes it red, but by the time this completes, Stopa sees the same blue shape and makes it green.

This problem opens an interesting question about how we perceive anything in life! If your friend is in front of you for example, how do you see them?

Well, light bounces around the room. Some of it hits your friend and reflects into your eyes. Your eyes then process the light, which activates signals in your brain. Those signals in turn activate more signals, until a thought pops up: “I see my friend”.

All of this takes time, right? It feels instant, but it takes time for light to move and for signals to activate in your brain. This means then, that everything you see is from the past!

When you try to solve problems like “Joe and Stopa change the same shape at the same time”, you realize at least computers always see the past. It’s awe-inspiring that our reality works the same. We just experienced a sliver of the programmer’s second insight: they learn to unravel the secrets of nature. ^[14]

Transcendence, II

As programmers master these insights, they take their work to heights we couldn’t have imagined before. Well, actually we don’t know this quite yet, because we programmers haven’t mastered the insights!

Giotto

If you paralleled the story of programmers with Renaissance painters, we’d be at the Giotto stage:

Giotto was early. Before him, most painters followed strict rules and their work was flat. He questioned assumptions, began to study nature, and made giant leaps in his lifetime ^[15]. Though his own work was still crude, in only a few generations Michelangelo would step on the scene.

That’s where the best programmers are today. Our work is crude, but we’re questioning assumptions and constantly looking for inspiration from nature. It’s so early that some of the best ideas are prime picking, and the state of the art evolves every year. We can already see astonishing results at the edges, like computers transcribing human speech with better accuracy or self-driving cars whizzing around roads.

Progress

We’re left in an exciting state. I’m curious to know what transcendence will look like in our field. Computers are faster than any natural organ in our body. They need no sleep, can work 24/7, can “distribute” their brains around the world, and run for millennia. What kind of surprises lie ahead for us? What kind of concepts will the Michelangelo programmers of the future be using?

These insights touch on what makes programming art. Programming isn’t a rote exercise, but an inquiry into reality and perception.

Art

Wow, what a journey we just went on.

Painters and Programmers

We learned that the brush forces the painter to notice details that we’d ignore. The painter’s goal is to affect us, and successive paintings help them unravel our subconscious drivers. Invariably they discover secrets of nature, and eventually transcend it.

Similarly, the computer language forces the programmer to notice details that we’d ignore. Their goal is to solve hard problems, and successive solutions help them unravel tools for human thought. Invariably they discover secrets of nature, and eventually transcend it.

Writers and Creators

It strikes me, that perhaps all creative pursuits are like this. This essay was born from a conversation, where I tried to explain why I found programming so invigorating. I thought I’d start with painting as the metaphor. However, putting words to paper had me discovering details I had ignored. I was plunged into book after book on art and even expanded my thinking on programming. ^[16]

There’s an obvious utility to the output of creation, but even if there wasn’t, I think there would be an overwhelming reason to do it anyway, just for the insights you get. ^[17]

I hope this got you thrilled about the beauty of making things.

Credits

Ernst Gombrich’s “Story of Art” influenced how I saw art throughout the ages. PG is a hero of mine, and his essays on hacking and painting continue to inspire me ^[18]. I am taken by Rich Hickey’s model for programming. His motivations for Clojure largely inspired the “Reality, II” section. ^[19]

Thanks to Joe Averbukh, Daniel Woelfel, Nicole Garcia Fischer, Mark Shlick, Lina Wang, Dennis Heihoff, Alex Reichert, Julien Odent, Irakli Popkhadze, Simon Chaffetz, Jacky Wang, Nino Parunashvili, Marek Golias for reviewing drafts of this essay