Compositing the Myths

How to make the images?

In my last blog post, “Is It Art?” I questioned whether making composite images from the output of other people’s GANs was an appropriate approach. (A GAN is a type of neural network for image generation). It seems like my human-machine collaboration experiment will be more personal and immediate if I am the one training the neural networks, rather than just selecting from the pre-existing output of other GANs.

I had made one of the four images I planned as “prototypes” as part of my practice-based research. I decided to dig a little more into making my own neural networks before proceeding with the next three images.

First try setting up a GAN

Trying to program GANs turned out to be extremely tricky, however. There is a lot of math, computer science and theoretical knowledge required. The scientific papers on the topic are quite daunting. Although several good implementations are available on Github, I quickly hit obstacles I didn’t know how to traverse.

One of the first obstacles was not having a suitable computer. For most machine learning work, a powerful computer with an Nvidia GPU is needed. Many of these models will take months to run on the CPU alone. Having heard that it’s possible to rent time on a virtual machine with a good GPU from Amazon, I set out to follow some tutorials to set that up. There is a delay of a couple of days  while Amazon reviews the request.

The next issue I had was getting anything to run on a remote server. The information I found on how to do this seems to be geared to people who are pretty experienced programmers. I think I got things running in the end, but had some troublesome errors and found that some of the python scripts in the repository had known issues that have not yet been fixed.

I realized that I am a little too new with machine learning to jump into the deep end right away. If it were just the theory or the math or just Python or just the libraries that were new to me, I could work my way through it, but since all of these things are new to me, taking them on all at once is not going to go well. I decided to start looking for help, and at the same time started looking at other machine learning techniques for image generation.

In the meantime, I proceeded making the next two images based on the general outlines I devised and sketched (see the “Four Prototype Images” blog post).

“Earth” Image

This image depicts the machines’ notion that they may have originated in the earth. I like that this is an inversion of the myths in which a heavenly influence originates from the sky. If the machines have a garbled understanding of how the earliest computers were made, they might gather that mining obscure minerals was part of it. There is also this idea (for example in Stephen King’s Maximum Overdrive) that an animating force could emerge from underground due to careless excavation.

I decided to make this image out of pictures that were used in the famous ImageNet dataset, commonly used to train neural networks like BigGAN. ImageNet contains over 14 million images categorized by subject. They seem to be mostly from the pre-Facebook and Instagram era, when resolutions were low and sensors were poor. I soon found that almost all the images come from Flickr or personal or academic websites from before 2007 and are usually not much more than 500 pixels on the long edge. Normally for making composites look real, the photographer starts with a good looking, high res photo of a background, and tries to capture as much of the subject and props “in camera” as possible, to reduce the amount of time needed in photoshop trying to make things look real. Since I may not be aiming for photorealism anyway, I decided to embrace this limitation, and see where it leads.

As I began work on the image, I found that ImageNet is a Stanford-based initiative. A few weeks ago, my co-supervisor Adam Tindale mentioned that while he was there, he witnessed some IT department employees wheeling a safe through the lab. In it was a digital copy of all of Stanford’s data from that year. They were taking it out to a secret location in the desert to bury it, so that in the event of a global calamity, a future civilization could one day find and revive their lost knowledge. This seems like a particularly enlightened form of reverse mining. We’ve taken all the readily accessible metals and minerals from the surface of the earth and scattered them. If our modern technology were to cease, it would be hard for a subsequent civilization to have a Bronze or Iron Age. Burying our data might not fix that problem, but could be a boost in the event there is a civilization that comes after a dark age.

I thought it would be interesting that in searching for their origins underground, a future machine civilization could uncover ImageNet, a well-organized catalogue of our world and civilization, and try to make sense of us using those images.

I did a few more sketches to try to flesh out my ideas for this composition. I considered having a few workers unearthing a tablet with strange markings at the bottom of a pit, something like a strip mine. Unfortunately, it isn’t usually possible to find such specific items in ImageNet. It is organized in a hierarchy using only nouns, so verbs like working, digging etc are absent. It is set up to be used as a classification hierarchy, rather than a set of keywords.

I thought about making a mountain in the reverse image of a strip mine – like a stepped or terraced pyramid. This is reflected in my initial sketch, in which I composited a scene from a strip mine.

Experimental sketch: how would I turn a strip mine into a mountain through compositing?

Through ImageNet I found the original Flickr album from a mining company and browsed through it while looking for images I could use while planning my composition. It is interesting that all of these images have copyright rules that restrict them from being modified or used in derivative works like mine. So images of what goes on under the earth aren’t public. These spaces are mediated by corporations who, judging by their Flickr album, have a very keen interest in controlling the perception of what goes in the ground, with workers and in nearby communities.

I spent quite a bit of time looking for mining images while planning this composition. It ended up being an important part of my research and making process. I found myself reflecting that all of my grandparents came to Canada from Europe for work in the gold mines in remote locations of northern Ontario. This mining was part of the colonial history of Canada.

I searched for images of veins in the rock that I could use to show the gold like circuit traces coming out of the ground. A lot of this was beyond what I am capable of drawing from scratch, or compositing from ImageNet. Nonetheless, it gave me ideas for the next set of images.

I settled on a composition and produced a more detailed sketch:

Final sketch before photo compositing

In this sketch, two figures (generated by a GAN) have unearthed a golden computer punch card (the preferred storage medium for data before the 1960s). The strip mine-like spiral ramp in the foreground goes down into the earth, which has a coppery glow. The mountain might be a pile of iron pellets. The golden sky is decorated in the style of religious panel paintings with a pattern depicting drones. This represents the machines’ ascendence from the ground into the sky. If I can, I will have a neural network evolve the outline of the drones in each row going upwards.

Since this is a prototype, I did not incorporate all the elements I dreamed up in the image below. It is composed of six photographs, with some hand-illustrated elements such as the shovel and the lighting effects on the foreground figures.

Composite of six photographs – a punch card being unearthed by the long-vanished humans during machine pre-history.

This is intended to be printed over burnished gold so that the computer card and the sun will reflect light brilliantly.

My first experiments with printing on a shiny surface were pretty disastrous.

Gerhard Richter would have approved of this print. I was disappointed.

There are several artists who have published books and videos about printing on unusual surfaces. With some experimentation, I was able to print with an inkjet printer or a shiny aluminum film, letting the white areas of the image allow the base material to shine through:

The sun and punch card reveal the shiny surface beneath.

As I mentioned in the post called “Physical Processes, Collaborations and Surprises”, making these digital images physical is an important part of this project. Having a reflective surface as the background of the print allows for a much higher contrast ration than you can get on a screen.

This image is not really something I want to show widely, but was a helpful part of my process in thinking through the questions of machine origin myths.