Making pictures

My process is to create raw materials using neural networks (GANs), composite them together by hand, and transfer the results to a gilded panel. For now, I am creating the designs inscribed on gold by hand in Illustrator.

The process starts with sketches and lots of experimentation with the outputs from the GANs. As I mentioned in a previous post, I recently built a larger dataset to train the GANs to produce better options. I was feeling limited by the GAN output I had.

I have been producing faces and bodies with separate neural networks. Our human minds are so finely attuned to the nuances of faces, that they benefit from the extra detail and exactness produced by a GAN trained on just faces.

Here’s an example of an image that didn’t seem worth printing. I didn’t do too much fine tuning of this image (e.g. to match the skin tone of the face to the body!). I liked the upward gaze of the face, and planned to have her contemplating a scene, but when assembled, I thought her look was too odd and her pose was too formal, and not very expressive. Nonetheless, it was one of the clearer figures and better faces from my GANs, so I didn’t have lots of choice.

Unsuccessful composite of face and body

I decided to build a better dataset and train the GANs some more, detailed in this blog post. The faces look promising, and I’ve been working on some new composites.

At the same time, I’ve been practicing my gilding. This is a pretty time-consuming step – building up the base and adding multiple layers, sanding them smooth and then very often encountering some other problem – cracks, scratches, or a base that’s too hard or too soft or too gluey or not gluey enough. If I rub through the gold while burnishing, it can be hard to patch without leaving obvious marks. As Zach Arias once said about photography, “We want to make big leaps, but what actually happens is gradual progress”. This seems to be true here as well. Each panel I make is better than the last, as I figure out what temperatures to use, and what brush, etc.

In the mornings I’ve been sketching compositions. I find I’m pretty foggy first thing in the morning, so the sketches can take unexpected turns. Through this process, I got the idea to portray the neural network’s architecture by inscribing it in the gold background. Over a few days this evolved:

Freehand drawing is not my forte, but it’s still a helpful way of thinking
Experimenting with giving the networks scale. In a digital drawing, I could include many more layers.

I also thought about working the nodes in a network into a lace, scrollwork or vine motif in the background that would blend organic and digital components, alluding to how life emerges from networks, and neural networks emerged from biological forms.

For the figure, I used a design created using transfer learning – a dataset built on portrait paintings then trained on renaissance faces. It produced an example that evoked a human figure, but with an interesting internal patterning suggesting a less literal and more symbolic portrayal of the figure.

I researched structures of neural networks and found some great resources. The Asimov Institute’s Neural Network Zoo (link), and Piotr MidgaƂ’s very insightful paper on medium about the value of visualizing in talking about network architecture (link).

The elements I wanted to combine (from left to right): a figure created by transfer learning (GAN) + neural network diagrams (source) + vine design

The network will emerge from the figure into the gold background, with the nodes of the network represented as leaves, and the complex interconnections between then represented by vines.

I am still working on the drawing while my gilded panels dry. Hoping to try punching (stippling) as well as engraving to get some shading on the leaves (nodes). Here is the panel so far:

The printed figure is mostly hidden under a layer of gold (which will be removed) and frisket – a protective film of latex that will be removed.

This is certainly my cleanest gilded panel so far, lacking the usual rips and tears, but there may be a problem with the surface – it’s not burnishing properly and I’m not sure why. No big leaps, then, but slow steady progress.