Park Generation using pix2pix

I heard about the pix2pix machine learning model recently, which can translate between domains like the following:

For example, by training on a large dataset of image pairs of building outlines and real building facades, the underlying model (a generative adversarial network) can learn to translate any arbitrary outline input into a generated ‘real’-looking building facade.

I work in the discipline of landscape architecture, where it is common to spend significant amounts of time creating drawings. After seeing this pix2pix model, I thought that it might be possible to automate this drawing process to some extent. One could draw a simpler outline and then use the model to automatically draw more complex features.

As a preliminary experiment, in order to apply this pix2pix model to drawings of parks, I created a small dataset (of 3 image pairs). The first two image pairs are used as training data, and the final outline was used as a test.The reason why the model can be trained is that the colors in the outline section of the pairs are consistent.

Here is the dataset, with the outline on the left and the park drawing on the right.

I trained a pix2pix model over the first two image pairs. I then tested using the outline of the third image pair. Here is the progression of the model, over this test image, as training proceeds over 200 epochs:

Here is a final comparison between the original outline versus the generated image versus the reference drawing.

Note that the model learned to associate curved shapes with water, likely learning that from the first image pair. The green patches become tree-like areas, drawing from both training image pairs. The quality of the generated image is not very high, due to limitations on the resolution that the model can output. Additionally, this park generation task is challenging because many of the features in the drawings are fine-grained: generating these precisely would require much more training data or the assistance of a human.