Fake zebra.

Last post had some funny Elons. Here's how I made them. Well, here's how a machine learning model made them with me serving as quality control.

Source.

A few months back, Zac linked this blog post about the new AI hotness, Stable Diffusion. Having recently upped my CUDA game, I decided to give this model a try.

This part is boring but contains highly-specific troubleshooting information, for casual consumption skip down to "a photograph of an astronaut riding a horse".

Video cards don't hot swap

Since I had Dall-e draw a 'neural graphics card', I asked Stable Diffusion to do the same. It probably needed more guidance.

I can't remember the last time I upgraded a video card without refreshing the rest of the system, but here I was yanking a 1060 and replacing it with a 3080. Windows was somewhat straightforward: remove hardware, uninstall nvidia tools, put in card, install. I didn't touch cuda, cudann, etc., so there may be some follow-up if and when I do ML in Windows.

For Ubuntu I just ripped and replaced the card, then tried to apt install the updated drivers. Alas, the installer halted on discovering that the 'nvidia-drm' kernel module was still running. The fix I found:

The lands between RTX and Python

There's a lot of software between nvidia drivers and a PyTorch script. On my box, all of it remained from the 1060 era, so I wasn't sure how they'd do with their underlying software (the graphics card driver) changing.

Updating PyTorch was previously kind of a pain, so I tried to avoid it, alas:


I was on cuda_11.5.r11.5 and my existing Torch install was for 10.2:


Some github bug thread mentioned that 3080s were not supported on that verison of cuda. The nvidia Getting Started Locally link told me to run:


That command only successfully installed torchaudio.

So I went back through the previous steps of checking the available wheels and pointing pip to the right one.


Basically what Getting Started Locally said, but with the specific Torch versions:


Voila:


And just to be sure, I checked that torch and cuda both were working:

Pip

There wasn't a requirements.txt that I could find, so I also needed to get:

Models and checkpoints

The readme said to run the model(?) downloader script, it took several hours. Next I needed model checkpoints, available after creating an account on Hugging Face and symlinking one of the files in the specified directory.

Quantize.py

Finally, I ran the Hello World script:


No astronaut, only:


From this, something in the Stable Diffusion stack uses a no-longer-supported library, but you can download the right one and update the various Python packages manually.


I haven't done a ton of Python at home or work until recently. IDLE and emacs aren't super great for it, so in searching for a proper IDE I came upon Spyder. It seems to hit the sweet spot of not being too invasive (Eclipse) while being helpful (autocomplete and robust syntax assistance). It's "a Python IDE for scientists" so -1 for being pretentious.

Sure, let's do more troubleshooting

Alas, the Spyder version in apt is apparently really old and chokes on a recent installation of Qt/QtPy. Since there's no standalone Linux installer, the solutions are:

• Use Conda. No thank you.
• Roll back Qt and/or use Docker. Nahhhhh.

I elected to go with option three. The exception pointed me to Spyder's install.py where the incompatible lines of code all seemed to be about scaling raster images. I don't expect to heavily use raster images in a Python IDE and probably have the screen real estate to accommodate full size ones, so I commented out these lines. I expected another wave of Qt exceptions when I fired 'er back up, but everything worked.

I tried the txt2img.py script that's basically Stable Diffusion's Hello World:


Was tripling my video memory not enough? The messages is a bit unclear, as was repeated on the torch forums:

abhinavdhere

I got "RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.76 GiB total capacity; 9.76 GiB already allocated; 21.12 MiB free; 9.88 GiB reserved in total by PyTorch)" I know that my GPU has a total memory of at least 10.76 GB, yet PyTorch is only reserving 9.88 GB. In another instance, I got a CUDA out of memory error in middle of an epoch and it said "5.50 GiB reserved in total by PyTorch" Why is it not making use of the available memory? How do I change this amount of memory reserved?

Running nvidia-smi told me that steady state gpu memory used was 340MiB / 12288MiB, so running this without a GUI wasn't going to make much difference.

Umais

I have the exact same question. I can not seem to locate any documentation on how pytorch reserves memory and the general information regarding memory allocation seems pretty scant. I'm also experiencing Cuda Out of Memory errors when only half my GPU memory is being utilised ("reserved").

Others seemed to experience the same thing (in 2020) and I'm not the only one who sees ambiguity in that memory math.

ptrblck

Yes, [multiple processes having their own copy of the model] makes sense and indeed an unexpectedly large batch in one process might create this OOM issue. You could try to reduce the number of workers or somehow guard against these large batches and maybe process them sequentially?

Ptrblck was addressing an issue with parallel processes sharing the GPU, but this was conceptually the same problem as mine. When I dialed down the txt2img batch size or output dimensions, I stopped hitting the OOM exception. Deep learning model sizes explode when you increase image height and width; all of those highly-connected nodes must connect to exponentially more peers. In my limited experience, batch size (number of images you feed in at a time) only scales memory usage by a multiple of the input/output dimensions. Maybe Stable Diffusion handles batching differently. According to this thread there may be some platform-layer memory management issues that could be resolved with additional updating.

An aside

Source.

It's perhaps worth noting that this exposes a major shortcoming of ML platforms, they don't leverage the lower half of the memory pyramid.

Source.

Hardware-accelerated machine learning (at the consumer level) runs everything out of GPU memory, this sets a fairly rigid boundary on problem size. If these platforms could wisely page out to computer main memory and virtual memory, that 12gb or even 48gb ceiling wouldn't be so hard.

It would be a little less challenging to support parallelism - connect identical 4090s and treat them a two processing cores with a single blob of mutually-accessible memory (mildly impacted by bus throughput). From the desktop machine learning enthusiast perspective, it means I could spend a paltry $1000 to double my model capacity. Alas, this doesn't seem to be supported.

And so the only way to throw money at the problem is to move from a $1000 gamer graphics card to a $10,000 48GB data center card. Or, of course, buy time at one such data center.

Models like Dall-e and Stable Diffusion have text interpretation components, graphics inference components, and generative components. I haven't peeled back the layers to know how separable these models are, but another theoretical solution is to partition the models under the hood. This requires that each step of the process be separable and would incur a performance hit of unloading/loading between stages of the pipeline.


After all that I was able to run the Hello World script from the docs:


My astronaut horseman was... less good than Dall-e? Well, deep learning is very sensitive to parameters so let's count successful execution as a win and try the other application of Stable Diffusion, img2img.


The input is an mspaint-tier drawing of a mountain and river. Combining that input image with a text prompt should produce something like this:



My results weren't quite as refined, but certainly in the ballpark:

txt2img

I ran the equestrinaut example with the default batch size (6) and had to dial down the output dimensions to not OOM my GPU. Spamming nvidia-smi, I saw a peak usage of about 9gb:


I tried 512x448 and 512x324, both exceeded my 12gb video memory. I should note that these runs were done before the batch size fix discussed above (for topicality), these dimensions would probably work with a small batch.


I switched up the prompt and found that 448x256 worked:



Rigid airships looked cool with Dall-e, so let's try:

img2img

For img2img I tried to channel both Andy's Seattle and the serene landscape of the github example. A zeppelin floats above Florence. And let's do an oil painting rather than a photo or cgi render.


My 3080 easily accommodated a 448x336 image, probably because I set the samples/iterations to one. The first pass gave me a simple-but-coherent take on my input image as well as something, well, abstract.


Using the successful one for a second pass, Stable Diffusion churned out a more colorful redraw as well as something significantly different. Alas, my zeppelin had turned into clouds.


A third pass on the more detailed image brought out some of the details:


Not quite Andy's postapocalypse/preinvasion Seattle, but it's a pretty good Hello World. The lack of detail could be the oil painting directive, an ambiguous text prompt, or something else.

From the Stable Diffusion sub that also has this instructional.

Style Transfer was a neat image-to-image application but compute-intensive and noisy. Dall-e Mini introduced (me to) a generative model using text inputs. Stable Diffusion can do all of these. For serious digital artists, it can be part of a traditional Photoshop/Illustrator-heavy pipeline, e.g. the image above. For the more casual user, it can 'imagine' things from a text prompt or stylize a photo/drawing:

From Zac.

Related - internal

Related - external