Inpainting using Stable Diffusion#

Inpainting is the task of replacing a selected part of an image by generated pixels that make the selection disappear. When working with scientific images, this could be seen as scientific misconduct. Be careful when applying this technique to your microscopy images.

In this notebook we demonstrate the technique using Stable Diffusion the example shown below is modified from here.

import PIL
import requests
import torch
from io import BytesIO
from skimage.io import imread
import stackview
import numpy as np
from diffusers import StableDiffusionInpaintPipeline

We set up a Stable Diffusion pipeline an load it to our graphics processing unit.

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-inpainting",
    # "runwayml/stable-diffusion-inpainting", 
    torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

We load our example image as numpy array.

np_init_image = imread("data/real_cat.png")

As huggingface hub models expect images a pillow images, we need to convert it first.

init_image = PIL.Image.fromarray(np_init_image)
init_image
../_images/50cc5c20349b3338b77309a70d3d8511375c5f83053328f5e459e83b89edc6ad.png
mask_np = imread("data/real_cat_mic_mask.tif")
mask_image = np.asarray([mask_np, mask_np, mask_np]).swapaxes(0, 2).swapaxes(0, 1)
mask_image = PIL.Image.fromarray((mask_image * 255).astype(np.uint8))
mask_image
../_images/b1df127dc70972b73d4d936401f31fb3458c342ec2074864434404b913e7ab94.png
prompt = "A black white cat fur"
image = pipe(prompt=prompt, 
             image=init_image, 
             mask_image=mask_image, 
             num_inference_steps=50, 
             width=512, 
             height=512,
             num_images_per_prompt=1,
            ).images[0]
C:\Users\haase\miniconda3\envs\genai2\Lib\site-packages\diffusers\models\attention_processor.py:1584: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  hidden_states = F.scaled_dot_product_attention(
np_image = np.array(image)
stackview.insight(np_image)
shape(512, 512, 3)
dtypeuint8
size768.0 kB
min0
max252

Exercise#

Use stackview.annotate to create a binary image selecting the cat and replace it with another microscope.