Inpainting using Stable Diffusion#
Inpainting is the task of replacing a selected part of an image by generated pixels that make the selection disappear. When working with scientific images, this could be seen as scientific misconduct. Be careful when applying this technique to your microscopy images.
In this notebook we demonstrate the technique using Stable Diffusion the example shown below is modified from here.
import PIL
import requests
import torch
from io import BytesIO
from skimage.io import imread
import stackview
import numpy as np
from diffusers import StableDiffusionInpaintPipeline
We set up a Stable Diffusion pipeline an load it to our graphics processing unit.
pipe = StableDiffusionInpaintPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-inpainting",
# "runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16
)
pipe = pipe.to("cuda")
We load our example image as numpy array.
np_init_image = imread("data/real_cat.png")
As huggingface hub models expect images a pillow images, we need to convert it first.
init_image = PIL.Image.fromarray(np_init_image)
init_image
mask_np = imread("data/real_cat_mic_mask.tif")
mask_image = np.asarray([mask_np, mask_np, mask_np]).swapaxes(0, 2).swapaxes(0, 1)
mask_image = PIL.Image.fromarray((mask_image * 255).astype(np.uint8))
mask_image
prompt = "A black white cat fur"
image = pipe(prompt=prompt,
image=init_image,
mask_image=mask_image,
num_inference_steps=50,
width=512,
height=512,
num_images_per_prompt=1,
).images[0]
C:\Users\haase\miniconda3\envs\genai2\Lib\site-packages\diffusers\models\attention_processor.py:1584: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
hidden_states = F.scaled_dot_product_attention(
np_image = np.array(image)
stackview.insight(np_image)
|
|
Exercise#
Use stackview.annotate to create a binary image selecting the cat and replace it with another microscope.