Qwen2-VL via KISSKI#
In this notebook we will use the vision language model Qwen/Qwen2-VL-7B-Instruct to inspect an image.
In this notebook we will use the KISSKI LLM service infrastructure. KISSKI is the German AI Service Center for Sensible and Critical Infrastructure. Before you can access it, you need to create an API key by filling out this form; make sure to check the box “API access to our chat service”.
import openai
from skimage.io import imread
import stackview
from image_utilities import numpy_to_bytestream
import base64
from stackview._image_widget import _img_to_rgb
import os
from IPython.display import display, Markdown
Example image#
First we load a microscopy image. With such an example we can test if the model was trained on scientific microscopy data.
hela_cells = imread("data/hela-cells-8bit.tif")
stackview.insight(hela_cells)
|
|
We will now send the image to the LLM server and ask it the some questions.
from image_utilities import prompt_kisski
res = prompt_kisski("what's in this image?", hela_cells)
display(Markdown(res))
This image appears to be a fluorescence microscopy image of cells. The blue areas are likely the cell nuclei stained with a blue fluorescent dye such as DAPI (4’,6-diamidino-2-phenylindole). The green structures surrounding the nuclei might represent cytoskeletal elements or other cellular components labeled with a green fluorescent dye, such as FITC (fluorescein isothiocyanate). The red dots could indicate a specific protein or cellular component visualized with a red fluorescent marker. The combination of these colors helps researchers identify and study different components and processes within the cells.
Exercise#
Ask the model to specifically describe what is in a selected colour channel. Repeat this exercise with a natural image such as “real_cat.png”.