Qwen2-VL#

In this notebook we will use the vision language model Qwen/Qwen2-VL-7B-Instruct to inspect an image.

We will use ScaDS.AI LLM infrastructure infrastructure at the Center for Information Services and High Performance Computing (ZIH) of TU Dresden. To use it, you must be connected via TU Dresden VPN.

import openai
from skimage.io import imread
import stackview
from image_utilities import numpy_to_bytestream
import base64
from stackview._image_widget import _img_to_rgb
import os
from IPython.display import display, Markdown

Example image#

First we load a microscopy image. With such an example we can test if the model was trained on scientific microscopy data.

hela_cells = imread("data/hela-cells-8bit.tif")
stackview.insight(hela_cells)
shape(512, 672, 3)
dtypeuint8
size1008.0 kB
min0
max255

We will now send the image to the LLM server and ask it the some questions.

from image_utilities import prompt_scads_llm

res = prompt_scads_llm("what's in this image?", hela_cells)

display(Markdown(res))

This image appears to be a microscopic view of cells, likely taken using fluorescence microscopy. The cells are stained with different colors to highlight specific structures or proteins.

  • The blue areas represent the nuclei of the cells, stained with a dye like DAPI or Hoechst.

  • The green areas likely represent the actin filaments, which are a part of the cell’s cytoskeleton.

  • The red areas could represent another protein or structure within the cells, possibly stained with a different dye.

This type of imaging is commonly used in biological research to study cell structure and function.

Exercise#

Ask the model to specifically describe what is in a selected colour channel. Repeat this exercise with a natural image such as “real_cat.png”.