{
"cells": [
{
"cell_type": "markdown",
"id": "1c7a3958-a52b-4446-8506-94f0c98863f6",
"metadata": {},
"source": [
"# Benchmarking spot counting using Vision Language Models\n",
"\n",
"There are some vision language models such as [moondream](https://huggingface.co/vikhyatk/moondream2) which are capable of counting objects in images by pointing at them. The model generates a list of point coordinates corresponding to locations in the image where prompted objects are. While the model is trained for natural images, it can be used to count bright blobs in dark images, too. This indicates that such models might be useful for microscopy image analysis. In this notebook we will benchmark how well it performs, first on an actual microscopy image of nuclei and furthermore on synthetic images the look similar."
]
},
{
"cell_type": "markdown",
"id": "38b84c3f-7a1d-4324-af3b-bb3be13f119f",
"metadata": {},
"source": [
"Installation (Windows):\n",
"* Download vips-dev-w64-all-8.16.1.zip from [here](https://github.com/libvips/build-win64-mxe/releases/tag/v8.16.1), unzip it, and add its subfolder `bin` to the PATH environment variable.\n",
"* `pip install einops pyvips`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ab4408ba-b33f-46d0-89cb-59133378f1ae",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"from PIL import Image\n",
"from image_utilities import numpy_to_bytestream, extract_json, generate_spots\n",
"from tqdm import tqdm\n",
"import stackview\n",
"\n",
"model = AutoModelForCausalLM.from_pretrained(\n",
" \"vikhyatk/moondream2\",\n",
" revision=\"2025-01-09\",\n",
" trust_remote_code=True,\n",
" # Comment to run on CPU. To use the GPU, you need about 5 GB of GPU Memory.\n",
" device_map={\"\": \"cuda\"}\n",
")"
]
},
{
"cell_type": "markdown",
"id": "18c7adae-d56b-452b-bba9-aed363bc831d",
"metadata": {},
"source": [
"## Human mitosis\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "22263f34-6be6-4a8f-9bbb-a5e138dacee4",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"