GPT 5 for bounding-box segmentation#

In this notebook we will use the vision language model GPT 5 to test if it supports drawing bounding boxes around objects.

import openai
from skimage.io import imread
import stackview
from image_utilities import extract_json
from prompt_utilities import prompt_openai

import json
import os
import pandas as pd
from skimage.io import imsave

Bounding box segmentation#

We first load an example dataset, a crop of the human_mitosis image from scikit-image.

import stackview
from skimage import data
import numpy as np

# Load the human mitosis dataset
image = data.human_mitosis()[:100, :100]

stackview.insight(image)
shape(100, 100)
dtypeuint8
size9.8 kB
min7
max88
reply = prompt_openai("""
Give me a json object of bounding boxes around ALL bright blobs in this image. Assume the image width and height are 1. 
The format should be like this: 

```json
[
    {'x':float,'y':float, 'width': float, 'height': float},
    {'x':float,'y':float, 'width': float, 'height': float},
    ...
]
```

If you think you can't do this accuratly, please try anyway.
""", image, model="gpt-5-2025-08-07")
print(reply)
bb = json.loads(extract_json(reply))
bb

new_image = stackview.add_bounding_boxes(image, bb)

images = [new_image]
[
  {"x": 0.02, "y": 0.01, "width": 0.10, "height": 0.10},
  {"x": 0.15, "y": 0.02, "width": 0.10, "height": 0.10},
  {"x": 0.28, "y": 0.06, "width": 0.11, "height": 0.11},
  {"x": 0.42, "y": 0.09, "width": 0.11, "height": 0.11},
  {"x": 0.57, "y": 0.08, "width": 0.12, "height": 0.12},
  {"x": 0.18, "y": 0.22, "width": 0.11, "height": 0.11},
  {"x": 0.33, "y": 0.26, "width": 0.11, "height": 0.11},
  {"x": 0.47, "y": 0.30, "width": 0.12, "height": 0.12},
  {"x": 0.22, "y": 0.48, "width": 0.13, "height": 0.13},
  {"x": 0.38, "y": 0.57, "width": 0.14, "height": 0.14},
  {"x": 0.55, "y": 0.62, "width": 0.14, "height": 0.14},
  {"x": 0.70, "y": 0.73, "width": 0.14, "height": 0.14}
]
new_image
shape(100, 100, 3)
dtypeuint8
size29.3 kB
min0
max255