Interactive slicing throug sub-sets of embedded image data

In this notebook we will explore a stack of images using an interactive scatterplot. For demonstration purposes, we explore a stack of teaching slides and a dimensionality reduced vision embedding of them. Every slide exists in two different layouts and you can spot these pairs in the plots because we have a layout column in our DataFrame.

import pandas as pd
import stackview
import os
import numpy as np
from skimage.io import imread

df = pd.read_csv("data.csv")
df["layout"] = df["layout"] - 1

# Show first few rows of the loaded DataFrame
df.head()

	filename	embedding	layout	x	y	z
0	11_RDM_page01.png	-0.05880768597126007,0.001273319125175476,0.24...	1	0.908017	0.603683	6.068271
1	11_RDM_page02.png	0.010850191116333008,0.17468787729740143,0.291...	1	0.748177	0.632403	6.475627
2	11_RDM_page03.png	-0.0703616738319397,0.17920269072055817,0.0797...	1	0.237475	-1.157015	6.497894
3	11_RDM_page04.png	0.03190414607524872,0.25511622428894043,0.0414...	1	0.144671	-1.233542	6.464936
4	11_RDM_page05.png	-0.02796362340450287,0.2169242799282074,-0.023...	1	0.209207	-1.191703	6.571118

df["layout"].unique()

array([1, 0])

We also define a helper function that loads all images mentioned in a dataframe into one big numpy array.

def get_images(selected_rows):
    # Load images for selected pages
    images = []
    for _, row in selected_rows.iterrows():
        img_path = os.path.join('images', row['filename'])
        img = imread(img_path)
        images.append(img)

    if len(images) == 0:
        return np.zeros((2,2,2))
    else:
        return np.asarray(images)

We can use the sliceplot function of stackview to visualize the embedding next to selected slides.

stackview.sliceplot(df, get_images(df), 
                    column_x="x", 
                    column_y="y", 
                    column_selection="layout")

When running this notebook locally, this plot will be shown and it is interactive.

Exercise

Explore the plot by dragging lines around islands of datapoints with the mouse. What content are these islands about?