Allowing language models to choose the right algorithm#
In this notebook we enable a language model to choose for the right algorithm. We define multiple segmentation algorithms / tools and then give the language model the choice which one to use given different inputs.
from langchain.memory import ConversationBufferMemory
from langchain.chat_models import ChatOpenAI
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.tools import tool
from skimage.io import imread
from bia_utilities import voronoi_otsu_labeling, local_minima_seeded_watershed
import stackview
Again, we define an image storage and a list of tools.
image_storage = {}
tools = []
@tools.append
@tool
def load_image(filename:str):
"""Useful for loading an image file and storing it."""
print("loading", filename)
image = imread(filename)
image_storage[filename] = image
return "The image is now stored as " + filename
We define two segmentation algorithms, one for segmenting bright objects and one for segmenting dark objects.
@tools.append
@tool
def segment_bright_objects(image_name):
"""
Useful for segmenting bright objects (such as nuclei) in an image
that has been loaded and stored before.
"""
print("segmenting (Voronoi-Otsu-Labeling)", image_name)
image = image_storage[image_name]
label_image = voronoi_otsu_labeling(image, spot_sigma=4)
label_image_name = "segmented_" + image_name
image_storage[label_image_name] = label_image
return "The segmented image has been stored as " + label_image_name
@tools.append
@tool
def segment_dark_objects(image_name):
"""
Useful for segmenting dark objects with bright border (e.g. cells with marked membranes) in an image
that has been loaded and stored before.
"""
print("segmenting (Local-minima-seeded watershed)", image_name)
image = image_storage[image_name]
label_image = local_minima_seeded_watershed(image, spot_sigma=10)
label_image_name = "segmented_" + image_name
image_storage[label_image_name] = label_image
return "The segmented image has been stored as " + label_image_name
@tools.append
@tool
def show_image(image_name):
"""Useful for showing an image that has been loaded and stored before."""
print("showing", image_name)
image = image_storage[image_name]
display(stackview.insight(image))
return "The image " + image_name + " is shown above."
We create some memory and a large language model based on OpenAI’s chatGPT.
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
llm=ChatOpenAI(temperature=0)
C:\Users\haase\miniconda3\envs\genai-gpu\Lib\site-packages\langchain_core\_api\deprecation.py:141: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.0.10 and will be removed in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.
warn_deprecated(
Given the list of tools, the large language model and the memory, we can create an agent.
agent = initialize_agent(
tools,
llm,
agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,
memory=memory
)
C:\Users\haase\miniconda3\envs\genai-gpu\Lib\site-packages\langchain_core\_api\deprecation.py:141: LangChainDeprecationWarning: The function `initialize_agent` was deprecated in LangChain 0.1.0 and will be removed in 1.0. Use Use new agent constructor methods like create_react_agent, create_json_agent, create_structured_chat_agent, etc. instead.
warn_deprecated(
This agent can then respond to prompts.
agent.run("Please load the image data/membranes.tif")
C:\Users\haase\miniconda3\envs\genai-gpu\Lib\site-packages\langchain_core\_api\deprecation.py:141: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use invoke instead.
warn_deprecated(
loading data/membranes.tif
'The image is now stored as data/membranes.tif'
agent.run("Show the image.")
showing data/membranes.tif
|
|
'The image data/membranes.tif has been shown.'
agent.run("Please segment the image data/membranes.tif")
segmenting (Local-minima-seeded watershed) data/membranes.tif
'The segmented image has been stored as segmented_data/membranes.tif'
agent.run("Please show the segmented data/membranes.tif image.")
showing segmented_data/membranes.tif
|
|
'The image segmented_data/membranes.tif is shown above.'
The segmentation does not look like a cell-segmentation. Thus, we should ask more specifically.
agent.run("Which algorithm did you use? Why did you use this algorithm?")
'I used an image segmentation algorithm to segment the image data/membranes.tif. This algorithm was chosen because it is effective at separating objects of interest in the image based on their characteristics, such as brightness or color, which can help in further analysis and processing of the image data.'
Note: The language model cannot see the image. Its tool selection depends on information you provided and information it acquired during the chat.
Exercise#
Run the code a couple of times and study if it always uses the same algorithm. Afterwards, rename “membranes.tif” to “bright_blobs.tif” and run the code again. If the algorithm chooses a different algorithm then, why?