Ollama

Ollama is a tool that downloads models to our computer and allows us to run them locally. Before executing the following code, you need to run ollama run gemma3:1b once, if you didn’t do this during setup. Also, depending on how you installed ollama, you may have to execute it in a terminal window using this command, before executing this notebook:

ollama serve

If you can’t recall which models you already downloaded, you can list them:

ollama list

As you will see, we access the local models offered via ollama using the OpenAI API as shown before. We just exchange the base_url and we do not need to provide an API-Key.

import openai
openai.__version__

'1.74.0'

def prompt_ollama(prompt:str, model="gemma3:1b"):
    """A prompt helper function that sends a prompt to 
    ollama and returns only the text response."""
    # setup connection to the LLM server
    client = openai.OpenAI(
        base_url = "http://localhost:11434/v1",
        api_key = "none" # not required by ollama
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # extract answer
    return response.choices[0].message.content

prompt_ollama("Hi!")

"Hi there! How's your day going so far? 😊 \n\nIs there anything you’d like to talk about or any I can help you with?"

Exercise

Explore the list of models offered by ollama. If you have time and free harddrive space, download some models and try them out. Use ollama run ... to download and try a model in the terminal. Use ollama list to list all models and ollama rm ... to delete a model.

client = openai.OpenAI(base_url = "http://localhost:11434/v1")

print("\n".join(sorted([model.id for model in client.models.list().data])))

bakllava:latest
deepseek-coder-v2:latest
deepseek-r1:1.5b
deepseek-r1:latest
gemma3:12b
gemma3:1b
gemma3:4b
granite3.2-vision:latest
llama3.1:latest
llama3.2-vision:11b
llama3.2-vision:latest
llama3.2:1b
llama3.2:latest
llava:13b
llava:latest
minicpm-v:latest
mistral:latest
moondream:latest
moondream:v2
mxbai-embed-large:latest
nomic-embed-text:latest
olfh/teuken-7b-instruct-commercial-v0.4:latest
phi3.5:latest
qwen2.5vl:7b
qwen3:4b
qwen3:8b