Ollama

Ollama is a tool that downloads models to our computer and allows us to run them locally. Before executing the following code, you need to run ollama run llama3:8b once, if you didn’t do this during setup. Also, depending on how you installed ollama, you may have to execute it in a terminal window using this command, before executing this notebook:

ollam serve

As you will see, we access the local models offered via ollama using the OpenAI API as shown before. We just exchange the base_url and we do not need to provide an API-Key.

import openai
openai.__version__

'1.35.14'

def prompt_ollama(message:str, model="llama3:8b"):
    """A prompt helper function that sends a message to ollama and returns only the text response."""
    if isinstance(message, str):
        message = [{"role": "user", "content": message}]
        
    # setup connection to the LLM
    client = openai.OpenAI()
    client.base_url = "http://localhost:11434/v1"
    client.api_key = "none"
    response = client.chat.completions.create(
        model=model,
        messages=message
    )
    
    # extract answer
    return response.choices[0].message.content

prompt_ollama("Hi!")

"Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?"

Exercise

Explore the list of models offered by ollama. If you have time and free harddrive space, download some models and try them out. Use ollama run ... to download and try a model in the terminal. Use ollama list to list all models and ollama rm ... to delete a model.