ScaDS.AI LLM endpoint

In this notebook we will use the ScaDS.AI LLM infrastructure infrastructure at the Center for Information Services and High Performance Computing (ZIH) of TU Dresden. To use it, you must be connected via TU Dresden VPN. Also this method uses the OpenAI API and we just change the base_url.

Read more

• ScaDS.AI LLM examples

import os
import openai
openai.__version__

'1.42.0'

def prompt_scadsai_llm(message:str, model="meta-llama/Meta-Llama-3.1-70B-Instruct"):
    """A prompt helper function that sends a message to ScaDS.AI LLM server at 
    ZIH TU Dresden and returns only the text response.
    """
    import os
    
    # convert message in the right format if necessary
    if isinstance(message, str):
        message = [{"role": "user", "content": message}]
    
    # setup connection to the LLM
    client = openai.OpenAI(base_url="https://llm.scads.ai/v1",
                           api_key=os.environ.get('SCADSAI_API_KEY')
    )
    response = client.chat.completions.create(
        model=model,
        messages=message
    )
    
    # extract answer
    return response.choices[0].message.content

prompt_scadsai_llm("Hi!")

"It's nice to meet you. Is there something I can help you with or would you like to chat?"

Exercise

List the models available in the endpoint and try them out by specifying them when calling prompt_scadsai_llm().

client = openai.OpenAI(base_url="https://llm.scads.ai/v1",
                       api_key=os.environ.get('SCADSAI_API_KEY'))

print("\n".join([model.id for model in client.models.list().data]))

mistral-7b-q4
llama-fallback-mistral
meta-llama/Meta-Llama-3.1-70B-Instruct
CohereForAI/c4ai-command-r-plus