Github Models Marketplace

If you have signed up to Github Marketplace Models, you can use their infrastructure for prompting LLMs. Depending on which LLM you are using, you need to approach them differently. They can be used using the same API key though.

def prompt_azure_openai(message:str, model="gpt-4o"):
    """A prompt helper function that sends a message to ollama and returns only the text response."""
    import os
    import openai
    if isinstance(message, str):
        message = [{"role": "user", "content": message}]
        
    # setup connection to the LLM
    client = openai.OpenAI(
        base_url = "https://models.inference.ai.azure.com/",
        api_key = os.environ.get("GH_MODELS_API_KEY") # potentially enter a different key from your environment
    )
    response = client.chat.completions.create(
        model=model,
        messages=message
    )
    
    # extract answer
    return response.choices[0].message.content

prompt_azure_openai("What is the capital of france?")

'The capital of France is Paris.'

Meta LLama 3.1

def prompt_azure(message:str, model="meta-llama-3.1-70b-instruct"):
    import os
    from azure.ai.inference import ChatCompletionsClient
    from azure.ai.inference.models import SystemMessage, UserMessage
    from azure.core.credentials import AzureKeyCredential
    
    client = ChatCompletionsClient(
        endpoint="https://models.inference.ai.azure.com",
        credential=AzureKeyCredential(os.environ.get("GH_MODELS_API_KEY")),
    )
    
    response = client.complete(
        messages=[
            SystemMessage(content="You are a helpful assistant."),
            UserMessage(content="What is the capital of France?"),
        ],
        temperature=1.0,
        top_p=1.0,
        max_tokens=1000,
        model=model
    )
    
    print(response.choices[0].message.content)

prompt_azure("What is the capital of france?")

The capital of France is Paris.

Microsoft Phi 3.5

The same function also works with other models

prompt_azure("What is the capital of france?", model="Phi-3.5-mini-instruct")

 The capital of France is Paris. It is not only the largest city in France but also a global center for art, fashion, gastronomy, and culture. Paris is known for its landmarks such as the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum, which is the world's largest art museum. The city is also one of the major economic and political hubs in Europe.

Exercise

Check out different models on the Github Models Marketplace and invoke another one, e.g. a Mistral model.