Github Models Marketplace

If you have signed up to Github Marketplace Models, you can use their infrastructure for prompting LLMs. Depending on which LLM you are using, you need to approach them differently. They can be used using the same API key though.

import openai

def prompt_github_models(prompt:str, model="gpt-4o"):
    """A prompt helper function that sends a prompt to the 
    Github Model LLM-server and returns only the text response."""
    import os
        
    # setup connection to the LLM-server
    client = openai.OpenAI(
        base_url = "https://models.inference.ai.azure.com/",
        api_key = os.environ.get("GH_MODELS_API_KEY")
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # extract answer
    return response.choices[0].message.content

prompt_github_models("What is the capital of france?")

'The capital of France is **Paris**.'

Other models

The same function also works with other models

prompt_github_models("What is the capital of france?", model="Phi-3.5-mini-instruct")

" The capital of France is Paris. It is not only the political and cultural center of the country but also well known for its fashion, art, and history. Paris is home to numerous historical landmarks including the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum, which is the world's largest art museum and a historic monument. The city's influence extends much beyond France's borders, playing a key role in global affairs through diplomacy, tourism, and as a hub for international business."

Exercise

Check out different models on the Github Models Marketplace and invoke another one, e.g. a Mistral model.