Skip to main content

Petals

Petals: https://github.com/bigscience-workshop/petals

Open In Colab

Pre-Requisites​

Ensure you have petals installed

pip install git+https://github.com/bigscience-workshop/petals

Usage​

Ensure you add petals/ as a prefix for all petals LLMs. This sets the custom_llm_provider to petals

from litellm import completion

response = completion(
    model="petals/petals-team/StableBeluga2", 
    messages=[{ "content": "Hello, how are you?","role": "user"}]
)

print(response)

Usage with Streaming​

response = completion(
    model="petals/petals-team/StableBeluga2", 
    messages=[{ "content": "Hello, how are you?","role": "user"}],
    stream=True
)

print(response)
for chunk in response:
  print(chunk)

Model Details​

Model NameFunction Call
petals-team/StableBelugacompletion('petals/petals-team/StableBeluga2', messages)
huggyllama/llama-65bcompletion('petals/huggyllama/llama-65b', messages)