from portkey_ai import Portkey
import os
lb_config = {
"strategy": { "mode": "loadbalance" },
"targets": [{
"provider": 'openai',
"api_key": os.environ["OPENAI_API_KEY"],
"weight": 0.1
},{
"provider": 'groq',
"api_key": os.environ["GROQ_API_KEY"],
"weight": 0.9,
"override_params": {
"model": 'llama3-70b-8192'
},
}],
}
client = Portkey(config=lb_config)
response = client.chat.completions.create(
messages=[{"role": "user", "content": "What's the meaning of life?"}],
model="gpt-4o-mini"
)
print(response.choices[0].message.content)
Implementing conditional routing:
from portkey_ai import Portkey
import os
openai_api_key = os.environ["OPENAI_API_KEY"]
groq_api_key = os.environ["GROQ_API_KEY"]
pk_config = {
"strategy": {
"mode": "conditional",
"conditions": [
{
"query": {"metadata.user_plan": {"$eq": "pro"}},
"then": "openai"
},
{
"query": {"metadata.user_plan": {"$eq": "basic"}},
"then": "groq"
}
],
"default": "groq"
},
"targets": [
{
"name": "openai",
"provider": "openai",
"api_key": openai_api_key
},
{
"name": "groq",
"provider": "groq",
"api_key": groq_api_key,
"override_params": {
"model": "llama3-70b-8192"
}
}
]
}
metadata = {
"user_plan": "pro"
}
client = Portkey(config=pk_config, metadata=metadata)
response = client.chat.completions.create(
messages=[{"role": "user", "content": "What's the meaning of life?"}]
)
print(response.choices[0].message.content)
The above example uses the metadata value user_plan to determine which model should be used for the query. This is useful for SaaS providers who offer AI through a freemium plan.
Harnessing Portkey AI Gateway for LLM integration
Portkey represents a significant innovation in LLM integration. It addresses critical challenges in managing multiple providers and optimizing performance. By providing an open-source framework that enables seamless interaction with various LLM providers, the project fills a significant gap in current AI development workflows.
The project thrives on community collaboration, welcoming contributions from developers worldwide. With an active GitHub community and open issues, Portkey encourages developers to participate in expanding its capabilities. The project’s transparent development approach and open-source licensing make it accessible for both individual developers and enterprise teams.
