Supported LLM providers
Supported LLM providers
The AI Enabler Proxy integrates with various Large Language Model (LLM) providers, enabling efficient request routing based on complexity and cost. This document outlines the supported providers and their available models for routing and proxying.
Provider overview
Below is a comprehensive table of supported providers, their API identifiers, any additional required fields, and the models they support for routing and proxying:
| Provider | API Identifier | Additional Required Fields | Routable Models | Proxyable Models |
|---|---|---|---|---|
| OpenAI | OPENAI | N/A | gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18, gpt-4o-2024-08-06 | ft:gpt-4-0613, ft:gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-4-0125-preview, gpt-3.5-turbo, ft:gpt-4o-2024-08-06, gpt-4-0314, gpt-4-32k-0613, gpt-4-turbo-preview, gpt-4-32k, gpt-4o-2024-05-13, gpt-4o-mini, gpt-3.5-turbo-1106, gpt-4-0613, gpt-4-turbo, gpt-3.5-turbo-0125, gpt-4, gpt-4o, gpt-4o-mini-2024-07-18, gpt-4-32k-0314, gpt-4-turbo-2024-04-09, gpt-4-1106-preview, gpt-4o-2024-08-06, ft:gpt-4o-mini-2024-07-18, ft:gpt-3.5-turbo-0125, ft:gpt-3.5-turbo-0613, ft:gpt-3.5-turbo-1106 |
| Azure | AZURE | url and apiVersion | gpt-4o-2024-08-06 | gpt-4o, gpt-3.5-turbo, gpt-3.5-turbo-0125, gpt-4-1106-preview, gpt-4-0613, gpt-4-0125-preview, gpt-4-32k-0613, gpt-4-turbo-2024-04-09, gpt-3.5-turbo-16k, gpt-4-32k, command-r-plus, gpt-4-turbo, gpt-3.5-turbo-1106, gpt-4o-mini, gpt-4o-2024-08-06, gpt-4, mistral-large-2402 |
| AzureAI | AZUREAI | url | N/A | mistral-small, mistral-large |
| Anthropic | ANTHROPIC | N/A | claude-3-5-sonnet-20240620 | claude-3-sonnet-20240229, claude-instant-1, claude-2, claude-3-5-sonnet-20240620, claude-3-opus-20240229, claude-instant-1.2, claude-3-haiku-20240307, claude-2.1 |
| Gemini | GEMINI | N/A | gemini-1.5-pro | gemini-1.5-pro-001, gemini-pro, gemini-gemma-2-27b-it, gemini-1.5-pro, gemini-gemma-2-9b-it, gemini-1.5-flash |
| VertexAI (Gemini) | VERTEXAIGEMINI | url | gemini-1.5-pro | gemini-1.0-ultra-001, gemini-1.0-pro-001, gemini-1.5-pro, gemini-1.5-flash, gemini-pro, gemini-1.5-pro-001, gemini-1.0-pro |
| VertexAI (Anthropic) | VERTEXAIANTHROPIC | url | claude-3-5-sonnet-20240620 | claude-3-haiku-20240307, claude-3-5-sonnet-20240620, claude-3-sonnet-20240229, claude-3-opus-20240229 |
| Groq | GROQ | N/A | llama3:8b, llama3:70b | llama3.1:8b, llama3:8b, llama3.1:70b, llama3:70b, llama2:70b, mixtral-8x7b-32768, gemma-7b-it |
| Mistral | MISTRAL | N/A | N/A | mistral-large-2407, mistral-large-2402, mixtral-8x7b-32768, mistral-medium-2312, open-mixtral-8x22b, codestral-2405, mistral-small, open-mistral-7b, open-mistral-nemo-2407, mistral-tiny |
| Ollama | OLLAMA | url | llama3:70b, llama3:8b, llama2:7b | llama3:70b, llama3:8b, llama2:7b, llama2:13b, llama2:70b |
| Cohere | COHERE | N/A | N/A | command-r-08-2024, command-r-plus-08-2024, command-r-plus, command-r, command-light |
| Databricks | DATABRICKS | url | llama3:70b | databricks-mpt-30b-instruct, databricks-mpt-7b-instruct, databricks-dbrx-instruct, llama3:70b, llama2:70b |
| Codestral | CODESTRAL | N/A | N/A | codestral-2405 |
This table provides a quick reference for the supported providers, their API identifiers, any additional fields required for configuration, and the models available for routing and proxying. For detailed information on configuring and using these providers with the CAST AI LLM Proxy, please refer to the getting started guide.
Updated about 1 month ago