Supported LLM providers
Supported LLM Providers
The CAST AI LLM Proxy integrates with various Large Language Model (LLM) providers, enabling efficient request routing based on complexity and cost. This document outlines the supported providers and their available models for routing and proxying.
Provider Overview
Below is a comprehensive table of supported providers, their API identifiers, any additional required fields, and the models they support:
| Provider | API Identifier | Additional Required Fields | Routable and Proxyable Models |
|---|---|---|---|
| OpenAI | OPENAI | N/A | gpt-4-0125-preview, gpt-4-0613, gpt-4-1106-preview, gpt-4o-2024-05-13, gpt-4-turbo-2024-04-09 |
| Azure | AZURE | url and apiVersion | command-r-plus, gpt-4-0613, gpt-4-1106-preview, gpt-4-turbo-2024-04-09, mistral-large-2402 |
| AzureAI | AZUREAI | url | mistral-small |
| Anthropic | ANTHROPIC | N/A | claude-2.1, claude-3-5-sonnet-20240620, claude-3-haiku-20240307, claude-3-opus-20240229, claude-3-sonnet-20240229 |
| Gemini | GEMINI | N/A | gemini-1.0-pro, gemini-1.5-flash, gemini-1.5-pro |
| VertexAI (Gemini) | VERTEXAIGEMINI | url | gemini-1.0-pro, gemini-1.5-flash, gemini-1.5-pro |
| VertexAI (Anthropic) | VERTEXAIANTHROPIC | url | claude-3-5-sonnet-20240620, claude-3-haiku-20240307, claude-3-opus-20240229, claude-3-sonnet-20240229 |
| Groq | GROQ | N/A | gemma-7b-it, llama3-70b-8192, llama3-8b-8192, mixtral-8x7b-32768 |
| Mistral | MISTRAL | N/A | codestral-2405, mistral-large-2402, mistral-medium-2312, mistral-small, mixtral-8x7b-32768, open-mistral-7b, open-mixtral-8x22b |
| Ollama | OLLAMA | url | llama3-70b-8192, llama3-8b-8192, llama2:7b, llama2:70b |
| Cohere | COHERE | N/A | command-light, command-r, command-r-plus |
| Databricks | DATABRICKS | url | databricks-dbrx-instruct |
| Codestral | CODESTRAL | N/A | codestral-2405 |
| OpenRouter | OPENROUTER | N/A | claude-3-haiku-20240307, command-r-plus, gpt-4o-2024-05-13 |
| Anyscale | ANYSCALE | N/A | gemma-7b-it |
This table provides a quick reference for the supported providers, their API identifiers, any additional fields required for configuration, and the models available for routing and proxying. For detailed information on configuring and using these providers with the CAST AI LLM Proxy, please refer to the getting started guide.
Updated 28 days ago