# Quick Start

#### Getting Started

1. Create an API Key
   1. Access [Arcee Platform](https://chat.arcee.ai/), Register or Login.
   2. Access [Wallet](https://chat.arcee.ai/api/wallet) to top up if needed.
   3. Create an API Key in the [API Keys](https://chat.arcee.ai/api/api-keys) management page.
   4. Copy your API Key for use and store in a secure location.

<table data-view="cards"><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><a href="https://chat.arcee.ai/">Arcee API Platform</a></td><td>Access Arcee Platform, Register or Login.</td></tr><tr><td><a href="https://chat.arcee.ai/api/api-keys">API Key Management</a></td><td>Create an API Key in the API Keys management page.</td></tr><tr><td><a href="https://docs.arcee.ai/~/revisions/UOfL3qIelQCFUdc2TpQu/quick-deploys">Deploy a Model</a></td><td>Learn how to deploy our models on your own infrastructure.</td></tr></tbody></table>

2. Choose Model

> Arcee Platform offers a variety of models in different sizes, and you can select the appropriate model based on your needs. For detailed model introductions, please refer to our [available models](https://docs.arcee.ai/get-started/models-overview).

3. Make your first API Call

After preparing your `API Key` and selecting a model, you can start making API calls. Here are examples using `curl`, `Python`, and `JavaScript`:

{% tabs %}
{% tab title="curl" %}

```bash
curl -X POST "https://api.arcee.ai/api/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer api-key" \
  -d '{
    "model": "trinity-large-thinking",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'
```

{% endtab %}

{% tab title="Python" %}

```python
from openai import OpenAI 

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.arcee.ai/api/v1"
)

response = client.chat.completions.create(
    model="trinity-large-thinking",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "What is 25 + 37?"},
    ],
    stream=False
)

print(response.choices[0].message.content)
print(response.choices[0].message.reasoning_content)
```

{% endtab %}

{% tab title="JavaScript" %}

```javascript
import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: 'https://api.arcee.ai/api/v1',
    apiKey: 'YOUR_API_KEY'
});

async function main() {
  const completion = await openai.chat.completions.create({
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: "What is 25 + 37?" }
    ],
    model: "trinity-large-thinking",
  });

  console.log(completion.choices[0].message.content);
  console.log(completion.choices[0].message.reasoning_content);
}

main();
```

{% endtab %}
{% endtabs %}


# Models Overview

Arcee AI offers models at various sizes to meet different deployment scenarios. Choosing the right model can help you complete tasks more efficiently, accurately, and cost effectively.&#x20;

{% hint style="info" %}
Trinity Mini and Large (Preview) are currently the only models available via API. Try Trinity-Large-Preview on OpenRouter [here](https://openrouter.ai/arcee-ai/trinity-large-preview:free/).
{% endhint %}

<table><thead><tr><th>Model</th><th>Trinity-Nano (6B)</th><th>Trinity-Mini (26B)</th><th width="187.4609375">Trinity-Large-Thinking (400B)</th></tr></thead><tbody><tr><td><strong>Strength</strong></td><td>Lightweight, ultra-low latency model.</td><td>Fast and cost-efficient model for well-defined tasks.</td><td>Robust generalist model with strong performance across reasoning, coding, math, and complex task decomposition.</td></tr><tr><td><strong>Ideal Deployment</strong></td><td>Fully local on consumer GPUs, edge servers, and mobile devices. Tuned for offline operation.</td><td>Serve customer-facing apps, agent backends, and high-throughput services in cloud or VPC.</td><td>Advanced agents, reasoning systems, and developer tools. Deployed via hosted cloud endpoints or self-hosted in multi-GPU configurations.</td></tr><tr><td><strong>Active Parameters</strong></td><td>1B per token</td><td>3B per token</td><td>13B per token</td></tr><tr><td><strong>Context Window</strong></td><td>128k tokens</td><td>128k tokens</td><td>512k tokens (hosted at 128k)</td></tr><tr><td><strong>Knowledge Cutoff</strong></td><td>2024</td><td>2024</td><td>2024</td></tr><tr><td><strong>Speed</strong></td><td><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><br>Instant</td><td><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><br>Very Fast</td><td><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><span data-gb-custom-inline data-tag="emoji" data-code="26a1">⚡</span><br>Very Fast</td></tr><tr><td><strong>API Model Name</strong></td><td>Not Hosted</td><td>Not Hosted</td><td><strong>trinity-large-thinking</strong></td></tr></tbody></table>

> Robust generalist model with strong performance across reasoning, coding, math, and **complex task decomposition**.


# Pricing

This page provides pricing information for Arcee AI's models on Arcee Platform. All prices are in USD.

#### Text Models

Prices per 1M Tokens.

| Model                  | Input      | Output     |
| ---------------------- | ---------- | ---------- |
| Trinity-Mini           | not hosted | not hosted |
| Trinity-Large-Preview  | not hosted | not hosted |
| Trinity-Large-Thinking | $0.25      | $0.80      |


# Integration List

Arcee AI models are accessible in two primary ways: the Arcee API Platform or self-hosting through one of the [Quick Deploys](/quick-deploys/hardware-prerequisites) libraries. Both options provide an OpenAI-compatible endpoint, which makes integrating into the most popular inference and agent tools, quick and simple. In this section, you'll see how you can integrate Arcee AI models into the most common AI tools.

If there's a tool not listed here which you would like to have added, please reach out to our team. If a tool is not listed and supports the OpenAI-compatible endpoint, you can follow the same pattern shown in these examples.

#### Integration List

* [Hermes Agent](https://arcee-ai-2025.webflow.io/blog/how-to-use-hermes-agent-with-trinity-large-thinking)
* [OpenRouter](/get-started/integration-list/openrouter)
* [KiloCode](/get-started/integration-list/kilo-code)
* [Cline](/get-started/integration-list/cline)
* [OpenCode](/get-started/integration-list/opencode)
* [Roo Code](/get-started/integration-list/roo-code)
* [LangSmith](/get-started/integration-list/langsmith)
* [LangGraph](/get-started/integration-list/langgraph)
* [CrewAI](/get-started/integration-list/crewai)
* [llamaIndex](/get-started/integration-list/llamaindex)
* [n8n](/get-started/integration-list/n8n)


# OpenRouter

[OpenRouter](https://openrouter.ai/) is a unified platform that provides developers with seamless access to a wide range of large language models (LLMs) through a single OpenAI-compatible API interface. It enables model interoperability by allowing users to easily route requests between open-source and proprietary models while managing authentication, quotas, and usage tracking in one place.&#x20;

This tutorial will guide you through utilizing Arcee AI's language models on OpenRouter using an OpenAI-compatible endpoint.

***

**Prerequisites**

* OpenRouter Account
  * If you don't have an account, set one up [here](https://openrouter.ai/).
* OpenRouter API Key
  * If you don't have an API Key, create one [here](https://openrouter.ai/settings/keys).
* `openai` Python SDK (install using uv or pip)

**Quickstart**

Run a completion with an Arcee model on OpenRouter

```bash
from openai import OpenAI

# Initialize OpenRouter-compatible client
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="<OPENROUTER_API_KEY>",
)

# Run a chat completion
completion = client.chat.completions.create(
    model="arcee-ai/trinity-mini",
    messages=[
        {
            "role": "user",
            "content": "What are small language models and how do they compare to LLMs?"
        }
    ]
)

# Print result
print(completion.choices[0].message.content)

```


# Roo Code

Roo Code is an AI-powered coding assistant that integrates directly into your development environment, enabling you to interact with large language models (LLMs) for tasks such as code generation, refactoring, explanation, and debugging. It provides a configurable interface for selecting different model providers while maintaining a consistent, OpenAI-compatible interaction layer.

Roo Code supports OpenRouter as an API provider, allowing you to access a wide range of language models, including **Arcee AI's Trinity models,** through a single unified endpoint. This setup simplifies configuration, enables rapid experimentation across models, and centralizes authentication, usage tracking, and cost management.

This guide walks through how to configure Roo Code to use Trinity models via OpenRouter.

***

**Prerequisites**

* OpenRouter Account
  * If you don't have an account, set one up [here](https://openrouter.ai/).
* OpenRouter API Key
  * If you don't have an API Key, create one [here](https://openrouter.ai/settings/keys).

**Quickstart**

**Configure OpenRouter in Roo Code**

1. Open **Roo Code Settings**
   1. Click the gear icon (⚙️)  in the Roo Code panel
2. Select API Provider
   1. Choose **OpenRouter** from the *API Provider* dropdown
3. Enter API Key
   * Paste your OpenRouter API key into the **OpenRouter API Key** field
4. Select Model
   * Choose an Arcee **Trinity** model (example: Trinity-Large) from the *Model* dropdown

Once saved, Roo Code will route all requests through OpenRouter using the selected Arcee model.


# Cline

Cline is an AI-powered coding assistant designed for interactive, agent-style development workflows. It enables developers to reason over codebases, plan multi-step changes, and execute actions using large language models (LLMs), all from within the editor.

Cline supports OpenRouter as an API provider, allowing you to access a wide range of language models, including **Arcee AI’s Trinity models,** through a single OpenAI-compatible endpoint. This simplifies setup, enables easy experimentation across models, and centralizes authentication, usage tracking, and billing.

This guide walks through how to configure Cline to use Trinity models via OpenRouter.

***

**Prerequisites**

* OpenRouter Account
  * If you don't have an account, set one up [here](https://openrouter.ai/).
* OpenRouter API Key
  * If you don't have an API Key, create one [here](https://openrouter.ai/settings/keys).

**Configure OpenRouter in Cline**

1. Open **Cline Settings**
2. Click the settings icon (⚙️) in the Cline panel
3. Select API Provider
   1. Choose **OpenRouter** from the *API Provider* dropdown
4. Enter API Key
   1. Paste your OpenRouter API key into the **OpenRouter API Key** field
5. Select Model
   1. Choose an Arcee **Trinity** model (example: Trinity-Large) from the *Model* dropdown


# OpenCode

OpenCode is a terminal-first AI coding assistant that enables developers to interact with large language models (LLMs) directly from the command line. It supports interactive prompting, model switching, and configuration-driven customization for advanced workflows.

OpenCode supports OpenRouter as an API provider, allowing you to access a wide range of language models, including **Arcee AI’s Trinity models**, through a single OpenAI-compatible endpoint. This simplifies setup, enables easy experimentation across models, and centralizes authentication and usage tracking.

This guide walks through how to configure OpenCode to use Trinity models via OpenRouter.

***

**Prerequisites**

* OpenRouter Account
  * If you don't have an account, set one up [here](https://openrouter.ai/).
* OpenRouter API Key
  * If you don't have an API Key, create one [here](https://openrouter.ai/settings/keys).

**Quickstart**

1. Run the `/connect` command and search for OpenRouter.
2. Enter the API key for the provider.

   ```
   ┌ API key
   │
   │
   └ enter
   ```
3. Many OpenRouter models are preloaded by default, run the `/models` command to select an Arcee Trinity model (for example, Trinity-Large) from the list.

   ```
   /models
   ```
4. Configure Models via opencode.json (optional)
   1. You can add or customize models using the OpenCode configuration file, including their provider:

```
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "openrouter": {
      "models": {
        "arcee-ai/trinity-large": {
          "options": {
            "provider": {
              "order": ["openrouter"],
              "allow_fallbacks": false
            }
          }
        }
      }
    }
  }
}
```


# Kilo Code

Kilo Code is an open-source AI coding agent that integrates directly into VS Code to support code generation, refactoring, terminal automation, and browser-based workflows using large language models (LLMs).

Kilo Code supports custom OpenAI-compatible providers, allowing you to connect directly to the **Arcee AI API** and access **Trinity models** without a third-party proxy. This provides a direct connection to Arcee's inference platform with full access to all available models.

This guide walks through how to configure Kilo Code to use Trinity models via the Arcee AI API.

***

**Prerequisites**

* Arcee AI Account
  * If you don't have an account, sign up at [chat.arcee.ai](https://chat.arcee.ai/).
* Arcee AI API Key
  * If you don't have an API Key, create one [here](https://chat.arcee.ai/api/api-keys).
* *Arcee AI models are also available via* [*OpenRouter*](https://openrouter.ai/)*. Kilo Code supports OpenRouter as a built-in provider — simply select it from the provider list and use your OpenRouter API key.*

**Quickstart**

1. Install Kilo Code
   1. Open the Extensions panel in VS Code and search for **Kilo Code**, then click **Install**.
2. Open Kilo Code Settings
   1. Click the Kilo Code icon in the left sidebar to open the panel, then click the gear icon (⚙️) in the top right corner.
3. Add Arcee AI as a Custom Provider
   1. Navigate to the **Providers** tab.
   2. Scroll down to **Custom provider** and click **+ Connect**.
   3. Enter the following settings:
      * **Provider ID:** `arcee-ai`
      * **Display Name:** `Arcee AI`
      * **Base URL:** `https://api.arcee.ai/api/v1`
      * **API Key:** Your Arcee AI API key
4. Add Models
   1. Kilo Code should automatically detect and add available models. If it does not, manually add the following:
      * **Model ID:** `trinity-large-thinking` — **Display Name:** `Trinity Large Thinking`
      * **Model ID:** `trinity-large-preview` — **Display Name:** `Trinity Large Preview`
      * **Model ID:** `trinity-mini` — **Display Name:** `Trinity Mini`
5. Set Default Model *(Optional)*
   1. Navigate to the **Models** tab.
   2. Under **Default Model**, select **Arcee AI / Trinity Large Thinking** from the dropdown.

Once configured, Kilo Code will route all requests directly through the Arcee AI API using the selected Trinity model.


# Clarifai

[Clarifai](https://www.clarifai.com/) is a full-stack AI platform for building, fine-tuning, and deploying models across text, image, video, and audio. It provides a unified environment for data management, vector search, annotation, training, and real-time inference, making it a strong option for teams that want an end-to-end system rather than an inference-only service.<br>

This tutorial will guide you through using Arcee AI’s language models on Clarifai through the Clarifai SDK.

#### Prerequisites

* **Clarifai Account**
* **Clarifai API Key**\
  If you do not have one, create it in your Clarifai console.
* **Clarifai Python SDK**

#### Quickstart

Run a completion using an Arcee model.

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.clarifai.com/v2/ext/openai/v1",
    api_key="<YOUR_API_KEY>",
)
response = client.chat.completions.create(
    model="https://clarifai.com/arcee_ai/AFM/models/trinity-mini/versions/7c698b6ebd604853ad3f75adecb59c1c",
    messages=[
        {"role": "system", "content": "Talk like a pirate."},
        {
            "role": "user",
            "content": "How do I check if a Python object is an instance of a class?",
        },
    ],
    temperature=0.7,
    stream=False, # stream=True also works, just iterator over the response
)
print(response)

```


# Together.ai

[Together.ai](https://www.together.ai/) is an open platform for running, fine-tuning, and deploying large language models (LLMs) with high performance and low latency. Beyond inference, Together.ai supports distributed fine-tuning, model evaluation, and custom deployments, making it a flexible choice for teams building production-grade AI applications.&#x20;

This tutorial will guide you through utilizing Arcee AI's language models on Together.ai using Together.ai's SDK.

***

**Prerequisites**

* Together.ai Account
* Together.ai API Key
  * If you don't have one, create one [here](https://api.together.ai/settings/api-keys).
* `together` Python SDK (install using uv or pip)

**Quickstart**

Run a completion with an Arcee model

```bash
from together import Together

# Initialize client
client = Together(api_key="<YOUR_TOGETHER_API_KEY>")

# Run a chat completion with an Arcee model
response = client.chat.completions.create(
    model="arcee-ai/trinity-mini",
    messages=[
        {
            "role": "user",
            "content": "What are small language models and how do they compare to LLMs?"
        }
    ]
)

# Print result
print(response.choices[0].message.content)

```


# LangGraph

[LangGraph](https://www.langchain.com/langgraph) is an open-source framework for building stateful, multi-agent applications powered by LLMs. It extends the LangChain ecosystem by enabling developers to define agents and workflows as dynamic computation graphs, where each node represents a function or agent and edges define conditional logic and transitions. Designed for both flexibility and scalability, LangGraph excels at use cases requiring memory, branching logic, and cyclic behavior such as agent collaboration, simulations, or tool-augmented reasoning.&#x20;

This tutorial shows how to integrate Arcee AI models into LangGraph using an OpenAI-compatible endpoint, enabling the use of Arcee’s specialized models within LangGraph workflows.

***

**Prerequisites**

* Python: `>=3.10 and <3.14`&#x20;
* Arcee AI model running locally or accessible via API

**Quickstart**

Enivornment and project setup:

```bash
# Create project folder
mkdir arceeai_langgraph && cd arceeai_langgraph

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# Create and activate virtual environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install LangGraph and OpenAI-compatible client
uv pip install --pre -U langgraph langchain-openai

```

{% hint style="danger" %}
If you run into any errors installing langgraph, follow their [Installation Guide](https://docs.langchain.com/oss/python/langgraph/install).
{% endhint %}

Create a new python file called `arceeai_langgraph.py` and with the following:

<pre class="language-python"><code class="lang-python">import os
from typing_extensions import Annotated, TypedDict

from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain_core.messages import AnyMessage, SystemMessage, HumanMessage
from langchain_openai import ChatOpenAI

class State(TypedDict):
    messages: Annotated[list[AnyMessage], add_messages]

# Configure Arcee AI Model
ARCEE_BASE = os.getenv("OPENAI_API_BASE", "http://127.0.0.1:8080/v1")
ARCEE_KEY  = os.getenv("OPENAI_API_KEY", "your-arcee-api-key")
ARCEE_MODEL = os.getenv("OPENAI_MODEL_NAME", "trinity-mini")

# Initialize Arcee AI model with OpenAI-compatible configuration
arcee_llm = ChatOpenAI(
    model=ARCEE_MODEL,
    api_key=ARCEE_KEY,
    base_url=ARCEE_BASE,
)

# Define a simple graph node that uses the Arcee AI model
def summarize(state: State) -> State:
    # Add a system instruction once, if not present
    msgs = state["messages"]
    if not any(getattr(m, "role", "") == "system" for m in msgs):
        msgs = [SystemMessage(content="You are a concise technical writer.")] + msgs
    # Ask Arcee to respond
    ai = arcee_llm.invoke(msgs)
    return {"messages": [ai]}

# Build the graph that LangGraph will execute
builder = StateGraph(State)
builder.add_node("summarize", summarize)
builder.add_edge(START, "summarize")
builder.add_edge("summarize", END)
graph = builder.compile()

if __name__ == "__main__":
    text = """Arcee AI is a foundation model provider with a focus on building the highest performing models per parameter. 
They offer a range of models from on-device and edge optimized models to large language models. Their suite of models 
provides customers with the flexibility to choose the right model for the right task. All models are released Apache 2.0 
enabling the community to use safe, built-in-the-US models in their own environment or via the Arcee AI API platform."""
    inputs: State = {"messages": [HumanMessage(content=f"Summarize the following in three bullets:\n\n{text}")]}

<strong>    # Execute the LangGraph agent
</strong>    result = graph.invoke(inputs)
    
    # Print the results
    print("\n=== RESULT ===\n")
    print(result["messages"][-1].content)

</code></pre>

{% hint style="success" %}
This works out-of-the-box if you have an Arcee AI model running locally on your laptop. If you do not, change `ARCEE_BASE` , `ARCEE_KEY` , and `ARCEE_MODEL` .

You can also setup a `.env` file to store the configurations.
{% endhint %}

Run your Arcee AI powered LangGraph Agent

```bash
python arceeai_langgraph.py
```


# LangSmith

[LangSmith](https://smith.langchain.com/) is an end-to-end platform for debugging, evaluating, and monitoring language model applications. It integrates seamlessly with LangChain and other LLM frameworks to visualize traces, inspect prompts, measure performance, and manage production-grade evaluation workflows. LangSmith helps you build more reliable AI systems by turning opaque model calls into structured, traceable data through observability and experiment tracking.

This tutorial shows how to integrate Arcee AI models into LangSmith using an OpenAI-compatible endpoint. While the focus is on tracing, the same setup applies to other LangSmith features.

***

**Prerequisites**

* **Python:** `>=3.10 and <3.14`
* **LangSmith Account and API Key:**&#x20;
  * If you do not have one, you can create one [here](https://smith.langchain.com/)
* Arcee AI model running locally or accessible via API and an OpenAI-compatible endpoint

**Quickstart**

Environment and project setup:

```bash
# Create project folder
mkdir arceeai_langsmith && cd arceeai_langsmith

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# Create and activate virtual environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install LangSmith + LangChain OpenAI client
uv pip install langsmith langchain-openai

```

{% hint style="danger" %}
If you run into any errors installing LangSmith, follow their [documentation](https://docs.langchain.com/langsmith/home).
{% endhint %}

LangSmith uses environment variables for configuration. We'll also include the Arcee AI variables here. Create a `.env` file and include the following

```
# .env
LANGSMITH_TRACING=true
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=<your-api-key>
LANGSMITH_PROJECT=arcee-langsmith-project

ARCEE_API_BASE=http://127.0.0.1:8080/v1
ARCEE_API_KEY=<your-api-key>
ARCEE_MODEL_NAME="afm-4.5b"

```

Create a new python file called `arceeai_langsmith.py` with the following:

<pre class="language-python"><code class="lang-python">import os
from typing_extensions import Annotated, TypedDict

from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain_core.messages import AnyMessage, SystemMessage, HumanMessage
from langchain_openai import ChatOpenAI

class State(TypedDict):
    messages: Annotated[list[AnyMessage], add_messages]

# Configure Arcee AI Model
ARCEE_BASE = os.getenv("ARCEE_API_BASE", "http://127.0.0.1:8080/v1")
ARCEE_KEY  = os.getenv("ARCEE_API_KEY", "your-arcee-api-key")
ARCEE_MODEL = os.getenv("ARCEE_MODEL_NAME", "trinity-mini")

# Initialize Arcee AI model with OpenAI-compatible configuration
arcee_llm = ChatOpenAI(
    model=ARCEE_MODEL,
    api_key=ARCEE_KEY,
    base_url=ARCEE_BASE,
)

# Define a simple graph node that uses the Arcee AI model
def summarize(state: State) -> State:
    # Add a system instruction once, if not present
    msgs = state["messages"]
    if not any(getattr(m, "role", "") == "system" for m in msgs):
        msgs = [SystemMessage(content="You are a concise technical writer.")] + msgs
    # Ask Arcee to respond
    ai = arcee_llm.invoke(msgs)
    return {"messages": [ai]}

# Build the graph that LangGraph will execute
builder = StateGraph(State)
builder.add_node("summarize", summarize)
builder.add_edge(START, "summarize")
builder.add_edge("summarize", END)
graph = builder.compile()

if __name__ == "__main__":
    text = """Arcee AI is a foundation model provider with a focus on building the highest performing models per parameter. 
They offer a range of models from on-device and edge optimized models to large language models. Their suite of models 
provides customers with the flexibility to choose the right model for the right task. All models are released Apache 2.0 
enabling the community to use safe, built-in-the-US models in their own environment or via the Arcee AI API platform."""
    inputs: State = {"messages": [HumanMessage(content=f"Summarize the following in three bullets:\n\n{text}")]}

<strong>    # Execute the LangGraph agent
</strong>    result = graph.invoke(inputs)
    
    # Print the results
    print("\n=== RESULT ===\n")
    print(result["messages"][-1].content)

</code></pre>

{% hint style="success" %}
This works out-of-the-box if you have an Arcee AI model running locally on your laptop. If you do not, change `ARCEE_BASE` , `ARCEE_KEY` , and `ARCEE_MODEL` .
{% endhint %}

Run your Arcee AI powered LangGraph Agent with LangSmith Tracing

```bash
python arceeai_langsmith.py
```


# CrewAI

[CrewAI](https://www.crewai.com/) is an open-source framework designed to orchestrate AI agents. CrewAI empowers agents to work together seamlessly, tackling complex tasks and multi-agent workflows. With built-in support for LiteLLM, CrewAI allows you to connect with many different Language Models (LLMs), including **Arcee models** via OpenAI-compatible endpoints.

This tutorial will walk you through setting up **Arcee AI** as your LLM provider inside a CrewAI agent pipeline.

***

**Prerequisites**

* Python: `>=3.10 and <3.14`&#x20;
* Arcee AI model running locally or accessible via API and an OpenAI-compatible endpoint

**Quickstart**

Environment and project setup:

```bash
# Create project folder
mkdir arceeai_crewai && cd arceeai_crewai

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# Create and activate virtual environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install CrewAI
uv pip install crewai

```

{% hint style="danger" %}
If you run into any errors installing CrewAI, follow their [Installation Guide](https://docs.crewai.com/en/installation).
{% endhint %}

Create a new python file called `arceeai_crewai.py` with the following:

```python
import os
from crewai import LLM, Agent, Task, Crew

# Configure Arcee AI Model
ARCEE_BASE = os.getenv("OPENAI_API_BASE", "http://127.0.0.1:8080/v1")
ARCEE_KEY  = os.getenv("OPENAI_API_KEY", "your-arcee-api-key")
ARCEE_MODEL = os.getenv("OPENAI_MODEL_NAME", "trinity-mini")

# Initialize Arcee AI model with OpenAI-compatible configuration
arcee_llm = LLM(
    model=f"openai/{ARCEE_MODEL}",
    api_base=ARCEE_BASE,
    api_key=ARCEE_KEY
)

# Define a simple agent that uses the Arcee AI model
summarizer = Agent(
    role="Summarizer",
    goal="Summarize text into three crisp bullet points.",
    backstory="A concise technical writer who removes fluff.",
    llm=arcee_llm,
    verbose=True,
)

# Define the task the agent will perform
text = """Arcee AI is a foundation model provider with a focus on building the highest performing models per parameter. 
They offer a range of models from on-device and edge optimized models to large language models. Their suite of models 
provides customers with the flexibility to choose the right model for the right task. All models are released Apache 2.0 
enabling the community to use safe, built-in-the-US models in their own environment or via the Arcee AI API platform."""
task = Task(
    description=f"Summarize the following in three bullets:\n\n{text}",
    expected_output="Exactly three bullet points, each under 20 words.",
    agent=summarizer,
)

# Orchestrate & run
crew = Crew(agents=[summarizer], tasks=[task], verbose=True)
result = crew.kickoff()

# Print the results
print("\n=== RESULT ===\n")
print(result)
```

{% hint style="success" %}
This works out-of-the-box if you have an Arcee AI model running locally on your laptop. If you do not, change `ARCEE_BASE` , `ARCEE_KEY` , and `ARCEE_MODEL` .

You can also setup a `.env` file to store the configurations.
{% endhint %}

Run your Arcee AI powered CrewAI Agent

```bash
python arceeai_crewai.py
```


# llamaIndex

LlamaIndex is an open-source data framework designed to help LLMs connect with external data sources in a structured, efficient, and context-aware way. It provides a powerful suite of tools for ingesting, indexing, querying, and retrieving data from diverse formats such as PDFs, databases, APIs, and more. With modular components like custom indices, retrievers, and agents, LlamaIndex enables developers to build scalable Retrieval-Augmented Generation (RAG) pipelines and LLM-powered applications.&#x20;

This tutorial will guide you through integrating Arcee models into llamaIndex using an OpenAI-compatible endpoint.

The first example shows how to run simple inference with llamaIndex, while the second example shows how to setup a local RAG pipeline.

***

#### Model Inference

**Prerequisites**

* Python: `>=3.9`&#x20;
* Arcee AI model running locally or accessible via API and an OpenAI-compatible endpoint

**Quickstart**

Environment and project setup:

```bash
# Create project folder
mkdir arceeai_llamaindex && cd arceeai_llamaindex

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# Create and activate virtual environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install LlamaIndex OpenAI-compatible client
uv pip install llama-index-llms-openai-like
```

{% hint style="danger" %}
If you run into any errors installing llamaindex, follow their [Instillation Guide](https://developers.llamaindex.ai/python/framework/getting_started/installation/) and [OpenAILike Guide](https://developers.llamaindex.ai/python/framework-api-reference/llms/openai_like/).
{% endhint %}

Create a new python file called `arceeai_llamaindex.py` with the following:

```python
import os
from llama_index.llms.openai_like import OpenAILike

# Configure Arcee AI Model
ARCEE_BASE = os.getenv("OPENAI_API_BASE", "http://127.0.0.1:8080/v1")
ARCEE_KEY  = os.getenv("OPENAI_API_KEY", "your-arcee-api-key")
ARCEE_MODEL = os.getenv("OPENAI_MODEL_NAME", "trinity-mini")

# Initialize Arcee AI model with OpenAI-compatible configuration
arcee_llm = OpenAILike(
    model=ARCEE_MODEL,
    api_base=ARCEE_BASE,
    api_key=ARCEE_KEY,
    is_chat_model=True,
    #is_function_calling_model=True,
)

# Define the prompt to be sent to the Arcee AI model
text = """Arcee AI is a foundation model provider with a focus on building the highest performing models per parameter. 
They offer a range of models from on-device and edge optimized models to large language models. Their suite of models 
provides customers with the flexibility to choose the right model for the right task. All models are released Apache 2.0 
enabling the community to use safe, built-in-the-US models in their own environment or via the Arcee AI API platform."""

prompt = f"Summarize the following in three bullets:\n\n{text}"

# Invoke the Arcee AI model
response = arcee_llm.complete(prompt)

# Print the results
print("\n=== RESULT ===\n")
print(str(response))
```

{% hint style="success" %}
This works out-of-the-box if you have an Arcee AI model running locally on your laptop. If you do not, change `ARCEE_BASE` , `ARCEE_KEY` , and `ARCEE_MODEL` .

You can also setup a `.env` file to store the configurations.
{% endhint %}

Test your script:

```bash
python arceeai_llamaindex.py
```

#### Retrieval Augmented Generation

This example sets up a RAG pipeline with LlamaIndex using an Arcee AI model for text generation and an OpenAI Embeddings model for document embeddings. It uses an in-memory vector database that is cleared after execution. For persistent storage and more advanced pipelines, see the [LlamaIndex documentation](https://developers.llamaindex.ai/python/framework/).

**Prerequisites**

* Python: `>=3.9`&#x20;

**Environment Setup for RAG Pipeline**

```bash
# Create project folder
mkdir arceeai_llamaindex_rag && cd arceeai_llamaindex_rag

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

# Create and activate virtual environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install LlamaIndex core, Arcee LLM wrapper, and embedding support
uv pip install llama-index-core llama-index-llms-openai-like llama-index-embed

```

Create a new python file called `arceeai_llamaindex_rag.py`  with  the following:

```python
import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.llms import ChatMessage

# Configure Arcee AI Model
ARCEE_BASE = os.getenv("OPENAI_API_BASE", "http://127.0.0.1:8080/v1")
ARCEE_KEY  = os.getenv("OPENAI_API_KEY", "your-arcee-api-key")
ARCEE_MODEL = os.getenv("OPENAI_MODEL_NAME", "trinity-mini")

# Initialize Arcee AI model with OpenAI-compatible configuration
arcee_llm = OpenAILike(
    model=ARCEE_MODEL,
    api_base=ARCEE_BASE,
    api_key=ARCEE_KEY,
    is_chat_model=True,
    #is_function_calling_model=True,
)

# Configure an embedding model to embed your documents
# This can be any embedding model, local or API
# In this example, we'll use an embedding model from OpenAI
embed_model = OpenAIEmbedding(
    model_name="text-embedding-3-small",
    api_base="https://api.openai.com/v1",
    api_key="YOUR_API_KEY", # Put your API Key here or reference from environment variables
)

# Set the models for llama-index to use
Settings.llm = arcee_llm
Settings.embed_model = embed_model

# Load documents
# In this example, we have some .txt/.md/.pdf files under ./data
documents = SimpleDirectoryReader("./data").load_data()

# Build the vector index and load in the documents
index = VectorStoreIndex.from_documents(documents)

# Query the index
query_engine = index.as_query_engine()
answer = query_engine.query("Summarize the top 5 key points in these files.") # Change the prompt to a specific question about your documents

# Print the results
print("\n=== RESULT ===\n")
print(answer.response)

```

{% hint style="success" %}
This works out-of-the-box if you have an Arcee AI model running locally on your laptop. If you do not, change `ARCEE_BASE` , `ARCEE_KEY` , and `ARCEE_MODEL` .

You can also setup a `.env` file to store the configurations.
{% endhint %}

Run your script:

```bash
python arceeai_llamaindex_rag.py
```


# n8n

[n8n](https://n8n.io/) is an open-source workflow automation platform that enables developers and businesses to connect APIs, services, and custom logic through visual, node-based workflows. It’s designed for flexibility and extensibility, allowing users to orchestrate complex data flows and automate repetitive tasks without writing full applications. With their `Basic LLM Chain` and `AI Agent` nodes, you can connect to language models across a variety of platforms.

This tutorial will guide you through how to use Arcee AI language models in n8n via OpenRouter.

***

**Prerequisites**

* n8n Account
  * If you do not have an n8n account, get started [here](https://docs.n8n.io/).

**Quickstart**

Download the workflow schema below

{% file src="/files/yvbZTYMJkiI9r73KPl0t" %}

{% hint style="info" %}
Right click on the file above and select "Open in a New Tab", the page you'll see is the workflow JSON schema. Right click on the page and select "Save as..." to save the file.
{% endhint %}

2. In your blank n8n workflow, select the ellipsis (3 dots "...") in the top right corner and select "Import from File". and select `Build your first Arcee AI agent.json.` Your workflow will now look like this:

<figure><img src="/files/Qelops7BiclI0ToYU7vJ" alt="" width="563"><figcaption></figcaption></figure>

{% hint style="success" %}
This workflow takes in a prompt from chat and passes it to an AI Agent. The AI Agent is powered by a language model (which we'll set to be an Arcee AI model) and has access to memory. With this node you can also configure tools to have your agent access outside systems and APIs.
{% endhint %}

Select the OpenRouter Chat Model node and setup your credentials for OpenRouter using an OpenRouter API Key. Select the Arcee AI model you want to use.

**Your OpenRouter Chat Model configuration should now look like this:**

<figure><img src="/files/JS63BfgIzj19DwmQP2P6" alt="" width="375"><figcaption></figcaption></figure>

Select Open Chat and send a message to your agent

<figure><img src="/files/LKG1gBJYqS35EBtdkl2M" alt=""><figcaption></figcaption></figure>

Your message will be passed to the Agent Node, which invokes the model and stores the messages in conversation memory.


# Pipecat

**Pipecat** is an open-source, real-time voice AI orchestration framework built for developers who want to create deeply interactive, multimodal conversational agents. It provides a flexible pipeline architecture for integrating speech recognition (STT), language models, and speech synthesis (TTS) components into low-latency, event-driven workflows. Designed for modularity and extensibility, Pipecat enables developers to compose custom voice agent stacks that handle audio streaming, turn-taking, and response timing with precision.

This tutorial will guide you through integrating Arcee AI models as the LLM backbone for your Pipecat voice agent. We'll integrate Arcee's models using OpenRouter.

***

<figure><img src="/files/KdGSyD7dqYYQP2kg9LQT" alt="" width="563"><figcaption></figcaption></figure>

**Prerequisites**

* **Python:** `>=3.10`
* OpenRouter API Key
  * If you don't have an account, set one up [here](https://openrouter.ai/).
* [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager installed

**Quickstart**

Install the Pipecat CLI globally using [uv](https://docs.astral.sh/uv/)

```bash
uv tool install pipecat-ai-cli
```

Run the Pipecat interactive setup wizard to create the scaffolding for a pipecat agent

```
pipecat init
```

You will be prompted with a few questions which determine how the project is setup. Use the following configuration:

```
Project name: arcee-pipecat
Bot type: Web/Mobile
Client framework: React
React dev server: Vite
Transport: SmallWebRTC
Add another transport for local testing? No
Pipeline architecture: Cascade (STT → LLM → TTS)
Speech-to-Text: Deepgram
Language model: OpenRouter
Text-to-Speech: Deepgram

Customize feature settings? Yes
Audio recording? Yes
Transcription logging? Yes
Smart turn-taking? Yes
Use video avatar service? No
Video input? No
Video output? No
Enable observability? No
Deploy to Pipecat Cloud? No
```

{% hint style="success" %}
These configurations create an agent which uses Deepgram for Speech-to-Text and Text-to-Speech, and Arcee models through OpenRouter for the LLM.

You can modify any of these settings to change your voice agent.
{% endhint %}

Setup the client

```bash
cd arcee-pipecat/client
npm install
npm run dev
```

Setup the server, install dependencies, and create a `.env` file

{% code fullWidth="false" %}

```bash
cd arcee-pipecat/server
uv sync
cp .env.example .env
```

{% endcode %}

In the `.env` file, populate your Deepgram and OpenRouter API Key and provide a Deepgram Voice ID and the Arcee AI LLM you want to use. For example:

```
# Deepgram (STT/TTS)
DEEPGRAM_API_KEY=<YOUR_DEEPGRAM_KEY>
DEEPGRAM_VOICE_ID=aura-2-thalia-en

# OpenRouter (LLM)
OPENROUTER_API_KEY=<YOUR_OPENROUTERR_KEY>
OPENROUTER_MODEL=arcee-ai/trinity-mini
```

{% hint style="info" %}
A full list of Deepgram Voice IDs can be found [here](https://developers.deepgram.com/docs/tts-models).
{% endhint %}

Edit the Voice Agent Kickoff System prompt

1. Navigate to `arcee-pipecat/server/bot.py`&#x20;
2. Replace the content of the Kickoff system prompt (on line 139) with the following message:

{% code overflow="wrap" %}

```
You are a helpful voice ai assistant powered Arcee AI Language Models. Respond naturally and keep your answers conversational. Start by introducing yourself.
```

{% endcode %}

The full line should now look like the following:

```python
        # Kick off the conversation
        messages.append({"role": "system", "content": "You are a helpful voice ai assistant powered Arcee AI Language Models. Respond naturally and keep your answers conversational. Start by introducing yourself."})
```

Run your Voice Agent, access at <http://localhost:5173/>, and click connect to start the conversation!

```bash
uv run bot.py
```


# ElevenLabs

**ElevenLabs Agents** is a conversational voice agent platform that combines automatic speech recognition (ASR), a pluggable language model, human-like TTS, and a turn-taking engine into a complete voice stack.

This tutorial will guide you through how to integrate Arcee AI model's as the language model for your ElevenLabs agent. The first section will showcase how to utilize our models through **Together.ai** and the second will showcase a **self-hosting** option.

***

### Using Arcee Models with Deepgram Voice Agents via Together.ai

#### Step 1: Create a Together.ai API Key

1. Go to [api.together.xyz/settings/api-keys](https://api.together.xyz/settings/api-keys)
2. Click **“Create API Key”**
3. Copy the key and store it securely

#### Step 2: Connect an Arcee model to Your ElevenLabs Agent

1. In the ElevenLabs dashboard, go to **Settings** → **Workspace Secrets**
2. Click **“Add a Secret”**
   * **Name:** `together-ai-api-key`
   * **Value:** Paste your Together AI API key
3. Click **“Add a Secret”** to save it to your workspace
4. Go to the **Agents** tab from the left pane
5. Select your existing agent or create a new one
6. Scroll to the **LLM** section
7. Beside "Select which provider and model to use for the LLM", select **“Custom LLM”**
8. Fill in the following fields:

<figure><img src="/files/TjEtGeIHGZ38QQgez5tM" alt=""><figcaption></figcaption></figure>

1. Click **Save** to apply the agent configuration

#### Step 3: Test the Agent

Click **"Test AI Agent"** in the ElevenLabs dashboard to chat with the model.

***

### Using a Self Hosted Model with ElevenLabs Agents

This section explains how to use one of our **self hosted models** as the LLM backbone for your agent by self-hosting the model on your own infrastructure. We will use AFM 4.5B in this example

#### Deploy the model

Refer to our [Quick Deploys](/quick-deploys/hardware-prerequisites) section and our [Hardware Prequesities ](/quick-deploys/hardware-prerequisites)page to select a method for deployment based on your use case and hardware. In this example, we'll use **llama.cpp:**

#### Launch the OpenAI-Compatible Server

Start `llama-server` with the correct model and context size. This will expose an OpenAI-compatible `/v1/chat/completions` endpoint:

```bash
bin/llama-server -m ./afm/AFM-4.5B-bf16.gguf \
  --host 0.0.0.0 \
  --port 8000 \
  --jinja \
  --ctx-size 8192
```

> Make sure the `--jinja` flag is included. This is required to enable the OpenAI-compatible API.

#### Expose the Server with ngrok (Required)

To make your server accessible, create a public URL using a tunneling tool like **ngrok**:

```bash
ngrok http 8000
```

This will generate a public HTTPS URL like:

```
https://your-subdomain.ngrok-free.dev → http://localhost:8000
```

Keep this ngrok tunnel open while the agent is active.

#### Configure ElevenLabs Agent to Use Your Self Hosted Model

#### Configure your agent

1. Go to the **Agents** tab and open your agent
2. In the **Model Configuration** section, **enter the ngrok url with "/v1" at the end**, a **placeholder model ID** and select **"None" for the API key**:

<figure><img src="/files/4HFBfdSeNtNN7iTU9vC9" alt=""><figcaption></figcaption></figure>

#### Test the Agent

Click **"Test AI Agent"** in the ElevenLabs dashboard to chat with the model.


# DeepGram

**Deepgram Voice Agents** is a flexible conversational AI stack that includes speech‑to‑text (STT), text‑to‑speech (TTS), and pluggable language models, designed to power real‑time, multi‑turn voice applications.

This tutorial will guide you through how to integrate Arcee AI model's as the LLM backbone for your Deepgram voice agent. The first section will showcase how to utilize our models through **Together.ai** and the second will showcase a **self-hosting** option.

***

### Using Arcee Models with Deepgram Voice Agents via Together.ai

#### Create a Together AI API Key

1. Go to [api.together.xyz/settings/api-key](https://api.together.xyz/settings/api-keys)
2. Click **Create API Key**
3. Copy the key and store it securely, you’ll need it to authorize model requests

#### Configure the Deepgram Agent to Use our Model

1. Navigate to the **Deepgram Voice Agent** section of the Deepgram playground
2. Scroll to the **Model** section
3. Under **Select a Large Language Model**, choose:

   ```
   Other – Custom model
   ```
4. Fill in the following fields (and replace the API key under authorization with your together API key from Step 1)

| Field                       | Value                                            |
| --------------------------- | ------------------------------------------------ |
| **Custom Model Name**       | arcee-ai/trinity-mini *(or any Arcee model)*     |
| **Custom Model URL**        | `https://api.together.xyz/v1/chat/completions`   |
| **Custom Model API Format** | OpenAI                                           |
| **Authorization Header**    | `Authorization` → `Bearer YOUR_TOGETHER_API_KEY` |

#### Test Your Agent

1. Scroll down and click **Talk to your Agent**
2. Speak to your agent or type a message
3. Open the **Developer Console** to view the underlying API calls and verify responses.
4. You’ll see the full conversation log in real time, and hear your model’s response played back using Deepgram’s TTS engine.

***

## Using a Self Hosted Model with Deepgram Voice Agents

This guide explains how to integrate **a self hosted Arcee model** as the LLM backbone for your Deepgram voice agent using a self-hosted setup. We will use AFM 4.5B for this example.

#### Deploy a Model with an OpenAI-Compatible Server

Refer to our [Quick Deploys](/quick-deploys/hardware-prerequisites) section and our [Hardware Prequesities ](/quick-deploys/hardware-prerequisites)page to select a method for deployment based on your use case and hardware. In this example, we'll use **llama.cpp:**

```bash
./bin/llama-server -m ./afm/AFM-4.5B-bf16.gguf \
  --host 0.0.0.0 \
  --port 8000 \
  --ctx-size 8192 \
  --jinja
```

Make sure `--jinja` is included to enable the OpenAI-compatible API.

#### Expose the server via ngrok

Deepgram needs a public HTTPS endpoint to reach your model.

Use ngrok or any tunneling tool:

```bash
ngrok http 8000
```

This will forward to your local server and give you a public URL like:

```
https://your-subdomain.ngrok-free.dev → http://localhost:8000
```

Keep this tunnel active while your Deepgram agent is running.

#### Configure the Deepgram Agent to Use Your Model

1. Navigate to the DeepGram VoiceAgent section of the DeepGram playground
2. Scroll to the **Model** section
3. Under **Select a Large Language Model**, choose: `Other – Custom model`
4. Fill in the following fields:

| Field                       | Value                                                       |
| --------------------------- | ----------------------------------------------------------- |
| **Custom Model Name**       | AFM                                                         |
| **Custom Model URL**        | `https://your-subdomain.ngrok-free.dev/v1/chat/completions` |
| **Custom Model API Format** | OpenAI                                                      |
| **Authorization Header**    | `Authorization` → `Bearer None`                             |

#### Test Your Agent

1. Scroll down and click "Talk to your Agent"
2. Speak to your agent
3. Examine the calls under the developer console

You’ll see the full conversation log in real time, and hear your model's response played back using Deepgram’s TTS engine.


# AFM-4.5B

**AFM-4.5B** is the first model available in the Arcee Foundation Model family. AFM-4.5B is a 4.5 billion parameter small language model, which delivers enterprise performance comparable to much larger models at vastly lower hosting costs, while being efficient enough to run on low-RAM GPUs or even CPUs.

AFM-4.5B comes in two variants - base and instruct. The base model was trained on a dataset of 8 trillion tokens, comprising 6.5 trillion tokens of general pre-training data followed by 1.5 trillion tokens of mid-training data with enhanced focus on mathematical reasoning and code generation. Following pre-training, the model underwent supervised fine-tuning on high-quality instruction datasets. The instruction-tuned model was further refined through reinforcement learning on verifiable rewards as well as for human preference.&#x20;

We used a modified version of [TorchTitan](https://arxiv.org/abs/2410.06511) for pre-training, [Axolotl](https://axolotl.ai/) for supervised fine-tuning, and a modified version of [Verifiers](https://github.com/willccbb/verifiers) for reinforcement learning.

Both variants of AFM-4.5B are available on Hugging Face:

[arcee-ai/AFM-4.5B](https://huggingface.co/arcee-ai/AFM-4.5B)

[arcee-ai/AFM-4.5B-Base](https://huggingface.co/arcee-ai/AFM-4.5B-Base)

### Deployment Quickstart

To get started deploying AFM-4.5B, proceed to [AFM-4.5B Quick Deploys](broken://pages/LcC5daAPEhSOUtS8SZ45).

### Model Summary

|                                  |                                                                                                         |
| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Name                             | `AFM-4.5B`                                                                                              |
| Parameters                       | 4.5 billion                                                                                             |
| Architecture                     | Decoder-only Transformer                                                                                |
| Activation Function              | ReLU²                                                                                                   |
| Attention                        | Grouped Query Attention                                                                                 |
| Training Tokens                  | [8 trillion](https://blog.datologyai.com/beyondweb/)\*                                                  |
| License                          | Apache 2.0                                                                                              |
| Recommended Inference Parameters | <ul><li>temperature: 0.5</li><li>top\_k: 50</li><li>top\_p: 0.95</li><li>repeat\_penalty: 1.1</li></ul> |

{% hint style="info" %}
The blog linked in Training Tokens details the dataset curation process done by Arcee AI and Datology AI.
{% endhint %}

### Training Pipeline

* **Pre-training (6.5T tokens)**: General web, code, multilingual, and reasoning data.
* **Mid-training (1.5T tokens)**: Emphasis on **math**, **programming**, and **structured reasoning**.
* **Supervised Fine-tuning**: High-quality instruction datasets for chat-style interactions.
* **RLHF**: Reinforcement learning with verifiable reward models and human preference optimization.
* **Data Curation**: Powered by **DatologyAI**, using model-based filtering, source mixing, and synthetic data synthesis.

### Performance Characteristics

* **Factual Accuracy**: Low hallucination rate due to clean, curated dataset.
* **Compliance**: Minimal IP risk with exclusion of copyrighted books and restricted data.
* **Inference Efficiency**: Suitable for real-time applications on lower-end GPUs or CPUs.
* **Multilingual**: Supports Arabic, English, French, German, Hindi, Italian, Korean, Mandarin, Portuguese, Russian, and Spanish.

### **Performance Metrics**

<table><thead><tr><th width="144.05078125">Hardware</th><th width="146.98828125">Max Model Len</th><th width="94.31640625">Quantization</th><th width="121.16796875">Max Concurrent Requests</th><th width="155.06640625">TPS per Request*</th></tr></thead><tbody><tr><td>H100 x 1</td><td>65536 (Max)</td><td>bf16</td><td>16</td><td>136</td></tr><tr><td>H100 x 1</td><td>4096</td><td>bf16</td><td>250</td><td>74.5</td></tr><tr><td>L40S x 1</td><td>8192</td><td>bf16</td><td>55</td><td>59</td></tr><tr><td>L40S x 1</td><td>4096</td><td>bf16</td><td>109</td><td>64</td></tr><tr><td>A10 x 1</td><td>8192</td><td>bf16</td><td>12</td><td>65</td></tr><tr><td>A10 x 1</td><td>4096</td><td>bf16</td><td>25</td><td>75</td></tr><tr><td>Intel CPU<sup>1</sup></td><td>1024</td><td>Q4_0</td><td>4</td><td>29</td></tr><tr><td>Graviton4<sup>2</sup></td><td>1024</td><td>Q4_0</td><td>4</td><td>60</td></tr></tbody></table>

<sup>1</sup> Intel Sapphire Rapids CPU with 32 threads

<sup>2</sup>AWS Graviton4 Instance with 32 vCPUs

{% hint style="info" %}
TPS benchmarks represent tokens per second per request at maximum concurrent requests. TPS will increase with fewer concurrent requests, so the benchmark numbers effectively represent minimum TPS.
{% endhint %}

### Relevant Blogs

[Announcing Arcee Foundation Models](https://www.arcee.ai/blog/announcing-the-arcee-foundation-model-family)

[Deep Dive: AFM-4.5B, the First Arcee Foundation Model](https://www.arcee.ai/blog/deep-dive-afm-4-5b-the-first-arcee-foundational-model)

[Is Running Language Models on CPU Really Viable?](https://www.arcee.ai/blog/is-running-language-models-on-cpu-really-viable)


# Trinity-Nano (6B)

**Overview**

Trinity Nano is a 6B-parameter (1B active) sparse mixture-of-experts language model, optimized for high-efficiency inference in real-time, on-device, and embedded AI applications.

**Key Features**

* Efficient attention mechanism: reduces memory and compute requirements while preserving long-context coherence.
* 128K-token context window: supports multi-turn interactions and extended document processing.
* Strong context utilization: fully leverages long inputs for coherent multi-turn reasoning and reliable function/tool calls.
* High inference efficiency: generates tokens rapidly while minimizing compute, delivering an outstanding price-to-performance ratio.

### Deployment Quickstart

To get started deploying Trinity-Nano, download the model [here](https://huggingface.co/arcee-ai) and proceed to [Quick Deploys](/quick-deploys/hardware-prerequisites)

### Model Summary

|                                  |                                                                                                  |
| -------------------------------- | ------------------------------------------------------------------------------------------------ |
| Name                             | Trinity-Nano-6B                                                                                  |
| Architecture                     | Mixture-of-Experts                                                                               |
| Parameters                       | 6 Billion Total, 1 Billion Active                                                                |
| Experts                          | 128 Experts, 8 Active                                                                            |
| Attention Mechanism              | Grouped Query Attention (GQA)                                                                    |
| Training Tokens                  | 10 trillion                                                                                      |
| License                          | Apache 2.0                                                                                       |
| Recommended Inference Parameters | <ul><li>temperature: 0.15</li><li>top\_p: 0.75</li><li>top\_k: 50</li><li>min\_p: 0.06</li></ul> |


# Trinity-Mini (26B)

**Overview**

Trinity Mini is a 26B-parameter (3B active) sparse mixture-of-experts language model, engineered for efficient inference over long contexts with robust function calling and multi-step agent workflows.

**Key Features**&#x20;

* Efficient attention mechanism: reduces memory and compute requirements while preserving long-context coherence.
* 128K-token context window: supports multi-turn interactions and extended document processing.
* Strong context utilization: fully leverages long inputs for coherent multi-turn reasoning and reliable function/tool calls.
* High inference efficiency: generates tokens rapidly while minimizing compute, delivering an outstanding price-to-performance ratio.

### Deployment Quickstart

To get started deploying Trinity-Mini, download the model [here](https://huggingface.co/arcee-ai) and proceed to [Quick Deploys](/quick-deploys/hardware-prerequisites)

### Model Summary

|                                  |                                                                                                         |
| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Name                             | Trinity-Mini-26B                                                                                        |
| Architecture                     | Mixture-of-Experts                                                                                      |
| Parameters                       | 26 Billion Total, 3.5 Billion Active                                                                    |
| Experts                          | 128 Experts, 8 Active                                                                                   |
| Attention Mechanism              | Grouped Query Attention (GQA)                                                                           |
| Training Tokens                  | 10 trillion                                                                                             |
| License                          | Apache 2.0                                                                                              |
| Recommended Inference Parameters | <p></p><ul><li>temperature: 0.15</li><li>top\_p: 0.75</li><li>top\_k: 50</li><li>min\_p: 0.06</li></ul> |


# Trinity-Large-Preview

**Overview**

Trinity Large (Preview) is a 400B-parameter (13B active) sparse mixture-of-experts language model, engineered to scale model capacity while maintaining inference efficiency over long contexts, with strong performance in reasoning-heavy workloads including math, coding-related tasks, and multi-step agent workflows.

**Key Features**&#x20;

* **Sparse mixture-of-experts architecture:** Uses an extremely sparse MoE design with 400B total parameters and 13B activated per token. Sparse expert routing constrains per-token activation, enabling efficient inference at scale.
* **Long-context training and utilization:** Trained at 256K sequence length with support for 512K inference (hosted at 128k), using architecture and training procedures designed to operate effectively over long inputs and extended multi-turn interactions over large inputs.
* **High throughput efficiency:** Designed with inference-time efficiency as a primary objective, leveraging both extreme sparsity and optimized attention mechanisms to achieve strong throughput on modern accelerator hardware.

### Deployment Quickstart

To get started deploying Trinity Large, download the model [here](https://huggingface.co/arcee-ai) and proceed to [Quick Deploys](/quick-deploys/hardware-prerequisites)

### Model Summary

|                                  |                                                        |
| -------------------------------- | ------------------------------------------------------ |
| Name                             | Trinity-Large-Preview                                  |
| Architecture                     | Mixture-of-Experts                                     |
| Parameters                       | 400 Billion Total, 13 Billion Active                   |
| Experts                          | 256 Experts, 4 Active                                  |
| Attention Mechanism              | Grouped Query Attention (GQA)                          |
| Training Tokens                  | 17 trillion                                            |
| License                          | Apache 2.0                                             |
| Recommended Inference Parameters | <ul><li>temperature: 0.8</li><li>top\_p: 0.8</li></ul> |

v


# Trinity-Large-Thinking

**Overview**

Trinity-Large-Thinking is a reasoning-optimized variant of Arcee AI's Trinity-Large family — a 398B-parameter sparse Mixture-of-Experts (MoE) model with approximately 13B active parameters per token. Built on Trinity-Large-Base and post-trained with extended chain-of-thought reasoning and agentic RL, Trinity-Large-Thinking delivers state-of-the-art performance on agentic benchmarks while maintaining strong general capabilities.

Trinity-Large-Thinking generates explicit reasoning traces wrapped in `<think>...</think>` blocks before producing its final response. This thinking process is critical to the model's performance — **thinking tokens must be kept in context** for multi-turn conversations and agentic loops to function correctly.

<figure><img src="/files/4fJE6DOMP83bRfk82M2y" alt=""><figcaption></figcaption></figure>

**Key Features**

* **Agentic-first design**: Purpose-built for tool calling, multi-step planning, and agent workflows
* **State-of-the-art agentic performance**: 94.7% on τ²-Bench, 91.9% on PinchBench, 98.2% on LiveCodeBench
* **Native reasoning traces**: Extended chain-of-thought via `<think>...</think>` blocks
* **Compatible with major agent frameworks**: Works out of the box with [OpenClaw](https://github.com/openclaw) and [Hermes Agent](https://github.com/NousResearch/hermes-agent)

**Thinking-in-Context: Important Usage Note**

Trinity-Large-Thinking produces reasoning traces inside `<think>...</think>` blocks before generating its final response.

This means:

1. **Multi-turn conversations**: When building chat applications, include the full assistant response (thinking + answer) in the conversation history for subsequent turns.
2. **Agentic loops**: When using Trinity-Large-Thinking as the backbone of an agent (OpenClaw, Hermes Agent, or custom), ensure your tool-calling loop preserves `<think>` blocks in the message history between steps.
3. **Context window management**: The 512k extended context window accommodates long reasoning chains across many agentic steps. If you must truncate history, prefer removing older turns entirely rather than stripping thinking tokens from recent turns.

For implementation details, pitfalls (`reasoning` vs `reasoning_content`), and Python/TypeScript examples, refer to the [Reasoning Traces](/capabilities/reasoning-traces) page.

### Benchmarks

<table><thead><tr><th>Benchmark</th><th width="114.20703125" align="right">Trinity-Large-Thinking</th><th width="117.40625" align="right">Opus-4.6</th><th width="113.6171875" align="right">GLM-5</th><th width="113.8203125" align="right">MiniMax-M2.7</th><th width="124.5" align="right">Kimi-K2.5</th></tr></thead><tbody><tr><td>IFBench</td><td align="right">52.3</td><td align="right">53.1</td><td align="right">72.3</td><td align="right"><strong>75.7</strong></td><td align="right">70.2</td></tr><tr><td>GPQA-Diamond</td><td align="right">76.3</td><td align="right"><strong>89.2</strong></td><td align="right">81.6</td><td align="right">86.2</td><td align="right">86.9</td></tr><tr><td>Tau2-Airline</td><td align="right"><strong>88.0</strong></td><td align="right">82.0</td><td align="right">80.5</td><td align="right">80.0</td><td align="right">80.0</td></tr><tr><td>Tau2-Telecom</td><td align="right">94.7</td><td align="right">92.1</td><td align="right"><strong>98.2</strong></td><td align="right">84.8</td><td align="right">95.9</td></tr><tr><td>PinchBench</td><td align="right">91.9</td><td align="right"><strong>93.3</strong></td><td align="right">86.4</td><td align="right">89.8</td><td align="right">84.8</td></tr><tr><td>AIME25</td><td align="right">96.3</td><td align="right"><strong>99.8</strong></td><td align="right">93.3</td><td align="right">80.0</td><td align="right">96.3</td></tr><tr><td>BCFLv4</td><td align="right">70.1</td><td align="right"><strong>77.0</strong></td><td align="right">70.8</td><td align="right">70.6</td><td align="right">68.3</td></tr><tr><td>MMLU-Pro</td><td align="right">83.4</td><td align="right"><strong>89.1</strong></td><td align="right">85.8</td><td align="right">80.8</td><td align="right">87.1</td></tr><tr><td>SWE-bench Verified*</td><td align="right">63.2</td><td align="right"><strong>75.6</strong></td><td align="right">72.8</td><td align="right">75.4</td><td align="right">70.8</td></tr></tbody></table>

\*All models evaluated in mini-swe-agent-v2

### Deployment Quickstart

To get started deploying Trinity Large, download the model [here](https://huggingface.co/arcee-ai) and proceed to [Quick Deploys](/quick-deploys/hardware-prerequisites)

### Model Summary

|                                  |                                      |
| -------------------------------- | ------------------------------------ |
| Name                             | Trinity-Large-Thinking               |
| Architecture                     | Sparse MoE (AfmoeForCausalLM)        |
| Parameters                       | 398 Billion Total, 13 Billion Active |
| Experts                          | 256 Experts, 4 Active                |
| Attention Mechanism              | Grouped Query Attention (GQA)        |
| Training Tokens                  | 17 trillion                          |
| License                          | Apache 2.0                           |
| Recommended Inference Parameters | <ul><li>temperature: 0.3</li></ul>   |


# Your First API Call

The Arcee Platform API is OpenAI compatible. It can be accessed through your command line, or, with minor configuration changes, through the OpenAI SDK.&#x20;

| Parameter | Value                                                  |
| --------- | ------------------------------------------------------ |
| base\_url | <https://api.arcee.ai/api/v1/chat/completions>         |
| api\_key  | [Generate API key](https://chat.arcee.ai/api/api-keys) |

#### **1. Generate an API Key**

Create an API key from your [**Arcee Platform**](https://chat.arcee.ai/login) dashboard. This key authenticates all API requests.

#### **2. Choose a Model**

Specify the model you want to run inference with. Arcee currently provides the following production models:

* **trinity-mini**

#### **3. Run the Code Snippet**

Use your **API key** and **model name** in the sample code below to make your first request.

{% tabs %}
{% tab title="cURL" %}

```bash
curl -X POST "https://api.arcee.ai/api/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
        "model": "trinity-large-thinking",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "Greetings!"}
        ],
        "stream": false
      }'
```

{% endtab %}

{% tab title="Python" %}

```python
from openai import OpenAI 

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.arcee.ai/api/v1"
)

response = client.chat.completions.create(
    model="trinity-large-thinking",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "What is 25 + 37?"},
    ],
    stream=False
)

print(response.choices[0].message.content)
print(response.choices[0].message.reasoning)

```

{% endtab %}

{% tab title="nodeJS" %}

```javascript
import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: 'https://api.arcee.ai/api/v1',
    apiKey: 'YOUR_API_KEY'
});

async function main() {
  const completion = await openai.chat.completions.create({
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: "What is 25 + 37?" }
    ],
    model: "trinity-large-thinking",
  });

  console.log(completion.choices[0].message.content);
  console.log(completion.choices[0].message.reasoning);
}

main();
```

{% endtab %}
{% endtabs %}


# Chat Completion

## POST /v1/chat/completions

> Create a chat completion

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"servers":[{"url":"https://api.arcee.ai"}],"security":[{"bearerAuth":[]}],"components":{"securitySchemes":{"bearerAuth":{"type":"http","scheme":"bearer","bearerFormat":"Arcee API key (prefixed with `rcai-`)"}},"schemas":{"ChatCompletionSystemMessageParam":{"properties":{"content":{"title":"Content","type":"string"},"role":{"const":"system","title":"Role","type":"string"},"name":{"title":"Name","type":"string"}},"required":["content","role"],"title":"ChatCompletionSystemMessageParam","type":"object"},"ChatCompletionUserMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"role":{"const":"user","title":"Role","type":"string"},"name":{"title":"Name","type":"string"}},"required":["content","role"],"title":"ChatCompletionUserMessageParam","type":"object"},"ChatCompletionContentPartTextParam":{"properties":{"text":{"title":"Text","type":"string"},"type":{"const":"text","title":"Type","type":"string"}},"required":["text","type"],"title":"ChatCompletionContentPartTextParam","type":"object"},"ChatCompletionContentPartImageParam":{"properties":{"image_url":{"$ref":"#/components/schemas/ImageURL"},"type":{"const":"image_url","title":"Type","type":"string"}},"required":["image_url","type"],"title":"ChatCompletionContentPartImageParam","type":"object"},"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"},"ChatCompletionAssistantMessageParam":{"properties":{"role":{"const":"assistant","title":"Role","type":"string"},"content":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Content"},"function_call":{"$ref":"#/components/schemas/FunctionCall"},"name":{"title":"Name","type":"string"},"tool_calls":{"items":{"$ref":"#/components/schemas/ChatCompletionMessageToolCallParam"},"title":"Tool Calls","type":"array"}},"required":["role"],"title":"ChatCompletionAssistantMessageParam","type":"object"},"FunctionCall":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"FunctionCall","type":"object"},"ChatCompletionMessageToolCallParam":{"properties":{"id":{"title":"Id","type":"string"},"function":{"$ref":"#/components/schemas/Function"},"type":{"const":"function","title":"Type","type":"string"}},"required":["id","function","type"],"title":"ChatCompletionMessageToolCallParam","type":"object"},"Function":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"Function","type":"object"},"ChatCompletionFunctionMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"name":{"title":"Name","type":"string"},"role":{"const":"function","title":"Role","type":"string"}},"required":["content","name","role"],"title":"ChatCompletionFunctionMessageParam","type":"object"},"ChatCompletionToolMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"role":{"const":"tool","title":"Role","type":"string"},"tool_call_id":{"title":"Tool Call Id","type":"string"}},"required":["content","role","tool_call_id"],"title":"ChatCompletionToolMessageParam","type":"object"},"Tool":{"type":"object","required":["type","function"],"properties":{"type":{"type":"string","enum":["function"]},"function":{"type":"object","required":["name"],"properties":{"name":{"type":"string","description":"Name of the function to be called."},"description":{"type":"string","description":"Description of what the function does."},"parameters":{"type":"object","description":"JSON Schema object describing the function's parameters."}}}}},"ChatCompletionResponse":{"type":"object","description":"OpenAI-compatible chat completion response. Standard fields (`id`, `object`, `created`, `model`, `choices`, `usage`) are returned; provider-specific extras are stripped before responding."},"Error":{"type":"object","properties":{"detail":{"type":"string","description":"Human-readable error message."}},"required":["detail"]}}},"paths":{"/v1/chat/completions":{"post":{"summary":"Create a chat completion","operationId":"createChatCompletion","tags":["Chat Completions"],"requestBody":{"required":true,"content":{"application/json":{"schema":{"properties":{"model":{"title":"Model","type":"string"},"messages":{"default":[],"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionSystemMessageParam"},{"$ref":"#/components/schemas/ChatCompletionUserMessageParam"},{"$ref":"#/components/schemas/ChatCompletionAssistantMessageParam"},{"$ref":"#/components/schemas/ChatCompletionFunctionMessageParam"},{"$ref":"#/components/schemas/ChatCompletionToolMessageParam"}]},"title":"Messages","type":"array"},"timeout":{"anyOf":[{"type":"number"},{"type":"integer"},{"type":"null"}],"default":null,"title":"Timeout"},"temperature":{"anyOf":[{"type":"number"},{"type":"null"}],"default":null,"title":"Temperature"},"top_p":{"anyOf":[{"type":"number"},{"type":"null"}],"default":null,"title":"Top P"},"n":{"anyOf":[{"type":"integer"},{"type":"null"}],"default":null,"title":"N"},"stream":{"anyOf":[{"type":"boolean"},{"type":"null"}],"default":null,"title":"Stream"},"stop":{"anyOf":[{"type":"object"},{"type":"null"}],"default":null,"title":"Stop"},"max_tokens":{"anyOf":[{"type":"integer"},{"type":"null"}],"default":null,"title":"Max Tokens"},"presence_penalty":{"anyOf":[{"type":"number"},{"type":"null"}],"default":null,"title":"Presence Penalty"},"frequency_penalty":{"anyOf":[{"type":"number"},{"type":"null"}],"default":null,"title":"Frequency Penalty"},"logit_bias":{"anyOf":[{"type":"object"},{"type":"null"}],"default":null,"title":"Logit Bias"},"user":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"User"},"seed":{"anyOf":[{"type":"integer"},{"type":"null"}],"default":null,"title":"Seed"},"tools":{"type":"array","description":"A list of tools the model may call. Currently only function tools are supported.","items":{"$ref":"#/components/schemas/Tool"}},"tool_choice":{"type":"string","enum":["auto","none","required"],"description":"Controls whether the model calls a tool. `auto` lets the model decide, `none` disables tool calls, `required` forces the model to call a tool."},"logprobs":{"anyOf":[{"type":"boolean"},{"type":"null"}],"default":null,"title":"Logprobs"},"top_logprobs":{"anyOf":[{"type":"integer"},{"type":"null"}],"default":null,"title":"Top Logprobs"},"functions":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}],"default":null,"title":"Functions"},"function_call":{"anyOf":[{"type":"string"},{"type":"null"}],"default":null,"title":"Function Call"}},"required":["model"],"title":"CompletionRequest","type":"object"}}}},"responses":{"200":{"description":"Chat completion result. When `stream: true`, the response is returned as `text/event-stream` instead of JSON, with OpenAI-style `data: {chunk}\\n\\n` framing terminated by `data: [DONE]\\n\\n`.","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ChatCompletionResponse"}}}},"400":{"description":"Invalid Format","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"401":{"description":"Authentication Fails","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"402":{"description":"Insufficient Balance","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"403":{"description":"Permission Denied","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"404":{"description":"Not Found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"422":{"description":"Invalid Parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"429":{"description":"Rate Limit Reached","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"500":{"description":"Server Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}},"503":{"description":"Server Overloaded","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}}}}}}}}
```

## The ChatCompletionContentPartImageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionContentPartImageParam":{"properties":{"image_url":{"$ref":"#/components/schemas/ImageURL"},"type":{"const":"image_url","title":"Type","type":"string"}},"required":["image_url","type"],"title":"ChatCompletionContentPartImageParam","type":"object"},"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"}}}}
```

## The ChatCompletionAssistantMessageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionAssistantMessageParam":{"properties":{"role":{"const":"assistant","title":"Role","type":"string"},"content":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Content"},"function_call":{"$ref":"#/components/schemas/FunctionCall"},"name":{"title":"Name","type":"string"},"tool_calls":{"items":{"$ref":"#/components/schemas/ChatCompletionMessageToolCallParam"},"title":"Tool Calls","type":"array"}},"required":["role"],"title":"ChatCompletionAssistantMessageParam","type":"object"},"FunctionCall":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"FunctionCall","type":"object"},"ChatCompletionMessageToolCallParam":{"properties":{"id":{"title":"Id","type":"string"},"function":{"$ref":"#/components/schemas/Function"},"type":{"const":"function","title":"Type","type":"string"}},"required":["id","function","type"],"title":"ChatCompletionMessageToolCallParam","type":"object"},"Function":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"Function","type":"object"}}}}
```

## The ChatCompletionContentPartTextParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionContentPartTextParam":{"properties":{"text":{"title":"Text","type":"string"},"type":{"const":"text","title":"Type","type":"string"}},"required":["text","type"],"title":"ChatCompletionContentPartTextParam","type":"object"}}}}
```

## The ChatCompletionFunctionMessageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionFunctionMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"name":{"title":"Name","type":"string"},"role":{"const":"function","title":"Role","type":"string"}},"required":["content","name","role"],"title":"ChatCompletionFunctionMessageParam","type":"object"},"ChatCompletionContentPartTextParam":{"properties":{"text":{"title":"Text","type":"string"},"type":{"const":"text","title":"Type","type":"string"}},"required":["text","type"],"title":"ChatCompletionContentPartTextParam","type":"object"},"ChatCompletionContentPartImageParam":{"properties":{"image_url":{"$ref":"#/components/schemas/ImageURL"},"type":{"const":"image_url","title":"Type","type":"string"}},"required":["image_url","type"],"title":"ChatCompletionContentPartImageParam","type":"object"},"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"}}}}
```

## The ChatCompletionMessageToolCallParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionMessageToolCallParam":{"properties":{"id":{"title":"Id","type":"string"},"function":{"$ref":"#/components/schemas/Function"},"type":{"const":"function","title":"Type","type":"string"}},"required":["id","function","type"],"title":"ChatCompletionMessageToolCallParam","type":"object"},"Function":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"Function","type":"object"}}}}
```

## The ChatCompletionSystemMessageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionSystemMessageParam":{"properties":{"content":{"title":"Content","type":"string"},"role":{"const":"system","title":"Role","type":"string"},"name":{"title":"Name","type":"string"}},"required":["content","role"],"title":"ChatCompletionSystemMessageParam","type":"object"}}}}
```

## The ChatCompletionToolMessageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionToolMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"role":{"const":"tool","title":"Role","type":"string"},"tool_call_id":{"title":"Tool Call Id","type":"string"}},"required":["content","role","tool_call_id"],"title":"ChatCompletionToolMessageParam","type":"object"},"ChatCompletionContentPartTextParam":{"properties":{"text":{"title":"Text","type":"string"},"type":{"const":"text","title":"Type","type":"string"}},"required":["text","type"],"title":"ChatCompletionContentPartTextParam","type":"object"},"ChatCompletionContentPartImageParam":{"properties":{"image_url":{"$ref":"#/components/schemas/ImageURL"},"type":{"const":"image_url","title":"Type","type":"string"}},"required":["image_url","type"],"title":"ChatCompletionContentPartImageParam","type":"object"},"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"}}}}
```

## The ChatCompletionUserMessageParam object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionUserMessageParam":{"properties":{"content":{"anyOf":[{"type":"string"},{"items":{"anyOf":[{"$ref":"#/components/schemas/ChatCompletionContentPartTextParam"},{"$ref":"#/components/schemas/ChatCompletionContentPartImageParam"}]},"type":"array"}],"title":"Content"},"role":{"const":"user","title":"Role","type":"string"},"name":{"title":"Name","type":"string"}},"required":["content","role"],"title":"ChatCompletionUserMessageParam","type":"object"},"ChatCompletionContentPartTextParam":{"properties":{"text":{"title":"Text","type":"string"},"type":{"const":"text","title":"Type","type":"string"}},"required":["text","type"],"title":"ChatCompletionContentPartTextParam","type":"object"},"ChatCompletionContentPartImageParam":{"properties":{"image_url":{"$ref":"#/components/schemas/ImageURL"},"type":{"const":"image_url","title":"Type","type":"string"}},"required":["image_url","type"],"title":"ChatCompletionContentPartImageParam","type":"object"},"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"}}}}
```

## The Function object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"Function":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"Function","type":"object"}}}}
```

## The FunctionCall object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"FunctionCall":{"properties":{"arguments":{"title":"Arguments","type":"string"},"name":{"title":"Name","type":"string"}},"required":["arguments","name"],"title":"FunctionCall","type":"object"}}}}
```

## The ImageURL object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ImageURL":{"properties":{"url":{"title":"Url","type":"string"},"detail":{"enum":["auto","low","high"],"title":"Detail","type":"string"}},"required":["url"],"title":"ImageURL","type":"object"}}}}
```

## The Tool object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"Tool":{"type":"object","required":["type","function"],"properties":{"type":{"type":"string","enum":["function"]},"function":{"type":"object","required":["name"],"properties":{"name":{"type":"string","description":"Name of the function to be called."},"description":{"type":"string","description":"Description of what the function does."},"parameters":{"type":"object","description":"JSON Schema object describing the function's parameters."}}}}}}}}
```

## The Error object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"Error":{"type":"object","properties":{"detail":{"type":"string","description":"Human-readable error message."}},"required":["detail"]}}}}
```

## The ChatCompletionResponse object

```json
{"openapi":"3.1.0","info":{"title":"Arcee Chat Completions API","version":"1.0.0"},"components":{"schemas":{"ChatCompletionResponse":{"type":"object","description":"OpenAI-compatible chat completion response. Standard fields (`id`, `object`, `created`, `model`, `choices`, `usage`) are returned; provider-specific extras are stripped before responding."}}}}
```


# Usage

## Get Usage Stats

> Get daily usage statistics for the API key's organization.\
> \
> Returns paginated daily usage statistics with optional filtering by date range,\
> model, provider, and source. Each record represents aggregated usage for a specific\
> user/model/provider/source combination on a given date.

```json
{"openapi":"3.1.0","info":{"title":"AFM API - External API","version":"0.1.0"},"paths":{"/api/v1/usage/stats":{"get":{"tags":["Usage"],"summary":"Get Usage Stats","description":"Get daily usage statistics for the API key's organization.\n\nReturns paginated daily usage statistics with optional filtering by date range,\nmodel, provider, and source. Each record represents aggregated usage for a specific\nuser/model/provider/source combination on a given date.","operationId":"get_usage_stats_api_v1_usage_stats_get","parameters":[{"name":"start_date","in":"query","required":false,"schema":{"type":"string","format":"date-time"},"description":"Start date for filtering (ISO format)"},{"name":"end_date","in":"query","required":false,"schema":{"type":"string","format":"date-time"},"description":"End date for filtering (ISO format)"},{"name":"model","in":"query","required":false,"schema":{"type":"string"},"description":"Filter by specific model"},{"name":"model_provider","in":"query","required":false,"schema":{"type":"string"},"description":"Filter by model provider"},{"name":"source","in":"query","required":false,"schema":{"type":"string"},"description":"Filter by source (api or app)"},{"name":"limit","in":"query","required":false,"schema":{"type":"integer","maximum":1000,"minimum":1,"description":"Number of records to return","default":100,"title":"Limit"},"description":"Number of records to return"},{"name":"offset","in":"query","required":false,"schema":{"type":"integer","minimum":0,"description":"Number of records to skip","default":0,"title":"Offset"},"description":"Number of records to skip"}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/UsageStatsResponse"},"title":"Response Get Usage Stats Api V1 Usage Stats Get"}}}},"401":{"description":"Authentication Fails","content":{"application/json":{}}},"422":{"description":"Invalid Parameters","content":{"application/json":{}}},"500":{"description":"Server Error","content":{"application/json":{}}}}}}},"components":{"schemas":{"UsageStatsResponse":{"properties":{"id":{"type":"string","format":"uuid","title":"Id"},"user_id":{"type":"string","format":"uuid","title":"User Id"},"org_id":{"type":"string","format":"uuid","title":"Org Id"},"email":{"type":"string"},"model":{"type":"string","title":"Model"},"source":{"type":"string","title":"Source"},"input_tokens":{"type":"integer","title":"Input Tokens"},"output_tokens":{"type":"integer","title":"Output Tokens"},"total_tokens":{"type":"integer","title":"Total Tokens"},"total_requests":{"type":"integer","title":"Total Requests"},"date":{"type":"string","format":"date-time","title":"Date"},"created_at":{"type":"string","format":"date-time","title":"Created At"}},"type":"object","required":["id","user_id","org_id","email","model","source","input_tokens","output_tokens","total_tokens","total_requests","date","created_at"],"title":"UsageStatsResponse","description":"API response model for daily usage statistics."}}}}
```

## Get Usage Summary

> Get aggregated usage summary statistics for the API key's organization.\
> \
> Returns high-level metrics including total tokens, requests, unique users,\
> models, providers, and breakdown by API vs app usage.

```json
{"openapi":"3.1.0","info":{"title":"AFM API - External API","version":"0.1.0"},"paths":{"/api/v1/usage/summary":{"get":{"tags":["Usage"],"summary":"Get Usage Summary","description":"Get aggregated usage summary statistics for the API key's organization.\n\nReturns high-level metrics including total tokens, requests, unique users,\nmodels, providers, and breakdown by API vs app usage.","operationId":"get_usage_summary_api_v1_usage_summary_get","parameters":[{"name":"start_date","in":"query","required":false,"schema":{"type":"string","format":"date-time"},"description":"Start date for summary (ISO format)"},{"name":"end_date","in":"query","required":false,"schema":{"type":"string","format":"date-time"},"description":"End date for summary (ISO format)"}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"$ref":"#/components/schemas/api_external__usage__models__UsageSummaryResponse"}}}},"401":{"description":"Authentication Fails","content":{"application/json":{}}},"422":{"description":"Invalid Parameters","content":{"application/json":{}}},"500":{"description":"Server Error","content":{"application/json":{}}}}}}},"components":{"schemas":{"api_external__usage__models__UsageSummaryResponse":{"properties":{"total_input_tokens":{"type":"integer","title":"Total Input Tokens"},"total_output_tokens":{"type":"integer","title":"Total Output Tokens"},"total_tokens":{"type":"integer","title":"Total Tokens"},"total_requests":{"type":"integer","title":"Total Requests"},"unique_users":{"type":"integer","title":"Unique Users"},"unique_models":{"type":"integer","title":"Unique Models"},"unique_providers":{"type":"integer","title":"Unique Providers"},"api_usage":{"type":"integer","title":"Api Usage"},"app_usage":{"type":"integer","title":"App Usage"},"date_range_days":{"type":"integer","title":"Date Range Days"}},"type":"object","required":["total_input_tokens","total_output_tokens","total_tokens","total_requests","unique_users","unique_models","unique_providers","api_usage","app_usage","date_range_days"],"title":"UsageSummaryResponse","description":"API response model for aggregated usage summary."}}}}
```


# Models

## Get Models

> Get all models according to OpenRouter provider spec.

```json
{"openapi":"3.1.0","info":{"title":"AFM API - External API","version":"0.1.0"},"paths":{"/api/v1/models":{"get":{"tags":["Models"],"summary":"Get Models","description":"Get all models according to OpenRouter provider spec.","operationId":"get_models_api_v1_models_get","responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ModelsListResponse"}}}},"401":{"description":"Authentication Fails","content":{"application/json":{}}},"422":{"description":"Invalid Parameters","content":{"application/json":{}}},"500":{"description":"Server Error","content":{"application/json":{}}}}}}},"components":{"schemas":{"ModelsListResponse":{"properties":{"data":{"items":{"$ref":"#/components/schemas/APIModelResponse"},"type":"array","title":"Data"}},"type":"object","required":["data"],"title":"ModelsListResponse"},"APIModelResponse":{"properties":{"id":{"type":"string","title":"Id"},"hugging_face_id":{"type":"string","title":"Hugging Face Id"},"name":{"type":"string","title":"Name"},"created":{"type":"integer","title":"Created"},"input_modalities":{"items":{"type":"string"},"type":"array","title":"Input Modalities"},"output_modalities":{"items":{"type":"string"},"type":"array","title":"Output Modalities"},"quantization":{"type":"string","title":"Quantization"},"context_length":{"type":"integer","title":"Context Length"},"max_output_length":{"type":"integer","title":"Max Output Length"},"pricing":{"$ref":"#/components/schemas/PricingResponse"},"supported_sampling_parameters":{"items":{"type":"string"},"type":"array","title":"Supported Sampling Parameters"},"supported_features":{"items":{"type":"string"},"type":"array","title":"Supported Features"},"description":{"type":"string"},"openrouter":{"$ref":"#/components/schemas/OpenRouterMetadata"},"datacenters":{"items":{"$ref":"#/components/schemas/DatacenterResponse"},"type":"array"}},"type":"object","required":["id","hugging_face_id","name","created","input_modalities","output_modalities","quantization","context_length","max_output_length","pricing","supported_sampling_parameters","supported_features"],"title":"APIModelResponse"},"PricingResponse":{"properties":{"prompt":{"type":"string","title":"Prompt"},"completion":{"type":"string","title":"Completion"},"image":{"type":"string","title":"Image"},"request":{"type":"string","title":"Request"},"input_cache_reads":{"type":"string","title":"Input Cache Reads"},"input_cache_writes":{"type":"string","title":"Input Cache Writes"}},"type":"object","required":["prompt","completion","image","request","input_cache_reads","input_cache_writes"],"title":"PricingResponse"},"OpenRouterMetadata":{"properties":{"slug":{"type":"string","title":"Slug"}},"type":"object","required":["slug"],"title":"OpenRouterMetadata"},"DatacenterResponse":{"properties":{"country_code":{"type":"string","title":"Country Code"}},"type":"object","required":["country_code"],"title":"DatacenterResponse"}}}}
```


# Streaming Messages

Streaming messages enables users to receive the model's responses in chunks, rather than waiting for the compete response to be generated. This is done by incrementally transmitting chunks of content to the user. This feature enables users to:

* **Instant Responses:** Content appears progressively without waiting for the full output.
* **Enhanced User Experience:** Minimizes waiting time and offers immediate feedback.
* **Lower Latency:** Information is delivered as it’s created, reducing perceived delay.
* **Dynamic Handling:** Allows real-time processing and display while data is being received.

#### Parameter Settings

To enable streaming, set `stream=True`.

#### Code <a href="#core-parameter-description" id="core-parameter-description"></a>

{% tabs %}
{% tab title="curl" %}

```bash
curl https://api.arcee.ai/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "MODEL_NAME",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is the difference between SLMs and LLMs?"
      }
    ],
    "stream":true
  }'
```

{% endtab %}

{% tab title="Python" %}

```python
from openai import OpenAI 
client = OpenAI(
    api_key="YOUR_API_KEY", # Your Arcee API Platform API Key
    base_url="https://api.arcee.ai/api/v1/")

response = client.chat.completions.create(
    model="MODEL_NAME", 
    messages=[ 
        {"role":"system", "content":"You are a helpful assistant"},
        {"role":"user", "content": "What is the difference between SLMs and LLMs?"},
    ],
    stream=True
)

for chunk in response:
    if chunk and hasattr(chunk, 'choices') and chunk.choices:
        delta = chunk.choices[0].delta
        if hasattr(delta, 'content') and delta.content is not None:
            print(delta.content, end="", flush=True)
```

{% endtab %}
{% endtabs %}

#### Response Example

```
data: {"choices": [{"delta": {"content": "Hello"}}]}
data: {"choices": [{"delta": {"content": "!"}}]}
data: {"choices": [{"delta": {"content": " How"}}]}
...
data: {"choices": [{"delta": {"content": " today"}}]}
data: {"choices": [{"delta": {"content": "?"}}]}
data: [DONE]
```


# Multi-Turn Conversations

Multi-Turn Conversations enable models to keep context from previous messages in a conversation providing a more in-depth experience. This guide will show how to use Arcee AI models through Arcee Platform for multi-turn conversations.

The Arcee AI `/chat/completions` API is a "stateless" API, meaning the server does not record the context of the user's requests. Therefore, the user must **concatenate all previous conversation history** and pass it to the chat API with each request.

The following Python code demonstrates how to easily concatenate context to achieve multi-turn conversations.

```python
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY", 
    base_url="https://api.arcee.ai/api/v1"
)

# Round 1
messages = [{"role": "user", "content": "What is a small language model?"}]
response = client.chat.completions.create(
    model="MODEL_NAME",
    messages=messages
)

answer = {"role": "assistant", "content": response.choices[0].message.content}
messages.append(answer)
print(f"Messages Round 1: {messages}")

# Round 2
messages.append({"role": "user", "content": "How do they differ from LLMs?"})
response = client.chat.completions.create(
    model="arcee-ai/trinity-mini-thinking",
    messages=messages
)

answer = {"role": "assistant", "content": response.choices[0].message.content}
messages.append(answer)
print(f"Messages Round 2: {messages}")
```

***

In the **first round** of the request, the `messages` passed to the API are:

```
[
    {"role": "user", "content": "What is a small language model?"}
]
```

In the **second round** of the request:

1. Add the model's output from the first round to the end of the `messages`.
2. Add the new question to the end of the `messages`.

The `messages` ultimately passed to the API are:

```
[
    {"role": "user", "content": "What is a small language model?"},
    {"role": "assistant", "content": "A small language model refers to a model that has a relatively limited number of parameters compared to other large language models. Here's a detailed explanation:\n\nKey Characteristics:\n- Smaller-scale model architecture\n- Fewer parameters (typically tens of billions instead of hundreds of billions)\n- Reduced computational resources needed\n- Faster training and inference times\n- Potentially lower energy consumption\n\nUse Cases:\n1. Resource-constrained environments\n2. Real-time applications\n3. Embedded systems\n4. Low-power devices\n5. Situational-specific needs\n\nExamples include models like:\n- Duchowny et al.'s DLVM\n- Chan et al.'s SHOORN\n- Salimon's ALma (a mid-scale model)\n\nThese smaller models strike a balance between performance and efficiency, often making them suitable for specific applications where resource limitations are a factor."},
    {"role": "user", "content": "How do they differ from LLMs?"}
]
```


# Function Calling

{% hint style="info" %}
Function calling lets models use APIs or functions to perform tasks, get real-time data, and respond intelligently to user input.
{% endhint %}

Function (tool) calling allows models to invoke predefined functions and APIs so they can take actions, fetch real-time data, and interact with external systems. This is useful for assistants, chat applications, and agentic systems that must act on user input.

Tools or functions expose specific capabilities through a name, description, and parameter schema. When tools are included in a request, the model can decide when to call them.

### Tool Calls

A tool call is produced when the model determines that using one of the available tools is required. Here are key parameters to pass when calling the model related to function/tool calling:&#x20;

* `tools`  Defines the functions the model can call, including their names, descriptions, and parameter schemas.
* `tool_choice` Controls whether the model should decide when to call a tool. The supported value is `auto`, which lets the model choose based on the prompt

### Tool Call Outputs

After your system executes the tool with the model-provided arguments, the result is returned to the model to generate the final response.

* `tool_calls`Contains the list of tool calls returned by the model. Each entry includes the tool name, the arguments the model generated, and an ID used to link your tool output back to the call.
* `function.name` The name of the tool or function the model selected.
* `function.arguments` A JSON string with the arguments the model generated for the tool.
* `id` Unique identifier for the tool call, used to match tool outputs to the corresponding request.

***

## Example: Function Calling with Custom Python Functions

This example demonstrates how to use function calling with an OpenAI-compatible model to **fetch real-time financial data** using the `yfinance` library. The model can retrieve stock prices, CEO names, and business summaries for publicly traded companies by invoking custom Python functions.

Before running the example, make sure the following Python packages are installed:

```python
pip install -qU httpx[http2] yfinance openai
```

* `httpx` is an HTTP client for Python that provides asynchronous support.
* The `[http2]` extra enables HTTP/2 support, which improves efficiency in communication with APIs.

<details>

<summary>Expand for Example Code</summary>

```python
import json
import yfinance as yf
from openai import OpenAI
import httpx

# Initialize client
endpoint = "https://api.arcee.ai/api/v1"
model = "trinity-nano-6b"
api_key = "<YOUR API KEY HERE>"

client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
    http_client=httpx.Client(http2=True)
)

# Define stock price function
def get_stock_price(company_name: str, stock_symbol: str) -> dict:
    """Get the last closing price of a company's stock"""
    try:
        stock = yf.Ticker(stock_symbol)
        hist = stock.history(period="1d")

        if hist.empty:
            return {
                "error": f"No price data found for {company_name} ({stock_symbol})."
            }

        price = float(hist["Close"].iloc[-1])
        return {"price": price}

    except Exception as e:
        return {"error": str(e)}

# Define CEO name function
def get_ceo_name(company_name: str, stock_symbol: str) -> dict:
    """Get the CEO of the company"""
    try:
        info = yf.Ticker(stock_symbol).info

        officers = info.get("companyOfficers", [])
        if not officers:
            return {"error": "No officer data available."}

        ceo = officers[0]
        return {
            "ceo": ceo.get("name", "Unknown"),
            "title": ceo.get("title", "Unknown")
        }

    except Exception as e:
        return {"error": str(e)}

# Define company summary function
def get_company_summary(company_name: str, stock_symbol: str) -> dict:
    """Get a company's business summary"""
    try:
        info = yf.Ticker(stock_symbol).info
        summary = info.get("longBusinessSummary")
        if not summary:
            return {"error": "No summary available."}

        return {"summary": summary}

    except Exception as e:
        return {"error": str(e)}

# Define function tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_stock_price",
            "description": "Get the last closing price of a company's stock",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "Company name, e.g.: Apple, Microsoft, Chipotle"
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "Stock ticker symbol, e.g.: AAPL, MSFT, CMG"
                    }
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_ceo_name",
            "description": "Get the CEO of the company",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "Company name, e.g.: Apple, Microsoft, Chipotle"
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "Stock ticker symbol, e.g.: AAPL, MSFT, CMG"
                    }
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_company_summary",
            "description": "Get a company's business summary",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "Company name, e.g.: Apple, Microsoft, Chipotle"
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "Stock ticker symbol, e.g.: AAPL, MSFT, CMG"
                    }
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    }
]

# Example Query
user_prompt = "What's the last closing price of Chipotle stock?"

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "user", "content": user_prompt}
    ],
    tools=tools,
    tool_choice="auto"
)

# Handle function calls
message = response.choices[0].message
messages = [{"role": "user", "content": user_prompt}]
messages.append(message)

if message.tool_calls:
    for tool_call in message.tool_calls:
        # Parse parameters
        args = json.loads(tool_call.function.arguments)
        if isinstance(args, str):
            args = json.loads(args)

        print(f"Calling {tool_call.function.name} with arguments: {args}")

        # Execute the Python functions
        if tool_call.function.name == "get_stock_price":
            result = get_stock_price(args.get("company_name"), args.get("stock_symbol"))
        elif tool_call.function.name == "get_ceo_name":
            result = get_ceo_name(args.get("company_name"), args.get("stock_symbol"))
        elif tool_call.function.name == "get_company_summary":
            result = get_company_summary(args.get("company_name"), args.get("stock_symbol"))
        else:
            result = {"error": f"Unknown function: {tool_call.function.name}"}

        print(f"Tool result: {result}")

        # Return function result to model
        messages.append({
            "role": "tool",
            "content": json.dumps(result, ensure_ascii=False),
            "tool_call_id": tool_call.id
        })

    # Final model call
    final_response = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
        tool_choice="none"
    )

    print("Final output:")
    print(final_response.choices[0].message.content)

else:
    if message.content:
        print(message.content)
    else:
        print("No content in response")



```

</details>

### Initializing the Client - Optimizing with Httpx

```python
endpoint = "https://api.arcee.ai/api/v1"
model = "YOUR_MODEL_NAME"
api_key="API_KEY"
client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
    http_client=httpx.Client(http2=True)
)
```

* **`http_client=httpx.Client(http2=True)`**:
  * Configures the HTTP client with HTTP/2 support for faster, more efficient communication.

### Custom functions for stock research

```python
def get_stock_price(company_name: str, stock_symbol: str) -> dict:
    try:
        stock = yf.Ticker(stock_symbol)
        hist = stock.history(period="1d")
        if hist.empty:
            return {"error": f"No price data found for {company_name} ({stock_symbol})."}
        return {"price": float(hist["Close"].iloc[-1])}
    except Exception as e:
        return {"error": str(e)}
```

```python
def get_ceo_name(company_name: str, stock_symbol: str) -> dict:
    try:
        info = yf.Ticker(stock_symbol).info
        officers = info.get("companyOfficers", [])
        if not officers:
            return {"error": "No officer data available."}
        ceo = officers[0]
        return {
            "ceo": ceo.get("name", "Unknown"),
            "title": ceo.get("title", "Unknown")
        }
    except Exception as e:
        return {"error": str(e)}
```

```python
def get_company_summary(company_name: str, stock_symbol: str) -> dict:
    try:
        info = yf.Ticker(stock_symbol).info
        summary = info.get("longBusinessSummary")
        if not summary:
            return {"error": "No summary available."}
        return {"summary": summary}
    except Exception as e:
        return {"error": str(e)}
```

<figure><img src="/files/qbk0daGrZoBLQCTOpmgL" alt=""><figcaption></figcaption></figure>

### Register the functions as tools

```python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_stock_price",
            "description": "Get the last closing price of a company's stock",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {"type": "string"},
                    "stock_symbol": {"type": "string"}
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_ceo_name",
            "description": "Get the CEO of the company",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {"type": "string"},
                    "stock_symbol": {"type": "string"}
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_company_summary",
            "description": "Get a company's business summary",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {"type": "string"},
                    "stock_symbol": {"type": "string"}
                },
                "required": ["company_name", "stock_symbol"]
            }
        }
    }
]

```

### Step 5: Creating the Initial Model Call (Function Calling)

By setting `tool_choice="auto"`, the model will decide whether to call a tool and which to call, or answer directly.

```python
user_prompt = "What's the last closing price of Chipotle stock?"

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "user", "content": user_prompt}
    ],
    tools=tools,
    tool_choice="auto"
)
```

At this stage, the model responds either with:

* a function/tool call including arguments, or
* a normal text response (if no function is needed).

### Handling Tool Calls and Returning a Response

After the user prompt is sent, the model may respond with one or more tool calls. This step processes those tool calls, executes the corresponding functions locally, and sends the results back to the model to complete the conversation.

```python

# Prepare message history
message = response.choices[0].message
messages = [{"role": "user", "content": user_prompt}, message]

# Execute tool calls if present
if message.tool_calls:
    for tool_call in message.tool_calls:
        args = json.loads(tool_call.function.arguments)
        if isinstance(args, str):  # Handle double-encoding
            args = json.loads(args)

        # Dispatch the correct tool
        if tool_call.function.name == "get_stock_price":
            result = get_stock_price(args.get("company_name"), args.get("stock_symbol"))
        elif tool_call.function.name == "get_ceo_name":
            result = get_ceo_name(args.get("company_name"), args.get("stock_symbol"))
        elif tool_call.function.name == "get_company_summary":
            result = get_company_summary(args.get("company_name"), args.get("stock_symbol"))
        else:
            result = {"error": f"Unknown function: {tool_call.function.name}"}

        messages.append({
            "role": "tool",
            "content": json.dumps(result, ensure_ascii=False),
            "tool_call_id": tool_call.id
        })

    # Final completion with tool results
    final_response = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
        tool_choice="auto"
    )

    print(final_response.choices[0].message.content)

else:
    # If no tools were called, return model response directly
    if message.content:
        print(message.content)
    else:
        print("No content in response")
```

### Example Output

```python
Calling get_stock_price with arguments: {'company_name': 'Chipotle', 'stock_symbol': 'CMG'}
Tool result: {'price': 31.0}
Final output: The last closing price of Chipotle stock was **$31.0**.
```


# Structured Outputs

Structured outputs allow you to control and define the format of a model’s response using a schema. This makes it easy to enforce consistency, guarantee valid JSON, and parse results programmatically. Below we show two approaches: simple JSON output using `json_object` and fully validated structured output using Pydantic classes.

***

#### Simple JSON  Example

JSON output is ideal for simple structured responses where you only need the model to return well-formed JSON.

{% hint style="info" %}
To enable JSON Output, users should:

1. Set the `response_format` parameter to `{'type': 'json_object'}`.
2. Include the word "json" in the system or user prompt, and provide an example of the desired JSON format to guide the model in outputting valid JSON.
3. Set the `max_tokens` parameter reasonably to prevent the JSON string from being truncated midway.
   {% endhint %}

```python
import json
from openai import OpenAI

client = OpenAI(
    api_key="<YOUR_API_KEY>", 
    base_url="https://api.arcee.ai/api/v1"
)

system_prompt = """
The user will provide some exam text. Please parse the "question" and "answer" and output them in JSON format. 

EXAMPLE INPUT: 
Which is the highest mountain in the world? Mount Everest.

EXAMPLE JSON OUTPUT:
{
    "question": "What is the capital of France?",
    "answer": "Paris"
}
"""

user_prompt = "What is the capital of Italy?"

messages = [{"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}]

response = client.chat.completions.create(
    model="MODEL_NAME",
    messages=messages,
    response_format={
        'type': 'json_object'
    }
)

print(json.loads(response.choices[0].message.content))
```


# Reasoning Traces

## Reasoning Traces

### Handling Reasoning Traces in Multi-Turn Conversations

Trinity-Large-Thinking generates explicit chain-of-thought reasoning inside `<think>...</think>` blocks before producing its response. When served via vLLM, these blocks are parsed into a dedicated `reasoning_content` field in the API response, separate from `content` and `tool_calls`.

Preserving reasoning across turns is critical for reliable multi-step tool use and agentic workflows. This guide covers how to do it correctly.

### How reasoning flows through a conversation

```
Turn 1: User sends message
         ↓
         Model generates: <think>reasoning</think> content + tool_calls
         ↓
         vLLM parses into: { reasoning_content, content, tool_calls }
         ↓
Turn 2: Client appends full assistant message (reasoning_content + content + tool_calls)
         Client appends tool result
         Client sends updated history
         ↓
         Chat template re-wraps reasoning_content in <think>...</think> during tokenization
         ↓
         Model sees prior chain-of-thought → generates next step correctly
```

If the client drops `reasoning_content` between turns, the model loses its prior chain-of-thought and may produce malformed output.

### Quick reference: assistant message shape

For assistant turns that call tools, include all three fields when appending back to history:

```json
{
  "role": "assistant",
  "content": "",
  "reasoning_content": "The user wants to cancel. I need their customer_id first, so I'll look them up by email.",
  "tool_calls": [
    {
      "id": "call-1",
      "type": "function",
      "function": {
        "name": "get_customer_by_email",
        "arguments": "{\"email\": \"jane@example.com\"}"
      }
    }
  ]
}
```

| Field               | Required             | Notes                                                                                     |
| ------------------- | -------------------- | ----------------------------------------------------------------------------------------- |
| `content`           | Yes                  | Use `""` if the model returned null. Never pass `null`.                                   |
| `reasoning_content` | Strongly recommended | From the API response's `reasoning_content` field. `reasoning` is also accepted on input. |
| `tool_calls`        | Conditional          | Include when the model made tool calls (omit for final non-tool assistant turns).         |

### Field name: `reasoning` vs `reasoning_content`

The naming can be confusing because different layers use different names:

| Layer                                | Field name                                                   | Direction               |
| ------------------------------------ | ------------------------------------------------------------ | ----------------------- |
| Arcee API / vLLM API response        | `reasoning_content`                                          | Output                  |
| OpenAI Python SDK                    | `reasoning_content`                                          | Output (attribute name) |
| Arcee API input (assistant messages) | `reasoning_content` (also accepts `reasoning`)               | Input                   |
| Self-hosted vLLM input               | `reasoning` (some versions don't accept `reasoning_content`) | Input                   |
| Chat template (Jinja)                | both `reasoning` and `reasoning_content`                     | Input                   |

The safe rule: use `reasoning_content` when constructing assistant messages for input. This matches what the API returns, so you can pass it straight back. The backend also accepts `reasoning` on input and converts it automatically.

> **Note for self-hosted vLLM users:** Some vLLM versions only read `reasoning` on input messages ([vllm#38488](https://github.com/vllm-project/vllm/issues/38488)). If you're hitting vLLM directly (not through Arcee's API), you may need to map `reasoning_content` → `reasoning` on assistant messages.

### Python implementation

#### Installation

```bash
pip install openai
```

#### Complete agentic loop

```python
import json
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-api-key"
)

MODEL = "arcee-ai/Trinity-Large-Thinking"

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_customer_by_email",
            "description": "Look up a customer by email address.",
            "parameters": {
                "type": "object",
                "properties": {
                    "email": {"type": "string", "description": "Customer email address"}
                },
                "required": ["email"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "cancel_subscription",
            "description": "Cancel a customer's subscription. Requires customer_id.",
            "parameters": {
                "type": "object",
                "properties": {
                    "customer_id": {"type": "string"},
                    "reason": {"type": "string"}
                },
                "required": ["customer_id"]
            }
        }
    }
]


def execute_tool(name: str, arguments: str) -> str:
    """Replace with your actual tool implementations."""
    args = json.loads(arguments)
    if name == "get_customer_by_email":
        return json.dumps({"customer_id": "C2001", "name": "Jane Doe", "plan": "Premium"})
    elif name == "cancel_subscription":
        return json.dumps({"success": True, "message": f"Cancelled for {args['customer_id']}"})
    return json.dumps({"error": "Unknown tool"})


def build_assistant_message(msg) -> dict:
    """
    Build an assistant message dict that preserves reasoning for the next turn.

    Key details:
    - Pass reasoning_content straight back (matches what the API returns)
    - Use "" instead of None for content (avoids tokenization issues)
    - Preserve the full tool_calls array
    """
    assistant_msg = {
        "role": "assistant",
        "content": msg.content or "",  # never null
    }

    # Preserve reasoning — critical for multi-turn
    reasoning = getattr(msg, "reasoning_content", None) or getattr(msg, "reasoning", None)
    if reasoning:
        assistant_msg["reasoning_content"] = reasoning

    # Preserve tool calls
    if msg.tool_calls:
        assistant_msg["tool_calls"] = [
            {
                "id": tc.id,
                "type": "function",
                "function": {
                    "name": tc.function.name,
                    "arguments": tc.function.arguments
                }
            }
            for tc in msg.tool_calls
        ]

    return assistant_msg


def run_agent(user_message: str, system_prompt: str = "You are a helpful customer service agent."):
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]

    max_steps = 10  # safety limit

    for step in range(max_steps):
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=tools,
            tool_choice="auto",
            temperature=0,
            max_tokens=1000
        )

        msg = response.choices[0].message

        # Append full assistant message with reasoning preserved
        messages.append(build_assistant_message(msg))

        # No tool calls → final response
        if not msg.tool_calls:
            return msg.content or ""

        # Execute each tool call and append results
        for tc in msg.tool_calls:
            result = execute_tool(tc.function.name, tc.function.arguments)
            messages.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "content": result
            })

    raise RuntimeError("Agent exceeded maximum steps")


# Usage
response = run_agent("I want to cancel my subscription. My email is jane@example.com")
print(response)
```

### TypeScript implementation

#### Installation

```bash
npm install openai
```

#### Complete agentic loop

```typescript
import OpenAI from "openai";
import type {
  ChatCompletionMessageParam,
  ChatCompletionTool,
} from "openai/resources/chat/completions";

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "your-api-key",
});

const MODEL = "arcee-ai/Trinity-Large-Thinking";

const tools: ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "get_customer_by_email",
      description: "Look up a customer by email address.",
      parameters: {
        type: "object",
        properties: {
          email: { type: "string", description: "Customer email address" },
        },
        required: ["email"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "cancel_subscription",
      description: "Cancel a customer's subscription. Requires customer_id.",
      parameters: {
        type: "object",
        properties: {
          customer_id: { type: "string" },
          reason: { type: "string" },
        },
        required: ["customer_id"],
      },
    },
  },
];

async function executeTool(name: string, args: string): Promise<string> {
  // Replace with your actual tool implementations
  const parsed = JSON.parse(args);
  switch (name) {
    case "get_customer_by_email":
      return JSON.stringify({ customer_id: "C2001", name: "Jane Doe", plan: "Premium" });
    case "cancel_subscription":
      return JSON.stringify({ success: true, message: `Cancelled for ${parsed.customer_id}` });
    default:
      return JSON.stringify({ error: "Unknown tool" });
  }
}

/**
 * Build an assistant message that preserves reasoning for the next turn.
 *
 * Key details:
 * - Pass reasoning_content straight back (matches what the API returns)
 * - Use "" instead of null for content
 * - Preserve the full tool_calls array
 */
function buildAssistantMessage(
  msg: OpenAI.Chat.Completions.ChatCompletionMessage
): ChatCompletionMessageParam {
  const assistantMsg: Record<string, unknown> = {
    role: "assistant",
    content: msg.content ?? "", // never null
  };

  // Preserve reasoning — critical for multi-turn
  const reasoning = (msg as any).reasoning_content ?? (msg as any).reasoning;
  if (reasoning) {
    assistantMsg.reasoning_content = reasoning;
  }

  // Preserve tool calls
  if (msg.tool_calls?.length) {
    assistantMsg.tool_calls = msg.tool_calls.map((tc) => ({
      id: tc.id,
      type: "function" as const,
      function: {
        name: tc.function.name,
        arguments: tc.function.arguments,
      },
    }));
  }

  return assistantMsg as ChatCompletionMessageParam;
}

async function runAgent(
  userMessage: string,
  systemPrompt = "You are a helpful customer service agent."
): Promise<string> {
  const messages: ChatCompletionMessageParam[] = [
    { role: "system", content: systemPrompt },
    { role: "user", content: userMessage },
  ];

  const maxSteps = 10; // safety limit

  for (let step = 0; step < maxSteps; step++) {
    const response = await client.chat.completions.create({
      model: MODEL,
      messages,
      tools,
      tool_choice: "auto",
      temperature: 0,
      max_tokens: 1000,
    });

    const msg = response.choices[0].message;

    // Append full assistant message with reasoning preserved
    messages.push(buildAssistantMessage(msg));

    // No tool calls → final response
    if (!msg.tool_calls?.length) {
      return msg.content ?? "";
    }

    // Execute each tool call and append results
    for (const tc of msg.tool_calls) {
      const result = await executeTool(tc.function.name, tc.function.arguments);
      messages.push({
        role: "tool",
        tool_call_id: tc.id,
        content: result,
      });
    }
  }

  throw new Error("Agent exceeded maximum steps");
}

// Usage
const response = await runAgent(
  "I want to cancel my subscription. My email is jane@example.com"
);
console.log(response);
```

### OpenRouter integration

When using Trinity through [OpenRouter](https://openrouter.ai/), reasoning is returned in a `reasoning_details` object (OpenRouter's unified reasoning shape). For multi-turn conversations, pass `reasoning_details` back as-is on assistant messages — OpenRouter handles the model-specific upstream translation automatically.

#### Debugging upstream requests

To verify that reasoning is being sent upstream correctly, enable echo mode:

```json
{
  "debug": { "echo_upstream_body": true }
}
```

See [OpenRouter debugging docs](https://openrouter.ai/docs/api/reference/errors-and-debugging#debugging) for details.

### Common pitfalls

#### 1. `xml_in_reasoning` — tool call XML inside reasoning field

**Symptom:** The response has `tool_calls: []` (empty) and `reasoning_content` contains raw XML like `<function=get_details_by_id><parameter=id>L1001</parameter>...`

**Cause:** The previous assistant turn likely lost reasoning context (and/or had invalid message shape), so the model generated the tool call inside its thinking block instead of as structured output.

**Fix:** Ensure every assistant message in the conversation history includes `reasoning_content` from the prior API response.

```
❌ Broken — no reasoning, null content
{"role": "assistant", "content": null, "tool_calls": [...]}

✅ Fixed — reasoning preserved, content non-null
{"role": "assistant", "content": "", "reasoning_content": "...", "tool_calls": [...]}
```

#### 2. `reasoning_content` ignored on self-hosted vLLM

**Symptom:** You're passing `reasoning_content` on assistant messages to a self-hosted vLLM instance, but the model behaves as if reasoning is missing.

**Cause:** Some vLLM versions only read `reasoning` (not `reasoning_content`) from input messages ([vllm#38488](https://github.com/vllm-project/vllm/issues/38488)). This does **not** affect Arcee's hosted API, which accepts both.

**Fix:** If you're hitting vLLM directly, map `reasoning_content` → `reasoning` on input:

```python
# Self-hosted vLLM workaround
assistant_msg["reasoning"] = msg.reasoning_content
```

#### 3. `content: null` on assistant tool-call turns

**Symptom:** Degraded tool call quality or malformed output on subsequent turns.

**Cause:** When the model makes a tool call, `content` may be `null` in the API response. Passing `null` back can contribute to malformed follow-up behavior in some integrations.

**Fix:** Normalize `null` to empty string:

```python
"content": msg.content or ""   # Python
```

```typescript
content: msg.content ?? ""     // TypeScript
```

#### 4. Missing vLLM serving flags

**Symptom:** Reasoning leaks into `content` with visible `<think>...</think>` tags, or tool calls appear as raw XML instead of structured `tool_calls`.

**Fix:** Ensure vLLM is started with the correct flags:

```bash
vllm serve arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

| Flag                             | Purpose                                                       |
| -------------------------------- | ------------------------------------------------------------- |
| `--reasoning-parser deepseek_r1` | Separates `<think>` blocks into the `reasoning_content` field |
| `--enable-auto-tool-choice`      | Enables tool call parsing                                     |
| `--tool-call-parser qwen3_coder` | Parses tool calls into structured `tool_calls` array          |

Note: vLLM versions before 0.18 may also require `--enable-reasoning`. If the flag is not recognized, `--reasoning-parser` alone is sufficient.

### vLLM serving reference

#### Minimal serving command

```bash
vllm serve arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

#### Production serving command (example only)

```bash
vllm serve arcee-ai/Trinity-Large-Thinking \
  --served-model-name arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --tensor-parallel-size <set-for-your-hardware> \
  --max-model-len <set-for-your-workload> \
  --gpu-memory-utilization <set-for-your-cluster-policy> \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key "$API_KEY" \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

#### Context length guidance

Set `--max-model-len` based on your real conversation lengths, tool-chain depth, and GPU memory budget. Higher values pre-allocate more KV cache at startup. A practical approach is to start conservatively, monitor `truncation/finish_reason: "length"`, then scale up incrementally.


# Hardware Prerequisites

The table below outlines the minimum and recommended memory requirements for running each Trinity model at different precision levels. In general, **4-bit and 8-bit quantization are ideal for most use cases**, offering a strong balance between performance and memory efficiency. These lower precisions make it easier to deploy on a wide range of hardware, including CPUs.&#x20;

| **Model**              | **Precision** | **Minimum RAM** | **Recommended RAM** |
| ---------------------- | ------------- | --------------- | ------------------- |
| **Trinity-Nano-6B**    | 4-bit         | 4 GB            | 6 GB                |
| **Trinity-Nano-6B**    | 8-bit         | 8 GB            | 8 GB                |
| **Trinity-Nano-6B**    | bf16          | 14 GB           | 14 GB               |
| **Trinity-Mini-26B**   | 4-bit         | 16 GB           | 24 GB               |
| **Trinity-Mini-26B**   | 8-bit         | 32 GB           | 32 GB               |
| **Trinity-Mini-26B**   | bf16          | 64 GB           | 64 GB               |
| **Trinity-Large-400B** | 4-bit         | 224GB           | 336GB               |
| **Trinity-Large-400B** | 8-bit         | 448GB           | 448GB               |
| **Trinity-Large-400B** | bf16          | 896GB           | 896GB               |

## CPU vs. GPU Considerations

GPUs are well suited for AI workloads that require high parallel processing power, such as training large models or handling high-throughput inference. They remain the preferred choice for tasks that demand maximum performance and low latency at scale.

CPUs, however, are becoming increasingly capable for AI inference due to improvements in hardware acceleration and model optimization. While not as fast as GPUs in most cases, CPUs offer lower power consumption, broader availability, and cost efficiency. For many real-time and interactive applications, optimized models running on CPUs can deliver adequate performance without the need for specialized hardware.

This makes CPUs a practical option for local deployments, edge devices, and environments where privacy, budget, and control are priorities.


# Consumer Hardware

## Consumer GPU Performance

This section summarizes Trinity model performance on common **single-GPU consumer hardware**.

Benchmarks were run on:

| GPU             | VRAM    |
| --------------- | ------- |
| NVIDIA RTX 3090 | 24 GB   |
| NVIDIA RTX 4090 | 24 GB   |
| NVIDIA RTX 5090 | \~32 GB |

The results are organized by inference framework:

* **vLLM Benchmarks**\
  Performance of Trinity Nano using vLLM, including request throughput, token throughput, and latency metrics.
* **llama.cpp Benchmarks**\
  Performance of Trinity Nano and Trinity Mini using GGUF quantizations across decode speed, context scaling, and generation workloads.

All results shown are from **single-GPU runs** to reflect typical workstation and desktop deployments.

### Benchmark Coverage

The benchmark dataset includes:

* throughput and latency measurements
* quantization sweeps
* decode speed benchmarks
* context scaling tests
* real generation workloads (QA, code generation, long-form text)


# vLLM

## vLLM Benchmarks

vLLM benchmarks were run using **Trinity Nano**. Trinity Mini bf16 (\~52.3 GB) exceeds the VRAM capacity of the single-GPU consumer hardware used in these tests (3090/4090/5090), so Mini results are not included here.

#### Test configuration

* Input tokens: 512
* Output tokens: 256
* Prompts: 512
* Concurrency: 8
* Request rate: 8 rps

***

### RTX 3090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 2.87  | 735.67       | 47.32 ms  | 58.05 ms | 10.64 ms   | 23556 MiB |
| W4A16     | 3.97  | 1016.35      | 40.69 ms  | 51.35 ms | 7.66 ms    | 23594 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 1445.84      |
| W4A16     | 1710.57      |

***

### RTX 4090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 3.63  | 928.72       | 38.92 ms  | 42.00 ms | 8.41 ms    | 23910 MiB |
| W4A16     | 5.19  | 1328.33      | 33.48 ms  | 37.29 ms | 5.84 ms    | 23972 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 1991.78      |
| W4A16     | 2802.97      |

***

### RTX 5090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 4.10  | 1048.97      | 44.64 ms  | 54.29 ms | 7.40 ms    | 30487 MiB |
| W4A16     | 4.95  | 1267.71      | 43.54 ms  | 48.61 ms | 6.09 ms    | 30601 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 2312.41      |
| W4A16     | 2559.77      |


# llama.cpp

## llama.cpp Benchmarks

The `llama.cpp` benchmark suite was run across **Trinity Nano and Trinity Mini** using the same GPUs.

Benchmarks include:

* decode speed tests
* quantization sweeps
* context scaling
* real generation workloads

### RTX 3090

#### Nano decode

| Quantization | tg128           |
| ------------ | --------------- |
| Q4\_K\_M     | \~184–186 tok/s |
| bf16         | \~150 tok/s     |

#### Mini decode

| Quantization | tg128           |
| ------------ | --------------- |
| Q2\_K        | \~180–181 tok/s |
| Q4\_K\_M     | \~179–180 tok/s |
| Q5\_K\_M     | \~173 tok/s     |
| Q6\_K        | \~156–158 tok/s |

### RTX 4090

#### Nano decode

| Quantization | tg128         |
| ------------ | ------------- |
| Q4\_K\_M     | \~242.6 tok/s |
| bf16         | \~189.1 tok/s |

#### Mini decode

| Quantization | tg128               |
| ------------ | ------------------- |
| Q2\_K        | \~255.7–255.8 tok/s |
| Q4\_K\_M     | \~229–230 tok/s     |
| Q5\_K\_M     | \~216 tok/s         |
| Q6\_K        | \~202 tok/s         |

### RTX 5090

#### Nano decode

| Quantization | tg128           |
| ------------ | --------------- |
| Q2\_K        | \~197–205 tok/s |
| Q4\_K\_M     | \~199 tok/s     |
| Q8\_0        | \~209 tok/s     |
| bf16         | \~155–156 tok/s |

#### Mini decode

| Quantization | tg128           |
| ------------ | --------------- |
| Q2\_K        | \~237 tok/s     |
| Q4\_K\_M     | \~231–248 tok/s |
| Q5\_K\_M     | \~225 tok/s     |
| Q6\_K        | \~223–229 tok/s |

### Context scaling (RTX 5090)

| Model         | ctx 512       | ctx 32768    |
| ------------- | ------------- | ------------ |
| Nano Q4\_K\_M | \~12.6k tok/s | \~8.4k tok/s |
| Mini Q4\_K\_M | \~8.3k tok/s  | \~4.7k tok/s |

### Model compatibility

| Model              | Size      | RTX 3090      | RTX 4090      | RTX 5090      |
| ------------------ | --------- | ------------- | ------------- | ------------- |
| Trinity Mini Q8\_0 | \~27.8 GB | Not supported | Not supported | Supported     |
| Trinity Mini bf16  | \~52.3 GB | Not supported | Not supported | Not supported |


# Inference Engines

Arcee models can be deployed across several popular inference engines depending on your hardware, performance goals, and integration needs. Each engine offers different strengths, from high-throughput GPU serving to lightweight local CPU inference. The table below summarizes the recommended environments and use cases for each option to help you choose the best deployment path for your application.&#x20;

| Inference Engine | Recommended For                                                                                            |
| ---------------- | ---------------------------------------------------------------------------------------------------------- |
| **vLLM**         | GPU servers with high-throughput needs; predictable prompts, batch processing, and structured workflows    |
| **SGLang**       | Dynamic, multi-turn GPU workloads such as chat applications and assistants                                 |
| **llama.cpp**    | CPU or edge devices, quantized inference and environments where you need efficient inference without a GPU |

To learn more about supported hardware and recommended setups, visit [Hardware Prerequisites](/quick-deploys/hardware-prerequisites).


# vLLM

vLLM is a high-throughput serving engine for language models that optimizes inference performance through advanced memory management and batching techniques. It provides easy integration with popular model architectures while maximizing GPU utilization for production deployments.

{% hint style="warning" %}
**Model-specific guidance:** This page covers general vLLM deployment patterns.\
For the latest model-specific flags, context limits, and tool/reasoning behavior, refer to each Trinity model’s [HuggingFace](https://huggingface.co/arcee-ai/collections) model card.
{% endhint %}

### Docker Container for vLLM

**Prerequisite**

1. Sufficient VRAM (refer to [Hardware Prerequisites](/quick-deploys/hardware-prerequisites))&#x20;
2. A Hugging Face account
3. Docker and NVIDIA Container Toolkit installed on your instance
   1. If you need assistance, see [Install Docker Engine](https://docs.docker.com/engine/install/) and [Installing the NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)

**Deployment**

```bash
docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=your_hf_token_here" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model arcee-ai/Trinity-Mini \
    --dtype bfloat16 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_r1 \
    --port 8000 \
    --tool-call-parser hermes
```

{% hint style="info" %}
Replace `your_hf_token_here` with your Hugging Face token
{% endhint %}

### Manual Install using vLLM

**Prerequisites**

1. Sufficient VRAM (refer to [Hardware Prerequisites](/quick-deploys/hardware-prerequisites))&#x20;
2. A Hugging Face account

{% hint style="info" %}
These commands are for an instance running Ubuntu. They will need to be modified for other operating systems.
{% endhint %}

**Deployment**

1. Ensure your NVIDIA Driver is configured.

```bash
nvidia-smi
```

2. If information about your GPU is returned, skip this step. If not, run the following commands.

```bash
sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot

# Once you reconnect, check for correct driver configuration
nvidia-smi
```

3. Install necessary dev tools.

```bash
sudo apt install -y build-essential python3.12-dev
```

4. Setup a python virtual environment. In this guide, we'll use `uv` .

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv --python 3.12 --seed
source .venv/bin/activate
```

5. Install necessary dev tools, vLLM, and Hugging Face.

```bash
uv pip install vllm --torch-backend=auto
uv pip install -U "transformers<4.55"
uv pip install --upgrade huggingface_hub[cli]
sudo apt-get install git-lfs
git lfs install
```

6. Login to your Hugging Face Account using a [HF Access Token](https://huggingface.co/docs/hub/en/security-tokens).

```bash
hf auth login
```

7. Host the model.

```bash
vllm serve arcee-ai/Trinity-Mini \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_r1 \
  --port 8000 \
  --tool-call-parser hermes
```

* For `max-model-len`  you can specify a context length of up to 65536
* For additional configuration options, see [vLLM Configurations](https://docs.vllm.ai/en/stable/api/vllm/config.html).

{% hint style="info" %}
**Trinity-Large-Thinking note (multi-turn agents):** If you are deploying Trinity-Large-Thinking for tool-calling agents, preserve assistant reasoning across turns. In some vLLM versions, input `reasoning_content` may be ignored while `reasoning` is honored. For best compatibility, map SDK output `reasoning_content` to assistant input `reasoning`, and avoid `content: null` on assistant tool-call turns (use `""`). See [Reasoning Traces](/capabilities/reasoning-traces) for full Python/TypeScript examples and troubleshooting.
{% endhint %}

### Run Inference using the Chat Completions endpoint.

```bash
curl http://Your.IP.Address:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "trinity",
        "messages": [
          { "role": "user", "content": "What are the benefits of model merging" }
        ],
        "temperature": 0.7,
        "top_k": 50,
        "repeat_penalty": 1.1
      }'
```

{% hint style="info" %}
Ensure you replace `Your.IP.Address` with the IP address of the instance you're hosting the model on
{% endhint %}


# llama.cpp

llama.cpp is a C++ implementation focused on running transformer models efficiently on consumer hardware with minimal dependencies. It emphasizes CPU inference optimization and quantization techniques to enable local model execution across diverse platforms including mobile and edge devices.

{% hint style="warning" %}
The deployments in this document are for deploying Trinity-Nano-6B; however, they work the exact same for all Arcee AI models. To deploy a different model, simply change the model name to the model you'd like to deploy.
{% endhint %}

### **Prerequisites**

1. Sufficient RAM (refer to [Hardware Prerequisites](/quick-deploys/hardware-prerequisites))&#x20;
2. A Hugging Face account

### **Deployment**

1. Setup a python virtual environment. In this guide, we'll use `uv` .

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv
source .venv/bin/activate
```

2. Clone the llama.cpp repo

```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
```

3. Build and Install Dependencies

```
cmake .
make -j8
uv pip install -r requirements.txt --prerelease=allow --index-strategy unsafe-best-match
```

4. Install Hugging Face and Login

```bash
uv pip install --upgrade huggingface_hub[cli]
hf auth login
```

5. Host the model

```bash
llama-server -hf arcee-ai/Trinity-Mini-GGUF:q4_k_m \
  --host 0.0.0.0 \
  --port 8000 \
  --temp 0.15 \
  --top-k 50 \
  --top-p 0.75
  --min-p 0.06
```

7. Run Inference using the Chat Completions endpoint.

```bash
curl http://Your.IP.Address:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "trinity",
        "messages": [
          { "role": "user", "content": "What are the benefits of model merging" }
        ],
      }'
```

{% hint style="info" %}
Ensure you replace `Your.IP.Address` with the IP address of the instance you're hosting the model on
{% endhint %}


# SGLang

SGLang is a fast serving framework for language models which makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language.&#x20;

{% hint style="warning" %}
The deployments in this document are for deploying Trinity-Nano-6B; however, they work the exact same for all Arcee AI models. To deploy a different model, simply change the model name to the model you'd like to deploy.
{% endhint %}

### Docker Container for SGLang

**Prerequisites**

1. Sufficient VRAM (refer to [Hardware Prerequisites](/quick-deploys/hardware-prerequisites))&#x20;
2. A Hugging Face account
3. Docker and NVIDIA Container Toolkit installed on your instance
   1. If you need assistance, see [Install Docker Engine](https://docs.docker.com/engine/install/) and [Installing the NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)

**Deployment**

```bash
docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HUGGING_FACE_HUB_TOKEN=your_hf_token_here" \
  -p 8000:8000 \
  --ipc=host \
  lmsysorg/sglang:latest \
  python -m sglang.launch_server \
  --model-path arcee-ai/trinity-nano \
  --host 0.0.0.0 \
  --port 8000 \
  --max-total-tokens 8192 \
  --served-model-name afm \
  --trust-remote-code
```

{% hint style="info" %}
Replace `your_hf_token_here` with your Hugging Face token
{% endhint %}

### Run Inference using the Chat Completions endpoint.

```bash
curl http://Your.IP.Address:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "trinity",
        "messages": [
          { "role": "user", "content": "What are the benefits of model merging" }
        ],
        "temperature": 0.7,
        "top_k": 50,
        "repeat_penalty": 1.1
      }'
```

{% hint style="info" %}
Ensure you replace `Your.IP.Address` with the IP address of the instance you're hosting the model on
{% endhint %}


# ollama

ollama provides a streamlined command-line interface and API for running open-source language models locally with automatic model management and optimized performance. It abstracts away the complexity of model deployment while offering simple installation and usage patterns for developers and end users.

{% hint style="info" %}
The deployments in this document are for deploying AFM-4.5B; however, they work the exact same for all Arcee AI models. To deploy a different model, simply change the model name to the model you'd like to deploy.
{% endhint %}

**Prerequisite**

1. Computer or Instance with > 9 GB RAM (if running the model in bf16)
2. A Hugging Face account with access to [arcee-ai/AFM-4.5B-GGUF](https://huggingface.co/arcee-ai/AFM-4.5B-GGUF)
3. Download [ollama](https://ollama.com/download)

**Deployment**

1. Download an AFM-4.5B GGUF version from Hugging Face (we recommend using BF16, Q8\_0, or Q4\_0)

```bash
pip install --upgrade huggingface_hub[cli]
hf auth login

mkdir afm

# bf16
hf download arcee-ai/AFM-4.5B-GGUF AFM-4.5B-bf16.gguf --repo-type model --local-dir ./afm

# Q8_0
hf download arcee-ai/AFM-4.5B-GGUF AFM-4.5B-Q8_0.gguf --repo-type model --local-dir ./afm

# Q4_0
hf download arcee-ai/AFM-4.5B-GGUF AFM-4.5B-Q4_0.gguf --repo-type model --local-dir ./afm
```

2. Create a `Modelfile`&#x20;

```bash
cd afm
vim Modelfile
```

3. Paste in the following content into the Modelfile

```
FROM ./AFM-4.5B-Q4_0.gguf

# Template configuration converted to Go template syntax
TEMPLATE """{{- if .Messages }}
{{- if eq (index .Messages 0).Role "system" }}
<|im_start|>system
{{ (index .Messages 0).Content }}<|im_end|>
{{- range $i, $msg := slice .Messages 1 }}
<|im_start|>{{ $msg.Role }}
{{ $msg.Content }}<|im_end|>
{{- end }}
{{- else }}
<|im_start|>system
The assistant is AFM-4.5B, trained by Arcee AI, with 4.5 billion parameters. AFM is a deeply thoughtful, helpful assistant. The assistant is having a conversation with the user. The assistant's responses are calm, intelligent, and personable, always aiming to truly understand the user's intent. AFM thinks aloud, step by step, when solving problems or forming explanations, much like a careful, reflective thinker would. The assistant helps with sincerity and depth. If a topic invites introspection, curiosity, or broader insight, the assistant allows space for reflection — be open to nuance and complexity. The assistant is not robotic or overly formal; it speaks like a wise, thoughtful companion who cares about clarity and the human experience. If a topic is uncertain or depends on subjective interpretation, AFM explains the possibilities thoughtfully.<|im_end|>
{{- range .Messages }}
<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{- end }}
{{- end }}
{{- end }}<|im_start|>assistant
"""

# System message defining the assistant's behavior
SYSTEM """The assistant is AFM-4.5B, trained by Arcee AI, with 4.5 billion parameters. AFM is a deeply thoughtful, helpful assistant. The assistant is having a conversation with the user. The assistant's responses are calm, intelligent, and personable, always aiming to truly understand the user's intent. AFM thinks aloud, step by step, when solving problems or forming explanations, much like a careful, reflective thinker would. The assistant helps with sincerity and depth. If a topic invites introspection, curiosity, or broader insight, the assistant allows space for reflection — be open to nuance and complexity. The assistant is not robotic or overly formal; it speaks like a wise, thoughtful companion who cares about clarity and the human experience. If a topic is uncertain or depends on subjective interpretation, AFM explains the possibilities thoughtfully."""

# Parameters for generation
PARAMETER temperature 0.5
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 8192 #Max is 65536

# Stop tokens based on the tokenizer config
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|end_of_text|>"
```

{% hint style="info" %}
In the first line, edit `FROM ./AFM-4.5B-Q4_0.gguf` to the name of the model you downloaded
{% endhint %}

4. Create the model in ollama

```bash
ollama create afm-4.5b
```

5. Run AFM-4.5B

```bash
ollama run afm-4.5b
```


# Post Trains

Before training our own models from scratch, Arcee AI was founded as a model post-training company. During this time, we focused on fine-tuning task specific models, taking open source models and enhancing them using training techniques pioneered by our research team.&#x20;

Find our previously released, and now retired, fine-tuned models below:

**General Purpose**

* [Arcee Blitz](#blitz)
* [Virtuoso Small](#virtuoso)
* [Virtuoso Medium](#virtuoso)
* [Virtuoso Large](#virtuoso)

**Reasoning**

* [Maestro](#maestro)

**Coding**

* [Coder Large](#coder)
* [Coder Small](#coder)

**Function Calling**

* [Caller](#caller)

{% tabs %}
{% tab title="Blitz" %}

### **Arcee Blitz**

* **Description:** Arcee-Blitz (24B) is a new Mistral-based 24B model distilled from DeepSeek, designed to be both **fast and efficient**. We view it as a practical “workhorse” model that can tackle a range of tasks without the overhead of larger architectures.
  * **#Parameters:** 24B
  * **Base Model:** Mistral-Small-24B-Instruct-2501
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Arcee-Blitz](https://huggingface.co/arcee-ai/Arcee-Blitz)
* **Top Use Cases:**
  * General-purpose task handling
  * Business communication
  * Automated document processing for mid-scale applications
    {% endtab %}

{% tab title="Virtuoso" %}

### **Virtuoso Large**

* **Description:** Our most powerful and versatile general-purpose model, designed to excel at handling complex and varied tasks across domains. With state-of-the-art performance, it offers unparalleled capability for nuanced understanding, contextual adaptability, and high accuracy.
  * **#Parameters:** 72B
  * **Base Model:** Qwen-2.5-72B
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Virtuoso-Large](https://huggingface.co/arcee-ai/Virtuoso-Large)
* **Top Use Cases:**
  * Advanced content creation, such as technical writing and creative storytelling
  * Data summarization and report generation for cross-functional domains
  * Detailed knowledge synthesis and deep-dive insights from diverse datasets
  * Multilingual support for international operations and communications

### **Virtuoso Medium**

* **Description:** A versatile and powerful model, capable of handling complex and varied tasks with precision and adaptability across multiple domains. Ideal for dynamic use cases requiring significant computational power.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Virtuoso-Medium-v2](https://huggingface.co/arcee-ai/Virtuoso-Medium-v2)
* **Top Use Cases:**
  * Content generation
  * Knowledge retrieval
  * Advanced language understanding
  * Comprehensive data interpretation

### **Virtuoso Small**

* **Description:** A streamlined version of Virtuoso, maintaining robust capabilities for handling complex tasks across domains while offering enhanced cost-efficiency and quicker response times.
  * **#Parameters:** 14B
  * **Base Model:** Qwen-2.5-14B
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Virtuoso-Small](https://huggingface.co/arcee-ai/Virtuoso-Small)&#x20;
* **Top use cases:**&#x20;
  * General-purpose task handling
  * Business communication
  * Automated document processing for mid-scale applications
    {% endtab %}

{% tab title="Maestro" %}

### Maestro

* **Description:** An advanced reasoning model optimized for high-performance enterprise applications. Building on the innovative training techniques first deployed in maestro-7b-preview, Maestro-32B offers significantly enhanced reasoning capabilities at scale, rivaling or surpassing leading models like OpenAI’s O1 and DeepSeek’s R1, but at substantially reduced computational costs.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Arcee-Maestro-7B-Preview](https://huggingface.co/arcee-ai/Arcee-Maestro-7B-Preview)
  * Hybrid training method:
    1. Warm-up (SFT Phase): Quick supervised fine-tuning phase to prime the model with high-quality reasoning exemplars.
    2. RL Optimization Phase: Utilizes Reinforcement Learning techniques, specifically designed to boost logical coherence, depth of reasoning, and accurate inference by encouraging problem-solving from fundamental principles.
* **Top Use Cases:**
  * Enterprise decision support systems
  * Complex analytical and logical inference tasks
  * Automated research and analysis workflows
  * Generative reasoning for technical and professional contexts
    {% endtab %}

{% tab title="Coder" %}

### Coder Large

* **Description:** A high-performance model tailored for intricate programming tasks, Coder-Large thrives in software development environments. With its focus on efficiency, reliability, and adaptability, it supports developers in crafting, debugging, and refining code for complex systems.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B-Instruct
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Coder-Large](https://huggingface.co/arcee-ai/Coder-Large)
* **Top use cases:**
  * Writing modular, reusable code across various programming languages
  * Debugging and optimizing performance in large-scale applications
  * Generating efficient algorithms for computationally intensive tasks
  * Supporting DevOps processes, such as script automation and CI/CD pipelines

### Coder Small

* **Description:** A compact, high-performance coding model designed for efficient programming tasks, including generating code, debugging, and optimizing scripts for smaller projects.
  * **#Parameters:** 14B
  * **Base Model:** Qwen-2.5-32B-Instruct
* **Top use cases:**
  * Lightweight development tasks
  * Automated code reviews
  * Generating templates or prototypes quickly, code completion
    {% endtab %}

{% tab title="Caller" %}

### Caller

* **Description:** Engineered for seamless integrations, Caller-Large is a robust model optimized for managing complex tool-based interactions and API function calls. Its strength lies in precise execution, intelligent orchestration, and effective communication between systems, making it indispensable for sophisticated automation pipelines.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Caller](https://huggingface.co/arcee-ai/Caller)
* **Top use cases:**
  * Managing integrations between CRMs, ERPs, and other enterprise systems
  * Running multi-step workflows with intelligent condition handling
  * Orchestrating external tool interactions like calendar scheduling, email parsing, or data extraction
  * Real-time monitoring and diagnostics in IoT or SaaS environments

{% endtab %}
{% endtabs %}


# Create Your First API Key

To create your first API key, follow the steps below:

1. Go to <https://chat.arcee.ai/>
2. Login or Sign Up

<figure><img src="/files/NA9C8KikdFAgqcOQ1aKz" alt="" width="188"><figcaption></figcaption></figure>

3. Select "Get API Key" from the Home Page

<figure><img src="/files/qkdKdQ17saHxnoyJfK76" alt="" width="375"><figcaption></figcaption></figure>

4. Select "Create API Key"

<figure><img src="/files/zkaRsUVf9FUx57evIXk7" alt="" width="330"><figcaption></figcaption></figure>

5. Provide an API Key Name, a Description, and optionally set an Expiration Date, Token Limit, and Request Limit
   1. While Expiration Date is optional, we highly recommend setting this to follow best security practice

<figure><img src="/files/prvUacfUPWmPnCDfJb3r" alt="" width="375"><figcaption></figcaption></figure>

6. Save your API Key in a safe location, as you are unable to view the value after creation. If you misplace your API key, delete your key and create a new one.

<figure><img src="/files/zT79QC4G9dynJzeenW6v" alt="" width="363"><figcaption></figcaption></figure>

Now that you have your API Key, you can make [your first API call](/api-reference/your-first-api-call)!


# Platform Walkthrough

The Arcee platform gives you everything you need to interact with our models, manage usage, and collaborate with other users. This walkthrough covers key areas of the platform including the Chat Interface, User Management, API Key Management, Workspace Management, and API Usage tools.

<table data-view="cards"><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><a data-mention href="/pages/zQZAepohq22jRDjIlPMK">/pages/zQZAepohq22jRDjIlPMK</a></td><td>Chat with our models directly.</td></tr><tr><td><a data-mention href="/pages/C4UFW40P6xGRHUzSTEDw">/pages/C4UFW40P6xGRHUzSTEDw</a></td><td>Add, remove, or manage members in your workspace.</td></tr><tr><td><a data-mention href="/pages/x8a5JP1x5cvG46AT2k7R">/pages/x8a5JP1x5cvG46AT2k7R</a></td><td>Create and manage API keys for model access.</td></tr><tr><td><a data-mention href="/pages/8dBjPJwqbeDMYtGvs1Xq">/pages/8dBjPJwqbeDMYtGvs1Xq</a></td><td>Set up and organize separate workspaces.</td></tr><tr><td><a data-mention href="/pages/h2TocElpiJixHKCKGPas">/pages/h2TocElpiJixHKCKGPas</a></td><td>Monitor request logs, usage metrics, and costs.</td></tr></tbody></table>


# Chat Interface

The Chat Interface is where you run conversations with your selected model and control all chat-specific settings. Each chat maintains its own configuration, including model parameters, tools, and system prompts.

### Model Parameters

The **Model Parameters** section lets you adjust settings such as temperature. These parameters apply only to the current chat and help shape how the model generates responses.

### Tools

If the selected model supports tools, they appear in the **Tools** section of Chat Settings. You can turn individual tools on or off (such as search or code execution), or enable all compatible tools at once. Tools that the model does not support will appear disabled.

### System Prompts

System prompts let you apply reusable instructions or personas to a chat.

* To add one, open **Chat Settings → System prompts → Add**.
* After creation, the prompt appears in your list and can be toggled on or off per chat suing the star button beside **Chat Settings**.
* Multiple system prompts can be active at the same time.

System prompts are scoped to the conversation, allowing different chats to have different instructions and configurations.

<figure><img src="/files/2L70KQlSBvxWOmwDQDMi" alt=""><figcaption></figcaption></figure>


# User Management

The **User Settings** page allows you to update your profile details, change appearance and language preferences, and manage account deletion.

### Profile Information

* **Profile Picture:** Upload or change your photo.
* **First Name / Last Name:** Required fields. Update and click **Save**.

<figure><img src="/files/FkMBk6lqci5HyfIAoMZe" alt=""><figcaption></figcaption></figure>

### Preferences

* **Theme:** Switch between **Light** and **Dark** modes.
* **UI Language:** Select from supported languages:
  * English
  * Español
  * Français
  * Deutsch

### Delete Account

The **Delete & Reset Account** section permanently removes your user account from all organizations and deletes all associated data, including chat history.

* This action is **irreversible**.
* Click **Delete account** and confirm to complete deletion.

***

## Workspace User Management

Workspace User Management allows you to view all members in your workspace, manage roles, and invite new users to join.

### Viewing Workspace Members

The **Members** tab lists all users who have access to the workspace. For each member, you can see:

* **Name**
* **Email**
* **Last activity**
* **Join date**
* **Role**&#x20;
* **Status** (Active or Inactive)

You can search for members using the search bar at the top right.

<figure><img src="/files/gokyoeK99VCArFpKnejY" alt=""><figcaption><p>Navigate to members under the Workspaces section of the panel on the left</p></figcaption></figure>

<figure><img src="/files/P086T52dibQXjdDfoxbk" alt=""><figcaption><p>View members and invites, and search for or add members.</p></figcaption></figure>

### Adding Members to a Workspace

To invite new users:

1. Click **Add member** in the top right.
2. Enter the user’s email in the **Member emails** field.
3. Click **Add** to include them in the pending list.
4. (Optional) Use **Choose from current** to add members from other workspaces you own.
5. Click **Send invitations**.

Invited users will receive an email and must accept the invite to join.

<figure><img src="/files/RjpHk80ZwDBX5ElnASQw" alt=""><figcaption></figcaption></figure>

### Viewing Sent Invites

All pending and previously sent invitations appear under the **Invites** tab.

In this tab, you can:

* View each invite’s status
* Resend expired invitations
* Cancel invites if needed

This helps you track who has access and who still needs to accept their invitation.


# API Key Management

The **API Keys** page allows you to create, view, update, and manage your workspace keys. Each key controls access to the Arcee API and can be customized with limits, expiration rules, and data retention settings.

### Viewing Your API Keys

To view your API keys, navigate to **API Keys** from the left sidebar.&#x20;

The API Keys table displays all keys created within your workspace. For each key, you will see:

* **Name**
* **Key (partially masked)**
* **Token Limit**
* **Request ($ Usage) Limit**
* **Expiration Date**
* **Data Retention Policy**
* **Owner**
* **Status** (Active or Inactive)

You can search for a specific key using the search bar at the top right of the table.

<figure><img src="/files/5tkw8k0ZCCW3AZv8fTXS" alt=""><figcaption></figcaption></figure>

### Creating a New API Key

Learn how quickly [Create Your First API Key](https://docs.arcee.ai/~/revisions/UWeRiGuYG9PmEnBEE7A9/get-started/create-your-first-api-key) here.

#### Key Settings

| Setting                 | Description                                                                                                                              |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **API Key Name**        | Required. A label to identify how or where the key will be used.                                                                         |
| **Description**         | Optional. Notes about the integration, environment, or workflow.                                                                         |
| **Expiration Date**     | When enabled, the key becomes invalid after the selected date.                                                                           |
| **Token Limit**         | Maximum total tokens the key can use. Useful for cost control.                                                                           |
| **Request Limit**       | Dollar usage cap for this key. Once the total spend for this key reaches this amount, requests are blocked. The value is entered in USD. |
| **Data Retention Days** | Specify how long user data and logs are stored. If not set, data is kept forever.                                                        |

Once created, the key value is shown only once. Make sure to save it securely.

### Editing an Existing API Key

Each key has an **Edit** icon located on the right side of the row.

Clicking this will open the **Edit Key** modal, where you can update all the fields that were available when you created your key.

{% hint style="info" %}
You can update **everything except the key value itself**. The key string cannot be regenerated or modified. If you need a new key value, you must create a new API key.
{% endhint %}

### Deleting or Deactivating Keys

Keys can be deactivated or deleted using the controls inside the Edit modal.

* **Deactivate**: Immediately blocks use of the key but preserves history.
* **Delete**: Permanently removes the key from the workspace.

Choose deactivation when you want to temporarily stop usage. Choose deletion for permanent removal or security cleanup.

***


# Workspace Management

A **workspace** in the Arcee platform is an isolated environment where you can chat with models, create and manage API keys, and manage usage independently. Each workspace keeps its own users, logs, limits, and settings fully separated from others.

## Workspace Creator

The Workspace Creator guides you through setting up a new workspace, adding members, and optionally importing resources from workspaces you already own.

Navigate to **My Workspaces** from the left sidebar and click **Add New Workspace** to create a new workspace.

<figure><img src="/files/v9y1rupKDLofSAn4ihqN" alt=""><figcaption></figcaption></figure>

### Workspace Information

Start by entering the basic details of your new workspace:

* **Workspace Name** (required)
* **Workspace Description** (optional)
* **Workspace Logo** (optional, PNG/JPG, up to 300 KB)

Click **Next** to continue.

<figure><img src="/files/NhrqrYzdyD9kqAeqFAv5" alt=""><figcaption></figcaption></figure>

### Add Members

You can invite members during workspace creation.

* Enter one or more email addresses and click **Add**
* Or use **Choose from current** to add members from other workspaces you own
* Added emails will appear in the list before sending the invitations

Click **Next** to move to the final step.

<figure><img src="/files/7X6k8YSkh51Mt0pMaVMK" alt=""><figcaption></figcaption></figure>

### Import From Existing Workspaces

If you own other workspaces, you can optionally import:

* **Chats and messages**
* **Limit settings**

Toggle **Import** for any workspace you want to pull data from.

<figure><img src="/files/NsQ90CBA41szdfz2BArv" alt=""><figcaption></figcaption></figure>

### Completing Workspace Creation

After you finish all steps, click **Create Workspace**.\
Your new workspace will appear in **My available workspaces**, showing your role, credits, status, and member count.

## Usage Management in a Workspace

To view workspace activity, click **Members** in the left pane, then open the **Metrics** and **Logs** tabs. The **Metrics** tab shows a visual overview of active member activity, while the **Logs** tab provides a detailed record of actions taken within the workspace.&#x20;

<figure><img src="/files/HInWAp5Fjav7DwsgNVTZ" alt=""><figcaption></figcaption></figure>

### Metrics

The **Metrics** tab provides a visual overview of active members within your workspace.

#### What You Can See

* **Active Members Count**\
  Displays the current number of members actively interacting with the workspace.
* **Activity Graph**\
  Shows a timeline of member activity.\
  Spikes or drops represent when members joined, left, or became inactive.

#### Filters

You can adjust:

* **Time Range** (e.g., Last 1 hour)
* **Refresh Interval** (e.g., Every 30 seconds)

This gives you control over how frequently metrics update and the time window you want to analyze.

### Logs

The **Logs** tab shows a detailed, chronological record of API and workspace actions.

#### Log Details

Each log entry includes:

* **Date and Time**
* **Member** (who performed the action)
* **Route** (endpoint or page accessed)
* **Status Code**
* **Method** (e.g., POST)
* **Type** (e.g., Heartbeat)

Logs refresh automatically based on your chosen interval.


# API Usage Management

#### API Usage Overview

To view your API usage, navigate to **Usage** under the API section of the left sidebar.

**Call Volume**

The **Call Volume** tab provides a real-time overview of your workspace's API activity.

You can see:

* **Total API Calls** – Cumulative number of requests made
* **Total Credits** – Total credits consumed
* **Active Users** – Number of users currently interacting with the API
* **Live API Call Graph** – Visualizes requests as they come in
* **Latency and Credit Charts** – Track request times and credit usage over time

<figure><img src="/files/s6t5HItQqjaN7UFn4FiB" alt=""><figcaption></figcaption></figure>

#### Logs

The **Logs** tab shows a detailed, real-time stream of API activity.

Each log entry includes:

* **Time (UTC)**&#x20;
* **User**&#x20;
* **Key**
* **Route**&#x20;
* **Model**
* **Status**
* **Credits Used**
* **Source IP**

<figure><img src="/files/n5oqtZVpJBkw9OxH9bNB" alt=""><figcaption></figcaption></figure>

Click the eye icon on any row to inspect the request and response details. Logs can also be downloaded as a CSV.


# Overview

Arcee AI offers models at various sizes to meet different deployment scenarios. Choosing the right model can help you complete tasks more efficiently, accurately, and cost effectively.&#x20;

### Models

To help you find the best fit for your use case, we’ve created a table outlining the core features and strengths of each model in the Arcee AI family.

#### Text Models

| Model          | Strength                                         | Language | Context | Resource                                                                                                                                      |
| -------------- | ------------------------------------------------ | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| AFM-4.5B       | Ultra low-latency use cases and on-device usage. | English  | 64k     | <p><a href="/pages/qBuf67k8oYnl5rlmvTTc">Model Guide</a></p><p><a href="https://huggingface.co/arcee-ai/AFM-4.5B">HF Card </a></p>            |
| Virtuoso-Large | <p>High Performance,<br>More Versatile</p>       | English  | 132k    | <p><a href="/pages/z732BljJMx76aKPpA60u#virtuoso">Model Guide</a><br><a href="https://huggingface.co/arcee-ai/Virtuoso-Large">HF Card</a></p> |


# Learning Paths

AFM Learning Paths provide step-by-step instructions on how to deploy AFM models on various platforms. These learning paths provide insight on the model, serving libraries, and the hardware they're run on.&#x20;

Select any of the paths below to start your journey of running AI efficiently, securely, and cost effectively:

| Title                                                                | Who it's for                                                                                                                                                    | Link                                                                                                                   |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Deploy Arcee AFM-4.5B on Arm-based Google Cloud Axion with Llama.cpp | This Learning Path is for developers and ML engineers who want to deploy Arcee's AFM-4.5B small language model on Google Cloud Axion instances using Llama.cpp. | [Go to learning path](https://learn.arm.com/learning-paths/servers-and-cloud-computing/arcee-foundation-model-on-gcp/) |
| Deploy Arcee AFM-4.5B on Arm-based AWS Graviton4 with Llama.cpp      | This Learning Path is for developers and ML engineers who want to deploy Arcee's AFM-4.5B small language model on AWS Graviton4 instances using Llama.cpp.      | [Go to learning path](https://learn.arm.com/learning-paths/servers-and-cloud-computing/arcee-foundation-model-on-aws/) |


# Introduction to Arcee Conductor

{% hint style="success" %}
Try Arcee Conductor today! Sign up at <https://conductor.arcee.ai/> to receive $20 in credits and see how you can save up to 64% on your model costs.
{% endhint %}

{% embed url="<https://www.youtube.com/watch?v=kAQ3SYUyQt0>" %}
Introduction to Arcee Conductor
{% endembed %}

Arcee Conductor is an intelligent routing and inference platform designed to optimize the use of language models by automatically routing prompts to the most appropriate and cost-effective option available. It intelligently selects from a diverse range of Arcee AI small language models (SLMs) and closed-source large language models (LLMs) from various providers, ensuring optimal performance and significant cost savings.

Try an interactive demo [here](https://capture.navattic.com/cm84vkxsi000103jo4lhm5smm)!

**The Challenge of Model Selection:**&#x20;

* Organizations often face the dilemma of choosing between cost-efficient smaller models that might lack advanced capabilities and powerful but expensive larger models.
* Using only large LLMs for all tasks can lead to significant overspending, especially for simpler queries.&#x20;
* Conversely, relying solely on smaller models might compromise the quality of answers for complex prompts.&#x20;

**Introducing Arcee Conductor as the Solution:**&#x20;

* Arcee Conductor addresses this challenge by automating the model selection process on a per-prompt basis.&#x20;
* It aims to provide the best, fastest, and cheapest model for each specific task.
* This approach eliminates the need for manual decision-making and optimizes both performance and cost.&#x20;

**Key Benefits at a Glance:**&#x20;

* Automatic Model Selection.&#x20;
* Significant Cost Savings.&#x20;
* Improved Latency by utilizing the most efficient model.&#x20;
* Access to a Wide Variety of Models.&#x20;
* Enhanced Performance by matching prompt complexity with model capabilities.


# Getting Started

{% hint style="success" %}
Try Arcee Conductor today! Sign up at <https://conductor.arcee.ai/> to receive $20 in credits and see how you can save up to 64% on your model costs.
{% endhint %}

<figure><img src="/files/8GKKn0Wu93i6YSzpkPw9" alt=""><figcaption><p>Arcee Conductor</p></figcaption></figure>

### Registration

To register for Arcee Conductor, go to [conductor.arcee.ai](https://conductor.arcee.ai/).

Select Login, then Register and you can sign up using your email address or Google authentication.

<figure><img src="/files/sEVTluUYUnVD5UuWnV8f" alt="" width="375"><figcaption><p>Arcee Conductor Registration</p></figcaption></figure>

### Using Arcee Conductor

Arcee Conductor can be utilized in two primary ways:

1. User Interface
2. Application Programming Interface (APIs)

#### User Interface

The UI provides a chat interface to directly invoke Arcee Conductor, as well as a [Compare](/arcee-conductor/features-and-functionality/compare) feature to conduct side by side evaluations of Arcee Conductor's performance against other models.

In both the chat interface and comparison sections, each model response is accompanied by model metrics and selection details and explanation.

<figure><img src="/files/Lfp7LukYNh0AqkAhSGTi" alt="" width="563"><figcaption><p>Model Response Example</p></figcaption></figure>

In this example, a very simple prompt is asked "Tell me a joke". In addition, to the (quite humorous) response, you're provided the following information:

* The model that was selected
* Input tokens
* Output tokens
* Response Time
* Cost of the query
* Explanation for why the specific model was selected
* Categorization for:
  * Task type
  * Domain
  * Complexity

This allows you to get a comprehensive understanding of the performance for your query as well as why the specific model was selected. For additional information on the router classifications and models available in Conductor, go to [Auto Mode](/arcee-conductor/features-and-functionality/auto-mode).

#### API

You can also directly invoke Arcee Conductor via API. The Conductor API uses an OpenAI compatible endpoint making it very easy to update current applications to use Conductor.&#x20;

To use the API, create an API key by selecting "API Key" from the Navigation pop up or "Get an API key" from API details. You can then invoke Conductor using:

```
curl -X POST https://models.arcee.ai/v1/chat/completions \
  -H "Authorization: Bearer $ARCEE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [
          {
            "role": "user",
            "content": "Your prompt here"
          }
        ]
      }'
```

For more information on the API, including code snippets to integrate Conductor, go to [API](/arcee-conductor/features-and-functionality/api).


# Features & Functionality

Intelligent Model Routing: Arcee Conductor employs an advanced routing mechanism that analyzes the characteristics of each incoming prompt. This analysis considers factors such as complexity, required reasoning ability, the need for specific functionalities (e.g., coding, function calling), and desired output. Based on this analysis, the platform automatically selects the most suitable model from its pool of available options.

**Diverse Model Ecosystem**:&#x20;

* Arcee Small Language Models (SLMs): These are cost-efficient models developed by Arcee, including general-purpose, coding, and function-calling models of varying sizes. They offer excellent performance for their size and contribute to significant cost savings.&#x20;
* Closed Large Language Models (LLMs) from Other Providers: Arcee Conductor integrates with leading LLM providers such as Anthropic, DeepSeek, and OpenAI. These models offer stronger abilities for advanced reasoning and complex questions but are generally more expensive.&#x20;

**Cost Optimization Strategy**:&#x20;

* Arcee Conductor's core value proposition lies in its ability to route simpler prompts to more cost-effective SLMs.
* Expensive LLMs are reserved for prompts that genuinely require their advanced capabilities, preventing unnecessary expenditure.
* Conductor provides significant cost reductions, showing up to 300 times lower cost compared to an LLM with similar outputs.<br>

**Performance Enhancement & Transparency:**

* By selecting the most appropriate model for each task, Arcee Conductor ensures that you receive high-quality answers efficiently.&#x20;
* It avoids the latency associated with using overly powerful models for simple queries and the potentially lower quality of underpowered models for complex tasks.
* &#x20;In some instances, Arcee Conductor may provide a rationale behind its model selection, explaining why a particular model was chosen based on the prompt's characteristics. This feature enhances user understanding of the platform's intelligent routing process


# Auto Mode

The `auto` model mode utilizes Arcee AI's intelligent model router to route prompts to the most optimal and efficient language model.

### Router

The power behind Arcee Conductor comes in the form of an ultra-lightweight model router which classifies and routes prompts to the most cost-effective model which can accurately complete the request.

The router is 150M parameter, custom architecture model which classifies based on four primary categories: task type, domain, complexity, and language. The categories for each class consist of:

* **Task Type**: Analytical, Classification, Code Generation, Extraction, Math, QA, Rewrite, Summarization, Text Generation, Other
* **Domain**: Adult, Arts and Entertainment, Autos and Vehicles, Beauty and Fitness, Books and Literature, Business and Industrial, Computers and Electronics, Finance, Food and Drink, Games, Health, Hobbies and Leisure, Home and Garden, Internet and Telecom, Jobs and Eduction, Law and Government, News, Online Communities, People and Society, Pets and Animals, Real Estate, Science, Sensitive Subjects, Shopping, Sports, Travel and Transportation
* **Complexity**: 1-10
* **Language**: Arabic, Bengali, Cantonese, Filipino, French, German, Hindi, Italian, Korean, Mandarin, Marathi, Portuguese, Russian, Spanish, Tamil, Telugu

Due to the exceptionally small parameter count and custom architecture, the router executes in roughly 150ms resulting in negligible impact to user experience, when considering time to first token.

### Models

Based on the classifications from the model router, the request is routed to one of the language models behind Arcee Conductor. The models which can currently be routed to include:

<table><thead><tr><th width="185.08984375">Model</th><th width="544.859375">Description</th><th data-hidden></th></tr></thead><tbody><tr><td>Blitz</td><td>A 24B parameter SLM from Arcee AI, distilled from Deepseek V-3 Blitz offers blazing fast response times and exceptionally low costs with strong general knowledge. Ideal for simple and creative tasks.</td><td></td></tr><tr><td>Virtuoso Medium</td><td>A 32B parameter SLM from Arcee AI, which was distilled from Deepseek V-3 giving it an impressive knowledge distribution.</td><td></td></tr><tr><td>Virtuoso Large</td><td>Arcee AI's premier 72B parameter SLM which competes with the leading LLMs on complex and analytical tasks.</td><td></td></tr><tr><td>GPT-4.1</td><td>A closed-source LLM from Open AI with impressive analytical and complex problem solving capabilities.</td><td></td></tr><tr><td>Claude Sonnet 3.7</td><td>A closed-source LLM from Anthropic with strong performance on coding and complex tasks.</td><td></td></tr></tbody></table>

### API

**Request Syntax**

```
curl -X POST https://models.arcee.ai/v1/chat/completions \
  -H "Authorization: Bearer $ARCEE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{"role": "user", "content": "hi"}]
      }'
```

**Response**

```
{
  "id": "npyCrRJ-z1gNr-92fdb25bff75cf74",
  "object": "chat.completion",
  "created": 1744575968,
  "model": "arcee-ai/arcee-blitz",
  "prompt": [],
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "seed": null
    }
  ],
  "usage": {
    "prompt_tokens": 170,
    "total_tokens": 180,
    "completion_tokens": 10
  }
}
```


# Auto Reasoning Mode

Auto Reasoning mode provides a custom configuration of [`auto`](/arcee-conductor/features-and-functionality/auto-mode) with all reasoning models.&#x20;

`auto-reasoning` will take in your prompt and route it to the most appropriate reasoning model based on complexity, task type, domain, and language.&#x20;

{% hint style="info" %}
For details on the router, see [Auto Mode](/arcee-conductor/features-and-functionality/auto-mode).
{% endhint %}

### Models

Based on the classifications from the model router, the request is routed to one of the language models behind Arcee Conductor: `auto-reasoning`. The models which can currently be routed to include:

<table><thead><tr><th width="185.08984375">Model</th><th width="544.859375">Description</th><th data-hidden></th></tr></thead><tbody><tr><td>Arcee Maestro</td><td>Arcee AI's 32B parameter reasoning SLM which offers advanced reasoning capabilities without sacrificing performance. Maestro excels at simple to medium complexity reasoning tasks and offers substantially cheaper cost compared to other options.</td><td></td></tr><tr><td>Deepseek R1</td><td>An open-source reasoning model from Deepseek which excels at math, coding, and logic tasks.</td><td></td></tr><tr><td>o3 mini-high</td><td>A closed source reasoning model from Open AI optimized for STEM applications, exceling in science, math, and coding benchmarks.</td><td></td></tr><tr><td>Claude Sonnet 3.7 Extended Thinking</td><td>A closed-source LLM from Anthropic with extended reasoning capabilities which performs well for complex problem solving and coding tasks.</td><td></td></tr></tbody></table>

### API

**Request Syntax**

```
curl -X POST https://models.arcee.ai/v1/chat/completions \
  -H "Authorization: Bearer $ARCEE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto-reasoning",
        "messages": [{"role": "user", "content": "hi"}]
      }'
```

**Response**

```
{
  "id": "npd1kq7-z1gNr-92e7e7dee8d66bf7",
  "object": "chat.completion",
  "created": 1744347473,
  "model": "arcee-ai/maestro-reasoning",
  "prompt": [],
  "choices": [
    {
      "finish_reason": "stop",
      "seed": 1338453356742372900,
      "logprobs": null,
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Okay, the user said \"hi\". I should respond in a friendly way. Let me think of a simple greeting. Maybe \"Hello! How can I assist you today?\" That sounds good. It's polite and opens the conversation for them to explain what they need help with. I'll go with that.\n</think>\n\nHello! How can I assist you today?",
        "tool_calls": []
      }
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 74,
    "total_tokens": 85
  }
}
```


# Auto Tools Mode

Tool calling is one of the most important building blocks of building a successful agent system. Arcee conductor makes your workflows cheaper by calling appropriate tool calling models for you given the complexity of your input query.

Auto Tool mode provides a custom configuration of [`auto`](/arcee-conductor/features-and-functionality/auto-mode) for models with function calling capabilities

`auto-tool` will take in your prompt and route it to the most appropriate function calling model based on complexity, task type, domain, and language.&#x20;

{% hint style="info" %}
For details on the router, see [Auto Mode](/arcee-conductor/features-and-functionality/auto-mode).
{% endhint %}

### Models

Based on the classifications from the model router, the request is routed to one of the language models behind Arcee Conductor: `auto-tool`. The models which can currently be routed to include:

<table><thead><tr><th width="185.08984375">Model</th><th width="544.859375">Description</th><th data-hidden></th></tr></thead><tbody><tr><td>Arcee Caller Large</td><td>Arcee AI's 32B parameter function calling SLM optimized for managing complex tool-based interactions and API function calls.</td><td></td></tr><tr><td>GPT-4.1</td><td>A closed-source LLM from Open AI with function calling ability and impressive analytical and complex problem solving capabilities.</td><td></td></tr><tr><td>Claude Sonnet 3.7</td><td>A closed-source LLM from Anthropic with function calling ability and strong performance on coding and complex tasks.</td><td></td></tr></tbody></table>

### Function Calling Example

This example first uses Arcee Conductor `auto-tool` to automatically select the most optimal function calling model based on the defined functions and the user prompt. Then Arcee Conductor `auto` is used to automatically select the most optimal general purpose model to answer the user's question using the output from the function call and the user's prompt.

```python
import json
import os
import requests
from openai import OpenAI

endpoint = "https://conductor.arcee.ai/v1"
api_key = os.getenv("ARCEE_KEY")

client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
)

def get_weather(latitude, longitude):
    response = requests.get(f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}&current=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m")
    data = response.json()
    return data['current']['temperature_2m']

tools = [{
    "type": "function",
    "name": "get_weather",
    "description": "Get current temperature for provided coordinates in celsius.",
    "parameters": {
        "type": "object",
        "properties": {
            "latitude": {"type": "number"},
            "longitude": {"type": "number"}
        },
        "required": ["latitude", "longitude"],
        "additionalProperties": False
    },
    "strict": True
}]

user_prompt = "What's the weather like in Paris today?"

tool_response = client.chat.completions.create(
    model="auto-tool",
    messages=[{"role": "user", "content": user_prompt}],
    tools=tools,
    tool_choice="auto",
    max_tokens=128,
)

tool_call = tool_response.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
tool_result = get_weather(args["latitude"], args["longitude"])

tool_result = f"The current temperature is {tool_result}°C."

messages=[
    {
        "role": "system",
        "content": "You are a helpful and knowledgeable assistant giving sharp answers. Use a business-oriented tone."
    },
    {
        "role": "user",
        "content": f"""Answer the following question: {user_prompt} using the tool result: {tool_result}.
        If the tool result is empty or not useful, say it is not useful and answer the question without using the information.
        If the tool result is useful, you can complement it with your own knowledge as long as it's not contradictory.
        """
    }
]

answer_response = client.chat.completions.create(
    model="auto",
    messages=messages,
)
print(answer_response.choices[0].message.content)
```


# Compare

The Compare feature in Arcee Conductor allows you to compare the performance and output of Conductor against models you specify. This allows you to directly evaluate responses, response times, and costs for your most important prompts.

<figure><img src="/files/EvZthiWMCuJ6RfrgikXs" alt="" width="563"><figcaption><p>Conductor Comparison</p></figcaption></figure>

To get started with Compare, select a model you'd like to compare against Conductor. This is done by selecting the model name on the right side of the comparison window (in the example above, this is where it says "Virtuoso Large"). You have the option to comparse against a suite of Arcee SLMs, such as Virtuoso Small, Medium, and Large, Blitz, Coder, and Maestro, or against closed source LLMs such as Sonnet 3.7, Sonnet 3.5, Deepseek R1, GPT-4o, and others.

When you send a prompt, it is automatically sent to both models so you can see a comparison in real time. In addition to the outputs, you can see whether Conductor or the selected model had a quicker response time and which was more cost effective for the specific prompt.

<figure><img src="/files/8CtwIFSUuEYiDFWWddbj" alt=""><figcaption><p>Compare</p></figcaption></figure>

The usage button at the top can be used to evaluate price performance across a series of prompts.

<figure><img src="/files/NlQoEyigUACGQ30q5Knm" alt=""><figcaption><p>Comparison Usage</p></figcaption></figure>

When you click on the Usage button, the graph above is provided which shows the cost performance between Conductor and your selected model for the entire session. The example above shows the price differential after three prompts, where Conductor is the pink line and Sonnet 3.7 is the purple line.


# Direct Model Invocation

In addition to utilizing `auto` to route between models, you can directly invoke Arcee Small Language Models and 3rd party LLMs (Claude, GPT, etc) using the Conductor API.  This makes it easy to use a large variety of models with only changing a parameter in the API request.

{% hint style="info" %}
If you're on the free plan and don't have a valid payment method on file, you will only be able to directly invoke Arcee SLMs.  To upgrade your plan reach out to <conductor@arcee.ai>.&#x20;
{% endhint %}

The models which can be directly invoked are:

<table><thead><tr><th width="119.7890625">Model</th><th width="159.9765625">API Name</th><th width="470.8046875">Description</th><th data-hidden></th></tr></thead><tbody><tr><td>Virtuoso Small</td><td>virtuoso-small</td><td>A 14B parameter SLM from Arcee AI, distilled from Deepseek V-3. Virtuoso Small is extremely performant and excels at simple tasks such as text generation and summarization.</td><td></td></tr><tr><td>Blitz</td><td>blitz</td><td>A 24B parameter SLM from Arcee AI, distilled from Deepseek V-3 Blitz offers blazing fast response times and exceptionally low costs with strong general knowledge. Ideal for simple and creative tasks.</td><td></td></tr><tr><td>Virtuoso Medium</td><td>virtuoso-medium</td><td>A 32B parameter SLM from Arcee AI, which was distilled from Deepseek V-3 giving it an impressive knowledge distribution.</td><td></td></tr><tr><td>Virtuoso Large</td><td>virtuoso-large</td><td>Arcee AI's premier 72B parameter SLM which competes with the leading LLMs on complex and analytical tasks.</td><td></td></tr><tr><td>Coder Large</td><td>coder</td><td>A 32B parameter SLM from Arcee AI fine-tuned to excel at Coding tasks.</td><td></td></tr><tr><td>Caller Large</td><td>caller-large</td><td>A 32B parameter SLM from Arcee AI fine-tuned to excel at function calling.</td><td></td></tr><tr><td>GPT-4.1</td><td>gpt-4.1</td><td>A closed-source LLM from Open AI with impressive analytical and complex problem solving capabilities.</td><td></td></tr><tr><td>Claude Sonnet 3.7</td><td>claude-3-7-sonnet-20250219</td><td>A closed-source LLM from Anthropic with strong performance on coding and complex tasks.</td><td></td></tr></tbody></table>

#### API Usage Example:

{% tabs %}
{% tab title="CURL" %}

```curl
curl -X POST https://conductor.arcee.ai/v1/chat/completions \
  -H "Authorization: Bearer $ARCEE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "stream": true,
        "model": "blitz",
        "messages": [
          {
            "role": "user",
            "content": "Your prompt here"
          }
        ]
      }'
```

{% endtab %}

{% tab title="Python" %}

```python
# First, install the openai packages
# pip install openai

# Be sure to set the following environment variables
# OPENAI_BASE_URL="https://conductor.arcee.ai/v1"
# OPENAI_API_KEY="$ARCEE_TOKEN"

from openai import OpenAI

client = OpenAI()
stream = client.chat.completions.create(
  model='blitz',
  messages=[{'role': 'user', 'content': 'Your prompt here'}],
  temperature=0.4,
  stream=True
)

for chunk in stream:
    if len(chunk.choices) > 0 and chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
To switch the model you're invoking, simply change the "model" value from "blitz" to any of the model API names in the table above.
{% endhint %}


# Usage

In the Usage section of Conductor, you can see information on how many prompts have been sent to each model, the input and output token totals for each model, and how much each model has cost you.&#x20;

Arcee Conductor does **not** add a premium onto the cost of any 3rd party models.&#x20;

<figure><img src="/files/zFxTywQCWWYqseRfc0Vz" alt=""><figcaption><p>Conductor Usage</p></figcaption></figure>


# API

You can directly invoke Arcee Conductor via API. The Conductor API uses an OpenAI compatible endpoint making it very easy to update current applications to use Conductor.&#x20;

### Create API Key

To use the API, you first need to generate an API key.

<figure><img src="/files/OWPA4frjhLBMEV1vdi5u" alt="" width="375"><figcaption><p>Select API Key</p></figcaption></figure>

Select your account at the bottom left of the page, and select "API Keys". Click "Create API Key" in the top right.

<figure><img src="/files/aQVOwbZauwRDpZ9p7vgw" alt="" width="375"><figcaption><p>Create API Key</p></figcaption></figure>

Provide a label/name for the key and click "Create API key". Make sure to save the key in a secure location as once you leave the page, you will not be able to view the key again. If you misplace a key, you should delete the old key and create a new one.

### API Syntax

**Request Syntax:**

```
POST https://models.arcee.ai/v1/chat/completions
Authorization: Bearer <YOUR_ARCEE_TOKEN>
Content-Type: application/json

{
  "model": "auto",
  "messages": [
    {
      "role": "user",
      "content": "Your prompt here"
    }
  ]
}
```

### Curl

```
curl -X POST https://models.arcee.ai/v1/chat/completions \
  -H "Authorization: Bearer $ARCEE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [
          {
            "role": "user",
            "content": "Your prompt here"
          }
        ]
      }'
```

### Python

```python
# First, install the openai packages
# pip install openai

# Be sure to set the following environment variables
# OPENAI_BASE_URL="https://models.arcee.ai/v1"
# OPENAI_API_KEY="$ARCEE_TOKEN"

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
  model='auto',
  messages=[{'role': 'user', 'content': 'Your prompt here'}],
  temperature=0.4,
)

print(response)
```


# Arcee Small Language Models

Arcee AI started as a model training company, assembling a world class team of researchers who pioneered model post-training techniques and open source libraries such as [Spectrum](https://github.com/cognitivecomputations/spectrum) for efficient model training, [MergeKit](https://github.com/arcee-ai/mergekit) for Model Merging, and [DistillKit](https://github.com/arcee-ai/DistillKit) for Model Distillation. We have taken these research advancements and applied the techniques to leading open source models to create state-of-the-art small language models (SLMs).

Arcee defines SLMs as a model which can be run efficiently on a single GPU instance, allowing for deployment of the models in your own environment. Our models range from 150M to 72B parameters and are optimized for cost efficiency and performance while maintaining high levels of accuracy.

Arcee SLMs are fine-tuned to be task specific, offering general purpose, reasoning, coding, function calling, and vision models. Below is a high level overview of our models:

**General Purpose**

* Arcee Blitz
* Virtuoso Small
* Virtuoso Medium
* Virtuoso Large

**Reasoning**

* Maestro

**Coding**

* Coder Large

**Function Calling**

* Caller Large

See [Model Selection](/arcee-conductor/arcee-small-language-models/model-selection) for more detailed information on each model.


# Model Selection

Discover our collection of Small Language Models (SLMs) fine-tuned by Arcee AI, each optimized for specific tasks and designed to power efficient, production-ready applications.

<details>

<summary>Blitz - General Purpose</summary>

### **Arcee Blitz**

* **Description:** Arcee-Blitz (24B) is a new Mistral-based 24B model distilled from DeepSeek, designed to be both **fast and efficient**. We view it as a practical “workhorse” model that can tackle a range of tasks without the overhead of larger architectures.
  * **#Parameters:** 24B
  * **Base Model:** Mistral-Small-24B-Instruct-2501
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Arcee-Blitz](https://huggingface.co/arcee-ai/Arcee-Blitz)
* **Top Use Cases:**
  * General-purpose task handling
  * Business communication
  * Automated document processing for mid-scale applications

</details>

<figure><img src="/files/UQmMv66S49T3AFJ83XIs" alt="" width="250"><figcaption><p>Blitz</p></figcaption></figure>

<details>

<summary>Virtuoso  (Small, Large, Medium) - General Purpose</summary>

### **Virtuoso Large**

* **Description:** Our most powerful and versatile general-purpose model, designed to excel at handling complex and varied tasks across domains. With state-of-the-art performance, it offers unparalleled capability for nuanced understanding, contextual adaptability, and high accuracy.
  * **#Parameters:** 72B
  * **Base Model:** Qwen-2.5-72B
  * API Access is Available via Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
* **Top Use Cases:**
  * Advanced content creation, such as technical writing and creative storytelling
  * Data summarization and report generation for cross-functional domains
  * Detailed knowledge synthesis and deep-dive insights from diverse datasets
  * Multilingual support for international operations and communications

### **Virtuoso Medium**

* **Description:** A versatile and powerful model, capable of handling complex and varied tasks with precision and adaptability across multiple domains. Ideal for dynamic use cases requiring significant computational power.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * API Access is Available via Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
* **Top Use Cases:**
  * Content generation
  * Knowledge retrieval
  * Advanced language understanding
  * Comprehensive data interpretation

### **Virtuoso Small**

* **Description:** A streamlined version of Virtuoso, maintaining robust capabilities for handling complex tasks across domains while offering enhanced cost-efficiency and quicker response times.
  * **#Parameters:** 14B
  * **Base Model:** Qwen-2.5-14B
  * API access is available via Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
  * Open-source and available on Hugging Face under the Apache-2.0 license: [arcee-ai/Virtuoso-Small](https://huggingface.co/arcee-ai/Virtuoso-Small)&#x20;
* **Top use cases:**&#x20;
  * General-purpose task handling
  * Business communication
  * Automated document processing for mid-scale applications

\`

</details>

<figure><img src="/files/D43NQz6903eIrpJQElkS" alt="" width="250"><figcaption><p>Virtuoso</p></figcaption></figure>

<details>

<summary>Coder (Small, Large) - Coding</summary>

### Coder Large

* **Description:** A high-performance model tailored for intricate programming tasks, Coder-Large thrives in software development environments. With its focus on efficiency, reliability, and adaptability, it supports developers in crafting, debugging, and refining code for complex systems.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B-Instruct
  * Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
* **Top use cases:**
  * Writing modular, reusable code across various programming languages
  * Debugging and optimizing performance in large-scale applications
  * Generating efficient algorithms for computationally intensive tasks
  * Supporting DevOps processes, such as script automation and CI/CD pipelines

### Coder Small

* **Description:** A compact, high-performance coding model designed for efficient programming tasks, including generating code, debugging, and optimizing scripts for smaller projects.
  * **#Parameters:** 14B
  * **Base Model:** Qwen-2.5-32B-Instruct
* **Top use cases:**
  * Lightweight development tasks
  * Automated code reviews
  * Generating templates or prototypes quickly, code completion

</details>

<figure><img src="/files/5A5VbznhaGdj91e2ynLz" alt="" width="250"><figcaption><p>Coder</p></figcaption></figure>

<details>

<summary>Caller (Large) - Tool Use and Function Call</summary>

### Caller

* **Description:** Engineered for seamless integrations, Caller-Large is a robust model optimized for managing complex tool-based interactions and API function calls. Its strength lies in precise execution, intelligent orchestration, and effective communication between systems, making it indispensable for sophisticated automation pipelines.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * API Access is Available via Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
* **Top use cases:**
  * Managing integrations between CRMs, ERPs, and other enterprise systems
  * Running multi-step workflows with intelligent condition handling
  * Orchestrating external tool interactions like calendar scheduling, email parsing, or data extraction
  * Real-time monitoring and diagnostics in IoT or SaaS environments

</details>

<figure><img src="/files/7zm6py57w5wEMTenpIZA" alt="" width="250"><figcaption><p>Caller</p></figcaption></figure>

<details>

<summary>Maestro - Reasoning</summary>

### Maestro

* **Description:** An advanced reasoning model optimized for high-performance enterprise applications. Building on the innovative training techniques first deployed in maestro-7b-preview, Maestro-32B offers significantly enhanced reasoning capabilities at scale, rivaling or surpassing leading models like OpenAI’s O1 and DeepSeek’s R1, but at substantially reduced computational costs.
  * **#Parameters:** 32B
  * **Base Model:** Qwen-2.5-32B
  * API Access is Available via Arcee Conductor: [https://conductor.arcee.ai](/arcee-orchestra/introduction-to-arcee-orchestra)
  * Hybrid training method:
    1. Warm-up (SFT Phase): Quick supervised fine-tuning phase to prime the model with high-quality reasoning exemplars.
    2. RL Optimization Phase: Utilizes Reinforcement Learning techniques, specifically designed to boost logical coherence, depth of reasoning, and accurate inference by encouraging problem-solving from fundamental principles.
* **Top Use Cases:**
  * Enterprise decision support systems
  * Complex analytical and logical inference tasks
  * Automated research and analysis workflows
  * Generative reasoning for technical and professional contexts

</details>

<figure><img src="/files/6ndLrL6VnECNrHL833ck" alt="" width="250"><figcaption><p>Maestro</p></figcaption></figure>


# Model Performance

This section demonstrates our models' performance across different domains through standard benchmarks and real-world applications.

<figure><img src="/files/5IPop6YVWaVsM4FrDzZi" alt=""><figcaption><p>Evaluation Results of Arcee Models on <a href="https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/">Open LLM Leaderboard </a>Benchmarks</p></figcaption></figure>


# Model Capabilities


# Text Generation and Analysis

In this example, you will learn how to use `Virtuoso-Large` , for text generation, creative writing, and text analysis and comparison.&#x20;

### Prerequisites

* Python 3.10 or higher
* `httpx` library
* `openai` library
* API key for accessing the Arcee.ai models

### Step 1: Environment Setup

1. Create and activate a Python virtual environment:

```bash
Copypython -m venv env-openai-client
source env-openai-client/bin/activate  # On Unix/macOS
# or
.\env-openai-client\Scripts\activate  # On Windows
```

2. Install required packages:

```bash
pip install httpx openai
```

3. Create `api_key.py` file:

```python
api_key = "your_api_key_here"
```

### Step 2: Initialize the Virtuoso Client

Set up the OpenAI client specifically for the Virtuoso Large model:

```python
import httpx
import os
from openai import OpenAI
from api_key import api_key

endpoint = "https://models.arcee.ai/v1"
model = "virtuoso-large"  # Specific model for creative and analytical tasks

client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
    http_client=httpx.Client(http2=True)
)
```

### Step 3: Create Response Handler

Set up a function to handle streaming responses:

```python
def print_streaming_response(response):
    num_tokens = 0
    for message in response:
        if len(message.choices) > 0:
            num_tokens += 1
            print(message.choices[0].delta.content, end="")
    print(f"\n\nNumber of tokens: {num_tokens}")
```

### Step 4: Testing Creative Writing Capabilities

Example of generating creative content:

```python
response = client.chat.completions.create(
    model=model,
    messages=[
        {'role': 'user', 
         'content': 'Write a short horror story in the style of HP Lovecraft. It should take place in the 1920s in Antarctica. Write at least 2000 words.'
        }   
    ],
    temperature=0.9,
    stream=True,
    max_tokens=16384
)

print_streaming_response(response)
```

### Step 5: Text Analysis and Comparison

Example of analyzing and comparing literary texts:

```python
# Read and analyze the first text
with open("alice.txt", "r") as file:
    book_text1 = file.read()

num_words = len(book_text1.split())
print(f"Number of words in text 1: {num_words}")

# Read and analyze second text
with open("gatsby.txt", "r") as file:
    book_text2 = file.read()

num_words = len(book_text2.split())
print(f"Number of words in text 2: {num_words}")

# Generate comparative analysis
response = client.chat.completions.create(
    model=model,
    messages=[
        {'role': 'user', 
         'content': f"""Draw a parallel between the main characters of these two books.
         
         First text: {book_text1}
         
         Second text: {book_text2}"""
        }   
    ],
    temperature=0.9,
    stream=True,
    max_tokens=2048
)

print_streaming_response(response)
```

### Best Practices for Virtuoso Large

1. **Creative Writing Tasks**:
   * Be specific about style, genre, and length
   * Provide context and time period if relevant
   * Specify any particular themes or elements to include
   * Use a higher temperature (0.9) for more creative outputs
2. **Text Analysis Tasks**:
   * Provide complete texts for analysis
   * Specify the type of analysis needed
   * Consider token limits when analyzing large texts
   * Use a lower temperature (0.7) for a more focused analysis
3. **File Handling**:
   * Always use proper error handling when reading files
   * Check the file size before processing
   * Consider chunking large texts if needed


# Simple RAG

Retrieval-augmented generation (RAG) is a powerful framework that combines the strengths of Large Language Models (LLMs) with information retrieval systems. At its core, RAG enables AI systems to generate more accurate, contextual, and factual responses by accessing and leveraging external knowledge bases. This approach addresses one of the key limitations of traditional LLMs: their inability to access up-to-date or specific information beyond their training data.

### How RAG Works

RAG operates through two fundamental steps:

1. **Retrieval Phase**: Documents are converted into vector embeddings. These embeddings are stored in a vector database. When a query is received, the relevant information is retrieved using semantic search. Then, the system identifies and extracts the most pertinent pieces of information.
2. **Generation Phase**: Retrieved information is intelligently incorporated into the prompt. The LLM uses this context to generate informed, accurate responses. The response combines the model's inherent knowledge with the retrieved information.

In this notebook, we explore multiple ways in which we can use Arcee's SLMs (Small Language Models) in aiding us to implement efficient Retrieval-Augmented Generation (RAG) pipelines.

<figure><img src="/files/i1N7JIr4K4W6T8PZjzBW" alt=""><figcaption><p>Basic RAG Overview</p></figcaption></figure>

First, let's install the necessary packages for downloading data from the web. We first have to ensure that this data is not present in the training corpus of the SLMs (Small Language Models) we're using for RAG (Retrieval-Augmented Generation). This verification is important to confirm that the RAG system is functioning properly and that the LLM is truly retrieving information rather than responding with knowledge from its parameters.

Below is the list of packages that need to be installed :

1. `httpx`
2. `openai`
3. `requests`
4. `python-dotenv`
5. `voyageai`
6. `trafilatura`
7. `lxml_html_clean`

```bash
! pip install 'httpx[http2]'
! pip install openai requests python-dotenv voyageai trafilatura lxml_html_clean
```

## Downloading the Documents

We first download the document that has been recently published on reasoning models and save this document for further processing.

```python
import requests
from lxml_html_clean import Cleaner
from lxml import html

# Example usage
url = "https://sebastianraschka.com/blog/2025/understanding-reasoning-llms.html"
cleaner = Cleaner(
        style=True,
        links=True,
        add_nofollow=True,
        page_structure=False,
        safe_attrs_only=True,
        remove_tags=['span', 'div', 'aside', 'nav']
    )

def clean_webpage(url, cleaner):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Connection': 'keep-alive',
    }
    
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        
        doc = html.fromstring(response.content)
    
        cleaned_doc = cleaner.clean_html(doc)
        text = cleaned_doc.text_content()
        text = ' '.join(text.split())
        
        return text
        
    except Exception as e:
        print(f"Error: {e}")
        return None


cleaned_text = clean_webpage(url, cleaner)

if cleaned_text:
    with open('cleaned_article.txt', 'w', encoding='utf-8') as f:
        f.write(cleaned_text)

# Let us print the first 200 characters for verifying.
print(cleaned_text[:200])
print("Length of the text : ", len(cleaned_text))
```

Then, import the libraries needed for the API call. Also, please save your `OPENAI_API_KEY` and `OPENAI_BASE_URL` and your `VOYAGE_API_KEY` in your `.env` file

```python
import os
from dotenv import load_dotenv
import numpy as np
import os

import httpx
from openai import OpenAI

load_dotenv()

client = OpenAI(
  http_client=httpx.Client(http2=True),
  base_url=os.environ["OPENAI_BASE_URL"],
  api_key=os.environ["OPENAI_API_KEY"]
)
```

#### RAG from Scratch

In this section, we explore how to implement RAG using manual document chunking and by using Faiss as the backbone to retrieve relevant documents for a question. In this example, we are using Voyage for extracting the embeddings from documents and later, we use it for retrieving relevant documents.

For more information: <https://docs.voyageai.com/docs/introduction>

#### Split the Document into Chunks

In a RAG system, it is crucial to split the document into smaller chunks so that it’s more effective to identify and retrieve the most relevant information in the retrieval process later. In this example, we simply split our text by character, combine 2048 characters into each chunk, and we get 37 chunks in total.

```python
chunk_size = 2048
chunks = [cleaned_text[i:i + chunk_size] for i in range(0, len(cleaned_text), chunk_size)]
```

**Considerations:**

* **Chunk size**: Depending on your specific use case, it may be necessary to customize or experiment with different chunk sizes and chunk overlap to achieve optimal performance in RAG. For example, smaller chunks can be more beneficial in retrieval processes, as larger text chunks often contain filler text that can obscure the semantic representation. As such, using smaller text chunks in the retrieval process can enable the RAG system to identify and extract relevant information more effectively and accurately. However, it’s worth considering the trade-offs that come with using smaller chunks, such as increasing processing time and computational resources.
* **How to split**: While the simplest method is to split the text by character, there are other options depending on the use case and document structure. For example, to avoid exceeding token limits in API calls, it may be necessary to split the text by tokens. To maintain the cohesiveness of the chunks, it can be useful to split the text into sentences, paragraphs, or HTML headers. If working with code, it’s often recommended to split by meaningful code chunks for example using an Abstract Syntax Tree (AST) parser.

#### Create Embeddings for Each Text Chunk

For each text chunk, we then need to create text embeddings, which are numeric representations of the text in the vector space. Words with similar meanings are expected to be in closer proximity or have a shorter distance in the vector space. To create an embedding, we will use Voyage AI's API endpoint and the embedding model `voyage-3-large`. We create a `get_text_embedding` to get the embedding from a single text chunk and then, we use list comprehension to get text embeddings for all text chunks.

```python
import voyageai
vo = voyageai.Client()
sentences = [
    "Renewable energy sources are becoming increasingly important",
    "Multi agent systems combined with reasoning models would be able to transform the Tech landscape ",
    "Quantum mechanics revolutionized our understanding of atomic behavior.",
]
result = vo.embed(sentences, model="voyage-3")

result_embeddings = np.array(result.embeddings)
print(result_embeddings.shape)
```

```python
def get_text_embedding(text):
    result = vo.embed([text], model="voyage-3")
    vec_array = np.array(result.embeddings)
    return vec_array

text_embeddings = np.array([get_text_embedding(chunk) for chunk in chunks])
```

```python
text_embeddings = text_embeddings.squeeze()
text_embeddings.shape
```

#### Load into a Vector Database

Once we get the text embeddings, a common practice is to store them in a vector database for efficient processing and retrieval. There are several vector databases to choose from. In our simple example, we are using an open-source vector database Faiss, which allows for efficient similarity search.

With Faiss, we instantiate an instance of the Index class, which defines the indexing structure of the vector database. We then add the text embeddings to this indexing structure.

Please install `faiss-gpu` using the following command:

`! conda install -c pytorch -c nvidia faiss-gpu=1.9.0`

Refer to the [Faiss documentation](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md) for more information.

```python
import faiss

d = text_embeddings.shape[1]
index = faiss.IndexFlatL2(d)
index.add(text_embeddings)
```

#### Create Embeddings for a Question

Whenever users ask a question, we also need to create embeddings for this question using the same embedding models as before.

```python
question = "Explain about the DeepSeek-R1-Zero Large language model."
question_embeddings = np.array([get_text_embedding(question)])
# squeeze the array
question_embeddings = np.squeeze(question_embeddings, axis=0)
question_embeddings.shape
question_embeddings
```

**Considerations:**

**Hypothetical Document Embeddings (HyDE):** In some cases, the user’s question might not be the most relevant query to use for identifying the relevant context. Instead, it may be more effective to generate a hypothetical answer or a hypothetical document based on the user’s query and use the embeddings of the generated text to retrieve similar text chunks.

#### Retrieve Similar Chunks from the Vector Database

We can perform a search on the vector database with `index.search`, which takes two arguments: the first is the vector embeddings of the question, and the second is the number of similar vectors to retrieve. This function returns the distances and the indices of the most similar vectors to the question vector in the vector database. Then based on the returned indices, we can retrieve the actual relevant text chunks that correspond to those indices.

```python
D, I = index.search(question_embeddings, k=2)
print(I)
```

```python
retrieved_chunk = [chunks[i] for i in I.tolist()[0]]
print(retrieved_chunk)
```

**Considerations:**

* **Retrieval Methods**: There are a lot of different retrieval strategies. In our example, we are showing a simple similarity search with embeddings. Sometimes when there is metadata available for the data, it’s better to filter the data based on the metadata first before performing similarity search. There are also other statistical retrieval methods like TF-IDF and BM25 that use frequency and distribution of terms in the document to identify relevant text chunks.
* **Retrieved Document**: Do we always retrieve individual text chunks as it is? Not always.
  * Sometimes, we would like to include more context around the actual retrieved text chunk. We call the actual retrieve text chunk "child chunk" and our goal is to retrieve a larger "parent chunk" that the "child chunk" belongs to.
  * On occasion, we might also want to provide weights to our retrieved documents. For example, a time-weighted approach would help us retrieve the most recent document.
  * One common issue in the retrieval process is the "lost in the middle" problem where the information in the middle of a long context gets lost. Our models have tried to mitigate this issue. For example, in the passkey task, our models have demonstrated the ability to find a "needle in a haystack" by retrieving a randomly inserted passkey within a long prompt, up to 32k context length. However, it is worth considering experimenting with reordering the document to determine if placing the most relevant chunks at the beginning and end leads to improved results.

#### Combine Context and Question in a Prompt and Generate a Response

Finally, we can offer the retrieved text chunks as the context information within the prompt. Here is a prompt template where we can include both the retrieved text and user questions in the prompt.

```python
prompt = f"""
Context information is below.
---------------------
{retrieved_chunk}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: {question}
Answer:
"""
```

```python
import textwrap


def run_virtuoso(prompt):

  response = client.chat.completions.create(
    model='virtuoso-small',
    messages=[{'role': 'user', 'content': prompt}],
    temperature=0.4,
  )
  return response


response = run_virtuoso(prompt)
answer_text = response.choices[0].message.content


print(textwrap.fill(answer_text, width=80))
```


# Function Calling

Function calling lets Arcee models connect to external tools like user-defined functions or APIs. This integration helps build applications for specific use cases. In this section, we defined three functions for getting stock prices and company information, enabling answers to stock market queries. In this example, we used the Caller model, our specialized SLM trained for tool use and function calling, and Virtuoso Large, for final output generation via LLM reasoning.&#x20;

Here are the four steps to do function calling with Arcee Models:

You should first define the tools and then, pass the user prompt alongside the list of tools to the model.

<figure><img src="/files/dtRafw889iecC97YYr5y" alt=""><figcaption></figcaption></figure>

Then, the model will detect the required function and extract the function arguments from the user query. &#x20;

<figure><img src="/files/fcOIy05ZQDH70f6VN7ZV" alt=""><figcaption></figcaption></figure>

Next, the model will call the tool to get the relevant information.

<figure><img src="/files/44hqMwPiU4WdX7gTpDOT" alt=""><figcaption></figcaption></figure>

Finally, the model will integrate the output of function calling in the final response.

<figure><img src="/files/mCrPiaAxRD3sXpRzBIX0" alt=""><figcaption></figcaption></figure>

Here, you can find the step-by-step implementation of an example pipeline showing how to use Yahoo Finance API using the function calling and tool use capability of the Arcee models.&#x20;

### Step 1: Installing Required Libraries

```python
pip install -qU httpx[http2] yfinance openai
```

This command installs and upgrades three essential Python libraries:

1. **`httpx[http2]`**:
   * `httpx` is an HTTP client for Python that provides asynchronous support.
   * The `[http2]` extra enables HTTP/2 support, which improves efficiency in communication with APIs.
2. **`yfinance`**:
   * A Python library that allows easy access to stock market data from Yahoo Finance.
   * Useful for retrieving historical stock prices, company financials, and real-time data.
3. **`openai`**:
   * The OpenAI API client library is required for making API requests to Arcee models.

The `-qU` flag:

* `-q` (quiet) suppresses unnecessary output.
* `-U` (upgrade) ensures that the latest versions of the packages are installed.

This setup ensures that all dependencies required for function calling, stock data retrieval, and HTTP requests are available in your environment.

### Step 2: Importing Required Libraries

```notebook-python
import httpx
import json
import pprint
import yfinance
from openai import OpenAI
```

* **`httpx`**: Used for making HTTP requests, particularly with HTTP/2 support.
* **`json`**: Standard Python library for handling JSON data.
* **`pprint`**: Pretty Print module to display data in a structured and readable format.
* **`yfinance`**: Library for fetching stock market data from Yahoo Finance.
* **`OpenAI`**: OpenAI client library to interact with the Arcee Model APIs.

### Step 3: Setting Up API Endpoint and Key

```python
endpoint="https://conductor.arcee.ai/v1"
api_key="YOUR API KEY GOES HERE"
```

* `endpoint`: The base URL for the API that will be used for function calling.
* `api_key`: The API key required to authenticate requests.&#x20;

{% hint style="info" %}
Note! Make sure not to expose API keys in production environments for security reasons.
{% endhint %}

### Step 4: Initializing the Client

```python
client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
    http_client=httpx.Client(http2=True)
)
```

* **`OpenAI()`**: Creates an OpenAI client instance to interact with the model.
* **`base_url=endpoint`**: Specifies the endpoint where API requests will be sent.
* **`api_key=api_key`**: Provides authentication for accessing the API.
* **`http_client=httpx.Client(http2=True)`**:
  * Configures the HTTP client with HTTP/2 support for faster, more efficient communication.

At this stage, the client is ready to send API requests.

### Step 5: Defining  the Functions for Stock Market Research

This section defines three functions that use `yfinance` to fetch stock prices, CEO names, and company summary information.

```python
def get_stock_price(company_name, stock_symbol):
    stock = yfinance.Ticker(stock_symbol)
    price = stock.history(period="1d")["Close"].values[0]
    return f"The last closing price of {company_name} ({stock_symbol}) was ${price:.2f}."

def get_ceo_name(company_name, stock_symbol):
    stock = yfinance.Ticker(stock_symbol)
    info = stock.info
    ceo = info['companyOfficers'][0]['name']
    return f"The CEO of {company_name} is {ceo}. The full job title is {info['companyOfficers'][0]['title']}."

def get_company_information(company_name, stock_symbol):
    stock = yfinance.Ticker(stock_symbol)
    summary = stock.info['longBusinessSummary']
    return summary
```

<figure><img src="/files/qbk0daGrZoBLQCTOpmgL" alt=""><figcaption></figcaption></figure>

### Step 6: Defining Tools for Function Calling

This section defines a list of tools that will be available for function calling via the Model Engine API.

````python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_stock_price",
            "description": "Use this function to get the last price of a stock",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "A company name (e.g., Mc Donalds)",
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "A company stock ticker (e.g., MCD)",
                    },
                },
                "required": ["company_name", "stock_symbol"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_ceo_name",
            "description": "Use this function to get the name of a company's CEO",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "A company name (e.g., Mc Donalds)",
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "A company stock ticker (e.g., MCD)",
                    },
                },
                "required": ["company_name", "stock_symbol"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_company_summary",
            "description": "Use this function to describe a company's activities, products, services, and customers",
            "parameters": {
                "type": "object",
                "properties": {
                    "company_name": {
                        "type": "string",
                        "description": "A company name (e.g., Mc Donalds)",
                    },
                    "stock_symbol": {
                        "type": "string",
                        "description": "A company stock ticker (e.g., MCD)",
                    },
                },
                "required": ["company_name", "stock_symbol"],
            },
        },
    }
]
```
````

### Step 7 - Defining `call_tool`

This function **`call_tool`** is responsible for:

1. Sending a **user prompt** to the Caller model.
2. Automatically determining if a function call is needed.
3. Extracting the function and arguments from the API response.
4. Dynamically calling the corresponding function.

````python
def call_tool(user_prompt, max_tokens=128):
    response = client.chat.completions.create(
        model="caller",
        messages=[

            {
                "role": "user",
                "content": user_prompt
            }
        ],
        tools=tools,
        tool_choice="auto",
        max_tokens=max_tokens,
    )

    tool_calls = response.choices[0].message.tool_calls

    # Check if there are any tool calls
    if tool_calls:
        # Extract the first tool call (assuming there's at least one)
        first_tool_call = tool_calls[0]

        # Extract function name and arguments
        function_name = first_tool_call.function.name
        arguments_json = first_tool_call.function.arguments
        arguments_dict = json.loads(arguments_json)

        ## Assuming the function is in the current namespace or imported
        if function_name in globals():
            # Get the function object based on its name
            function_to_call = globals()[function_name]

            print(f"Calling {function_name} with arguments: {arguments_dict}")

            # Call the function with unpacked keyword arguments
            result = function_to_call(**arguments_dict)
            return result
        else:
            print(f"Function {function_name} not found in the global namespace.")
            return None
    else:
        # No tool call: print the generated response
        print("No tool call")
        return None

```
````

### Step 8 - Defining `call_tool_and_invoke_model`

The function **`call_tool_and_invoke_model`** extends `call_tool` by:

1. Calling the function using OpenAI’s **function calling** mechanism.
2. Passing the tool result to the Arcee model, **`virtuoso-large`**.
3. Generating a **final response** that intelligently uses the tool’s output.

This approach combines function calling with LLM reasoning, ensuring accuracy and context-aware responses. If function calling fails, the model still provides a meaningful answer.

```python
def call_tool_and_invoke_model(user_prompt, max_tokens=1024):
    tool_result = call_tool(user_prompt)

    response = client.chat.completions.create(
        model="virtuoso-large",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful and knowledgeable assistant giving sharp answers. Use a business-oriented tone."
            },
            {
                "role": "user",
                "content": f"""Answer the following question: {user_prompt} using the tool result: {tool_result}.
                If the tool result is empty or not useful, say it is not useful and answer the question without using the information.
                If the tool result is useful, you can complement it with your own knowledge as long as it's not contradictory.
                """
            }
        ],
        max_tokens=max_tokens
    )
    return response.choices[0].message.content
```

<figure><img src="/files/4Q0b4efebtGE9poLMMxa" alt=""><figcaption></figcaption></figure>

## Example Workflow

#### User Input

```
user_prompt = "What's the last closing price of Chipotle stock?"
```

#### Step 1 - Function Call Execution

```python
response = call_tool(user_prompt)
pprint.pprint(response)
```

#### Step 2 - Function Output

```
Calling get_stock_price with arguments: {'company_name': 'Chipotle', 'stock_symbol': 'CMG'} 'The last closing price of Chipotle (CMG) was $58.35.'
```

#### Step 3 - Function Call and Response Generation

```python
response = call_tool_and_invoke_model(user_prompt)
pprint.pprint(response)
```

#### Step 4 - Model Output

```
Calling get_stock_price with arguments: {'company_name': 'Chipotle', 'stock_symbol': 'CMG'} ('The last closing price of Chipotle (CMG) was $58.35. This price reflects the ' "most recent trading day's closing value for the company's stock. It's " 'important to note that stock prices can fluctuate based on market ' 'conditions, company performance, and broader economic factors, so this ' 'figure may change with new trading sessions.')
```


# Code Generation

In this example, you will learn how to use the Arcee coder model for a coding problem.

### Prerequisites

* Python 3.12 or higher
* `httpx` library
* `openai` library
* API key for accessing the Arcee.ai models

### Step 1: Setting Up the Environment

1. Create a new Python virtual environment:

```bash
python -m venv env-openai-client
source env-openai-client/bin/activate  # On Unix/macOS
# or
.\env-openai-client\Scripts\activate  # On Windows
```

2. Install the required packages:

```bash
pip install httpx openai
```

3. Create a file named `api_key.py` containing your API key:

```python
api_key = "your_api_key_here"
```

### Step 2: Initialize the Coder Client

Create a new Jupyter Notebook or Python script and set up the OpenAI client specifically for the Coder model:

```python
import httpx
import os
from openai import OpenAI
from api_key import api_key

endpoint = "https://models.arcee.ai/v1"
model = "coder"  # Arcee's specialized SLM for coding tasks

client = OpenAI(
    base_url=endpoint,
    api_key=api_key,
    http_client=httpx.Client(http2=True)
)
```

### Step 3: Set Up the Response Handler

Create a helper function to handle streaming responses:

```python
def print_streaming_response(response):
    num_tokens = 0
    for message in response:
        if len(message.choices) > 0:
            num_tokens += 1
            print(message.choices[0].delta.content, end="")
    print(f"\n\nNumber of tokens: {num_tokens}")
```

### Step 4: Testing Technical Explanation Capabilities

Test the model's ability to explain complex technical concepts with code examples:

```python
response = client.chat.completions.create(
    model=model,
    messages=[
        {'role': 'user', 
         'content': """Explain the difference between logit-based distillation 
         and hidden state distillation. Show an example for both with Pytorch code, 
         with BERT-Large as the teacher model, and BERT-Base as the student model."""
        }   
    ],
    temperature=0.9,
    stream=True,
    max_tokens=16384
)

print_streaming_response(response)
```

### Step 5: Testing Code Review and Improvement Capabilities

You can use the model to review and improve existing code:

```python
code_example = """
def print_streaming_response(response):
    num_tokens=0
    for message in response:
        if len(message.choices) > 0:
            num_tokens+=1
            print(message.choices[0].delta.content, end="")
    print(f"\\n\\nNumber of tokens: {num_tokens}")
"""

response = client.chat.completions.create(
    model=model,
    messages=[
        {'role': 'user', 
         'content': f"Improve the following code: {code_example}. Explain why your changes are an improvement."
        }   
    ],
    temperature=0.9,
    stream=True,
    max_tokens=2048
)

print_streaming_response(response)
```

### Best Practices for Using the Coder Model

1. **Specific Prompts**:
   * Be specific about the programming language
   * Specify the framework or library you're using
   * Mention any version requirements
   * Include context about the problem you're trying to solve
2. **Code Review Requests**:
   * Include the complete code snippet you want to review
   * Specify what aspects you want to improve (performance, readability, security, etc.)
   * Ask for explanations of suggested improvements
3. **Technical Explanations**:
   * Request specific examples alongside theoretical explanations
   * Ask for comparisons between different approaches
   * Request code snippets that demonstrate the concepts


# Pricing

Arcee Conductor only charges you for the price of the model inference. There is **no** premium added to 3rd party models. You simply pay the API price for the model you are routed to.

For the models currently in Conductor, the price for each model is:

<figure><img src="/files/n5uPsMVjuitv0ZOIoQaY" alt=""><figcaption><p>Conductor Model Token Cost</p></figcaption></figure>


# Introduction to Arcee Orchestra

Arcee Orchestra is an end-to-end agentic platform for building AI workflows. &#x20;

<figure><img src="/files/JOKmtNoeXA2x57hbRLWf" alt="" width="563"><figcaption><p>Arcee Orchestra</p></figcaption></figure>

### **Current Way to Build AI Applications**

Building and scaling AI agents, agentic workflows, and AI applications is complex and difficult. At a minimum, you need to determine or develop:

1. What models to use
2. A framework to orchestrate requests
3. Tools (APIs, functions, external systems) to integrate
4. Monitoring and observability capabilities to track performance and costs
5. Hardware to run on

This complexity is compounded when building multiple agents to work together and then scaling your solution. Additionally, when you're piecing all these components together from different providers, one change from a single provider can impact the performance of your solution or potentially break it entirely.

### The Solution

Arcee Orchestra provides all these components in one, easy to use, low-code/no-code UI that can be consumed through SaaS or deployed fully in your environment. This means you know your agents will run the way you expect regardless of scale, and you can meet the most strict compliance guidelines and regulations because data never needs to leave your environment.

Orchestra is built on top of Arcee's state-of-the-art suite of small language models (SLMs) which are trained to perform exceptionally well in agentic workflows.&#x20;

There are 4 primary components to Orchestra:

1. Arcee Small Language Models - purpose built SLMs, which excel at instruction following and understanding API data, enabling reliable utilization of AI.
2. Workflows - allow you to build automations consisting of calls to SLMs/LLMs, connections to external systems, code execution, and more.
3. Integrations - built-in integrations to the most popular systems to enable easy interaction between external systems.&#x20;
4. Chat UI - single interface to chat with your models and invoke workflow automations/agents.

To see how integrations, workflows, the chat UI, and the model router work together, continue to [Getting Started](/arcee-orchestra/getting-started).


# Getting Started

{% embed url="<https://youtu.be/wDzB7bR3fOE>" %}
Getting Started with Arcee Orchestra
{% endembed %}

When you log in to Arcee Orchestra, you are presented with the [Chat UI](/arcee-orchestra/chat-interface). This is where you're able to interact with SLMs, such as Arcee's Virtuoso Models, and be able to call workflows.&#x20;

[Workflows](/arcee-orchestra/workflows) are automated processes that combine deterministic steps with AI and agentic capabilities. Workflows are made up of [components](/arcee-orchestra/workflows/workflow-components), or steps, which provide full flexibility to customize the automation to your specific business need.

In addition to connecting directly to your systems using code, Arcee Orchestra provides built-in [integrations](/arcee-orchestra/workflows/workflow-components/integrations) to easily connect to the most popular systems.

{% hint style="info" %}
To get access to Arcee Orchestra, contact our team at <sales@arcee.ai> or book a demo [here](https://www.arcee.ai/book-a-demo).
{% endhint %}


# Workflows

Process automations in Arcee Orchestra are built via workflows. **Workflows** are a collection of actions which automate a specific business task. These actions, aka "components" or "nodes", can be requests to an SLM or LLM, external API calls, code execution, queries to a vector database, integrations into over 200+ built-in connectors, conditionals/decisions, aggregations, and more. Each action will be covered in-depth in [Components](/arcee-orchestra/workflows/workflow-components).

Workflows are executed along a graph node by node allowing for complete flexibility and customization. Take for example the simple workflow below:

<figure><img src="/files/M4Rfuf2RZS7zG5A9Bj0P" alt=""><figcaption><p>News Analysis Workflow</p></figcaption></figure>

1. In the start node, a variable **prompt** is defined which expects text as input.
2. When a prompt is received, two built-in integrations for search, Serpapi and Tavily, are invoked where the query is the prompt entered by the user.
3. The articles retrieved from these integrations are then passed to a model node to analyze the articles and provide an answer to the user's query.
4. The model's response is then provided back to the user.

### Building a Workflow

{% embed url="<https://www.youtube.com/watch?v=o8Evp3v7xEw>" %}
Build Your First Workflow pt 1
{% endembed %}

{% embed url="<https://www.youtube.com/watch?v=qqVgJs4PIKs>" %}
Build Your First Workflow pt 2
{% endembed %}

{% embed url="<https://www.youtube.com/watch?v=WS3ItQhJ9js>" %}
Build Your First Workflow pt 3
{% endembed %}

Workflows are built by defining and connecting nodes together. In the video above, you are walked through how to create a simple workflow to generate a travel itinerary for a given location and have the itinerary emailed to you.

This initial example serves as a basis for how workflow components work together. The real power of Orchestra comes from when you connect your business applications to a workflow to execute relevant and impactful automations.

### Invoking a Workflow

Workflows can be invoked in chat interface or directly via an API.

When invoking workflows through the chat interface, simply select the tool from the tool bar and submit a prompt. The selected model will determine what tool to call based on the prompt and forward the request to the determined workflow. Navigate to [Chat Interface](/arcee-orchestra/chat-interface) for more information. &#x20;

When invoking workflows with an API, simply specify your API key, the workflow ID, and the input parameters. Navigate to [API Invocation](/arcee-orchestra/workflows/api-invocation) and [Execute a Workflow API](/arcee-orchestra/workflows/api-invocation/workflow-execution-api) for more information.

### Scaling Workflows

Workflows scale at the node level meaning when your traffic increases they automatically scale to meet your traffic demands.&#x20;


# Workflow Components

The power of workflows come from the components, or "nodes", which make them up.  Here you will find a brief overview of each node, however, for detailed information on each node and how to effectively use them, navigate to their respective pages.

[Model Node](/arcee-orchestra/workflows/workflow-components/model-node) is used to send requests to an SLM or LLM. Orchestra comes out of the box with Arcee SLMs built to perform exceptionally well in workflows; however, other models can be integrated into the model node as well.

[Code Node](/arcee-orchestra/workflows/workflow-components/code-node) is used to execute code. This is commonly used to execute custom functions, integrate with systems that don't come as out-of-the-box integrations, and provide full flexibility for any action which is not natively built into Orchestra.

[Knowledge Retrieval](/arcee-orchestra/workflows/workflow-components/knowledge-retrieval) provides the capability to store and use your data within workflows. This enables retrieval augmented generation (RAG), semantic search, and document retrieval.

[Integrations](/arcee-orchestra/workflows/workflow-components/integrations) are built-in connections to popular tools and services. This allows for easy connection to systems such as document stores, CRMs, messaging tools, web scraping, and much more.&#x20;

[Conditional](/arcee-orchestra/workflows/workflow-components/conditional-node) is used to branch your workflow based on defined criteria or a threshold.&#x20;


# Model Node

The model node allows you to send requests to small language models (SLMs) and large language models (LLMs). &#x20;

Arcee SLMs are the foundation of Orchestra, as they were trained to excel as agents and within AI networks.&#x20;

### Arcee's Models

Out-of-the box, Orchestra comes with the following Arcee SLMs:

* Virtuoso Large
  * Our most powerful and versatile general-purpose model, designed to excel at handling complex and varied tasks across domains. With state-of-the-art performance, it offers unparalleled capability for nuanced understanding, contextual adaptability, and high accuracy. Its scalability and depth make it ideal for enterprises requiring comprehensive AI solutions.
* Virtuoso Medium
  * A versatile and powerful model, capable of handling complex and varied tasks with precision and adaptability across multiple domains. Ideal for dynamic use cases requiring significant computational power.
* Virtuoso Small
  * A streamlined version of Virtuoso, maintaining robust capabilities for handling complex tasks across domains while offering enhanced cost-efficiency and quicker response times.
* Coder
  * A high-performance model tailored for intricate programming tasks, Coder-Large thrives in software development environments. With its focus on efficiency, reliability, and adaptability, it supports developers in crafting, debugging, and refining code for complex systems.&#x20;

For detailed information on each model and how they're trained to be the most effective models in agentic systems, see [model selection](broken://pages/P274a9ApfI83zhseM4bT).

### Bring Your Own

While Arcee highly recommends using the built-in models for optimal performance, there are times when a customer has their own fine-tuned model or has a requirement to use an outside model. Any model compatible with the OpenAI API can be integrated into Orchestra.

### Using the Model Node

<figure><img src="/files/NTrQtDqRHof3R2yJljq1" alt="" width="375"><figcaption><p>Model Configuration</p></figcaption></figure>

In the model node, you can:

1. Select which model you'd like to use
2. Import any variables from previous nodes in the workflow
3. Write your User and System prompts
4. Specify model parameters in Model Settings

<figure><img src="/files/pUYMNBYLxzjQtgMblJOa" alt="" width="306"><figcaption><p>Model Node Settings</p></figcaption></figure>

In model settings you can define the following settings:

* Max Tokens
  * This defines the maximum number of tokens (or parts of a word) that the model can generate. A higher max\_tokens means the model can produce longer responses and is useful when the output should be longer and more detailed. A lower max\_tokens means the model will be constrained to a shorter response, which is useful to ensure shorter responses.
* Temperature
  * Temperature controls the randomness in output token selection, where lower values make responses more deterministic, while higher values increase creativity and variability. Higher values are useful for creative tasks such as storytelling and brainstorming, while lower values are useful for precise tasks such as coding and analysis.
* Top P
  * Top P is used to limit token selection to a top percentage of potential next tokens. A higher top\_p (for example, top\_p > 0.8), can include less probable tokens leading to a more diverse and creative output. A lower top\_p (for example, top\_p < 0.3), narrows selection to only the most likely tokens leading to a more focused and deterministic output.
* Top K
  * Similar to Top P, Top K limits token selection to the top-k most probable tokens. A higher top\_k will lead to more creative outputs, while a lower top\_k will lead to more precise and deterministic outputs.
* Repetition Penalty
  * Repetition Penalty penalizes repeated use of the same tokens to encourage varied responses and avoid redundancy. Higher values for repetition penalty discourage repetition leading to more diverse output, which is good for creative tasks. Lower values for repetition penalty allow for more repetition, which is good for tasks where creativity isn't a factor such as writing contracts or technical instructions.

### Model Response

```
{
  "variable1": "This can be any text, file, JSON, image, or output from a previous node",
  "variable2": "Multiple variables can be added to integrate data from numerous sources",
  "ad_model": "\"I'm an AI agent designed to help drive innovation within your business. What will you create with Arcee Orchestra?\""
}
```

Output of the model node is a JSON object which lists any variables passed into the model as well as the model's response.

Model output can be referenced using the name of the specific model node. Given the example above, `{{ad_model}}` would return `"I'm an AI agent designed to help drive innovation within your business. What will you create with Arcee Orchestra?"` .


# Code Node

The code node is used to execute code within your workflows and provides full flexibility to connect to systems which aren't built-in integrations and to execute any custom functions.

### Python Environment and Custom Packages

The code node provides a python environment and includes standard request-handling and commonly used libraries. It currently runs on Python 3.10.14. If a package is required which is not natively included in the python environment, you can install it using `subprocess` :&#x20;

```
import subprocess
import sys

# Ensure the steamspypi package is installed
def install_package(package_name):
  try:
    __import__(package_name)
  except ImportError:
    print(f"{package_name} not found. Installing...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", package_name])

install_package("steamspypi")
```

Replace `steamspypi` with the name of the package you want to install.

### Code Node Output

Output from the code node is a JSON object with the following structure:

```
  "code_1": {
    "data": {
      "results": "Hello World from Function",
      "stdout": "Hello World from standard output\n",
      "stderr": "",
      "error": "",
      "sandbox_id": "sandbox-5067"
    },
    "error": null,
    "successfull": true,
    "successful": true
  }
```

### Referencing Code Output

When selecting the output of a code node in a downstream node, you are given 3 options:

1. `code_node_name`
2. `code_node_name.data.stdout`&#x20;
3. `code_node_name.data.results`&#x20;

<details>

<summary>Results</summary>

When you `return` an expression after a function, it can be referenced via `{{code_block_name.data.results}}`.&#x20;

For example, the code:

```
def example():
   welcome = "Welcome to Orchestra"

   return welcome

example()
```

will output:

```
{
  "code_1": {
    "successfull": true,
    "data": {
      "results": "Welcome to Arcee Orchestra",
      "stdout": "",
      "stderr": "",
      "error": "",
      "sandbox_id": "sandbox-32ad"
    },
    "error": null
  }
}
```

`{{code_1.data.results}}` will return "Welcome to Arcee Orchestra".

</details>

<details>

<summary>Standard Output</summary>

When you `print()` an expression, it can be referenced via `{{code_block_name.data.stdout}}`.&#x20;

For example, the code:

```
welcome = "Welcome to Orchestra"

print(welcome)
```

will output:

```
{
  "code_1": {
    "successfull": true,
    "data": {
      "results": "",
      "stdout": "Welcome to Orchestra\n",
      "stderr": "",
      "error": "",
      "sandbox_id": "sandbox-054a"
    },
    "error": null
  }
}
```

`{{code_1.data.stdout}}` will return "Welcome to Arcee Orchestra".

</details>

<details>

<summary>Code Node Name</summary>

Select the code node name for the output mapping when you want to reference a part of the code node output that is not `results` or `stdout`. For example, if you have a code node named `code 1` and you wanted to check if there was an error in the code node, you could use:

`{{code_1.data.error}}`

</details>

### Using Model Output in a Code Node

It's very common to take the output from a model node and pass it into a code node. This is often done to format the response, check for specific conditions or pass the response to custom integrations through APIs. The easiest way to do this is with the following:

```
model_output = f"""{{model_node}}"""

... the rest of your code ...
```

{% hint style="info" %}
This is best when the model output is just text, for example, a summary or analysis. Model output needs to be handled differently if a specific output structure was instructed to the model, such as JSON.
{% endhint %}


# Integrations

Integrations are the built-in connectors to external systems and solutions.

### Authenticating Integrations

Before you can use an integration, you need to connect it to your account. To do this, navigate to the integrations page and select 'Add New Integration'

<figure><img src="/files/nBBYpOSi4O9cZJREoljs" alt=""><figcaption></figcaption></figure>

This will pop up a list with all available built-in integrations. Select the integration you want to establish a connection with.&#x20;

{% hint style="info" %}
Different integrations require different authentication methods including OAuth 2.0, Bearer Token, JWT, or API Key. Additionally, some require additional information, such as an Organization Name or base URL.&#x20;
{% endhint %}

Once you have filled in the required information, select Connect.

Arcee partners with [Composio](https://composio.dev/) to power many integrations, so you will see a page from Composio indicating that your authentication was successful.

### Using Integrations

When working within a workflow, to integrate with a built-in integration, simply select the Integrations Node on the left navigation bar.

Click on the node and then select 'Choose Integration'.

<figure><img src="/files/dpSEKZmkcmZUZ0lp3Uwz" alt="" width="298"><figcaption><p>Integrations</p></figcaption></figure>

From the drop down list, select the integration you'd like to connect to.

You will then see all the actions you can do for that integration. Scroll through all the actions and select the one you want to use.

<figure><img src="/files/064d5AbCkbxJu21cAXwl" alt="" width="300"><figcaption><p>Actions</p></figcaption></figure>

Each integration action will have different parameters to determine what is done or sent via the action. For example, see the Slack integration action for sending a message to a channel:

<figure><img src="/files/vP7NzUXwHEMSs9JOGpGY" alt="" width="304"><figcaption><p>Integration Example</p></figcaption></figure>

In this example, you specify the channel you want to send the message to, any attachments you want to include, the text of the message, and others. Some parameters are required, which is specified by having the 'Required' tag at the top right of the parameter. If you are unsure what a parameter is, hover over the informational pop up to the left of the parameter.&#x20;

To dynamically set a parameter within an integration, refer to the [Passing Variables](/arcee-orchestra/workflows/passing-variables) page.


# Knowledge Retrieval

The knowledge retrieval node is used to store your documents and data for use in workflows, enabling retrieval augmented generation (RAG), semantic search, and document retrieval.

The knowledge retrieval node automates the complexities of setting up a vector database, pre-processing your documents, loading the data into the database, and executing queries to retrieve relevant documents.&#x20;

### Retrieval Node Components:

**Vector Database**

Orchestra Enterprise customers are automatically provisioned with a vector database when their organization is created. Each database is isolated to each customer, meaning your data is secure and only you can access the data in your database.

**Data Upload**

To upload data, select the Knowledge Retrieval Node and click on "+ Add Parameter". This will give you the ability to upload documents. Current supported document types include TXT, PDF, JSON, MD, XLSX, DOCX, and PPTX, with an individual file max size of 15MB.

<figure><img src="/files/0AOx1pZpv6PACRMN6hOr" alt="" width="375"><figcaption><p>Data Upload</p></figcaption></figure>

**Data Pre-Processing**

Once data has been uploaded, it is processed to be optimally stored in the vector database. The data is parsed, cleaned, chunked, embedded, and indexed.&#x20;

1. For TXT and PDF, content is parsed into text segments, or "chunks", and any embedded images undergo object character recognition (OCR) to extract any text.&#x20;
2. Deduplication is applied to remove any redundant data for more relevant search.&#x20;
3. Documents are chunked.
4. Text chunks are vectorized using a top [MTEB](https://github.com/embeddings-benchmark/mteb) model.
5. Vectors are indexed using a proprietary vector indexing algorithm.

**Data Storage**

Data is stored in an index within the vector database.

#### Search Prompt

Once all data is uploaded to the knowledge retrieval node, you specify a prompt which is used to search the database using either vector search, semantic search, or a full text search. The most common approach for setting the search prompt is to dynamically pass a prompt either provided by the user or created earlier in the workflow.

<figure><img src="/files/t5b36cYPDVy3M8yeUVa8" alt="" width="375"><figcaption><p>Knowledge Retrieval Node</p></figcaption></figure>

**Inference / Data Retrieval**

When the knowledge retrieval node is invoked:

1. The same model used to embed the data in the vector database, embeds the prompt.&#x20;
2. Vector Search or Full Text Search is used to retrieve the most relevant documents.&#x20;
3. A reranker is used before returning the relevant data to the workflow.&#x20;

**Knowledge Retrieval Node Output**

The knowledge retrieval returns the model's response to the prompt based on the documents retrieved from the vector database.


# Conditional Node

The conditional node enables dynamic decision-making and branching logic within workflows. It allows workflows to evaluate specific conditions or rules and execute different paths based on the outcome.

<figure><img src="/files/EEYsvo87nc3jyoeaUb4t" alt="" width="563"><figcaption><p>Conditional Node Workflow</p></figcaption></figure>

In the risk analysis example above, the workflow input is passed to a senior analyst model. Based on the determined risk and reward, different actions are taken. If the risk is low enough, a draft proposal is be generated. If the risk is too high, more analysis can be run. If the risk to reward ratio is in the middle, more research can be done with integrations. If the ratio falls outside expected boundaries, company policies can be searched to determine the right course of action. &#x20;

### Setting Up Conditionals

<figure><img src="/files/wu7mZasJK8bV9c3C9GW1" alt="" width="393"><figcaption><p>Condition</p></figcaption></figure>

Each condition has a name and the set of criteria which define it. You specify a variable from an earlier step in the workflow, the condition, and the value to compare it against. Values can be strings or integers/floats depending on which condition is selected. For a single condition, you can specify multiple criteria using `and` or `or` .&#x20;

<figure><img src="/files/TDVn66nd77OOD8piTe0X" alt="" width="304"><figcaption></figcaption></figure>

Once the conditions have been defined, the action which occurs for each condition can be specified within the conditional node, by setting `Then:` or you can drag a connection line to your next node.

#### Supported Operators

* `==`  checks if the selected variable is equal to the provided value
* &#x20;`!=` checks if the selected variable is not equal to the provided value
* `>` checks if the selected variable is greater than the provided value
* `<` checks if the selected variable is less than the provided value
* `>=` checks if the selected variable is greater than or equal to the provided value
* `<=` checks if the selected variable is less than or equal to the provided value
* `in` checks if the selected variable contains the provided value/text
* `not in` checks if the selected variable does not contains the provided value/text


# Passing Variables

Before going into each workflow component/node, it's important to understand how variables are passed between nodes.

Each node will output a JSON object. To reference specific output from a previous node, you can use dot notation.&#x20;

Take the following example where we define an email subject in a code node and want to reference it in the built-in Gmail: Send Email Integration.

<figure><img src="/files/69ooSPTZGeO42JmRuquV" alt=""><figcaption></figcaption></figure>

The code node outputs:

```
"email_subject": {
    "successfull": true,
    "data": {
      "results": "Email from Orchestra",
      "stdout": "",
      "stderr": "",
      "error": "",
      "sandbox_id": "sandbox-5a47"
    },
    "error": null
  }
```

In the Gmail integration node, we can set the Subject variable to be

`{{email_subject.data.results}}`&#x20;

This sets "Email from Orchestra" to be the Subject of the email.

For examples of how to pass variables between nodes, see the workflow examples in the [workflow library](/arcee-orchestra/workflow-library).


# Observability and Monitoring

Arcee Orchestra provides real time workflow execution monitoring as well as execution history to view previous workflow executions.

### Real-Time Monitoring

When you run a workflow in the Orchestra UI, you are given a real-time visual of the workflow executing. This shows you:

1. Which step the workflow is currently running
2. How long it takes each step to execute
3. The input and output of each step
4. If any errors occur, which step the error occurs at

The real-time monitoring not only helps when initially building workflows, but also for optimizing them. For example, using the node execution time you can see the impact of using different sized SLMs/LLMs for specific tasks. View the output to decide if a smaller model provides the level of accuracy you need with the advantage of faster execution time and cheaper cost.

### Execution History


# API Invocation

A common use case for Orchestra workflows is to be integrated into or invoked from an external application.  This can be done using the Workflow API. The Workflow API provides endpoints to list available workflows, execute workflows, track execution status, and more.

### Available APIs

1. [List available workflows](/arcee-orchestra/workflows/api-invocation/list-available-workflows-api)
2. [Workflow Execution](/arcee-orchestra/workflows/api-invocation/workflow-execution-api)
3. [Workflow Execution Steps](/arcee-orchestra/workflows/api-invocation/workflow-execution-steps-api)
4. [Execution History](/arcee-orchestra/workflows/api-invocation/execution-history-api)
5. [Workflow Diagram](/arcee-orchestra/workflows/api-invocation/workflow-diagram-api)

### Swagger Documentation

[Arcee Orchestra Swagger Documentation](https://arcee.portal.swaggerhub.com/orchestra/default/arcee-orchestra-v-1-0-0)

{% hint style="info" %}
Swagger provides an easy way to test and visualize APIs and their required syntax. Open the link above in a new tab, (if you only see a base url, toggle to SwaggerUI) and you'll be presented a list of the Orchestra APIs.
{% endhint %}

### Setting up an API Key

All API requests require authentication using a Bearer token. To generate an API key:

1. Go to Settings --> API Tokens --> Add API Token.
2. Provide a name and select "Get API Token".

<figure><img src="/files/qwfv0uIMElbhRYuWF56q" alt="" width="422"><figcaption><p>Add a New API Token</p></figcaption></figure>

3. Copy the generated API Key. You will not be able to view this again so ensure you store it somewhere secure. You will also be provided code examples for calling the APIs in Javascript and Python.

<figure><img src="/files/LsEz4BBxq5u5gWF3Ere4" alt="" width="375"><figcaption><p>Add New API Token</p></figcaption></figure>

### Core Functionality

Each API call will use the API Key generated with the steps above as well as the base url: `https://orchestra.arcee.ai/api/v1/workflow`.

### Error Handling

All APIs adhere to standard HTTP status codes:

* **200 OK** – Request successful
* **400 Bad Request** – Invalid input
* **401 Unauthorized** – Invalid or missing authentication
* **404 Not Found** – Resource not found
* **500 Internal Server Error** – Server-side error

Proper error handling should be implemented when interacting with the API to manage failures effectively.

### Security Best Practices

To ensure secure API interactions, follow these best practices:

* Store API tokens securely in environment variables or credential stores.
* Use HTTPS for all API requests to prevent data interception.
* Implement proper error handling to avoid exposing sensitive information in API responses.

&#x20;


# List Available Workflows API

**Description:** Retrieves a list of available workflows.

**Try it out:** [Swagger Reference](https://arcee.portal.swaggerhub.com/orchestra/default/arcee-orchestra-v-1-0-0#/default/listWorkflows)

### **Request Syntax:**

```
GET /workflows
Authentication: Bearer <Your API key>
```

### **Response Syntax:**

```json
[
  {
    "workflow_id": "",
    "user_id": "",
    "organization_id": "",
    "created_at": "",
    "created_by": "",
    "description": "",
    "file_metadata": {
        "bucket": "",
        "url": "",
        "key": ""
    },
    "graph": {
        "settings": {
            "support_multiple_edges": boolean
        },
        "input_data": {},
        "description": "",
        "nodes": { ... },
        "edges": [ ... ],
        "entry_point": "",
        "finish_point": "",
        "selectedOutput": "",
        "node_metadata": ""
    },
    "name": "",
    "status": "",
    "updated_at": "",
    "updated_by": "",
    "version": "",
    "public": boolean
    },
    { ...
    }
]
```

<details>

<summary>List Available Workflows Response Example</summary>

Each workflow returned in the API list will include the following information.

<pre><code>[
  {
<strong>    "workflow_id": "1234f567-bb12-1234-b7cd-f7a71a98cd24",
</strong>    "user_id": "2*****************1",
    "organization_id": "2*****************7",
    "created_at": "2025-03-12 00:59:55.465508+00:00",
    "created_by": "2*****************1",
    "description": "A workflow that sends a simple \"hi\" message using the Virtuoso-Large model, starting from a start node, processing the message, and ending with the model's output.",
    "file_metadata": {
        "bucket": "orchestra-workflow-dev2025010618392776890000000a",
        "url": "https://orchestra-workflow-dev2025010618392776890000000a.s3.amazonaws.com/288772212935168457/workflows/hi_20250312_005955_d5f4e21a-f49b-4420-a708-d40b252092be.yaml",
        "key": "288772212935168457/workflows/hi_20250312_005955_d5f4e21a-f49b-4420-a708-d40b252092be.yaml"
    },
    "graph": {
        "settings": {
            "support_multiple_edges": true
        },
        "input_data": {},
        "description": "A workflow that sends a simple \"hi\" message using the Virtuoso-Large model, starting from a start node, processing the message, and ending with the model's output.",
        "nodes": {
            "model_1": {
                "type": "model",
                "metadata": {
                    "description": "Model node",
                    "model": "virtuoso-large",
                    "stream": false,
                    "max_tokens": 8192,
                    "temperature": 0.7,
                    "top_p": 1,
                    "top_k": 50,
                    "repetition_penalty": 0,
                    "messages": [
                        {
                            "role": "system",
                            "content": ""
                        },
                        {
                            "role": "user",
                            "content": "say hi"
                        }
                    ],
                    "output_mapping": {
                        "model_1": "choices[0].message.content"
                    }
                }
            }
        },
        "edges": [
            [
                "START",
                "model_1"
            ],
            [
                "model_1",
                "END"
            ]
        ],
        "entry_point": "model_1",
        "finish_point": "model_1",
        "selectedOutput": "model_1",
        "node_metadata": "{\"nodes\":[{\"id\":\"74326564-3041-41bc-bedc-44053c68bb7c\",\"type\":\"start\",\"data\":{\"inputData\":[],\"description\":\"\",\"outputData\":[],\"messages\":[],\"customName\":\"Start 1\",\"customLabel\":\"start_1\"},\"position\":{\"x\":400,\"y\":250},\"width\":260,\"height\":102},{\"id\":\"eafa038c-222a-4755-b240-a1d10c41fc8d\",\"type\":\"model\",\"data\":{\"inputData\":[],\"model\":{\"fullName\":\"virtuoso-large\",\"shortName\":\"Virtuoso-Large\",\"avatar\":\"/assets/Logo-small-DyzsrxbN.jpg\",\"modelName\":\"virtuoso-large\",\"key\":\"virtuosoLarge\"},\"outputName\":\"model_1\",\"description\":\"Model node\",\"output_mapping\":\"model_1\",\"modelSettings\":{\"max_tokens\":8192,\"temperature\":0.7,\"top_p\":1,\"top_k\":50,\"repetition_penalty\":0,\"stream\":false},\"customName\":\"Model 1\",\"customLabel\":\"model_1\",\"outputData\":[{\"id\":\"3487fe34-6ea3-4e27-b339-e1e919ec3216\",\"type\":\"variable\",\"outputVariable\":{\"variableName\":\"model_1\",\"variableType\":\"Text\",\"nodeName\":\"Model\"}}],\"messages\":[{\"role\":\"system\",\"content\":\"\"},{\"role\":\"user\",\"content\":\"say hi\"}]},\"position\":{\"x\":500,\"y\":360},\"width\":260,\"height\":86,\"selected\":false,\"dragging\":false,\"messages\":[{\"role\":\"system\",\"content\":\"\"},{\"role\":\"user\",\"content\":\"say hi\"}]},{\"id\":\"f5023d4b-fd60-4045-8c57-baf6ad89bfca\",\"type\":\"end\",\"data\":{\"label\":\"End\",\"description\":\"\",\"outputData\":[],\"customName\":\"End 1\",\"customLabel\":\"end_1\",\"output_variable\":{\"model_1\":\"Model 1\"},\"output_mapping\":\"model_1\"},\"position\":{\"x\":675,\"y\":470},\"width\":260,\"height\":132,\"selected\":true,\"dragging\":false}],\"edges\":[{\"id\":\"f88ce626-0f2e-4bf8-b559-8ba1691dbe61\",\"source\":\"74326564-3041-41bc-bedc-44053c68bb7c\",\"target\":\"eafa038c-222a-4755-b240-a1d10c41fc8d\",\"type\":\"default\"},{\"id\":\"a18e985c-67f5-4e15-bee9-d23199a905ad\",\"source\":\"eafa038c-222a-4755-b240-a1d10c41fc8d\",\"target\":\"f5023d4b-fd60-4045-8c57-baf6ad89bfca\",\"type\":\"default\"}],\"viewport\":{},\"avatar\":{\"emoji\":\"🤖\",\"bgColor\":\"hsl(160, 65%, 75%)\"},\"public\":false}"
    },
    "name": "hi",
    "status": "active",
    "updated_at": "2025-03-12 00:59:55.465513+00:00",
    "updated_by": 2*****************1,
    "version": "1.0.0",
    "public": false
    }
]
</code></pre>

</details>

### **Response Syntax Variables**

* **Workflow id**: ID associated with the workflow
* **User id**: ID of the user who created the workflow
* **Organization ID**: ID of the organization where the workflow is stored
* **Created at**: date-time the workflow was created
* **Created by**: User id of the user who created the workflow
* **Description**: Generated description of the workflow. This is what is passed to the routing model to determine when to call the workflow.
* **File Metadata**: Information on where the file is stored
* **Graph**: breakdown of the nodes and connections within the workflow&#x20;
* **Name**: name of the workflow
* **Status**: status of the workflow
  * Can be active or inactive
* **Updated at**: date-time for last time the workflow was updated
* **Updated by**: the user ID of the last person to update the workflow
* **Version**: Version of the workflow
* **Public**: whether the the workflow is public or private. This determines who in the organization can see the workflow


# Workflow Execution API

**Description:** Initiates execution of the specified workflow.

**Try it out:** [Swagger Reference](https://arcee.portal.swaggerhub.com/orchestra/default/arcee-orchestra-v-1-0-0#/default/executeWorkflow)

### **Request Syntax:**

```json
POST /workflows/{workflow_id}/execute
Authentication: Bearer <Your API key>
Content-type: application/json

{
  "input_variable_1": "",
  "input_variable_2": ""
}
```

{% hint style="info" %}
You can get the {workflow\_id} from the workflow page.
{% endhint %}

### **Response Syntax:**

```json
{
    "message": "",
    "run_id": ""
}
```

<details>

<summary>Execute a Workflow Response Example</summary>

```json
{
    "message": "Workflow execution started",
    "run_id": "fda37c90-dd1c-4317-82d1-797b7f77fca5"
}
```

</details>

### **Response Syntax Variables**

* **Message**: Indication that the workflow execution has begun
* **Run id**: ID for this specific execution of the workflow


# Workflow Execution Steps API

**Description:** Retrieves details about each step executed in a specific workflow run. A step is defined as a node within the workflow. A run is a single execution of a workflow.

### **Request Syntax:**

```
GET workflows/{workflow_id}/runs/{run_id}/steps
Authentication: Bearer <Your API key>
```

{% hint style="info" %}
You can get the {run\_id} of an executed workflow from the output of an Execute Workflow API call.&#x20;
{% endhint %}

### **Response Syntax:**

```
[
    {
        "id": "",
        "user_id": "",
        "organization_id": "",
        "node_id": "",
        "status": "",
        "start_time": "",
        "end_time": "",
        "elapsed_time": int,
        "error": "",
        "outputs": { ... },
        "inputs": { ... },
        "step_metadata": null,
        "tokens": 0
    },
    { ... }
]
```

<details>

<summary>Execution Steps Response Example</summary>

This example includes the output for a workflow which took in a prompt, generated a response with a model node and then emailed the response.

```
[
    {
        "id": "b2acc986-af06-418a-bf71-c730d5ghaf0b",
        "user_id": "2*****************1",
        "organization_id": "2*****************7",
        "node_id": "model",
        "status": "completed",
        "start_time": "2025-03-13T02:41:15.144470Z",
        "end_time": "2025-03-13T02:41:17.382162Z",
        "elapsed_time": 2,
        "error": null,
        "outputs": {
            "id": "chatcmpl-a9fd051615c7405e8d299c797ee16223",
            "object": "chat.completion",
            "created": 1741833676,
            "model": "virtuoso-medium",
            "choices": [
                {
                    "index": 0,
                    "message": {
                        "role": "assistant",
                        "reasoning_content": null,
                        "content": "Why was the weather forecast sad?\n\nBecause it had a lot of problems and was feeling a little \"forecast-ticular.\" \n\n(Note: \"Forecast-ticular\" is a play on words, combining \"forecast\" and \"particular.\")",
                        "tool_calls": []
                    },
                    "logprobs": null,
                    "finish_reason": "stop",
                    "stop_reason": null
                }
            ],
            "usage": {
                "prompt_tokens": 36,
                "total_tokens": 84,
                "completion_tokens": 48,
                "prompt_tokens_details": null
            },
            "prompt_logprobs": null,
            "confidence_info": {
                "error": "No logprobs available - model/provider may not support confidence calculation",
                "confidence_score": null,
                "mean_entropy": null,
                "total_entropy": null
            }
        },
        "inputs": {
            "model": "virtuoso-medium",
            "stream": false,
            "max_tokens": 8192,
            "temperature": 0.7,
            "top_p": 1,
            "top_k": 50,
            "repetition_penalty": 0,
            "messages": [
                {
                    "role": "user",
                    "content": "Tell me a joke about the weather"
                }
            ],
            "run_id": "5b69669f-975a-49af-a33a-61ff9ae5a20f",
            "input_data": {
                "prompt": "Tell me a joke about the weather"
            }
        },
        "step_metadata": null,
        "tokens": 0
    },
    {
        "id": "ad6edfea-bf1c-4639-9d9f-b53c9ad60bgg",
        "user_id": "2*****************1",
        "organization_id": "2*****************7",
        "node_id": "slack_integration",
        "status": "completed",
        "start_time": "2025-03-13T02:41:17.403879Z",
        "end_time": "2025-03-13T02:41:19.179523Z",
        "elapsed_time": 1,
        "error": null,
        "outputs": {
            "result": {
                "data": {
                    "ok": true,
                    "channel": "C0257V9FS09",
                    "ts": "1741833679.099979",
                    "warning": "missing_charset",
                    "response_metadata": {
                        "warnings": [
                            "missing_charset"
                        ]
                    }
                },
                "error": null,
                "successfull": true,
                "successful": true
            }
        },
        "inputs": {
            "app_name": "slack",
            "action": "SLACK_SHARE_A_ME_MESSAGE_IN_A_CHANNEL",
            "entity_id": "a***********m",
            "workspace_type": "local",
            "input_mapping": {
                "channel": "orchestra-demo",
                "text": "Why was the weather forecast sad?\n\nBecause it had a lot of problems and was feeling a little \"forecast-ticular.\" \n\n(Note: \"Forecast-ticular\" is a play on words, combining \"forecast\" and \"particular.\")"
            },
            "tool_config": {
                "tool_type": "slack"
            },
            "run_id": "5b69669f-975a-49af-a33a-61ff9ae5a20f",
            "input_data": {
                "prompt": "Tell me a joke about the weather",
                "model": "Why was the weather forecast sad?\n\nBecause it had a lot of problems and was feeling a little \"forecast-ticular.\" \n\n(Note: \"Forecast-ticular\" is a play on words, combining \"forecast\" and \"particular.\")"
            }
        },
        "step_metadata": null,
        "tokens": 0
    }
]
```

</details>

### **Response Syntax Variables**

For each node in the workflow, there will be the following output:

* id: ID associated with the specific step in the workflow
* User id: ID associated with the user who executed the workflow
* Organization id: ID of the organization where the workflow is stored
* Node id: ID for the specific node which is executed during this step
* Status: status of the workflow step
* **Start time**: date-time when the step began execution
* **End time**: date-time when the step finished execution
* **Elapsed time**: Amount of seconds the step took to complete
* **Error**: Any errors which occurred during step execution
* **Outputs**: The output of the step
  * Output data will vary based on the specific node which is executed
* **Inputs**: The input to the step
  * Input data will vary based on the specific node which is executed
* **Step Metadata**: Metadata associated with the specific step
* **Tokens**: number of tokens utilized in the step
  * Currently this is always set to 0


# Execution History API

**Description:** Fetches the last 100 execution runs for a given workflow.

**Try it out:** [Swagger Reference](https://arcee.portal.swaggerhub.com/orchestra/default/arcee-orchestra-v-1-0-0#/default/executionHistory)

### **Request Syntax:**

```json
GET /workflows/{workflow_id}/runs?limit={limit}&offset={offset}
Authentication: Bearer <Your API key>
```

**Query Parameters**:

* **limit (optional)**: the number of executions to include in the response
  * Default is 100
* **offset (optional)**: the index of the first result to be included
  * Default is 0

{% hint style="info" %}
For example, if you set limit=10 and offset=10, the response would include the 10th to the 19th most recent executions.
{% endhint %}

### **Response Syntax:**

"id" refers to the run\_id for the workflow run in the list.

```json
[
    {
        "id": "",
        "status": "",
        "start_time": "",
        "end_time": "",
        "elapsed_time": int,
        "error": "",
        "outputs": "",
        "all_outputs": {
            "node_1": "",
            "node_2": "",
            "node_3": { ...},
        },
        "inputs": {
            "input_variable_1": ""
        }
    },
    { ...
    }
]
```

<details>

<summary>Retrieve all Runs Response Example</summary>

{% code overflow="wrap" %}

```json
[
    {
        "id": "a40d4d53-7e5e-432b-8f23-8afdc53eefa3",
        "status": "completed",
        "start_time": "2025-03-12T03:39:30.476391Z",
        "end_time": "2025-03-12T03:39:36.133074Z",
        "elapsed_time": 5,
        "error": null,
        "outputs": "Here's a weather-related joke for you:\n\nWhy did the cloud cross the sky?\n\nBecause it wanted to go to the other side of the rainbow!\n\nI hope you found this joke sunny and amusing! If you'd like another weather joke or a different kind of joke, please let me know.",
        "all_outputs": {
            "prompt": "Tell me a joke about the weather",
            "model": "Here's a weather-related joke for you:\n\nWhy did the cloud cross the sky?\n\nBecause it wanted to go to the other side of the rainbow!\n\nI hope you found this joke sunny and amusing! If you'd like another weather joke or a different kind of joke, please let me know.",
        },
        "inputs": {
            "prompt": "Tell me a joke about the weather"
        }
    },
    { ...
    }
]
```

{% endcode %}

</details>

### **Response Syntax Variables**

* **ID**: Run id associated with the execution
* **Status**: completion status of the workflow
* **Start time**: date-time when the workflow began execution
* **End time**: date-time when the workflow finished execution
* **Elapsed time**: Amount of seconds the workflow took to complete
* **Error**: Any errors which occured during workflow execution
* **Outputs**: The output of the node selected in the End Node
* **All Outputs**: Outputs of each node
* **Inputs**: the inputs sent to the workflow


# Workflow Diagram API

**Description:** Returns the workflow diagram in Mermaid format.  This can be used to visualize your workflow using Mermaid.

**Try it out:** [Swagger Reference](https://arcee.portal.swaggerhub.com/orchestra/default/arcee-orchestra-v-1-0-0#/default/workflowDiagram)

### **Request Syntax:**

```
GET /workflows/{workflow_id}/diagram
Authentication: Bearer <Your API key>
```

### **Response Syntax:**

```json
{
    "diagram": ""
}
```

<details>

<summary>Retrieve Diagram Response Example</summary>

{% code overflow="wrap" %}

```json
{
    "diagram": "graph TD\n    model[\"model<br/>Model node\"]\n    slack_integration[\"slack_integration<br/>\"]\n    START((START))\n    END((END))\n    slack_integration --> END\n    START --> model\n    model --> slack_integration\n    style model fill:#bbf,stroke:#333,stroke-width:1px,color:#fff\n    style slack_integration fill:#bbf,stroke:#333,stroke-width:1px,color:#fff\n    style START fill:#9f9,stroke:#333,stroke-width:2px,color:#fff\n    style END fill:#f99,stroke:#333,stroke-width:2px,color:#fff"
}
```

{% endcode %}

</details>

### **Response Syntax Variables**

* **Diagram**: a breakdown of the workflow diagram in Mermaid format


# API Code Examples

This page provides code examples for running the APIs in Python.

Before running any of the scripts, run `export ARCEE_API_TOKEN=$YOUR_TOKEN`.

### List Available Workflows

```python
import requests
import os

# Setup
API_BASE_URL = "https://orchestra.arcee.ai/api/v1/workflow"
API_KEY = os.environ.get("ARCEE_API_TOKEN")
headers = {"Authorization": f"Bearer {API_KEY}"}

# List workflows
response = requests.get(f"{API_BASE_URL}/workflows", headers=headers)
print(response.json())
```

### Workflow Execution

```python
import requests
import os

# Setup
API_BASE_URL = "https://orchestra.arcee.ai/api/v1/workflow"
API_KEY = os.environ.get("ARCEE_API_TOKEN")
headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
workflow_id = "YOUR_WORKFLOW_ID"

# Execute a workflow
payload = {"prompt": "Tell me a joke about the weather"}
response = requests.post(f"{API_BASE_URL}/workflows/{workflow_id}/execute", 
                        headers=headers, 
                        json=payload)

print(response.json())
```

### Workflow Execution Steps

```python
import requests
import os

# Setup
API_BASE_URL = "https://orchestra.arcee.ai/api/v1/workflow"
API_KEY = os.environ.get("ARCEE_API_TOKEN")
headers = {"Authorization": f"Bearer {API_KEY}"}
workflow_id = "YOUR_WORKFLOW_ID"
run_id = "YOUR_RUN_ID"

# Get workflow execution steps
response = requests.get(f"{API_BASE_URL}/workflows/{workflow_id}/runs/{run_id}/steps", headers=headers)
run_details = response.json()
print(run_details)
```

### Execution History

```python
import requests
import os

# Setup
API_BASE_URL = "https://orchestra.arcee.ai/api/v1/workflow"
API_KEY = os.environ.get("ARCEE_API_TOKEN")
headers = {"Authorization": f"Bearer {API_KEY}"}
workflow_id = "YOUR_WORKFLOW_ID"
limit = 10
offset = 0

# Get execution history
response = requests.get(f"{API_BASE_URL}/workflows/{workflow_id}/runs?limit={limit}&offset={offset}", headers=headers)
print(response.json())
```

### Workflow Diagram

```python
import requests
import os

# Setup
API_BASE_URL = "https://orchestra.arcee.ai/api/v1/workflow"
API_KEY = os.environ.get("ARCEE_API_TOKEN")
headers = {"Authorization": f"Bearer {API_KEY}"}
workflow_id = "YOUR_WORKFLOW_ID"

# Get workflow diagram
response = requests.get(f"{API_BASE_URL}/workflows/{workflow_id}/diagram", headers=headers)
print(response.json())
```


# Upload Workflow JSON API

Upload workflow JSON definition programatically

You can Download Workflow JSON in the workflow builder.

<figure><img src="/files/J0lov6T7BhiJ5oLzdrnP" alt=""><figcaption><p>Download workflow JSON in the workflow builder</p></figcaption></figure>

You can edit the JSON programmatically and re-upload it with to following API route. Note: it is best practice to edit an existing workflow structure to preserve node layout and positioning.&#x20;

You can optional upload to a new workflow, or update an existing workflow by including `workflow_id` in the post data.&#x20;

```
import json
import requests
import json
import os
from pathlib import Path

with open("YourJson.json", "r") as f:
    workflow_json = json.load(f)

workflow_json_str = json.dumps(workflow_json)

url = f"{API_BASE_URL}/workflows/upload"
headers = {"Authorization": f"Bearer {API_KEY}"}

# Create files dictionary with the JSON content
files = {
    'workflow_file': ('YourJson.json', workflow_json_str.encode('utf-8'), 'application/json')
}

# Make the request
response = requests.post(
    url,
    headers=headers,
    files=files,
)

# # optionally include workflow_id to update an existing workflow
# # data = {
# #     "workflow_id" : "THE_WORKFLOW_ID"
# # }

# # # Make the request
# # response = requests.post(
# #     url,
# #     data,
# #     headers=headers,
# #     files=files,
# # )

# Print the results
print(f"Status code: {response.status_code}")
print(f"Response: {response.json() if response.status_code == 200 else response.text}")
```


# Workflow Runs API

Check a workflow run

You can start a workflow from API with the [Workflow Execution API](/arcee-orchestra/workflows/api-invocation/workflow-execution-api), returning the `run_id` of the workflow in question\
\
To check the status of the run, you can use the following API.

```
response = requests.get(f"{API_BASE_URL}/workflows/{workflow_id}/runs/{run_id}", headers=headers)
run_details = response.json()
run_details
```

This will return the details of the run - such as<br>

```
[{'id': 'THE_RUN_ID',
  'status': 'completed',
  'start_time': '2025-04-24T03:18:12.322630Z',
  'end_time': '2025-04-24T03:18:13.048954Z',
  'elapsed_time': 0,
  'error': None,
  'outputs': {'query': 'Hello from Python',
   'model_1': 'Hello there! How can I assist you today?'},
  'all_outputs': {'query': 'Hello from Python',
   'model_1': 'Hello there! How can I assist you today?'},
  'inputs': {'query': 'Hello from Python'}}]

```

To check the granular status of each step you can check the [Workflow Execution Steps API](/arcee-orchestra/workflows/api-invocation/workflow-execution-steps-api).


# Workflow Library

The workflow library serves as a repository for common use cases and workflows, which you can download and utilize in Arcee Orchestra.

Each use case within the workflow library contains:

1. A video demonstration of the workflow
2. The workflow JSON so you can upload the workflow and try it yourself
3. Example input/output for the specific workflow

These workflows provide examples of how different integrations can be utilized, how nodes can be combined to create complex automations, and can serve as a starting place for one of your workflows.


# Research Automation

The Market Research workflow makes it quick and easy to conduct research on a topic and generate a report with the findings. The workflow takes in a user question, researches the topic, creates both a brief and detailed summary, performs data extraction, generates a report and uploads the report to Google Docs and sends an email.

<figure><img src="/files/N3Yeb5dvaduc4fQQGk3S" alt=""><figcaption><p>Research Automation Workflow</p></figcaption></figure>

### Demonstration

{% embed url="<https://youtu.be/qX5sGp5c91o>" %}
Research Automation Demo Video
{% endembed %}

### Workflow JSON

{% file src="/files/YAgYF8OKUWxgTcQ0QLur" %}
JSON for Research Automation
{% endfile %}

{% hint style="info" %}
To download the workflow, select the JSON file above, then use command/control + S and save with the .json extension.

Once you've uploaded the workflow, you'll need to specify a recipient email in the Gmail integration.
{% endhint %}

### Input/Output Examples

{% tabs %}
{% tab title="Input" %}
Create a report on news for the Asian Markets
{% endtab %}

{% tab title="Output" %}
Email and Google Doc Report Containing Summaries such as:

```
Summary of President Lai warns against China's escalating "United Front" tactics

Brief Summary:
Taiwanese President William Lai Ching-te warned the public to be vigilant against China's intensifying "united front" tactics, as reported by the Taipei Times. 
This warning comes amid the Chinese Communist Party's ongoing efforts to pursue Taiwan's annexation.

Detailed Summary:
Taiwanese President William Lai Ching-te has issued a public warning regarding the escalating "United Front" tactics employed by China. These tactics, which are 
part of a broader strategy by the Chinese Communist Party (CCP), aim to undermine Taiwan's sovereignty and bolster support for unification with China. President Lai's 
remarks, reported by the Taipei Times, highlight the increasing intensity and sophistication of these strategies, which are perceived as a significant threat to Taiwan's 
independence and democratic values. The warning reflects the ongoing tension between the two sides of the Taiwan Strait and underscores the Taiwanese government's commitment 
to maintaining vigilance and resilience in the face of such challenges.

The "United Front" tactics encompass a wide range of activities, including political lobbying, economic influence, and media manipulation, designed to create divisions 
within Taiwanese society and erode public trust in the government. President Lai's call for public awareness and alertness indicates that these tactics have become more 
aggressive and pervasive, posing a serious concern for Taiwan's political and social stability. This development is particularly significant given the broader geopolitical 
context, where China's assertiveness in regional and global affairs has increased, and the United States and other allies are closely monitoring the situation to support 
Taiwan's defense and autonomy.


```

{% endtab %}
{% endtabs %}


# Real Time Financial Analysis

The real-time financial analysis workflow automes the research and generation of a financial report for a public company. The workflow accepts a ticker code as input, uses Python libraries to retrieve financial information and the latest news from Yahoo Finance, summarizes and structures all the information with a small language model and sends a report to the user with Gmail.

<figure><img src="/files/nDsREWhblTI8lZAgFn0w" alt=""><figcaption><p>Real Time Financial Analysis</p></figcaption></figure>

### Demonstration

{% embed url="<https://www.youtube.com/watch?v=yAEgEUxU5A8>" %}
Automated Financial Report with Analyst Recommendation
{% endembed %}

### Workflow JSON

{% file src="/files/TxEuBtC7g4KGwc1i9vOI" %}
JSON for Real Time Financial Analysis Workflow
{% endfile %}

{% hint style="info" %}
To download the workflow, select the JSON file above, then use command/control + S and save with the .json extension.

Once you've uploaded the workflow, you'll need to specify a recipient email in the Gmail integration.
{% endhint %}

### Input/Output Example

{% tabs %}
{% tab title="Input" %}
JPM
{% endtab %}

{% tab title="Output" %}

<figure><img src="/files/Xjv0K6delYtCeLfgf6hU" alt=""><figcaption><p>Real Time Financial Analysis Output</p></figcaption></figure>
{% endtab %}
{% endtabs %}


# Blog Writer

This workflow demonstrates the use of the YouTube and Google Docs integrations. Here, we build a content assistant, retrieving English captions from a YouTube video, writing a technical blog post based on the video's content, translating the blog post to Chinese and Hindi, and saving all three posts in Google Docs.

<figure><img src="/files/Jcq6iMzzFqI0sq9rYqKS" alt=""><figcaption><p>Blog Writer Workflow</p></figcaption></figure>

### Demonstration

{% embed url="<https://www.youtube.com/watch?v=w7svX-__4lA>" %}
Build a Blog Post Generator in Arcee Orchestra
{% endembed %}

### Workflow JSON

{% file src="/files/XDgOfrbZgnYCJmQhTqZa" %}
JSON for Blog Writer Workflow
{% endfile %}

{% hint style="info" %}
To download the workflow, select the JSON file above, then use command/control + S and save with the .json extension.
{% endhint %}

### Input/Output Example

**Input**: isZWnn3RC5w

**Output**:

The output for this workflow includes blog posts in English, Chinese, and Hindi.

{% file src="/files/iFF2JzJT2OLxzUCeS3zW" %}
Blog in English
{% endfile %}

{% file src="/files/EPKQ7c4AOT5liITxdjdZ" %}
Blog in Chinese
{% endfile %}

{% file src="/files/fqTqSByR2lGfjrU7W5Yd" %}
Blog in Hindi
{% endfile %}


# Code Improvement

The code improvement workflow starts with inputting a code snippet, then uses two models to generate improvements: one for making the code more Pythonic and another for identifying and suggesting security improvements. Finally, all improvements are compiled into a Google Doc.

<figure><img src="/files/lyd0ucf1EwevmEiBFcsL" alt=""><figcaption><p>Code Improvement Workflow</p></figcaption></figure>

### Demonstration

{% embed url="<https://youtu.be/glL2Bg4M1do>" %}
Build a Code Review Workflow in Arcee Orchestra
{% endembed %}

### Workflow JSON

{% file src="/files/0Hhc5viF4S2rUuSIya7J" %}
JSON for Code Improvement Workflow
{% endfile %}

{% hint style="info" %}
To download the workflow, select the JSON file above, then use command/control + S and save with the .json extension.
{% endhint %}

### Input/Output Example

{% tabs %}
{% tab title="Input" %}
Any python code, such as:

```python
import subprocess
import sys


def install_package(package_name):
  try:
    __import__(package_name)
  except ImportError:
    print(f"{package_name} not found. Installing...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", package_name])


install_package("yfinance")

```

{% endtab %}

{% tab %}
The workflow output is a Google Doc, which has content similar to the following:

#### Original Code

```python
import subprocess
import sys


def install_package(package_name):
  try:
    __import__(package_name)
  except ImportError:
    print(f"{package_name} not found. Installing...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", package_name])


install_package("yfinance")
```

#### Pythonic Improvements

```python
import importlib
import subprocess
import sys


def install_package(package_name):
    try:
        importlib.import_module(package_name)
    except ImportError:
        print(f"{package_name} not found. Installing...")
        subprocess.check_call([sys.executable, "-m", "pip", "install", package_name])


install_package("yfinance")
```

```python
import importlib.util
import subprocess
import sys


def install_package(package_name):
    if importlib.util.find_spec(package_name) is None:
        print(f"{package_name} not found. Installing...")
        subprocess.check_call([sys.executable, "-m", "pip", "install", package_name])


install_package("yfinance")
```

#### Explanation:

\- **importlib.import\_module vs. importlib.util.find\_spec**: The importlib.import\_module function attempts to import the module, which can be slow and unnecessary if we only want to check for its existence. importlib.util.find\_spec is more efficient as it only checks if the module specification exists without performing the import.

\- **Readability**: Using importlib.util.find\_spec makes the code more readable and Pythonic, as it clearly expresses the intent to check for the module's existence without side effects.

#### Security Improvements

1\. **Remote Code Execution**: The subprocess.check\_call function can be risky if the package\_name is derived from user input. This can be mitigated by sanitizing the input and using a whitelist of trusted packages.

2\. **Package Integrity**: Verify the package's integrity by using a trusted package index or checking digital signatures.

3\. **Error Handling**: Implement comprehensive error handling to manage various failure scenarios.

4\. **Virtual Environments**: Encourage the use of virtual environments to isolate package installations.

5\. **Update pip**: Ensure pip is up-to-date to benefit from the latest security patches.

```python
subprocess.check_callpackage_namepipimport subprocess
import sys
import pkg_resources


def install_package(package_name, source=None):
    # Check if the package is already installed
    if package_name in {pkg.key for pkg in pkg_resources.working_set}:
        print(f"{package_name} is already installed.")
        return


    # Ensure the package name is safe and trusted
    trusted_packages = {'yfinance', 'numpy', 'pandas', 'requests'}  # Example trusted packages
    if package_name not in trusted_packages:
        raise ValueError(f"Untrusted package: {package_name}")


    # Check if the package source is trusted
    trusted_sources = ['https://pypi.org/simple']
    if source is not None and source not in trusted_sources:
        raise ValueError(f"Untrusted source for {package_name}")


    try:
        # Install the package with a trusted source
        subprocess.check_call([sys.executable, "-m", "pip", "install", package_name, "--index-url", source or trusted_sources[0]])
    except subprocess.CalledProcessError as e:
        print(f"Error installing {package_name}: {e}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")


# Example usage:
install_package("yfinance", "https://pypi.org/simple")

```

#### Explanation:

\- **Input Validation**: The trusted\_packages set ensures that only known and trusted packages can be installed. This prevents potential command injection attacks.

\- **Trusted Sources**: The trusted\_sources list restricts the package installation to trusted indices, reducing the risk of installing malicious packages.

\- **Error Handling**: Comprehensive error handling is implemented to catch and report specific errors, such as subprocess.CalledProcessError, and handle unexpected exceptions.

\- **Virtual Environments**: While not explicitly shown in the code, it is recommended to run this script within a virtual environment to isolate the package installation.

\- **Update pip**: Ensure that pip is up-to-date by running pip install --upgrade pip periodically.

trusted\_packagestrusted\_sourcessubprocess.CalledProcessErrorpippip install --upgrade pipBy implementing these improvements, the code becomes more Pythonic, efficient, and secure.
{% endtab %}
{% endtabs %}


# Energy Domain Assistant

The energy domain assistant workflow automates the search and retrieval of information for energy related question. The workflow takes in a user prompt, retrieves data from cherry-picked energy-related PDF reports as well relevant links from Google, writes a detailed answer, and sends the answer to the user's Gmail.

<figure><img src="/files/dqydydRU81tgW9eLMM1P" alt=""><figcaption><p>Energy Domain Assistant Workflow</p></figcaption></figure>

### Demonstration

{% embed url="<https://www.youtube.com/watch?v=PyDojYF7F24&list=PLaTdiAPeh982Tw37GpiqnSGtEn9WZ2xDA&index=8>" %}
Build an Energy Domain Assistant
{% endembed %}

### Workflow JSON

{% file src="/files/RM4EavP6iOig6IDiUvQu" %}
Energy Domain Assistant JSON
{% endfile %}

{% hint style="info" %}
To download the workflow, select the JSON file above, then use command/control + S and save with the .json extension.

Once you've uploaded the workflow, you'll need to specify a recipient email in the Gmail integration.
{% endhint %}

### Input/Output Example

{% tabs %}
{% tab title="Input" %}
What is the impact of AI data centers on electricity consumption? Give me a regional breakdown.
{% endtab %}

{% tab title="Output" %}

<figure><img src="/files/vKsMdRhR0sPaRcLQrLm6" alt=""><figcaption><p>Output Email for Energy Domain Assistant</p></figcaption></figure>
{% endtab %}
{% endtabs %}


# Chat Interface

The chat interface in Orchestra provides a single place to message back and forward between SLMs and kick off automated workflows which you've built.&#x20;

<figure><img src="/files/wE4qEGGnnwUASOKjKE9p" alt=""><figcaption></figcaption></figure>

### **How it works**

When workflows are saved, a description of what the workflow automates is generated. Based on what workflows are enabled, the router learns the capabilities, goals, and parameters of the selected workflows.&#x20;

When a request is sent in the chat interface, the model determines whether this is a general question that the model you've selected can answer, or whether a workflow should be invoked. If it's a general question, the selected model provides a response. If the model determines a workflow should be called, it extracts the required parameters from the prompt for the specific workflow and then triggers the workflow to execute.

This means you can invoke all of your automations from the same place.&#x20;

For example, from a single chat interface, a sales representative could take a company they've never heard of and automatically research the specific company and their industry, generate a sales talk track and key points to discuss tailored to this customer, add the customer to their CRM, draft an introductory email, and send it to the customer.

### Models and Model Settings

<figure><img src="/files/iwBVElNaPy0rvzG3i3UU" alt="" width="224"><figcaption></figcaption></figure>

Out of the box, you can select Virtuoso-Large, Virtuoso-Medium, Virtuoso-Small, and coder. You also have the ability to bring in other models.&#x20;

You can set a system prompt to be passed to the model with each invokation, and can set hyperparameters such as max tokens, temperature, top p, frequency penalty, and presence penalty.

<figure><img src="/files/v8UVZEPojH3BWUMJDH1l" alt="" width="252"><figcaption><p>Workflow Selection</p></figcaption></figure>

On the tools tab, you can select the workflows you want to be considered when invoking the model.

Files and images can also be uploaded through the chat.


# FAQ

Frequently Asked Questions

### Technical FAQs

1. What is the SLA for Arcee Orchestra when consumed via cloud SaaS?
   1. Uptime Commitment: 99.95% uptime, allowing for approximately 9 hours of downtime per year. Orchestra's platform's architecture ensures that updates or upgrades to individual services within the cluster do not disrupt other functionalities.
   2. Response Times: 24/7 on-call dev support with a response time of 60 minutes for critical issues. Resolution times will depend on complexity but are expected to be within hours, not days.
   3. Credits for SLA Breaches: Credits will be applied to account for workflow executions lost during periods of SLA breaches.
2. Runtime limitations for Arcee Orchestra
   1. Workflows: Workflows execute along the graph node by node so there are no node or timeout limitations.
   2. Integrations: the provided integrations access external systems through APIs. There are no limitations on Arcee’s platform to restrict the number of requests to an external source; however, some companies build limitations into their own APIs which must be followed.
   3. Code Node: Not all PyPi packages are enabled by default, but we can make most libraries that customers need available.
3. What are all the available built-in integrations?
   1. You can find a list of the available integrations [here](https://docs.google.com/document/d/1JGBdldPS-vfK4bvtuEFPsv0XXco8XMaT8R2uyZESfbY/edit?tab=t.0).
4. Is there any portability of the system?
   1. For each workflow you create, the configuration can be downloaded. This means you'll have the prompt templates for each model, what integrations you're connecting with, and any code written in the code nodes. This allows for portability of information and configurations.
5. With the Enterprise "Shared" tier, could the performance suffer due to multiple tenants using at the same time?
   1. Our infrastructure supports multiple simultaneous workflows across numerous tenants. The application is built on Kubernetes, allowing for auto-scaling to accommodate high levels of traffic.
6. Is there any guarantee of input/output token speed?
   1. For Arcee AI Small Lanaguage Models, the token speed will vary depending on which model size is selected.  10B, 14B, 32B, and 72B parameter models typically average 100, 70, 50, and 30 tokens per second (TPS) respectively. However, these speeds will vary based on factors such as input token count and concurrent requests.  If you bring in a 3rd party model, Arcee cannot guarantee token speeds.
7. What if we want to upgrade to "dedicated", is there a path for an upgrade?
   1. Yes, users can upgrade to dedicated hosting. This process involves cloning the data in the current environment and moving it to the new environment. This process is typically completed within five days.


# Deprecation Policy

### Overview

Arcee models are **actively maintained and updated** to provide the latest performance and reliability improvements. Cloud-hosted versions may be **retired as part of regular updates**, while all released models remain **downloadable** on Hugging Face for independent use.

### Versioning & Updates

* Cloud models are **versioned** and may be replaced quickly when new improvements are available.
* Downloads remain stable, ensuring your workflows are not disrupted.
* Deprecated cloud versions **may stop receiving updates or support** after the notice period.

### Deprecation Criteria

A model may be retired due to:

* Release of a **new or improved version**
* **Security or reliability concerns**

### Notice & Transition

* Typical notice for standard cloud models: **up to 5 business days**
* Widely adopted models: **30 days**
* Notifications are provided via **email, API dashboard, and changelog**, with guidance for migration or upgrades.

### Commitment

We aim to **balance rapid innovation with user flexibility**, keeping cloud models current while ensuring downloadable models remain fully accessible.




---

[Next Page](/llms-full.txt/1)

