Reasoning Traces

Reasoning Traces

Handling Reasoning Traces in Multi-Turn Conversations

Trinity-Large-Thinking generates explicit chain-of-thought reasoning inside <think>...</think> blocks before producing its response. When served via vLLM, these blocks are parsed into a dedicated reasoning_content field in the API response, separate from content and tool_calls.

Preserving reasoning across turns is critical for reliable multi-step tool use and agentic workflows. This guide covers how to do it correctly.

How reasoning flows through a conversation

Turn 1: User sends message
         ↓
         Model generates: <think>reasoning</think> content + tool_calls
         ↓
         vLLM parses into: { reasoning_content, content, tool_calls }
         ↓
Turn 2: Client appends full assistant message (reasoning_content + content + tool_calls)
         Client appends tool result
         Client sends updated history
         ↓
         Chat template re-wraps reasoning_content in <think>...</think> during tokenization
         ↓
         Model sees prior chain-of-thought → generates next step correctly

If the client drops reasoning_content between turns, the model loses its prior chain-of-thought and may produce malformed output.

Quick reference: assistant message shape

For assistant turns that call tools, include all three fields when appending back to history:

{
  "role": "assistant",
  "content": "",
  "reasoning_content": "The user wants to cancel. I need their customer_id first, so I'll look them up by email.",
  "tool_calls": [
    {
      "id": "call-1",
      "type": "function",
      "function": {
        "name": "get_customer_by_email",
        "arguments": "{\"email\": \"[e-mail]\"}"
      }
    }
  ]
}

Field	Required	Notes
content	Yes	Use "" if the model returned null. Never pass null.
reasoning_content	Strongly recommended	From the API response's reasoning_content field. reasoning is also accepted on input.
tool_calls	Conditional	Include when the model made tool calls (omit for final non-tool assistant turns).

Field name: reasoning vs reasoning_content

The naming can be confusing because different layers use different names:

Layer	Field name	Direction
Arcee API / vLLM API response	reasoning_content	Output
OpenAI Python SDK	reasoning_content	Output (attribute name)
Arcee API input (assistant messages)	reasoning_content (also accepts reasoning)	Input
Self-hosted vLLM input	reasoning (some versions don't accept reasoning_content)	Input
Chat template (Jinja)	both reasoning and reasoning_content	Input

The safe rule: use reasoning_content when constructing assistant messages for input. This matches what the API returns, so you can pass it straight back. The backend also accepts reasoning on input and converts it automatically.

Note for self-hosted vLLM users: Some vLLM versions only read reasoning on input messages (vllm#38488). If you're hitting vLLM directly (not through Arcee's API), you may need to map reasoning_content → reasoning on assistant messages.

Python implementation

Installation

pip install openai

Complete agentic loop

import json
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="YOUR_API_KEY"
)

MODEL = "arcee-ai/Trinity-Large-Thinking"

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_customer_by_email",
            "description": "Look up a customer by email address.",
            "parameters": {
                "type": "object",
                "properties": {
                    "email": {"type": "string", "description": "Customer email address"}
                },
                "required": ["email"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "cancel_subscription",
            "description": "Cancel a customer's subscription. Requires customer_id.",
            "parameters": {
                "type": "object",
                "properties": {
                    "customer_id": {"type": "string"},
                    "reason": {"type": "string"}
                },
                "required": ["customer_id"]
            }
        }
    }
]


def execute_tool(name: str, arguments: str) -> str:
    """Replace with your actual tool implementations."""
    args = json.loads(arguments)
    if name == "get_customer_by_email":
        return json.dumps({"customer_id": "C2001", "name": "Jane Doe", "plan": "Premium"})
    elif name == "cancel_subscription":
        return json.dumps({"success": True, "message": f"Cancelled for {args['customer_id']}"})
    return json.dumps({"error": "Unknown tool"})


def build_assistant_message(msg) -> dict:
    """
    Build an assistant message dict that preserves reasoning for the next turn.

    Key details:
    - Pass reasoning_content straight back (matches what the API returns)
    - Use "" instead of None for content (avoids tokenization issues)
    - Preserve the full tool_calls array
    """
    assistant_msg = {
        "role": "assistant",
        "content": msg.content or "",  # never null
    }

    # Preserve reasoning — critical for multi-turn
    reasoning = getattr(msg, "reasoning_content", None) or getattr(msg, "reasoning", None)
    if reasoning:
        assistant_msg["reasoning_content"] = reasoning

    # Preserve tool calls
    if msg.tool_calls:
        assistant_msg["tool_calls"] = [
            {
                "id": tc.id,
                "type": "function",
                "function": {
                    "name": tc.function.name,
                    "arguments": tc.function.arguments
                }
            }
            for tc in msg.tool_calls
        ]

    return assistant_msg


def run_agent(user_message: str, system_prompt: str = "You are a helpful customer service agent."):
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message}
    ]

    max_steps = 10  # safety limit

    for step in range(max_steps):
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=tools,
            tool_choice="auto",
            temperature=0,
            max_tokens=1000
        )

        msg = response.choices[0].message

        # Append full assistant message with reasoning preserved
        messages.append(build_assistant_message(msg))

        # No tool calls → final response
        if not msg.tool_calls:
            return msg.content or ""

        # Execute each tool call and append results
        for tc in msg.tool_calls:
            result = execute_tool(tc.function.name, tc.function.arguments)
            messages.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "content": result
            })

    raise RuntimeError("Agent exceeded maximum steps")


# Usage
response = run_agent("I want to cancel my subscription. My email is [e-mail]")
print(response)

TypeScript implementation

Installation

npm install openai

Complete agentic loop

import OpenAI from "openai";
import type {
  ChatCompletionMessageParam,
  ChatCompletionTool,
} from "openai/resources/chat/completions";

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "your-api-key",
});

const MODEL = "arcee-ai/Trinity-Large-Thinking";

const tools: ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "get_customer_by_email",
      description: "Look up a customer by email address.",
      parameters: {
        type: "object",
        properties: {
          email: { type: "string", description: "Customer email address" },
        },
        required: ["email"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "cancel_subscription",
      description: "Cancel a customer's subscription. Requires customer_id.",
      parameters: {
        type: "object",
        properties: {
          customer_id: { type: "string" },
          reason: { type: "string" },
        },
        required: ["customer_id"],
      },
    },
  },
];

async function executeTool(name: string, args: string): Promise<string> {
  // Replace with your actual tool implementations
  const parsed = JSON.parse(args);
  switch (name) {
    case "get_customer_by_email":
      return JSON.stringify({ customer_id: "C2001", name: "Jane Doe", plan: "Premium" });
    case "cancel_subscription":
      return JSON.stringify({ success: true, message: `Cancelled for ${parsed.customer_id}` });
    default:
      return JSON.stringify({ error: "Unknown tool" });
  }
}

/**
 * Build an assistant message that preserves reasoning for the next turn.
 *
 * Key details:
 * - Pass reasoning_content straight back (matches what the API returns)
 * - Use "" instead of null for content
 * - Preserve the full tool_calls array
 */
function buildAssistantMessage(
  msg: OpenAI.Chat.Completions.ChatCompletionMessage
): ChatCompletionMessageParam {
  const assistantMsg: Record<string, unknown> = {
    role: "assistant",
    content: msg.content ?? "", // never null
  };

  // Preserve reasoning — critical for multi-turn
  const reasoning = (msg as any).reasoning_content ?? (msg as any).reasoning;
  if (reasoning) {
    assistantMsg.reasoning_content = reasoning;
  }

  // Preserve tool calls
  if (msg.tool_calls?.length) {
    assistantMsg.tool_calls = msg.tool_calls.map((tc) => ({
      id: tc.id,
      type: "function" as const,
      function: {
        name: tc.function.name,
        arguments: tc.function.arguments,
      },
    }));
  }

  return assistantMsg as ChatCompletionMessageParam;
}

async function runAgent(
  userMessage: string,
  systemPrompt = "You are a helpful customer service agent."
): Promise<string> {
  const messages: ChatCompletionMessageParam[] = [
    { role: "system", content: systemPrompt },
    { role: "user", content: userMessage },
  ];

  const maxSteps = 10; // safety limit

  for (let step = 0; step < maxSteps; step++) {
    const response = await client.chat.completions.create({
      model: MODEL,
      messages,
      tools,
      tool_choice: "auto",
      temperature: 0,
      max_tokens: 1000,
    });

    const msg = response.choices[0].message;

    // Append full assistant message with reasoning preserved
    messages.push(buildAssistantMessage(msg));

    // No tool calls → final response
    if (!msg.tool_calls?.length) {
      return msg.content ?? "";
    }

    // Execute each tool call and append results
    for (const tc of msg.tool_calls) {
      const result = await executeTool(tc.function.name, tc.function.arguments);
      messages.push({
        role: "tool",
        tool_call_id: tc.id,
        content: result,
      });
    }
  }

  throw new Error("Agent exceeded maximum steps");
}

// Usage
const response = await runAgent(
  "I want to cancel my subscription. My email is [e-mail]"
);
console.log(response);

OpenRouter integration

When using Trinity through OpenRouter, reasoning is returned in a reasoning_details object (OpenRouter's unified reasoning shape). For multi-turn conversations, pass reasoning_details back as-is on assistant messages — OpenRouter handles the model-specific upstream translation automatically.

Debugging upstream requests

To verify that reasoning is being sent upstream correctly, enable echo mode:

{
  "debug": { "echo_upstream_body": true }
}

See OpenRouter debugging docs for details.

Common pitfalls

1. xml_in_reasoning — tool call XML inside reasoning field

Symptom: The response has tool_calls: [] (empty) and reasoning_content contains raw XML like <function=get_details_by_id><parameter=id>L1001</parameter>...

Cause: The previous assistant turn likely lost reasoning context (and/or had invalid message shape), so the model generated the tool call inside its thinking block instead of as structured output.

Fix: Ensure every assistant message in the conversation history includes reasoning_content from the prior API response.

❌ Broken — no reasoning, null content
{"role": "assistant", "content": null, "tool_calls": [...]}

✅ Fixed — reasoning preserved, content non-null
{"role": "assistant", "content": "", "reasoning_content": "...", "tool_calls": [...]}

2. reasoning_content ignored on self-hosted vLLM

Symptom: You're passing reasoning_content on assistant messages to a self-hosted vLLM instance, but the model behaves as if reasoning is missing.

Cause: Some vLLM versions only read reasoning (not reasoning_content) from input messages (vllm#38488). This does not affect Arcee's hosted API, which accepts both.

Fix: If you're hitting vLLM directly, map reasoning_content → reasoning on input:

# Self-hosted vLLM workaround
assistant_msg["reasoning"] = msg.reasoning_content

3. content: null on assistant tool-call turns

Symptom: Degraded tool call quality or malformed output on subsequent turns.

Cause: When the model makes a tool call, content may be null in the API response. Passing null back can contribute to malformed follow-up behavior in some integrations.

Fix: Normalize null to empty string:

"content": msg.content or ""   # Python

content: msg.content ?? ""     // TypeScript

4. Missing vLLM serving flags

Symptom: Reasoning leaks into content with visible <think>...</think> tags, or tool calls appear as raw XML instead of structured tool_calls.

Fix: Ensure vLLM is started with the correct flags:

vllm serve arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Flag	Purpose
--reasoning-parser deepseek_r1	Separates <think> blocks into the reasoning_content field
--enable-auto-tool-choice	Enables tool call parsing
--tool-call-parser qwen3_coder	Parses tool calls into structured tool_calls array

Note: vLLM versions before 0.18 may also require --enable-reasoning. If the flag is not recognized, --reasoning-parser alone is sufficient.

vLLM serving reference

Minimal serving command

vllm serve arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Production serving command (example only)

vllm serve arcee-ai/Trinity-Large-Thinking \
  --served-model-name arcee-ai/Trinity-Large-Thinking \
  --dtype bfloat16 \
  --tensor-parallel-size <set-for-your-hardware> \
  --max-model-len <set-for-your-workload> \
  --gpu-memory-utilization <set-for-your-cluster-policy> \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key "$API_KEY" \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Context length guidance

Set --max-model-len based on your real conversation lengths, tool-chain depth, and GPU memory budget. Higher values pre-allocate more KV cache at startup. A practical approach is to start conservatively, monitor truncation/finish_reason: "length", then scale up incrementally.