Structured Outputs

Enabling JSON Output allows users to get structured responses from Arcee AI models. The JSON Output ensures the model outputs valid JSON strings.

To enable JSON Output, users should:

1. Set the response_format parameter to {'type': 'json_object'}.

2. Include the word "json" in the system or user prompt, and provide an example of the desired JSON format to guide the model in outputting valid JSON.

3. Set the max_tokens parameter reasonably to prevent the JSON string from being truncated midway.

JSON Output Example

import json
from openai import OpenAI

client = OpenAI(
    api_key="afm-13cf46d35fd48a6aa2da4c8d62424de8", 
    base_url="https://api.arcee.ai/api/v1"
)

system_prompt = """
The user will provide some exam text. Please parse the "question" and "answer" and output them in JSON format. 

EXAMPLE INPUT: 
Which is the highest mountain in the world? Mount Everest.

EXAMPLE JSON OUTPUT:
{
    "question": "What is the capital of France?",
    "answer": "Paris"
}
"""

user_prompt = "What is the capital of Italy?"

messages = [{"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}]

response = client.chat.completions.create(
    model="arcee-ai/AFM-4.5B",
    messages=messages,
    response_format={
        'type': 'json_object'
    }
)

print(json.loads(response.choices[0].message.content))