Guides

Run an OpenAI Batch API job on open models: change one line

If you already have a script that uploads a JSONL file to OpenAI's Batch API, waits, and downloads results, you don't need to rewrite it to try an open model. Batchrate implements the same Files and Batches calls. You change the base_url, swap the model id in your JSONL, and run the same script.

This guide walks through that change, what stays the same, the few things that differ, and how to handle failed lines.

The one-line change

Install the official SDK. There is no Batchrate package.

pip install openai

import os
from openai import OpenAI

# Before:
# client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# After:
client = OpenAI(
    api_key=os.environ["BATCHRATE_API_KEY"],   # br_live_...
    base_url="https://api.batchrate.ai/v1",
)

Batch calls (client.files.create, client.batches.create, client.batches.retrieve, and client.files.content) use that client. A direct client.chat.completions.create does not: POST /v1/chat/completions returns unknown_route. Those requests belong in the JSONL file.

Update the model id in your JSONL

Each line of a batch file is one request, in the same shape OpenAI uses:

{"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen3.6-35b-a3b","max_tokens":300,"messages":[{"role":"user","content":"Summarize this support ticket in two sentences: ..."}]}}
{"custom_id":"req-2","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen3.6-35b-a3b","max_tokens":300,"messages":[{"role":"user","content":"Summarize this support ticket in two sentences: ..."}]}}

The chat model on the price list:

It has a 32,768-token context, and that includes the reply. As with OpenAI, every line in one file uses the same model and the same url. If you want to run two models, upload two files. A model id is accepted when it is enabled; the pricing page and GET /v1/models list the ones you can submit.

If your existing files are already built for OpenAI, a few lines of Python will swap the model:

import json

def retarget(src, dst, model="qwen3.6-35b-a3b"):
    with open(src) as fin, open(dst, "w") as fout:
        for line in fin:
            row = json.loads(line)
            row["body"]["model"] = model
            fout.write(json.dumps(row) + "\n")

retarget("openai_batch.jsonl", "batch.jsonl")

A complete run

import os, time, json
from openai import OpenAI

client = OpenAI(api_key=os.environ["BATCHRATE_API_KEY"],
                base_url="https://api.batchrate.ai/v1")

f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
b = client.batches.create(
    input_file_id=f.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"job": "ticket-summaries"},
)

while b.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(60)
    b = client.batches.retrieve(b.id)
print(b.status, b.request_counts)

results = {}
if b.output_file_id:
    for line in client.files.content(b.output_file_id).text.splitlines():
        row = json.loads(line)
        results[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]
print(len(results), "results")

completion_window must be "24h". In practice results typically come back within hours, but write your code around the 24h window rather than a specific time. When I wire this into a pipeline, I poll every minute or two and let the job run in the background.

What's different from OpenAI

A few things are worth knowing before you move a real job:

• Batch only. A direct POST /v1/chat/completions returns unknown_route. Anything you'd send synchronously has to go in a JSONL file.
• File limits. An input file can have up to 5,000 lines and 10 MB. Bigger jobs get split across several files and batches (see below).
• Context. 32,768 tokens including the reply, so very long prompts need trimming or chunking.
• Thinking is off by default. To turn it on, set "chat_template_kwargs": {"enable_thinking": true} in the line body, and give max_tokens much more room (2,000 or more). Thinking tokens count toward max_tokens and are billed as output.
• Account limits. Before you add credit, an account can run 3 batches at a time and submit about 10M tokens per 24 hours. After the first top-up that rises to 100 running batches and 1B tokens per 24 hours.

You can check the cost of a file before you start it. GET /v1/files/{id}/estimate returns an estimate. Output tokens are estimated from each line's max_tokens, so set it to something realistic. price_microusd is millionths of a dollar.

est = client.get(f"/files/{f.id}/estimate", cast_to=object)
print(est["input_tokens"], est["output_tokens"], est["price_microusd"])

Structured output with json_schema

If you parse model output downstream, use response_format with a JSON schema. The output is constrained while it's written, so message.content is valid JSON that matches the schema, unless it gets cut off by max_tokens.

{"custom_id":"t-1","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen3.6-35b-a3b","max_tokens":200,"messages":[{"role":"user","content":"Ticket: 'Charged twice for October.' Return the category and urgency."}],"response_format":{"type":"json_schema","json_schema":{"name":"ticket","strict":true,"schema":{"type":"object","properties":{"category":{"type":"string","enum":["billing","bug","account","other"]},"urgency":{"type":"string","enum":["low","medium","high"]}},"required":["category","urgency"],"additionalProperties":false}}}}}

Most common schema keywords work (enum, anyOf, $ref, pattern, and so on). oneOf, allOf, and uniqueItems are rejected at upload with the line number, so rewrite oneOf as anyOf.

Splitting big jobs

import json

def split_jsonl(path, prefix, max_lines=5000, max_bytes=10_000_000):
    parts, buf, size = [], [], 0
    for line in open(path, "rb"):
        if buf and (len(buf) >= max_lines or size + len(line) > max_bytes):
            parts.append(buf); buf, size = [], 0
        buf.append(line); size += len(line)
    if buf: parts.append(buf)
    names = []
    for i, chunk in enumerate(parts):
        name = f"{prefix}-{i:04d}.jsonl"
        open(name, "wb").writelines(chunk)
        names.append(name)
    return names

Upload each part and create one batch per file.

Retrying failed lines by custom_id

You're only charged for requests that complete. Failed lines go to the batch's error file with response: null and an error object. If a batch hits the end of its 24h window, unfinished lines show up there with code batch_expired. Either way, pull the custom_ids and resubmit just those lines:

def retry_file(batch, input_path, out_path):
    if not batch.error_file_id:
        return None
    failed = {json.loads(l)["custom_id"]
              for l in client.files.content(batch.error_file_id).text.splitlines()}
    with open(input_path) as fin, open(out_path, "w") as fout:
        for line in fin:
            if json.loads(line)["custom_id"] in failed:
                fout.write(line)
    return out_path if failed else None

Check error.code before retrying. batch_expired means the 24h window ended before that line finished. It was not charged, and the line can be resent as-is. An upload the API rejects does not create a batch: the response code is invalid_file and the message names the line. Fix that line, then upload again.

What it costs

Example numbers: 50,000 ticket summaries, about 800 input tokens and 150 output tokens each, on qwen3.6-35b-a3b.

New accounts get 10,000,000 free tokens after email verification, with no card needed, which is enough to run a real test file before you decide anything.

Getting started

1. Sign up at batchrate.ai and create an API key in the console.
2. Change base_url and the model id.
3. Run one small file, compare the results to what you get today, then move the bigger jobs.

Full reference: batchrate.ai/docs.