Skip to content

Run your prompts on Sheshnag

Who this is for: someone runs a Sheshnag deployment and gave you its URL. You want to send it a batch of prompts and get results back.

Verified against code: 2026-08-26.

Sheshnag speaks OpenAI's Batch API, so if you have working OpenAI batch code the migration is a base_url change and a different key. This page covers what the dashboard cannot teach you: pointing code at the deployment, and driving it from a script.

The dashboard teaches the file format itself — it shows a sample line, validates what you upload before you commit to it, and explains each error. If you are submitting your first batch, do it there once and read this afterwards.

What you need
The deployment's URL e.g. https://sheshnag.example.edu. From whoever runs it.
A personal API key Dashboard → API Keys → create one. The raw gk-… value is shown once; copy it then.

Your key authenticates you, not a machine. It is not the same thing as the worker keys providers use — those live under the Provider portal and are only accepted on worker endpoints.

Try one prompt first

The dashboard's Playground tab runs a single prompt on any catalogue model with no JSONL and no API key. Under the hood it is a one-line batch — upload, validate, queue, run — and the result shows the answer, the tokens used, the worker that served it, and the exact JSONL line that produced it. Copy that line, add one per prompt, and you have your first batch file.

Switch the tab to Grid to run one prompt set across up to five arms — a different model, temperature, seed or system prompt per arm — and read the answers side by side, with the first word that differs from a base column highlighted. A grid is a group of batches, one per arm, tagged with a shared metadata.grid_id; each shows up on the Batches tab like any other.

Sweep quants fills the arms for you when the pool holds the same weights at more than one quantization: one arm per quantization, smallest first, every other setting copied from the first arm so the answer is about the quantization and nothing else. The arms are held to a single engine, since the same weights on two engines measures both at once and the table cannot show which moved. Quantizations the pool cannot serve, and any past the five-arm limit, are named under the button rather than dropped — a sweep missing its cheapest arm reads as a result about quality when it is a result about what was staged. The export carries each arm's quantization and lineage.

The model picker labels each entry loaded (in memory on an online worker), on disk, loads on first use (the first prompt pays one model load), or unavailable (no online worker has it). Only the last cannot be picked.

Set Logprobs to top 5, 10 or 20 to see the answer as the model saw it: every token shaded by its probability, a click on any token showing the alternatives it weighed, and a count of flip-prone positions — places where the top two candidates sit within one bf16 ulp (0.125 nats), so a different kernel, batch shape or GPU could have chosen the other. The switch is offered only when an online runtime for the model returns log-probabilities (vLLM, llama.cpp, Ollama ≥ 0.12.11). In grid mode it applies to every arm and the export carries the per-token data.

Share as image, beside a single-prompt answer, turns the run into a PNG for a post or a slide. The card shows the model, the parameters on the line, the system prompt, the prompt, the answer and the token counts; the worker and the batch id stay off it. Choose Light or Dark, 1200 or 1080 pixels wide, and the whole answer or its first N lines. With logprobs on, the answer is the shaded token strip, and numbered callouts show the alternatives at chosen positions: up to three of the least certain are picked for you, and clicking a token in the preview — or pressing Enter on it — adds or removes one. Around marked tokens trims a long answer to the stretch the callouts explain. Download PNG saves the image at 2× resolution; Copy image puts it on the clipboard.

Point your code at it

The base URL is your deployment's host plus /v1:

from openai import OpenAI

client = OpenAI(
    api_key="gk-...",                                  # your personal key
    base_url="https://sheshnag.example.edu/v1",        # not api.openai.com
)

Or with curl:

curl -H "Authorization: Bearer gk-..." \
     https://sheshnag.example.edu/v1/models

GET /v1/models is the cheapest way to prove your key and URL work. It also tells you which models you may actually request — see below.

body.model must be a catalogue id, not a raw model name

Every line's body.model has to be one of the ids from GET /v1/models, like mistral-7b-instruct-q4km — not mistral:7b and not gpt-4o-mini. There is deliberately no path to run an uncatalogued model, and the id must be the same on every line of the file. A model that is not in the catalogue fails validation with unsupported_model.

Submit, poll, download

Three calls. The whole loop:

# 1. Upload the JSONL
f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")

# 2. Create the batch
batch = client.batches.create(
    input_file_id=f.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

# 3. Poll until it settles
import time
TERMINAL = {"completed", "failed"}
while batch.status not in TERMINAL:
    time.sleep(10)
    batch = client.batches.retrieve(batch.id)

if batch.status == "failed":
    raise SystemExit(f"batch failed: {batch.id}")

# 4. Download results
results = client.files.content(batch.output_file_id)
open("results.jsonl", "wb").write(results.read())

Poll every 10–30 seconds. Batches are asynchronous by design and can sit waiting for a worker with the right model and enough VRAM; there is no benefit to polling faster.

The status difference that will bite you

Sheshnag has five batch statuses, and they are not quite OpenAI's:

Status Meaning
validating The file is being checked. Returned immediately on create.
validated The file is good and the batch is waiting for a worker.
in_progress A worker is running it.
completed Done — output_file_id is populated.
failed Terminal. Either validation rejected the file, or execution failed three times.

Two differences from OpenAI worth knowing:

  • validated is extra. OpenAI has no such status. Code that switches on status and assumes anything not in_progress or completed is an error will misbehave here. Treat validating and validated as "keep waiting".
  • finalizing, expired, cancelling and cancelled never occur. There is no cancel endpoint. If your code branches on them, those branches are dead.

When validation fails

Validation is asynchronous — batches.create returns straight away with validating, and the batch reaches validated or failed on its own. A failed batch carries per-line errors, each with a code (invalid_json, missing_field, duplicate_custom_id, unknown_fields, unsupported_model, …), the line number, and the offending field.

Two things to know before you go hunting:

  • At most 100 errors are stored, but the reported total is the real count. A file with 5,000 broken lines lists 100 of them. Fix the pattern, not the list.
  • The dashboard is better at this than the API. It groups errors by kind, shows the first few line numbers for each, and suggests a fix. If a batch fails validation, open it there rather than parsing error JSON.

You can also watch validation live rather than polling: GET /v1/batches/{batch_id}/events is a Server-Sent Events stream.

Parameters, and what survives

body on each line is an ordinary OpenAI chat-completions request, but not every parameter reaches the model — it depends on which runtime serves it. stream is always rejected, since a batch cannot stream.

See also