Jobs and batches

A job runs up to 1,000 endpoint requests for you in the background. Use one when a task needs many items ("these 300 videos"), a whole list ("every comment"), or a request that can take a while, such as a long transcript. Each item is exactly the request its endpoint answers synchronously: same parameters, same price, same caches, and its result is the same envelope, so the code that reads a single response reads a job's results too. Failed items cost nothing.

Create a job

Send the requests with POST /v1/jobs. Each one names an endpoint by ID (the x-socialtap-endpoint of its operation in the API reference) and its parameters, as you would send them to the endpoint, plus your own custom_id:

curl -X POST 'https://staging.socialtap.dev/v1/jobs?wait=60' \
  -H "x-api-key: $SOCIALTAP_API_KEY" \
  -H "idempotency-key: $(uuidgen)" \
  -H 'content-type: application/json' \
  --data '{
  "requests": [
    {
      "custom_id": "row-1",
      "endpoint": "youtube.transcript",
      "params": {
        "url": "dQw4w9WgXcQ"
      }
    }
  ]
}'

const res = await fetch("https://staging.socialtap.dev/v1/jobs?wait=60", {
  method: "POST",
  headers: { "x-api-key": process.env.SOCIALTAP_API_KEY ?? "", "idempotency-key": crypto.randomUUID(), "content-type": "application/json" },
  body: JSON.stringify({
    "requests": [
      {
        "custom_id": "row-1",
        "endpoint": "youtube.transcript",
        "params": {
          "url": "dQw4w9WgXcQ"
        }
      }
    ]
  }),
});
const body = await res.json();
// 200: body.results holds the first page of results; 202: still running, poll body.data.id.
console.log(res.status, body.data.status, body.data.request_counts);

Every item is checked before anything runs: an unknown endpoint or parameter, a missing required parameter, a duplicate custom_id or a list option on a single-item endpoint fails the whole request with invalid_input, and its param names the field, such as requests[3].params.url. Nothing is charged.

Synchronous batches. With wait (up to 1 minute), the request waits for the job to end. If it ends in time, the answer is 200 with the job and the first page of its results in results (continue from results_page.next_cursor). Otherwise it is 202 with the job as it is, and its URL in the location header.

Any endpoint, in the background. A single endpoint request with the header Prefer: respond-async becomes a one-request job: the answer is 202 with the job instead of the result. Its one item's custom_id is request.

Walk whole lists

On a list endpoint, max_pages (up to 1,000 per item) walks the list for you with its cursor: every page is one result with its page_index, priced as one page. The walk stops at the end of the list, at max_pages, or when the job's credits run out. A job holds at most 10,000 pages in total.

The job

Poll GET /v1/jobs/{id}: it is free, and wait long-polls until the job ends. Or subscribe a webhook to job.completed.

A job is queued until its first item starts, running while items run, and completed once every item has ended: there is no failed job, items fail one by one, so check request_counts. A canceled job is canceling until its running items finish, then canceled. A job that reaches expires_at (its deadline, 1 day after creation) ends expired: items that had not started expire, running ones finish.

Results

GET /v1/jobs/{id}/results lists results in the order they completed, cursor-paged, while the job runs and after. Keep the last page.next_cursor to read only what arrived since. Each result is one page of one item:

• result is exactly the synchronous response's envelope (data, meta, and page on list endpoints).
• error is exactly the synchronous error object: branch on error.code as you would for a single request.
• An item that ended without a fetched page (canceled, expired) has one result with that status and no result.
• ?status=failed lists the failures, to resubmit them in a new job.

Once the job ends, results_url is a signed link to every result as JSON Lines (one result per line), valid 1 hour and usable without an API key: hand it to a download tool or another service. Get the job again for a fresh link.

Results are kept 7 days after the job ends and job records 30 days; after that the job answers job_not_found.

Credits

1. Reserve. Creating a job sets max_credits aside: by default the most its items can cost (each item's maximum price, times its max_pages). Lower it to cap what the job may spend. If your balance can't cover it, the request fails with insufficient_credits and nothing runs; if the key has a spend limit, max_credits must also fit in what the limit leaves this interval, else spend_limit_reached.
2. Charge. Each page is charged its own price as it ends, exactly like a single request: cache hits and failures are free. A page that the remaining budget can't pay for does not run: its item is canceled with an insufficient_credits error whose details.max_credits says the job's cap stopped it, or spend_limit_reached when the key's spend limit did. Pages count against the key that created the job, exactly as its own requests do.
3. Release. When the job ends, whatever is still set aside goes back to your balance. credits.charged is what the job cost.

Keys and test mode

Creating a job needs a key with the jobs scope, and endpoints too: a job's items are endpoint requests. A test key's jobs run every item in test mode: each result is exactly the recorded answer the endpoint gives that test key, free, and nothing is set aside or charged. Test jobs are separate from live ones: a test key lists and reads its own, a live key never sees them.

Cancel

POST /v1/jobs/{id}/cancel stops a job: items that haven't started are canceled (free), items already running finish and are charged. Canceling a job that has ended changes nothing.

Limits and fairness

A job runs up to 10 items at a time, and all of an account's jobs together up to 25, so a large job never crowds out your own synchronous requests. When SocialTap is busy, a job's items wait for capacity instead of failing; an item whose platform fails is retried up to 3 times before it fails. The unfinished jobs of an account hold at most 50,000 pages; above that, creating a job answers rate_limited: wait for running jobs to end.

Retry safely

Send an idempotency-key with every create: if the connection drops, repeat the request with the same key and body and you get the job the first request created, never a second one. See idempotency.