# Jobs and batches

Docs

Run many requests, or one long list walk, in the background: same prices and caches as single requests, with results you can stream, download or receive by webhook.

A job runs up to 1,000 endpoint requests for you in the background. Use one when a task needs many items ("these 300 videos"), a whole list ("every comment"), or a request that can take a while, such as a long transcript. Each item is exactly the request its endpoint answers synchronously: same parameters, same price, same caches, and its result is the same envelope, so the code that reads a single response reads a job's results too. Failed items cost nothing.

| Route | What it does | Key scope |
| --- | --- | --- |
| `POST /v1/jobs` | Run many requests (or one long one) as a job | `jobs` |
| `GET /v1/jobs` | List jobs, newest first | `jobs` |
| `GET /v1/jobs/{id}` | Get a job (free to poll) | `jobs` |
| `GET /v1/jobs/{id}/results` | List a job's results as they complete | `jobs` |
| `GET /v1/jobs/{id}/results.jsonl` | Download every result of a job (JSON Lines) | none (signed URL) |
| `POST /v1/jobs/{id}/cancel` | Cancel a job | `jobs` |

## Create a job

Send the requests with `POST /v1/jobs`. Each one names an endpoint by ID (the `x-socialtap-endpoint` of its operation in the [API reference](https://staging.socialtap.dev/docs/reference/index.md)) and its parameters, as you would send them to the endpoint, plus your own `custom_id`:

Shell:

```shell
curl -X POST 'https://staging.socialtap.dev/v1/jobs?wait=60' \
  -H "x-api-key: $SOCIALTAP_API_KEY" \
  -H "idempotency-key: $(uuidgen)" \
  -H 'content-type: application/json' \
  --data '{
  "requests": [
    {
      "custom_id": "row-1",
      "endpoint": "youtube.transcript",
      "params": {
        "url": "dQw4w9WgXcQ"
      }
    }
  ]
}'
```

TypeScript:

```typescript
const res = await fetch("https://staging.socialtap.dev/v1/jobs?wait=60", {
  method: "POST",
  headers: { "x-api-key": process.env.SOCIALTAP_API_KEY ?? "", "idempotency-key": crypto.randomUUID(), "content-type": "application/json" },
  body: JSON.stringify({
    "requests": [
      {
        "custom_id": "row-1",
        "endpoint": "youtube.transcript",
        "params": {
          "url": "dQw4w9WgXcQ"
        }
      }
    ]
  }),
});
const body = await res.json();
// 200: body.results holds the first page of results; 202: still running, poll body.data.id.
console.log(res.status, body.data.status, body.data.request_counts);
```

| Field | Type | Description |
| --- | --- | --- |
| `requests` | `object[]` | The requests to run, 1 to max_requests; custom_id must be unique within the job. |
| `requests[].custom_id` | `string` | Your ID for the item (letters, digits, _ and -), echoed on its results. |
| `requests[].endpoint` | `string` | The endpoint ID (youtube.transcript). |
| `requests[].params` | `object` | The endpoint's query params, as you would send them to it (strings, numbers or booleans). |
| `requests[].max_pages?` | `integer` or `null` | List endpoints only: walk up to this many pages with the cursor (default 1). Each page is a result and costs a page's price. |
| `max_age?` | `integer` or `null` | Accept cached results up to this many seconds old for every item (free cache hits), like the endpoints' max_age. |
| `max_credits?` | `integer` or `null` | Most the job may spend; reserved at creation. Default: the sum of every item's maximum price (times its max_pages). Items that would go over it are canceled. |
| `webhook_url?` | `string` or `null` | Also deliver this job's job.\* events to this URL: the URL of one of the account's webhook endpoints (its secret signs them). |
| `metadata?` | `object` | Up to metadata_max_keys string pairs of your own, returned with the job and in its events. |

Every item is checked before anything runs: an unknown endpoint or parameter, a missing required parameter, a duplicate `custom_id` or a list option on a single-item endpoint fails the whole request with [`invalid_input`](https://staging.socialtap.dev/docs/errors/index.md#invalid_input), and its `param` names the field, such as `requests[3].params.url`. Nothing is charged.

**Synchronous batches.** With `wait` (up to 1 minute), the request waits for the job to end. If it ends in time, the answer is `200` with the job and the first page of its results in `results` (continue from `results_page.next_cursor`). Otherwise it is `202` with the job as it is, and its URL in the `location` header.

**Any endpoint, in the background.** A single endpoint request with the header `Prefer: respond-async` becomes a one-request job: the answer is `202` with the job instead of the result. Its one item's `custom_id` is `request`.

## Walk whole lists

On a list endpoint, `max_pages` (up to 1,000 per item) walks the list for you with its cursor: every page is one result with its `page_index`, priced as one page. The walk stops at the end of the list, at `max_pages`, or when the job's credits run out. A job holds at most 10,000 pages in total.

## The job

Poll `GET /v1/jobs/{id}`: it is free, and `wait` long-polls until the job ends. Or subscribe a [webhook](https://staging.socialtap.dev/docs/webhooks/index.md) to `job.completed`.

| Field | Type | Description |
| --- | --- | --- |
| `object` | `string` | Always job.<br>One of `job` |
| `id` | `string` | The job ID (job_…). |
| `status` | `string` | queued: nothing started yet. running. canceling: canceled, running items finishing. completed: every item ended (check request_counts: there is no job-level failure). canceled. expired: the deadline passed; unfinished items expired.<br>One of `queued`, `running`, `canceling`, `completed`, `canceled`, `expired` |
| `request_counts` | `object` |  |
| `request_counts.total` | `integer` |  |
| `request_counts.queued` | `integer` | Not started. |
| `request_counts.running` | `integer` | Started (fetching, waiting for a retry or walking pages). |
| `request_counts.succeeded` | `integer` |  |
| `request_counts.failed` | `integer` |  |
| `request_counts.canceled` | `integer` |  |
| `request_counts.expired` | `integer` |  |
| `credits` | `object` |  |
| `credits.max` | `integer` | max_credits: the most the job may spend. |
| `credits.reserved` | `integer` | Credits still held for items not ended (0 once the job ends). |
| `credits.charged` | `integer` | Credits charged so far (items' prices; failed items are free). |
| `max_age` | `integer` or `null` | The job's max_age; null when it accepts no cached results. |
| `metadata` | `object` | Your metadata; {} when none. |
| `webhook_url` | `string` or `null` | The job's own webhook URL; null when none. |
| `created_at` | `string` (date-time) |  |
| `started_at` | `string` (date-time) or `null` | When the first item started; null before. |
| `finished_at` | `string` (date-time) or `null` | When the job ended; null while it runs. |
| `cancel_requested_at` | `string` (date-time) or `null` |  |
| `expires_at` | `string` (date-time) | The deadline: items not ended by then expire. |
| `results_url` | `string` or `null` | Every result as JSON Lines (one JobResult per line), signed: works without the API key until results_url_expires_at. Null until the job ends and after results_expire_at. |
| `results_url_expires_at` | `string` (date-time) or `null` |  |
| `results_expire_at` | `string` (date-time) or `null` | When results are deleted; null while the job runs. |

A job is `queued` until its first item starts, `running` while items run, and `completed` once every item has ended: there is no failed job, items fail one by one, so check `request_counts`. A canceled job is `canceling` until its running items finish, then `canceled`. A job that reaches `expires_at` (its deadline, 1 day after creation) ends `expired`: items that had not started expire, running ones finish.

## Results

`GET /v1/jobs/{id}/results` lists results in the order they completed, cursor-paged, while the job runs and after. Keep the last `page.next_cursor` to read only what arrived since. Each result is one page of one item:

| Field | Type | Description |
| --- | --- | --- |
| `object` | `string` | Always job_result.<br>One of `job_result` |
| `custom_id` | `string` |  |
| `endpoint` | `string` |  |
| `page_index` | `integer` | 0 for a single-item endpoint and a list's first page; n for the list's page n+1. |
| `status` | `string` | The page's outcome; an item's status is its last result's. canceled and expired results carry no result (error says why when it is not the job's cancel or deadline).<br>One of `succeeded`, `failed`, `canceled`, `expired` |
| `status_code` | `integer` or `null` | The HTTP status the synchronous request answered; null when it did not run. |
| `result` | `object` or `null` | Exactly the synchronous success envelope; null unless succeeded. |
| `result.data` | `any` | The endpoint's data: its resource, or a list page's items. |
| `result.page?` | `object` or `null` | List endpoints only. |
| `result.page.has_more` | `boolean` | True when another page exists; then next_cursor is set. An empty page with has_more true is not the end. |
| `result.page.next_cursor` | `string` or `null` | Pass as cursor, with the same other parameters, to get the next page; null at the end. |
| `result.page.next_cursor_expires_at` | `string` (date-time) or `null` | When next_cursor stops working, RFC 3339 UTC; null at the end. Don't store cursors. |
| `result.page.end_reason` | `string` or `null` | Why there are no more pages; null while has_more is true. end: the list ended, or reached since/until. platform_limit: the platform stops serving further pages although the list goes on. max_pages: SocialTap's cap on pages in one walk. New values may be added. A short or empty page is not the end: only has_more false is.<br>One of `end`, `platform_limit`, `max_pages` |
| `result.meta` | `object` |  |
| `result.meta.request_id` | `string` | Unique request ID; include it when contacting support. |
| `result.meta.platform` | `string` | Platform ID of the endpoint (youtube, tiktok, …). |
| `result.meta.source` | `string` | Which upstream strategy produced the data. |
| `result.meta.is_cached` | `boolean` | True when served from the cache (free); see max_age. |
| `result.meta.credits_charged` | `integer` | Credits charged for this request (0 for errors and cache hits). |
| `result.meta.latency_ms` | `integer` | Time SocialTap spent on the request, in milliseconds. |
| `result.meta.fetched_at` | `string` (date-time) | When the data was fetched from the platform, RFC 3339 UTC (older than the request for cache hits). |
| `result.meta.input` | `object` or `null` | What the request's url or handle resolved to: kind, platform ID and canonical URL, so you can confirm what was fetched; null for searches. |
| `result.meta.credits_remaining` | `integer` or `null` | Account balance after this request (the x-credits-remaining header); for a busy account an estimate that may not yet subtract concurrent requests (about a second of usage), never below the real balance; null when SocialTap could not read it. |
| `result.meta.client_request_id` | `string` or `null` | Your x-client-request-id, echoed; null when you sent none. |
| `result.meta.warnings` | `object[]` | Things to act on that did not fail the request: a deprecated endpoint, partial data. \[\] when none. |
| `result.meta.mode` | `string` or `null` | live, or test for a test key's answer: a recorded response (fixture), free and deterministic; null when SocialTap could not tell.<br>One of `live`, `test` |
| `error` | `object` or `null` | Exactly the synchronous error object; null on success and for the job's own cancel or deadline. |
| `error.code` | `string` | Machine-readable error code (see the error codes list); stable. |
| `error.message` | `string` | What went wrong and how to fix it, for humans and agents; wording may change. |
| `error.request_id` | `string` | Unique request ID; include it when contacting support. |
| `error.status` | `integer` | The HTTP status of the response, repeated for clients that only see the body. |
| `error.retryable` | `boolean` | True when repeating the same request later may succeed (after details.retry_after when present). |
| `error.param` | `string` or `null` | The query parameter or header the error is about; null when it is not about one input. |
| `error.remediation` | `string` | What to do about this code, one sentence. |
| `error.doc_url` | `string` | URL of this code's documentation. |
| `error.details` | `object` | The code's dynamic values (the error codes list documents each code's keys); {} when none. |
| `error.charged` | `boolean` | Whether the request was charged: always false, failed requests are free. |
| `completed_at` | `string` (date-time) |  |

- `result` is exactly the synchronous response's envelope (`data`, `meta`, and `page` on list endpoints).
- `error` is exactly the synchronous error object: branch on `error.code` as you would for a single request.
- An item that ended without a fetched page (canceled, expired) has one result with that status and no `result`.
- `?status=failed` lists the failures, to resubmit them in a new job.

Once the job ends, `results_url` is a signed link to every result as JSON Lines (one result per line), valid 1 hour and usable without an API key: hand it to a download tool or another service. Get the job again for a fresh link.

Results are kept 7 days after the job ends and job records 30 days; after that the job answers [`job_not_found`](https://staging.socialtap.dev/docs/errors/index.md#job_not_found).

## Credits

1. **Reserve.** Creating a job sets `max_credits` aside: by default the most its items can cost (each item's maximum price, times its `max_pages`). Lower it to cap what the job may spend. If your balance can't cover it, the request fails with [`insufficient_credits`](https://staging.socialtap.dev/docs/errors/index.md#insufficient_credits) and nothing runs; if the key has a spend limit, `max_credits` must also fit in what the limit leaves this interval, else [`spend_limit_reached`](https://staging.socialtap.dev/docs/errors/index.md#spend_limit_reached).
2. **Charge.** Each page is charged its own price as it ends, exactly like a single request: cache hits and failures are free. A page that the remaining budget can't pay for does not run: its item is `canceled` with an [`insufficient_credits`](https://staging.socialtap.dev/docs/errors/index.md#insufficient_credits) error whose `details.max_credits` says the job's cap stopped it, or [`spend_limit_reached`](https://staging.socialtap.dev/docs/errors/index.md#spend_limit_reached) when the key's spend limit did. Pages count against the key that created the job, exactly as its own requests do.
3. **Release.** When the job ends, whatever is still set aside goes back to your balance. `credits.charged` is what the job cost.

## Keys and test mode

Creating a job needs a key with the `jobs` scope, and `endpoints` too: a job's items are endpoint requests. A test key's jobs run every item in [test mode](https://staging.socialtap.dev/docs/test-mode/index.md): each result is exactly the recorded answer the endpoint gives that test key, free, and nothing is set aside or charged. Test jobs are separate from live ones: a test key lists and reads its own, a live key never sees them.

## Cancel

`POST /v1/jobs/{id}/cancel` stops a job: items that haven't started are `canceled` (free), items already running finish and are charged. Canceling a job that has ended changes nothing.

## Limits and fairness

A job runs up to 10 items at a time, and all of an account's jobs together up to 25, so a large job never crowds out your own synchronous requests. When SocialTap is busy, a job's items wait for capacity instead of failing; an item whose platform fails is retried up to 3 times before it fails. The unfinished jobs of an account hold at most 50,000 pages; above that, creating a job answers [`rate_limited`](https://staging.socialtap.dev/docs/errors/index.md#rate_limited): wait for running jobs to end.

## Retry safely

Send an `idempotency-key` with every create: if the connection drops, repeat the request with the same key and body and you get the job the first request created, never a second one. See [idempotency](https://staging.socialtap.dev/docs/idempotency/index.md).

---

[HTML version](https://staging.socialtap.dev/docs/jobs/) · [Docs index for AI agents](https://staging.socialtap.dev/llms.txt) · [All docs in one file](https://staging.socialtap.dev/llms-full.txt)