Async tasks
Async tasks
Add a pingback URL to a search to queue it instead of waiting for the answer. Send 100 at once on a plan of 40, and all 100 are accepted.
How it works
- Queue a search: a normal search with a
pingback_url. It returns at once with202and a task id. - We run it as search capacity frees up, up to your plan's concurrency at a time, in turns with other customers.
- When it ends we call your URL with the id and the status: the pingback.
- You read the answer with GET /v1/tasks/{id}.
Why queue
Live searches above your plan's concurrency are refused with
429. Queued searches are never refused for that reason, so you can send a whole batch and collect the answers as they come.Good to know
- Credits are held when a search is queued and charged only for the pages that come back. A failed search costs nothing.
- Searches usually start within seconds. A failing search is tried up to 3 times.
- A search that cannot run within an hour ends with
task_expired, and its credits go back. - Up to 10,000 of your searches can wait or run at once; more get
429 queue_full. - Answers are kept for 7 days.