Refine one mask from clicks and/or a box
const url = 'https://api.stack.foldbun.com/v1/refine';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"model":"sam3","image":{"id":"img_3b1f0c5e9a7d42c8b6e1f0a9d8c7b6a5f4e3d2c1b0a998877665544332211000"},"points":[{"x":412,"y":230,"positive":true},{"x":80,"y":40,"positive":false}]}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://api.stack.foldbun.com/v1/refine \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "model": "sam3", "image": { "id": "img_3b1f0c5e9a7d42c8b6e1f0a9d8c7b6a5f4e3d2c1b0a998877665544332211000" }, "points": [ { "x": 412, "y": 230, "positive": true }, { "x": 80, "y": 40, "positive": false } ] }'Interactive refinement of a single object: positive and negative points, optionally a box. Limits: at most 20
points. Billed per call as one decode (refine, 1 per call), however many points and whether or not a box is
given, plus the image encoding (encode_image, waived on a cache hit; refining an image that was just
segmented is normally a hit). A client that re-sends all points after every click therefore pays the same for each click.
Interactive priority: if the call has not finished within 2 s of submission, it fails with HTTP 504 and error code upstream_timeout (retryable).
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Header Parameters
Section titled “Header Parameters”1 to 128 printable ASCII characters. Within an account, a key is remembered for 24 hours: repeating the request
with the same body returns the original response (header Idempotent-Replayed: true) without charging or
running the call again. The same key with a different body is 409 idempotency_conflict. See the Overview.
Request Bodyrequired
Section titled “Request Bodyrequired”Image, clicks and/or box to refine one mask.
Refinement request. At least one point or a box is required. Without a box, at least one point must be positive. The endpoint keeps no state between calls: send all points of the object with every call.
object
Model used for the refinement.
The image holding the object.
object
Handle returned by POST /v1/images (or by an earlier call): img_ plus the SHA-256 of the sanitized image. Valid only inside the uploading account.
Reserved (later). Currently 400 unsupported.
Standard base64 of a JPEG, PNG or WebP file; a data: prefix is tolerated.
Positive and negative clicks (at most 20).
A click on the image. Coordinates must be non-negative and lie inside the image (image.width x image.height).
object
Horizontal position in pixels of the oriented original.
Vertical position in pixels of the oriented original.
True: the point lies on the object. False: the point lies on something to exclude.
Box around the object (optional).
Optional tuning of the result. The refined object is an object prompt: it is always returned, and candidates of multimask are never merged or compared for overlap. The vertex limit and the budgets of SegmentOptions apply.
object
Return up to 3 candidate masks instead of the best one.
Long side of the mask grid in pixels (64 to 4096), used by mask output only. The grid is never larger than the image, so its long side is the smaller of this value and the image’s long side; mask.size holds the actual size.
Geometry of each instance. bbox, score and area are always returned; geometry that is not requested is not computed.
object
polygon: simplified outline with holes (Instance.polygon). mask: pixel mask (Instance.mask). []: no geometry.
Polygon output. Outlines are traced on the cleaned mask at image resolution (above 4096 px on the long side, on a
grid of 4096 px and scaled) and simplified with the Douglas-Peucker algorithm. tolerance and tolerancePx are
one setting: giving both in one object is 400 invalid_request, and in layered options the more specific layer wins.
object
Simplification tolerance as a fraction of the image diagonal (sqrt(image.width² + image.height²)): the largest distance between a simplified edge and the traced outline. The default can change; set the value explicitly where it matters.
Simplification tolerance in image pixels, instead of tolerance. 0 keeps every outline vertex (subject to the vertex limit).
Return the holes of each part. false returns outer rings only.
Largest number of parts (separate pieces) per instance. The largest parts are kept.
Mask output, used when geometry includes mask.
object
coco: COCO compressed RLE of the whole mask, lossless, readable with pycocotools. webp: lossless WebP of the
tight crop around the object; smaller than coco only for very detailed outlines such as hair, slower to produce,
and limited by the WebP pixel budget (see SegmentOptions).
Mask cleanup, applied to every instance before the size filters and the geometry output. Areas are pixel areas of the mask.
object
Parts (separate pieces) other than the largest one that are smaller than this are removed; the largest part is always kept. null keeps every part.
object
Threshold as a fraction of the image area (image.width x image.height).
Threshold in square image pixels.
Holes smaller than this are filled. null keeps every hole.
object
Threshold as a fraction of the image area (image.width x image.height).
Threshold in square image pixels.
Example
{ "model": "sam3", "image": { "id": "img_3b1f0c5e9a7d42c8b6e1f0a9d8c7b6a5f4e3d2c1b0a998877665544332211000" }, "points": [ { "x": 412, "y": 230, "positive": true }, { "x": 80, "y": 40, "positive": false } ]}Responses
Section titled “Responses”The refined object (up to 3 candidates with options.multimask), in the same shape as a segmentation result without dropStats. CBOR with Accept: application/cbor.
object
Call id call_<uuid>; quote it to support.
Id of the model that served the call.
The oriented original the coordinates refer to. Absent for text-only embeddings.
object
Image handle: img_ plus the SHA-256 of the sanitized bytes.
Width of the oriented original in pixels (EXIF orientation applied).
Height of the oriented original in pixels (EXIF orientation applied).
Size of the stored image in bytes.
Media type of the stored image (the type of the upload).
Credits this call charged. 0 for failed calls and calls covered by an included daily allowance. Internal accounts are metered but not debited, so cost is then the price of the call.
Account balance in credits after the call (unchanged for internal accounts).
Server-side timing of this call in milliseconds. queueMs is the time from submission until a GPU node took the job (queueing, including a re-queue after a node failure); processingMs is from there until the result was back (image fetch, inference, and the node-to-server round trips). inferenceMs, imageFetchMs and serverMs break this down further. Network time between the client and the API is not included.
object
Milliseconds from submission until a GPU node took the job.
Milliseconds from there until the result was back.
Model time on the GPU node as the node measured it, without the image fetch. Never larger than processingMs. Absent when the node did not report it.
Time the node spent obtaining the image bytes. 0 when the image was sent with the job or the node did not need it (encode cache hit). Absent when the node did not report it.
Whole time inside the API server: request received, until just before the response is written. Includes reading the request body, image preparation, admission, queueMs, processingMs and billing settlement. Absent only in an idempotent replay of a response stored by an older server.
How the call was priced.
object
One line per priced unit of the call.
object
Price-book unit, as in GET /v1/models prices[].unit (for example encode_image, prompt, refine, image).
Units used.
List price of one unit.
Price of the line: unitPrice * quantity, or 0 when waived. On a call that is not charged (trial, included daily allowance) lines still show the list price.
True for the image-encoding fee when the node already held the encoding (cache: hit); amount is then 0 and the fee that would have applied is unitPrice * quantity.
Price of the call (sum of the line amounts, cut down to the balance when capped).
Whether the account balance was debited. False for Playground trial calls and for calls covered by an included daily allowance (then pool and allowance are set); true for billed calls. Internal accounts are metered at the list price and never debited, so charged is false and total is the price.
The pool whose included daily allowance covered the call (public or free). Null for billed calls (including calls beyond the allowance), trial calls and internal accounts.
Today’s (UTC) included calls of pool, counting this one. Present only when pool is set.
object
Calls used today (UTC), including this one.
Included calls per UTC day.
Present and true only when the price exceeded the balance and the debit was cut down to it.
Segmented objects in score order (see SegmentOptions).
One segmented object. All coordinates are image pixels unless noted.
object
Index into prompts that produced the instance (segment only); for a merged instance, the lowest index of its members.
The group of the prompt, when the request set one.
The text prompt (sam3 text prompts), _exemplar for exemplar boxes. Absent on instances of point prompts. Absent in POST /v1/refine responses.
Confidence of the instance. For text-prompt and exemplar instances it is the detection confidence; for point, box (sam-lite) and refine instances it is the predicted IoU of the mask, the model’s own estimate of how well the mask fits the object. A merged instance has the highest score of its members.
[x1, y1, x2, y2]: tight box around the cleaned instance. x2 and y2 are exclusive, so x2 - x1 is the width in pixels.
Pixel area of the cleaned instance.
Merged instances only: the number of instances combined.
When geometry includes polygon: the parts, largest first. Empty when no part forms a ring of 3 vertices (objects one or two pixels thin).
One part (separate piece) of a polygon. Rings are flat integer arrays [x0, y0, x1, y1, ...] in image pixels. Each
vertex is the centre of an outline pixel, so 0 <= x <= image.width - 1 and 0 <= y <= image.height - 1. A ring has
at least 3 distinct vertices and is open (the first vertex is not repeated). Outer rings run clockwise on screen
(positive shoelace area with y pointing down), holes counter-clockwise. A simplified ring can intersect itself.
object
The outer ring.
Holes, largest first. Absent when the part has none or polygon.holes is false.
Present when the vertex limit (8,192 per instance) required a larger tolerance than requested: the tolerance actually applied, in image pixels.
The whole mask on the mask grid as COCO compressed RLE (column-major). {"size", "counts"} is a valid COCO RLE object for pycocotools.
object
COCO compressed RLE.
[height, width] of the mask grid: the image scaled to a long side of at most options.maskSize, never enlarged. Scale mask coordinates by image.width / size[1] horizontally and image.height / size[0] vertically.
COCO compressed run-length string.
The tight crop around the object on the mask grid, as a lossless WebP. Object pixels are white and the rest black; a channel value of 128 or more is object.
object
Lossless WebP crop.
[height, width] of the mask grid, as for coco.
[x, y] of the top-left corner of the crop on the mask grid.
[width, height] of the crop on the mask grid.
Base64 WebP; a byte string in CBOR responses.
budget: the response budget was used up, and the instance carries no geometry. vertices: the largest part alone exceeds the vertex limit, and the instance has no polygon. bbox, score and area are always kept.
POST /v1/segment only: one entry per prompt that lost instances, in prompt order; empty when nothing was lost.
What one prompt lost, by reason. Every counter is present (0 when none).
object
Index into prompts.
Candidates below filter.minScore that scored at least report.lowScoreFloor (0.1 when it is null); models emit many near-zero scores, which are not counted.
Dropped by filter.minSize.
Dropped by filter.minArea.
Dropped by nms.
Beyond the 300 candidates per request or beyond maxInstances.
Returned without geometry because the response budget was used up (omitted: budget); these are still in instances.
Only with report.droppedBoxes: dropped instances, highest score first, at most 300.
One dropped instance (report.droppedBoxes).
object
Index into prompts.
The step that dropped it; same names as in DropStats.
Score of the instance.
[x1, y1, x2, y2] as in Instance.bbox. For lowScore, limit and some tooSmall entries the instance was dropped before its mask was traced, and the box comes from the low-resolution mask of the model: approximate, and possibly larger than the object.
hit: the image encoding was reused and the encode fee waived. Not a promise; see the Overview.
Image encoding fee charged for this call; 0 on a cache hit.
Example
{ "image": { "contentType": "image/jpeg" }, "instances": [ { "mask": { "encoding": "coco" }, "omitted": "budget" } ], "dropped": [ { "reason": "lowScore" } ], "cache": "hit"}object
Call id call_<uuid>; quote it to support.
Id of the model that served the call.
The oriented original the coordinates refer to. Absent for text-only embeddings.
object
Image handle: img_ plus the SHA-256 of the sanitized bytes.
Width of the oriented original in pixels (EXIF orientation applied).
Height of the oriented original in pixels (EXIF orientation applied).
Size of the stored image in bytes.
Media type of the stored image (the type of the upload).
Credits this call charged. 0 for failed calls and calls covered by an included daily allowance. Internal accounts are metered but not debited, so cost is then the price of the call.
Account balance in credits after the call (unchanged for internal accounts).
Server-side timing of this call in milliseconds. queueMs is the time from submission until a GPU node took the job (queueing, including a re-queue after a node failure); processingMs is from there until the result was back (image fetch, inference, and the node-to-server round trips). inferenceMs, imageFetchMs and serverMs break this down further. Network time between the client and the API is not included.
object
Milliseconds from submission until a GPU node took the job.
Milliseconds from there until the result was back.
Model time on the GPU node as the node measured it, without the image fetch. Never larger than processingMs. Absent when the node did not report it.
Time the node spent obtaining the image bytes. 0 when the image was sent with the job or the node did not need it (encode cache hit). Absent when the node did not report it.
Whole time inside the API server: request received, until just before the response is written. Includes reading the request body, image preparation, admission, queueMs, processingMs and billing settlement. Absent only in an idempotent replay of a response stored by an older server.
How the call was priced.
object
One line per priced unit of the call.
object
Price-book unit, as in GET /v1/models prices[].unit (for example encode_image, prompt, refine, image).
Units used.
List price of one unit.
Price of the line: unitPrice * quantity, or 0 when waived. On a call that is not charged (trial, included daily allowance) lines still show the list price.
True for the image-encoding fee when the node already held the encoding (cache: hit); amount is then 0 and the fee that would have applied is unitPrice * quantity.
Price of the call (sum of the line amounts, cut down to the balance when capped).
Whether the account balance was debited. False for Playground trial calls and for calls covered by an included daily allowance (then pool and allowance are set); true for billed calls. Internal accounts are metered at the list price and never debited, so charged is false and total is the price.
The pool whose included daily allowance covered the call (public or free). Null for billed calls (including calls beyond the allowance), trial calls and internal accounts.
Today’s (UTC) included calls of pool, counting this one. Present only when pool is set.
object
Calls used today (UTC), including this one.
Included calls per UTC day.
Present and true only when the price exceeded the balance and the debit was cut down to it.
Segmented objects in score order (see SegmentOptions).
One segmented object. All coordinates are image pixels unless noted.
object
Index into prompts that produced the instance (segment only); for a merged instance, the lowest index of its members.
The group of the prompt, when the request set one.
The text prompt (sam3 text prompts), _exemplar for exemplar boxes. Absent on instances of point prompts. Absent in POST /v1/refine responses.
Confidence of the instance. For text-prompt and exemplar instances it is the detection confidence; for point, box (sam-lite) and refine instances it is the predicted IoU of the mask, the model’s own estimate of how well the mask fits the object. A merged instance has the highest score of its members.
[x1, y1, x2, y2]: tight box around the cleaned instance. x2 and y2 are exclusive, so x2 - x1 is the width in pixels.
Pixel area of the cleaned instance.
Merged instances only: the number of instances combined.
When geometry includes polygon: the parts, largest first. Empty when no part forms a ring of 3 vertices (objects one or two pixels thin).
One part (separate piece) of a polygon. Rings are flat integer arrays [x0, y0, x1, y1, ...] in image pixels. Each
vertex is the centre of an outline pixel, so 0 <= x <= image.width - 1 and 0 <= y <= image.height - 1. A ring has
at least 3 distinct vertices and is open (the first vertex is not repeated). Outer rings run clockwise on screen
(positive shoelace area with y pointing down), holes counter-clockwise. A simplified ring can intersect itself.
object
The outer ring.
Holes, largest first. Absent when the part has none or polygon.holes is false.
Present when the vertex limit (8,192 per instance) required a larger tolerance than requested: the tolerance actually applied, in image pixels.
The whole mask on the mask grid as COCO compressed RLE (column-major). {"size", "counts"} is a valid COCO RLE object for pycocotools.
object
COCO compressed RLE.
[height, width] of the mask grid: the image scaled to a long side of at most options.maskSize, never enlarged. Scale mask coordinates by image.width / size[1] horizontally and image.height / size[0] vertically.
COCO compressed run-length string.
The tight crop around the object on the mask grid, as a lossless WebP. Object pixels are white and the rest black; a channel value of 128 or more is object.
object
Lossless WebP crop.
[height, width] of the mask grid, as for coco.
[x, y] of the top-left corner of the crop on the mask grid.
[width, height] of the crop on the mask grid.
Base64 WebP; a byte string in CBOR responses.
budget: the response budget was used up, and the instance carries no geometry. vertices: the largest part alone exceeds the vertex limit, and the instance has no polygon. bbox, score and area are always kept.
POST /v1/segment only: one entry per prompt that lost instances, in prompt order; empty when nothing was lost.
What one prompt lost, by reason. Every counter is present (0 when none).
object
Index into prompts.
Candidates below filter.minScore that scored at least report.lowScoreFloor (0.1 when it is null); models emit many near-zero scores, which are not counted.
Dropped by filter.minSize.
Dropped by filter.minArea.
Dropped by nms.
Beyond the 300 candidates per request or beyond maxInstances.
Returned without geometry because the response budget was used up (omitted: budget); these are still in instances.
Only with report.droppedBoxes: dropped instances, highest score first, at most 300.
One dropped instance (report.droppedBoxes).
object
Index into prompts.
The step that dropped it; same names as in DropStats.
Score of the instance.
[x1, y1, x2, y2] as in Instance.bbox. For lowScore, limit and some tooSmall entries the instance was dropped before its mask was traced, and the box comes from the low-resolution mask of the model: approximate, and possibly larger than the object.
hit: the image encoding was reused and the encode fee waived. Not a promise; see the Overview.
Image encoding fee charged for this call; 0 on a cache hit.
default
Section titled “default”The error envelope. The HTTP status follows the code; only retryable: true errors are worth repeating
unchanged. A failed call costs 0.
| Status | Codes |
|---|---|
| 400 | invalid_request, invalid_image (the image passed the upload checks but could not be decoded), unsupported |
| 401 | unauthorized |
| 402 | insufficient_balance |
| 403 | model_locked |
| 404 | not_found, image_not_found |
| 409 | idempotency_conflict, request_in_progress |
| 413 | payload_too_large |
| 429 | rate_limited, daily_limit_reached, too_many_batches (all with Retry-After) |
| 500 | internal_error |
| 502 | upstream_error, bad_model_output |
| 503 | model_unavailable, model_busy, unavailable (all with Retry-After) |
| 504 | upstream_timeout |
model_unavailable also covers a short loss of contact with the only GPU worker serving the model: for about a
minute after the loss, interactive calls answer it at once with a short Retry-After (5 s), while sync and
batch calls stay queued until their own deadline instead of failing.
Meaning and retry guidance for each code: Errors.
Body of every error response.
object
The error.
object
Stable machine-readable error code. Branch on this, not on message. Listed with the HTTP statuses under Responses.
Human-readable explanation. The wording can change.
True when repeating the same request unchanged can succeed.
Suggested wait before a retry in milliseconds; 0 when there is no hint.
Example
{ "error": { "code": "model_busy", "message": "model sam3 is busy; retry shortly", "retryable": true, "retryAfterMs": 5000 }}Headers
Section titled “Headers”Whole seconds to wait before retrying; present on 429 and 503.
Server-generated request id (16 hexadecimal characters); present on every response. Quote it when contacting support.