SAM 3 (sam3)

sam3 is SAM 3 (Segment Anything Model 3) by Meta. It segments by concept: a short text prompt or an exemplar box returns a mask for every matching instance in the image. It also segments single objects that the caller marks with points, a box or clicks. Stack runs the concept detector and the single-object tracker of SAM 3 on one shared image encoding, so a concept call and a click call on the same image can reuse one cached encoding.

Model ID	sam3 (default model of POST /v1/refine)
Model	SAM 3 (Meta)
Weights	Official SAM 3 release by Meta (facebook/sam3)
Use cases	Concept segmentation, Interactive segmentation
Endpoints	POST /v1/segment, POST /v1/refine
Pool	Paid: billed from the balance
Billing units	encode_image (waived on a cache hit), prompt, point, refine. Units and prices: Billing
Licence	SAM License

How it works

A call has two stages: the image is encoded once, and every prompt is then decoded separately. Billing follows these two stages.

Image encoding. The image is resized to 1008 × 1008 pixels and passed through the image encoder, a vision transformer (ViT) trunk that turns the image into feature maps. This is the largest part of the work and is billed once per call as encode_image. The result is cached per account and image, on a best-effort basis (deleted after at most 7 days); when a later call finds the encoding in the cache, the stage is skipped and the fee is waived. The concept detector and the single-object tracker of SAM 3 read the same encoding, so a concept call and a click call on the same image can share one cache entry. The cached encoding is stored compactly, so a result on a cache hit can differ very slightly from a fresh encode.

Concept prompts. Each text prompt is one decode of the concept detector. Exemplar boxes are not decoded on their own: all boxes of the call, positive and negative, are added to the query of every text prompt. A call with boxes and no text prompt forms one visual query from all its boxes and is one decode. In a decode, a fusion encoder conditions the 72 × 72 = 5,184 image tokens on the query, a DETR-style decoder with 200 object queries proposes instances, a mask head produces a mask for each proposal, and a presence score gates the result, so that an absent concept returns nothing. The whole decode runs again for every query, which is why each text prompt is billed (prompt), and a box-only call is billed once. Exemplar boxes add no charge because they are part of the decode of each text prompt. The length of a text prompt does not change the price: the output of the text encoder is cached per string, and the decode costs the same for any prompt.

Point prompts. Each point prompt in POST /v1/segment is one decode of the single-object tracker: the point is encoded as a prompt, and a mask decoder returns the best mask for that one object. Each point is therefore billed as one point, and its instance carries no label.

Refinement. POST /v1/refine is one tracker decode for one object. All points and the optional box are inputs of that single pass, so a call is billed once as refine, however many points it carries. With options.multimask, the same pass returns up to 3 candidate masks.

Each instance carries a simplified polygon with holes in image pixels; a pixel mask (COCO RLE or lossless WebP) is returned on request. See Post-processing. The score of a concept instance is the detection confidence; the score of a point or refine instance is the predicted IoU of the mask.

In each use case

Concept segmentation

POST /v1/segment with text prompts, exemplar boxes, or both. The number of instances is not known in advance.

• A text prompt names a concept in 1 to 200 characters. Every instance of the concept is returned as its own object with a score, and label carries the prompt text.
• An exemplar box [x1, y1, x2, y2] around one object returns the boxed object and the objects that look like it (“find more like this”). positive defaults to true; with positive: false the box is a negative exemplar (“not like this”). All exemplar boxes of a call, positive and negative, condition every text prompt in the same call. Without a text prompt, all boxes together form one visual query whose instances have the label _exemplar, and that query needs at least one positive box. A call whose boxes are all negative needs a text prompt; points do not count. Boxes and text prompts can be mixed. Exemplar boxes suit things that are hard to name or that have many look-alikes.
• filter.minScore (default 0.5), size and area filters, nms and options.maxInstances (1 to 300, default 100) filter and limit the instances. See Post-processing.

Use case guide and example call: Concept segmentation.

Interactive segmentation

sam3 marks single objects in two ways.

• Points in POST /v1/segment. A prompt { "point": { "x", "y" } } returns exactly one instance: the best mask of the object under the point. The instance has no label, and its score is the predicted IoU. A point is a positive mark only; negative points belong to POST /v1/refine. Points can be mixed with text prompts and exemplar boxes in the same call. filter, nms and options.maxInstances do not apply to point instances, and exemplar boxes never affect points.
• A box in POST /v1/segment is not an object marker. It stays an exemplar and returns look-alikes as well. To segment the one object inside a box, send the box to POST /v1/refine.
• Clicks and a box in POST /v1/refine. Up to 20 positive and negative points and an optional box describe one object. The response holds the best mask, or up to 3 candidate masks with options.multimask. The whole call is one decode. The call has interactive priority with a 2 second deadline. sam3 is the default model of this endpoint.

Comparison with sam-lite: Interactive segmentation.

Use case guide and example call: Interactive segmentation.

Limits and notes

• 1 to 50 prompts per segment call, counting text prompts, exemplar boxes and points together; there is no separate cap on points. At most 10 exemplar boxes. A call needs at least one text prompt, positive exemplar box or point; a call whose boxes are all negative needs a text prompt. Text prompts of 1 to 200 characters. options.maxInstances up to 300.
• At most 20 points per refine call, with at least one point or a box.
• Boxes and points must be non-negative and lie inside the image.
• Image limits: see Images. All coordinates are pixels of the oriented original. Polygons and boxes are in image pixels. A requested mask lies on a grid whose long side is at most options.maskSize (default 2048 px): scale it by mask.size.
• Latency: recorded Playground calls had timing.processingMs between about 390 and 560 ms. The figure depends on the image, on queueing and on node load, and it is not a guarantee.