# SAM 3 (sam3)

`sam3` is SAM 3 (Segment Anything Model 3) by Meta. It segments by concept: a short text prompt or an exemplar box returns a mask for every matching instance in the image. It also segments single objects that the caller marks with points, a box or clicks. Stack runs the concept detector and the single-object tracker of SAM 3 on one shared image encoding, so a concept call and a click call on the same image can reuse one cached encoding.

| | |
|---|---|
| Model ID | `sam3` (default model of `POST /v1/refine`) |
| Model | SAM 3 (Meta) |
| Weights | Official SAM 3 release by Meta (`facebook/sam3`) |
| Use cases | [Concept segmentation](https://stack.foldbun.com/docs/use-cases/concept-segmentation/), [Interactive segmentation](https://stack.foldbun.com/docs/use-cases/interactive-segmentation/) |
| Endpoints | `POST /v1/segment`, `POST /v1/refine` |
| Pool | Paid: billed from the balance |
| Billing units | `encode_image` (waived on a cache hit), `prompt`, `point`, `refine`. Units and prices: [Billing](https://stack.foldbun.com/docs/billing/#price-list) |
| Licence | SAM License |

## How it works

A call has two stages: the image is encoded once, and every prompt is then decoded separately. Billing follows these two stages.

**Image encoding.** The image is resized to 1008 × 1008 pixels and passed through the image encoder, a vision transformer (ViT) trunk that turns the image into feature maps. This is the largest part of the work and is billed once per call as `encode_image`. The result is cached per account and image, on a best-effort basis (deleted after at most 7 days); when a later call finds the encoding in the cache, the stage is skipped and the fee is waived. The concept detector and the single-object tracker of SAM 3 read the same encoding, so a concept call and a click call on the same image can share one cache entry. The cached encoding is stored compactly, so a result on a cache hit can differ very slightly from a fresh encode.

**Concept prompts.** Each text prompt is one decode of the concept detector. Exemplar boxes are not decoded on their own: all boxes of the call, positive and negative, are added to the query of every text prompt. A call with boxes and no text prompt forms one visual query from all its boxes and is one decode. In a decode, a fusion encoder conditions the 72 × 72 = 5,184 image tokens on the query, a DETR-style decoder with 200 object queries proposes instances, a mask head produces a mask for each proposal, and a presence score gates the result, so that an absent concept returns nothing. The whole decode runs again for every query, which is why each text prompt is billed (`prompt`), and a box-only call is billed once. Exemplar boxes add no charge because they are part of the decode of each text prompt. The length of a text prompt does not change the price: the output of the text encoder is cached per string, and the decode costs the same for any prompt.

**Point prompts.** Each point prompt in `POST /v1/segment` is one decode of the single-object tracker: the point is encoded as a prompt, and a mask decoder returns the best mask for that one object. Each point is therefore billed as one `point`, and its instance carries no `label`.

**Refinement.** `POST /v1/refine` is one tracker decode for one object. All points and the optional box are inputs of that single pass, so a call is billed once as `refine`, however many points it carries. With `options.multimask`, the same pass returns up to 3 candidate masks.

Each instance carries a simplified polygon with holes in image pixels; a pixel mask (COCO RLE or lossless WebP) is returned on request. See [Post-processing](https://stack.foldbun.com/docs/post-processing/). The `score` of a concept instance is the detection confidence; the `score` of a point or refine instance is the predicted IoU of the mask.

## In each use case

### Concept segmentation

`POST /v1/segment` with text prompts, exemplar boxes, or both. The number of instances is not known in advance.

- A text prompt names a concept in 1 to 200 characters. Every instance of the concept is returned as its own object with a score, and `label` carries the prompt text.
- An exemplar box `[x1, y1, x2, y2]` around one object returns the boxed object and the objects that look like it ("find more like this"). `positive` defaults to `true`; with `positive: false` the box is a negative exemplar ("not like this"). All exemplar boxes of a call, positive and negative, condition every text prompt in the same call. Without a text prompt, all boxes together form one visual query whose instances have the label `_exemplar`, and that query needs at least one positive box. A call whose boxes are all negative needs a text prompt; points do not count. Boxes and text prompts can be mixed. Exemplar boxes suit things that are hard to name or that have many look-alikes.
- `filter.minScore` (default 0.5), size and area filters, `nms` and `options.maxInstances` (1 to 300, default 100) filter and limit the instances. See [Post-processing](https://stack.foldbun.com/docs/post-processing/).

Use case guide and example call: [Concept segmentation](https://stack.foldbun.com/docs/use-cases/concept-segmentation/).

### Interactive segmentation

`sam3` marks single objects in two ways.

- **Points in `POST /v1/segment`.** A prompt `{ "point": { "x", "y" } }` returns exactly one instance: the best mask of the object under the point. The instance has no `label`, and its `score` is the predicted IoU. A point is a positive mark only; negative points belong to `POST /v1/refine`. Points can be mixed with text prompts and exemplar boxes in the same call. `filter`, `nms` and `options.maxInstances` do not apply to point instances, and exemplar boxes never affect points.
- **A box in `POST /v1/segment` is not an object marker.** It stays an exemplar and returns look-alikes as well. To segment the one object inside a box, send the box to `POST /v1/refine`.
- **Clicks and a box in `POST /v1/refine`.** Up to 20 positive and negative points and an optional box describe one object. The response holds the best mask, or up to 3 candidate masks with `options.multimask`. The whole call is one decode. The call has interactive priority with a 2 second deadline. `sam3` is the default model of this endpoint.

Comparison with [`sam-lite`](https://stack.foldbun.com/docs/models/sam-lite/): [Interactive segmentation](https://stack.foldbun.com/docs/use-cases/interactive-segmentation/#supported-models).

Use case guide and example call: [Interactive segmentation](https://stack.foldbun.com/docs/use-cases/interactive-segmentation/).

## Limits and notes

- 1 to 50 prompts per segment call, counting text prompts, exemplar boxes and points together; there is no separate cap on points. At most 10 exemplar boxes. A call needs at least one text prompt, positive exemplar box or point; a call whose boxes are all negative needs a text prompt. Text prompts of 1 to 200 characters. `options.maxInstances` up to 300.
- At most 20 points per refine call, with at least one point or a box.
- Boxes and points must be non-negative and lie inside the image.
- Image limits: see [Images](https://stack.foldbun.com/docs/images/). All coordinates are pixels of the oriented original. Polygons and boxes are in image pixels. A requested mask lies on a grid whose long side is at most `options.maskSize` (default 2048 px): scale it by `mask.size`.
- Latency: recorded Playground calls had `timing.processingMs` between about 390 and 560 ms. The figure depends on the image, on queueing and on node load, and it is not a guarantee.