# Interactive segmentation

Interactive segmentation returns the mask of an object that the caller marks: a point on the object, a box around it, or a set of positive and negative clicks. Each mark gives exactly one object. Typical uses are click-to-select in annotation tools and image editors, and pipelines that already know where the objects are, for example boxes from a detector. To find every instance of a concept without marking each one, see [Concept segmentation](https://stack.foldbun.com/docs/use-cases/concept-segmentation/).

## Mark objects with points

A point marks one location on an object and returns one mask for that object, the best of the model's candidates. A call to `POST /v1/segment` can carry many points: it returns one instance per point, in the order of `prompts`, and the image is encoded once for all of them. A point is always a positive mark, because a lone negative point selects nothing. Negative points belong to refinement, where they trim a mask (see [Refine one object with clicks](#refine-one-object-with-clicks)).

![Dog lying on grass with a blue mask over the whole dog and a green dot on its chest.](../../../../assets/docs/sam-lite-point.webp)

*`sam-lite`: recorded Playground result for one positive click (green dot). The image is the result of a `/v1/refine` call with `points`; the score is the `score` of the single instance.*

With `sam3`, points can share a call with text prompts and exemplar boxes. Each point still returns exactly one instance, without a `label`. `filter`, `nms` and `options.maxInstances` apply to the text and exemplar instances only, and exemplar boxes never affect points. For example, a point on a car together with the text prompt `person` returns the car and every person in the image.

## Mark objects with boxes

With `sam-lite`, a box returns the mask of the object inside it. Boxes and points can be mixed in one call, and each prompt gives one instance. `sam-lite` does not use `positive`: a box always marks the object, and a point is always positive.

![Kitchen scene with a blue mask over a kettle inside a green box.](../../../../assets/docs/sam-lite-box.webp)

*`sam-lite`: recorded Playground result for one box (green outline), cropped to the surroundings of the box. The image is the result of a `/v1/refine` call with `box`; the score is the `score` of the single instance.*

With `sam3`, a box in `POST /v1/segment` is an exemplar: it returns the boxed object together with the objects that look like it (see [Exemplar boxes](https://stack.foldbun.com/docs/use-cases/concept-segmentation/#exemplar-boxes)). To get only the boxed object, send the box to `POST /v1/refine`, as in the next section.

## Refine one object with clicks

`POST /v1/refine` segments or refines one object from positive and negative points, optionally with a box. A positive point marks the object, and a negative point marks what to leave out. A call needs at least one point or a box and accepts up to 20 points. It has interactive priority with a 2 second deadline. `sam3` is the default model; `sam-lite` takes the same body.

The whole call is one decode for one object: the points and the box are inputs of the same pass, so a call is billed once, whatever the number of points (see [How SAM calls are billed](https://stack.foldbun.com/docs/billing/#how-sam-calls-are-billed)). By default the response holds the best mask, as a polygon. `options.multimask` returns up to 3 candidate masks instead, which helps when one click is ambiguous, for example a click on a shirt that could mean the shirt or the person.

A typical sequence: a box around a car with one positive point on it marks the car. When the mask spills onto a neighbouring object or the background, the next call adds a negative point there, and that area leaves the mask.

An editor sends the full set of clicks with every call: the endpoint keeps no state between calls. While the image encoding stays cached, repeated calls on the same image are fast and pay no encoding fee.

## Output options

The default output is a polygon in image pixels. Pixel masks, the polygon tolerance and fragment and hole cleanup are options: `options.defaults` and `prompts[].options` in `POST /v1/segment`, and `options.output` and `options.cleanup` in `POST /v1/refine`. Every point and box prompt is an object prompt, so it is always returned and `filter` does not apply to it. See [Post-processing](https://stack.foldbun.com/docs/post-processing/).

## Supported models

| | [`sam3`](https://stack.foldbun.com/docs/models/sam3/#interactive-segmentation) | [`sam-lite`](https://stack.foldbun.com/docs/models/sam-lite/#interactive-segmentation) |
|---|---|---|
| Model | SAM 3 | SAM 2.1 Hiera-S |
| Marks one object in `/v1/segment` | Point | Point or box |
| A box in `/v1/segment` | Exemplar: returns look-alikes | The object inside the box |
| Text prompts in the same call | Yes | No |
| `/v1/refine` | Points and an optional box; up to 3 candidates | Points and an optional box; up to 3 candidates |
| List price (credits) | 0.5 per image encoding + 0.02 per point in `/v1/segment`, or 0.02 per `/v1/refine` call | 0.2 per image encoding + 0.01 per point or box in `/v1/segment`, or 0.01 per `/v1/refine` call |
| Pool | Paid: billed from the balance | Free: the first 1,000 calls per UTC day after the first top-up, then billed from the balance |
| Recorded processing time | About 390 to 560 ms (recorded concept-prompt calls) | About 180 ms |
| Pick when | The same call also needs text or exemplar prompts, or the image is also segmented by concept with `sam3` | Cost and latency matter most, and every object is marked by the caller |

Both models use the image encoding cache: a `/v1/refine` call right after a `/v1/segment` call on the same image with the same model normally pays no image encoding. See [Billing](https://stack.foldbun.com/docs/billing/#sam-image-encoding-fee-and-cache-hits). For how each model produces these results, see [How it works](https://stack.foldbun.com/docs/models/sam3/#how-it-works) for `sam3` and [How it works](https://stack.foldbun.com/docs/models/sam-lite/#how-it-works) for `sam-lite`.

## Example call

One click on the dog in the first figure, sent to `POST /v1/refine` with `sam-lite`:

```bash
curl https://api.stack.foldbun.com/v1/refine \
  -H "Authorization: Bearer $FOLDBUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sam-lite",
    "image": { "id": "img_3b73d4112efca712359a1b2c89922b8494fac4a8f0b033c4dfc5205c3ec95b12" },
    "points": [{ "x": 760, "y": 560, "positive": true }]
  }'
```

The response is abridged: the polygon ring is shortened.

```json
{
  "id": "call_01a1913c-4de0-7f15-afbd-d26a0d0c347f",
  "model": "sam-lite",
  "image": {
    "id": "img_3b73d4112efca712359a1b2c89922b8494fac4a8f0b033c4dfc5205c3ec95b12",
    "width": 1512,
    "height": 1008,
    "bytes": 378383,
    "contentType": "image/jpeg"
  },
  "instances": [
    {
      "score": 0.9844,
      "bbox": [382, 102, 1171, 912],
      "area": 351204,
      "polygon": [
        { "outer": [602, 103, 771, 131, 910, 224, 1096, 421, 1170, 633, 1051, 811, 904, 911, 612, 905, 441, 801, 383, 583, 436, 301, 522, 168] }
      ]
    }
  ],
  "cache": "miss",
  "imageFee": 0.2,
  "cost": 0,
  "balance": 0,
  "timing": { "queueMs": 0, "processingMs": 186 },
  "billing": {
    "lines": [
      { "unit": "encode_image", "quantity": 1, "unitPrice": 0.2, "amount": 0.2, "waived": false },
      { "unit": "refine", "quantity": 1, "unitPrice": 0.01, "amount": 0.01, "waived": false }
    ],
    "total": 0.21,
    "charged": false,
    "pool": "free",
    "allowance": { "used": 1, "limit": 1000 }
  }
}
```

### Reading the response

- `instances` holds one entry: the mask of the dog. A refine response has no `prompt` and no `label`; a point instance from `/v1/segment` has `prompt` and no `label`.
- `score` 0.9844 is the predicted IoU of the mask, the model's own estimate of how well the mask fits the object. With `options.multimask`, `instances` holds up to 3 candidates to choose from.
- `bbox` and `polygon` are in pixels of the oriented original, 1512 × 1008 here (`image.width`, `image.height`). `bbox` is `[x1, y1, x2, y2]` with x2 and y2 exclusive, and `area` is the pixel area of the mask. `polygon` is a list of parts, largest first; each part has an `outer` ring and, if it has holes, `holes`, all flat integer arrays `[x0, y0, x1, y1, ...]`. A pixel mask is returned only on request (see [Images](https://stack.foldbun.com/docs/images/#masks)).
- `cache` is `miss` and `imageFee` is 0.2: the image was encoded in this call. The next click on the same image normally returns `hit` with `imageFee` 0.
- `billing.charged` is `false` and `pool` is `free`: the call was the first of 1,000 included calls today, so `cost` is 0 while the lines show the list price.

All fields: [API reference](https://stack.foldbun.com/docs/api/operations/refine/).

**Cost of this call.** At list prices, 0.2 (`encode_image`) + 0.01 (`refine`) = 0.21 credits on a new image, and 0.01 credits for each later click while the encoding stays cached. The call above was covered by the free pool's daily allowance and charged nothing. Unit prices: [Billing](https://stack.foldbun.com/docs/billing/#price-list).

## Endpoints and reference

| Endpoint | In this use case | Reference |
|---|---|---|
| `POST /v1/segment` | Many objects in one call: one instance per point, and with `sam-lite` per box | [API reference](https://stack.foldbun.com/docs/api/operations/segment/) |
| `POST /v1/refine` | One object from positive and negative points and an optional box: the best mask, or up to 3 candidates | [API reference](https://stack.foldbun.com/docs/api/operations/refine/) |

- [`sam3`](https://stack.foldbun.com/docs/models/sam3/) and [`sam-lite`](https://stack.foldbun.com/docs/models/sam-lite/): how each model works, its limits and measured latency.
- [How SAM calls are billed](https://stack.foldbun.com/docs/billing/#how-sam-calls-are-billed): units and worked examples, including a click-by-click editing session.
- [Images](https://stack.foldbun.com/docs/images/#coordinates): the pixel space of points, boxes and masks.
- [Concept segmentation](https://stack.foldbun.com/docs/use-cases/concept-segmentation/): every instance of a concept from a text prompt or an exemplar box.