> ## Documentation Index
> Fetch the complete documentation index at: https://openrouter.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Find and Mark Objects in Any Image

> Get bounding boxes and points for any object in a photo with Moondream's /detect and /point commands, then draw them on the image

Use this guide when you need to know where something is in an image, not only whether it is there: to count items, crop them out, check a shelf or a workbench, or tell an agent where to click.

## What you're building

![A workshop bench with purple boxes around three screwdrivers and purple dots on two hammers](https://mintcdn.com/openrouter-d02e98a0/ij25mP7SkzsfeYUD/assets/cookbook/image-understanding/find-objects-in-images/find-objects-in-images.webp?fit=max\&auto=format\&n=ij25mP7SkzsfeYUD\&q=85\&s=1fc473493092d034ccb97ff595439666)

Boxes around every screwdriver and a point on each hammer, from two requests to Moondream 3.1. Each request took under two seconds.

Moondream models answer two slash commands in an ordinary [Responses API](https://openrouter.ai/docs/api_reference/responses/overview.md) request. Start the text of a user message with `/detect {"object":"screwdriver"}` and the model returns a bounding box for every match. Start it with `/point {"object":"hammer"}` and it returns one point per match. Both come back as JSON in the `output_text` part of the `message` output item, with coordinates from 0 to 1 relative to the image, so you need no detection API, schema, or prompt engineering.

By the end, you will have a script that:

1. Asks for bounding boxes around one kind of object.
2. Asks for a point on each instance of another.
3. Draws the boxes and points on the photo and saves it as a PNG.

> [!TIP]
>
> Building with a coding agent? Paste the URL of this page into Claude Code, Codex, or Cursor and ask it to highlight the objects a user types in over the photo they upload, using this recipe.

## Before you start

You need:

* An OpenRouter API key available as `OPENROUTER_API_KEY`
* Bun, or Node.js 22.18 or newer (both run TypeScript files directly)
* `sharp` to load and draw on the image: `npm install sharp`
* A JPEG, PNG, WebP, or GIF photo. To follow along exactly, download the [workbench photo](https://openrouter.ai/docs/assets/cookbook/image-understanding/find-objects-in-images/workbench.jpg) used on this page and save it as `workbench.jpg`.

This guide uses `moondream/moondream3.1-9b-a2b`. `moondream/moondream3-preview` takes the same commands. See the [Moondream 3.1](https://openrouter.ai/moondream/moondream3.1-9b-a2b) model page for pricing, and [Image Understanding](https://openrouter.ai/docs/guides/overview/multimodal/image-understanding.md) for the ways to send images.

## Step 1: Find every match with /detect

Create `find-objects.ts`. The `loadImage` helper turns the photo upright and reads its format: Moondream reads the stored pixels and ignores the EXIF orientation tag that phone cameras set, so a phone photo would otherwise come back with coordinates for a sideways image. The `locate` helper sends one image and one command, and returns the array of matches. The command must be the first thing in the text part, and its argument must be a JSON object with an `object` key.

```ts expandable lines theme={null}
import sharp from "sharp";

const apiKey = process.env.OPENROUTER_API_KEY;
if (!apiKey) throw new Error("Set OPENROUTER_API_KEY first.");

const MODEL = "moondream/moondream3.1-9b-a2b";

type Box = { x_min: number; y_min: number; x_max: number; y_max: number };
type Point = { x: number; y: number };
type Image = { data: Buffer; mime: string; width: number; height: number };

const MIME_TYPES: Record<string, string> = {
  jpeg: "image/jpeg",
  png: "image/png",
  webp: "image/webp",
  gif: "image/gif",
};

async function loadImage(imagePath: string): Promise<Image> {
  const { data, info } = await sharp(imagePath)
    .autoOrient()
    .toBuffer({ resolveWithObject: true });
  const mime = MIME_TYPES[info.format];
  if (!mime) {
    throw new Error(`Unsupported image format: ${info.format}. Use JPEG, PNG, WebP, or GIF.`);
  }
  return { data, mime, width: info.width, height: info.height };
}

async function locate(
  image: Image,
  skill: "detect" | "point",
  object: string,
): Promise<unknown[]> {
  const response = await fetch("https://openrouter.ai/api/v1/responses", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: MODEL,
      input: [
        {
          role: "user",
          content: [
            { type: "input_text", text: `/${skill} ${JSON.stringify({ object })}` },
            {
              type: "input_image",
              image_url: `data:${image.mime};base64,${image.data.toString("base64")}`,
              detail: "auto",
            },
          ],
        },
      ],
    }),
  });

  if (!response.ok) throw new Error(await response.text());

  const result = await response.json();
  const content: string = result.output
    .find((item: { type: string }) => item.type === "message")
    .content.find((part: { type: string }) => part.type === "output_text").text;
  const key = skill === "detect" ? "objects" : "points";
  const matches = JSON.parse(content)[key];
  if (!Array.isArray(matches)) {
    throw new Error(`Expected "${key}" in the response, got: ${content}`);
  }
  return matches;
}

const image = await loadImage(process.argv[2] ?? "workbench.jpg");

const objects = (await locate(image, "detect", "screwdriver")) as Box[];
console.log(`detect screwdriver: ${objects.length} boxes`);
console.log(JSON.stringify(objects, null, 2));
```

Run it with `bun find-objects.ts` or `node find-objects.ts`:

```text lines theme={null}
detect screwdriver: 3 boxes
[
  {
    "x_min": 0.2119140625,
    "y_min": 0.41162109375,
    "x_max": 0.3193359375,
    "y_max": 0.60400390625
  },
  {
    "x_min": 0.27734375,
    "y_min": 0.36865234375,
    "x_max": 0.32421875,
    "y_max": 0.57666015625
  },
  {
    "x_min": 0.3125,
    "y_min": 0.3671875,
    "x_max": 0.359375,
    "y_max": 0.58984375
  }
]
```

Each box is the top-left (`x_min`, `y_min`) and bottom-right (`x_max`, `y_max`) corner as a fraction of the image width and height, measured from the top-left of the image. When nothing matches, the model returns `{"objects": []}`.

The `object` value can be a short description as well as a noun. `{"object":"red-handled screwdriver"}` returns one box, around the red one only.

## Step 2: Mark each match with /point

`/point` returns one point on each match instead of a box, which is all you need to count things or to pass a click target to an agent. Add these lines to the end of the script:

```ts lines theme={null}
const points = (await locate(image, "point", "hammer")) as Point[];
console.log(`point hammer: ${points.length} points`);
console.log(JSON.stringify(points, null, 2));
```

```text lines theme={null}
point hammer: 2 points
[
  {
    "x": 0.54296875,
    "y": 0.4375
  },
  {
    "x": 0.50390625,
    "y": 0.5234375
  }
]
```

## Step 3: Draw the results on the image

Multiply the fractions by the image size to get pixels, build an SVG with a rectangle per box and a circle per point, and let `sharp` lay it over the photo. Add this to the end of the script:

```ts expandable lines theme={null}
const { width, height } = image;

const rects = objects.map((b) => {
  const x = b.x_min * width;
  const y = b.y_min * height;
  const w = (b.x_max - b.x_min) * width;
  const h = (b.y_max - b.y_min) * height;
  return `<rect x="${x}" y="${y}" width="${w}" height="${h}" fill="none" stroke="#6467f2" stroke-width="5"/>`;
});

const dots = points.map(
  (p) =>
    `<circle cx="${p.x * width}" cy="${p.y * height}" r="14" fill="#6467f2" stroke="white" stroke-width="4"/>`,
);

const overlay = Buffer.from(
  `<svg width="${width}" height="${height}" xmlns="http://www.w3.org/2000/svg">${rects.join("")}${dots.join("")}</svg>`,
);

await sharp(image.data)
  .composite([{ input: overlay }])
  .png()
  .toFile("annotated.png");

console.log(`Saved annotated.png (${width}x${height})`);
```

```text lines theme={null}
Saved annotated.png (1360x768)
```

Open `annotated.png` to see the image at the top of this page. To crop a match out instead of drawing it, pass the same pixel values to `sharp(image.data).extract({ left: x, top: y, width: w, height: h })`, rounded to whole numbers.

## Troubleshooting

Moondream returns the same error for every malformed command: `Provider returned error`, with `"message":"Invalid request."` inside `error.metadata.raw`.

| Error | Cause | Fix |
| - | - | - |
| `Invalid request.` | The argument is plain text, such as `/detect screwdriver` | Send a JSON object: `/detect {"object":"screwdriver"}` |
| `Invalid request.` | The key is not `object`, or the value is an array | Use one `object` string per request, and send one request per object type |
| `Invalid request.` | The message has no image, or has more than one | Attach exactly one image to the message |
| `Expected "objects" in the response, got: {"object": "screwdriver"}` | The text does not start with the command, for example `Please /detect …` or a leading space | Put the command first in the text part |
| `Unexpected non-whitespace character after JSON` from `JSON.parse` | The request asked in plain language ("give me bounding boxes"), so the model answered in free text | Use `/detect` or `/point` to get JSON back |
| `{"objects": []}` | Nothing in the image matched the description | Try a more common name for the object, or a shorter description |
| `Unsupported image format: heif` | The photo is AVIF, or another format outside JPEG, PNG, WebP, and GIF | Convert it to JPEG first |
| `moondream/moondream3 is not a valid model ID` | Wrong model slug | Use `moondream/moondream3.1-9b-a2b` or `moondream/moondream3-preview` |

## Check your work

The script should print three boxes for the screwdrivers and two points for the hammers, then save `annotated.png` with a box around each screwdriver and a dot on each hammer. Point it at your own photo with `bun find-objects.ts my-photo.jpg` and change the two `object` strings to what you want to find. Small, tightly packed items can be missed, so check the count against the image before you rely on it.