Shop from a picture the shopper shows

Send the picture your shopper shows with their message, and Agentcard searches for the product in it. A viewer who points their phone at a jacket on screen and says “where can I get that jacket” gets cards for that jacket, not for every jacket.

agentcard.ts

const turn = buy.turn('where can I get that jacket', {
  image: { url: 'https://frames.yourapp.com/show/s01e04/00-12-31.jpg' },
});

Agentcard reads the picture and the shopper’s words together and names the product the picture shows. The search then runs on the shopper’s words with that product in place of “that”: “where can I get the Patagonia Down Sweater Jacket”. The cards, the reply and the conversation’s history all read those words.

Send the picture as data

Send a picture your app holds in memory as base64 data, with its type:

agentcard.ts

const turn = buy.turn('how much is this', {
  image: { data: frame.toString('base64'), media_type: 'image/jpeg' },
});

Agentcard takes image/jpeg, image/png, image/webp and image/gif, up to 3 MB once decoded. A full-resolution phone photo can be larger, so scale the picture down before you send it.

Send a picture with no words

A shopper who only shows a picture still gets a search. buy.turn needs the shopper’s words, so give it a short ask with the picture:

agentcard.ts

const turn = buy.turn('find this', {
  image: { url: 'https://frames.yourapp.com/show/s01e04/00-12-31.jpg' },
});

An app that calls client.buy itself sends the picture with no ask at all, and Agentcard searches for the product the picture shows, as if the shopper had written “find” and its name:

agentcard.ts

const turn = await client.buy({
  image: { url: 'https://frames.yourapp.com/show/s01e04/00-12-31.jpg' },
  external_user_id: 'user-42',
  resume: true,
});

Handle an unclear picture

Agentcard searches for a product only when it is sure what the picture shows. A picture of a crowded shelf, or of several products when the words do not say which, leaves the search on the shopper’s words alone, so “where can I get that jacket” searches for jackets. A picture with no words that Agentcard cannot read is refused, and the shopper’s next message with words searches as usual.

Check what the turn sends

The picture goes with the turn’s first request for products, and with nothing else. A pick of a card, a yes to a total and every later message send no picture, so a shopper who taps a card never uploads the frame again. Agentcard reads the picture during the turn and never keeps it. An app that calls client.buy itself sends the picture as the image field of the request. A request that sends image with confirm is refused.

Fix a refused picture

buy.turn checks the picture’s address, its type and the size of its data before any call, so a picture those checks refuse throws a TypeError naming the field. buy.turn('that', { image: { url: 'http://frames.yourapp.com/a.jpg' } }) throws:

TypeError: Pass image.url as an https URL of at most 2048 characters.

Send the picture from an https address instead:

agentcard.ts

const turn = buy.turn('that', { image: { url: 'https://frames.yourapp.com/a.jpg' } });

Those checks are the SDK’s own, and POST /buy refuses a picture the SDK passes: the request takes 3,000,000 decoded bytes, where buy.turn takes 3,145,728, so a picture between the two throws nothing and the call answers with a 400. Handle the request’s refusal as well as the TypeError. A request your app sends itself gets a 400 that names the field:

{
  "error": "image.data is at most 3 MB decoded",
  "code": "invalid_request",
  "hint": "POST /buy { \"ask\": \"where can I get that jacket\", \"image\": { \"url\": \"https://...\" } }"
}

Error	What to do
`image.url must be an https URL`	Send the picture from an `https` address, or as data.
`image takes exactly one of image.url or image.data`	Send the address or the data, not both.
`image.data is at most 3 MB decoded`	Scale the picture down and send it again.
`image.media_type must be one of image/jpeg, image/png, image/webp, image/gif`	Convert the picture to one of those types.
`image goes with an ask, never with confirm`	Send the picture with the shopper’s message, and confirm without it.
`image.embedding: embeddings are not accepted yet; send image.url or image.data`	Send the picture itself instead of a vector your app computed.
`ask is required: Agentcard could not tell what the picture shows, so send the shopper's words with it`	Send the picture again with the shopper’s words.

Getting started

Guides

Vault

Issuing

Commerce SDK

Purchase API

Partners

Shop from a picture the shopper shows

Send the picture as data

Send a picture with no words

Handle an unclear picture

Check what the turn sends

Fix a refused picture

​Send the picture as data

​Send a picture with no words

​Handle an unclear picture

​Check what the turn sends

​Fix a refused picture

Send the picture as data

Send a picture with no words

Handle an unclear picture

Check what the turn sends

Fix a refused picture