App backend IO that iterates through the catalog, but some APIs return 429 errors

VTEX in your native language 🌎 Portuguese
, , , ,
1

The app fetches all SKUs from the catalog — approximately 49k+ products — and for each product we need to retrieve its information, current price, and send the data to another API.

I’m using events to process the SKUs in chunks of 400 items, which would generate roughly 122 events.

import { MAX_ITEMS_PER_PAGE } from '../../../constants';

export async function example(ctx: Context) {
  const {
    clients: {
      catalogClient,
    },
  } = ctx

  const {
    range: { total }
  } = await catalogClient.getProductsAndSkus({
    _from: 1,
    _to: 1
  })

  const skuIds: number[] = []
  const totalIterations = Math.ceil(total / MAX_ITEMS_PER_PAGE)

  for (let i = 0; i < totalIterations; i++) {
    const skus = await catalogClient.getAllSKUIDs({
      page: i + 1,
      pageSize: MAX_ITEMS_PER_PAGE
    })

    skuIds.push(...skus)
  }

  const chunkSize = 400; // Set the chunk size to 400
  const numberOfChunks = Math.ceil(skuIds.length / chunkSize);

  for (let i = 0; i < numberOfChunks; i++) {
    const chunk = skuIds.slice(i * chunkSize, (i + 1) * chunkSize);

    await ctx.clients.events.sendEvent('', 'process-skus', {
      skuIds: chunk
    })
  }

  ctx.status = 200
  ctx.body = 'In progress'
}

However, I’m running into issues with some VTEX APIs due to throttling (400 requests per minute).

Is there any example app available?
Has anyone had to implement a backend app on IO that needs to handle large amounts of data?

2

In that case, wouldn’t it be possible for you to create a timeout for querying data from this API? The way you did it, a 429 error is likely to occur, which means too many requests were made.