I’ve been dealing with this problem for several years and wanted to know if anyone has managed to find a solution.
When I integrate the catalog via APIs (and also when I do it directly through the VTEX portal), it takes me between 3 to 4 hours to upload a catalog of 10,000 SKUs with low traffic, and sometimes it takes more than a day.
There are no errors, I can see that the products are being integrated, although sometimes some are pending approval — and I notice that when I update prices, for example, it doesn’t allow customers to buy the product at the old price, but this isn’t reflected either on the PDP or in search.
According to VTEX’s own documentation on rate limiting, there are request limits per IP, per account, per API route, and per other components of a request. These limits vary throughout the day and are not disclosed by VTEX.
With that in mind, the best approach would be to rely on FIFO queue-based integrations, with multiple workers processing your queue and monitoring the response headers to handle the rate limit for a given request.
Doing a rough calculation, 10,000 SKUs in 3 hours comes out to 55 SKUs per minute, which is quite low.
Are you integrating directly? Are you integrating using queues? What limits is your integration following?
thank you so much for the welcome and the reply! @andremiani
so we send them directly integrated through the API, the items arrive exactly as you said in a queue (I’m not sure if it’s a static FIFO or if there are multiple workers inside vtex, since I don’t see them going in order)
however the problem is that it takes a very long time for the products to be published on the sites once they arrive in the vtex catalog.
I believe the delay issue is due to indexing or caching. After sending your products via API, they enter the indexing queue.
This documentation here has a nice explanation of how indexing works here at VTEX. But to summarize: a product, after being sent for indexing, sits in the queue for 2 minutes, and in general, indexing takes an average of 4 minutes per product. This 4-minute window is an average estimate of the time the indexer takes to update all product information across all the trade policies it is offered under.
There are several factors that can make indexing take a bit longer, and there is also a time limit for how long a product can remain in the indexing queue — if that limit is exceeded, the product is sent to a retry queue and a few more attempts will be made every 15 minutes.
And in this documentation here, you’ll find an explanation of how our cache works. That documentation also includes a handy table with average cache times, and for catalog changes this time can be up to 2 hours:
Regarding the total time to send around 10,000 SKUs to the Catalog in VTEX, this could be related to a direct integration (without the use of queues). If the integration with VTEX is already happening using some queue system, you would just need to increase the number of workers to increase the number of items processed per minute — because if the calculation of 10,000 SKUs in 3 hours, equivalent to 55 SKUs per minute, is correct, there is a bottleneck somewhere.
Regarding the indexing time of a product in VTEX, after the integration has been completed and all the necessary data has been sent to VTEX, that’s what @KarinaMota explained and there isn’t much that can be done about it.
