How to export the data of Masterdata V2 , if we do have more than 1lac documents present in MasterData V2.
Is there any wayout by which we can do the same ?
Like Masterdata V1, we can export the documents in Excel. Can`t we do the same in V2 . ?
via Scroll API of V2, we can fetch the documents but there is limitation of 1k record at a time.
@Rakshit suggest to use the X-VTEX-MD-TOKEN, with that you can extract all the data, this token saves all the configuration of the API that you have made in the first request. You can hit that token unlimited times with the following URL: https://[…]/api/dataentities/CL/scroll?_token=TOKEN. Hope it can help you ! In case you need help, send me a message ! I highly recommend the API.
hey @EduHuiza , I tried that already, with that there is an limitation by which you can hit the API by max 10 times consecutively and after that it gives the error of “maximum hit reaches”. then we have to wait for around 10minutes for next successful API call, after that again it give the previous data as a result. That`s the problem.
And in every hit it gives of 1000 data at a time, which means 10 times x 1000 data = 10k data . But I do have almost 115k data in V2, that is the problem.
Please suggest in that case !
I think that the scroll method that was suggested here should work. I had a similiar problem that i solved with that.
The limitation to that is that you can only have 10 scrolls tokens concurrent. But you can use one token to get all your documents.
Here is a code example:
let MD_TOKEN = ''
let hasMoreData = true
const responseData: unknown[][] = []
while (hasMoreData) {
// eslint-disable-next-line no-await-in-loop
const { data, mdToken } = await MDClient.scroll({
fields,
size: 1000,
sort,
where,
mdToken: MD_TOKEN !== '' ? MD_TOKEN : undefined,
})
responseData.push(data)
// The first call is made without the token, then the first response gives us that token
// We use that token to make the next calls
MD_TOKEN = MD_TOKEN !== '' ? MD_TOKEN : mdToken
if (data.length === 0) {
hasMoreData = false
}
}
const documents = responseData.flat()