How to export the data of Masterdata V2 , if we do have more than 1lac documents present in MD

How to export the data of Masterdata V2 , if we do have more than 1lac documents present in MasterData V2.
Is there any wayout by which we can do the same ?

Like Masterdata V1, we can export the documents in Excel. Can`t we do the same in V2 . ?

via Scroll API of V2, we can fetch the documents but there is limitation of 1k record at a time.

Please suggest me the solution
@tyler @georgebrindeiro

Hi there, just curious where did you find export (in excel format) API for Masterdata V1?

Besides this tutorial, didn’t find any resolution.

hey @sirev ,

this is the way where you can take the export of all documents in Excel Format in MasterData V1.
Screenshot for your reference.

Hey Mohitkumar !

@Rakshit suggest to use the X-VTEX-MD-TOKEN, with that you can extract all the data, this token saves all the configuration of the API that you have made in the first request. You can hit that token unlimited times with the following URL: https://[…]/api/dataentities/CL/scroll?_token=TOKEN. Hope it can help you ! In case you need help, send me a message ! I highly recommend the API.

hey @EduHuiza , I tried that already, with that there is an limitation by which you can hit the API by max 10 times consecutively and after that it gives the error of “maximum hit reaches”. then we have to wait for around 10minutes for next successful API call, after that again it give the previous data as a result. That`s the problem.
And in every hit it gives of 1000 data at a time, which means 10 times x 1000 data = 10k data . But I do have almost 115k data in V2, that is the problem.
Please suggest in that case !

Hello @mohitkumar !

I think that the scroll method that was suggested here should work. I had a similiar problem that i solved with that.

The limitation to that is that you can only have 10 scrolls tokens concurrent. But you can use one token to get all your documents.

Here is a code example:

    let MD_TOKEN = ''

    let hasMoreData = true
    const responseData: unknown[][] = []

    while (hasMoreData) {
      // eslint-disable-next-line no-await-in-loop
      const { data, mdToken } = await MDClient.scroll({
        fields,
        size: 1000,
        sort,
        where,
        mdToken: MD_TOKEN !== '' ? MD_TOKEN : undefined,
      })

      responseData.push(data)

      // The first call is made without the token, then the first response gives us that token
      // We use that token to make the next calls
      MD_TOKEN = MD_TOKEN !== '' ? MD_TOKEN : mdToken

      if (data.length === 0) {
        hasMoreData = false
      }
    }

    const documents = responseData.flat()