How to decompress users_exports output files inside nodejs script (IOW, non-interactive)

[shane.bryan]

My use case is a script being written to migrate some fields from user_metadata to app_metadata for all users. Given some tenants I need to run this against have over 1000 users, I have to use the /api/v2/jobs APIs to export all users, fetch the resulting file upon completion and then import the updated user objects (with upsert: true).

I have tried both axios and fetch libraries to get the exported users file (.../my-tenant-name.json.gz...) from within my nodejs script, each trying to decompress the response.data contents as gzip, compress and br. I’ve tried setting responseType to ‘arrayBuffer’ as well as using response.data.toString(‘utf-8’)…

I seem to always get errors during the decompression steps… for example, using the zlib node module:

Error: incorrect header check

I am sure the file exists, I’ve got code polling the jobs endpoint waiting for either state === 'completed' or an error state/condition. If I console.log() the location and paste it into the browser it works, as other similar questions have identified. My problem is I need this to be all non-interactive and processed in code.

Here are two of the several ways I have made calls to get the export output file:

const response = await axios.get(exportResult.location, { responseType: 'arrayBuffer' });

and…

const response = await fetch(exportResult.location).then(res => zlib.unzipSync(res.body.toString()));

Any suggestions greatly appreciated!

[thameera]

Hey @shane.bryan, the export files are compressed with gzip. Can you try zlib.gunzipSync() instead of zlib.unzipSync()?

[shane.bryan]

Thanks for asking, but yes, that was one I also tried. I only showed two examples above if the many variations I’ve tried.

For now I’ve given up on decompressing it in memory and am now just saving to local disk and then reading it in via createReadStream, piping through zlib.createGunzip() and ndjson.parse().

It’s not ideal but at least I can move on for now and maybe come back to this if anyone comes up with suggestions I haven’t tried yet.

[thameera]

Creating a binary buffer from the axios result and then passing that to zlib.gunzip seems to work. I tested with the below code:

const run = async (exportLocation) => {
  const res = await axios.get(exportLocation, { responseType: 'arraybuffer' })
  const buf = Buffer.from(res.data, 'binary')
  const gunzipped = zlib.gunzipSync(buf)

  console.log(gunzipped.toString())
}

Hopefully that should work in your case too.