How do I get around the 'get users' 1000 limitation?

[noelt.dolan]

Hi all,

Currently I have a limited number for users, just a couple of hundred, but expect in the coming months for this to rapid increase. As part of my current process, using /api/v2/users endpoint, I can page through, requesting 100 entries each time which all works fine. However with expected increase in mind, I noticed in the api documentation there is a limit of 1000 even with paging! This seems unnecessary, and very restrictive for what I’m sure is a pretty common request. So my question is, using either the SDK or just standard API request, how can I retrieve ALL my users, whether it’s 200, 500, 1000, 5000, etc…?

For context, I’m request all to check for a specific boolean key/value in user_metadata, which at any point can have true|false|undefined. The number of users for any of these possibilities could, as user numbers increase, be easily 1000+

For referennce, Auth0 Management API v2
400 You can only page through the first 1000 records.
See https://auth0.com/docs/users/search/v3/view-search-results-by-page#limitation.

Thank you in advance,

Noel

[rueben.tiow]

Hi @noelt.dolan,

I recommend using the User Import/Export extension or the export job to get the complete list of your users without limit.

This workaround is also mentioned in our Auth0 Management API v2 documentation.

Thanks,
Rueben

[noelt.dolan]

The above suggested solution, ‘export job’, worked, but had it’s own quirks, see below for how I handled this in typescript. It may be useful and help someone else avoid some pain with this. Assumption that you’ve already setup appropriate auth0 variables AUTH0… When I says quirks, I’m specifically referring to questionable way in which the json response is delivered.

import { GetConnectionsStrategyEnum, ManagementClient  } from 'auth0';
import { setTimeout } from 'timers/promises';

const management = new ManagementClient({
  domain: AUTH0_USER_MANAGEMENT_DOMAIN,
  clientId: AUTH0_USER_MANAGEMENT_CLIENT_ID,
  clientSecret: AUTH0_USER_MANAGEMENT_CLIENT_SECRET,
});

const doGetAuth0Users = async() => {
  // To make a request to export our users, we first need to find the connection id for our 'auth0' account.
  const connections = await management.connections.getAll({ 
    strategy: [GetConnectionsStrategyEnum.auth0],
   });
  const { data } = connections;

  // there should just be exactly one entry so 'safe' to use 'find'...
  const auth0connection = data.find((d) => d.strategy === GetConnectionsStrategyEnum.auth0 && d.name === 'Username-Password-Authentication');

  // now, submit export users job request...
  const exportUsersJob = await management.jobs.exportUsers({
    connection_id: auth0connection?.id,
    format: 'json',
    fields: [
      { name: 'user_id', export_as: 'userId'  },
      { name: 'email', export_as: 'emailAddress' },
      { name: 'user_metadata.account_id', export_as: 'accountId' },
    ]
   });

  // now wait for export users job status completed...
  const users: any[] = [];
  let jobCompleted = false;
  while (!jobCompleted) {
    await setTimeout(5000); // wait 5 seconds for job to complete...
    const job = await management.jobs.get({ id: exportUsersJob.data.id });

    if (job.data.status === 'completed') {      
      const res = await fetch(job.data.location); // job.data.location is a .gz file on AWS S3
      
      const blob = await res.blob(); // this blob is a compressed gz stream, so we need to decompress it...

      const ds = new DecompressionStream('gzip');
      const decompressedStream = blob.stream().pipeThrough(ds);

      // bizarrely, we don't get a proper 'json' response for this, instead we get X text rows, each of which has to be parsed into an JSON object!
      const rows = (await new Response(decompressedStream).text()).split('\n');

      for (let i = 0; i < rows.length; i++) {
        if (!(rows[i]) || rows[i]?.length === 0 ) { // oh! it also returns an empty line at the bottom of the response!
          continue;
        }
        const user = JSON.parse(rows[i] || '{}' );

        users.push(user);
      }

      jobCompleted = true; // this should end the while loop as job request is completed!
    }
  }
  console.log(`users found: ${users.length}`);
  console.table(users);
}

doGetAuth0Users();