How do I get around the 'get users' 1000 limitation?

Hi all,

Currently I have a limited number for users, just a couple of hundred, but expect in the coming months for this to rapid increase. As part of my current process, using /api/v2/users endpoint, I can page through, requesting 100 entries each time which all works fine. However with expected increase in mind, I noticed in the api documentation there is a limit of 1000 even with paging! This seems unnecessary, and very restrictive for what I’m sure is a pretty common request. So my question is, using either the SDK or just standard API request, how can I retrieve ALL my users, whether it’s 200, 500, 1000, 5000, etc…?

For context, I’m request all to check for a specific boolean key/value in user_metadata, which at any point can have true|false|undefined. The number of users for any of these possibilities could, as user numbers increase, be easily 1000+

For referennce, Auth0 Management API v2
400 You can only page through the first 1000 records.
See https://auth0.com/docs/users/search/v3/view-search-results-by-page#limitation.

Thank you in advance,

Noel

Hi @noelt.dolan,

I recommend using the User Import/Export extension or the export job to get the complete list of your users without limit.

This workaround is also mentioned in our Auth0 Management API v2 documentation.

Thanks,
Rueben

The above suggested solution, ‘export job’, worked, but had it’s own quirks, see below for how I handled this in typescript. It may be useful and help someone else avoid some pain with this. Assumption that you’ve already setup appropriate auth0 variables AUTH0… When I says quirks, I’m specifically referring to questionable way in which the json response is delivered. :thinking:

import { GetConnectionsStrategyEnum, ManagementClient  } from 'auth0';
import { setTimeout } from 'timers/promises';

const management = new ManagementClient({
  domain: AUTH0_USER_MANAGEMENT_DOMAIN,
  clientId: AUTH0_USER_MANAGEMENT_CLIENT_ID,
  clientSecret: AUTH0_USER_MANAGEMENT_CLIENT_SECRET,
});

const doGetAuth0Users = async() => {
  // To make a request to export our users, we first need to find the connection id for our 'auth0' account.
  const connections = await management.connections.getAll({ 
    strategy: [GetConnectionsStrategyEnum.auth0],
   });
  const { data } = connections;

  // there should just be exactly one entry so 'safe' to use 'find'...
  const auth0connection = data.find((d) => d.strategy === GetConnectionsStrategyEnum.auth0 && d.name === 'Username-Password-Authentication');

  // now, submit export users job request...
  const exportUsersJob = await management.jobs.exportUsers({
    connection_id: auth0connection?.id,
    format: 'json',
    fields: [
      { name: 'user_id', export_as: 'userId'  },
      { name: 'email', export_as: 'emailAddress' },
      { name: 'user_metadata.account_id', export_as: 'accountId' },
    ]
   });

  // now wait for export users job status completed...
  const users: any[] = [];
  let jobCompleted = false;
  while (!jobCompleted) {
    await setTimeout(5000); // wait 5 seconds for job to complete...
    const job = await management.jobs.get({ id: exportUsersJob.data.id });

    if (job.data.status === 'completed') {      
      const res = await fetch(job.data.location); // job.data.location is a .gz file on AWS S3
      
      const blob = await res.blob(); // this blob is a compressed gz stream, so we need to decompress it...

      const ds = new DecompressionStream('gzip');
      const decompressedStream = blob.stream().pipeThrough(ds);

      // bizarrely, we don't get a proper 'json' response for this, instead we get X text rows, each of which has to be parsed into an JSON object!
      const rows = (await new Response(decompressedStream).text()).split('\n');

      for (let i = 0; i < rows.length; i++) {
        if (!(rows[i]) || rows[i]?.length === 0 ) { // oh! it also returns an empty line at the bottom of the response!
          continue;
        }
        const user = JSON.parse(rows[i] || '{}' );

        users.push(user);
      }

      jobCompleted = true; // this should end the while loop as job request is completed!
    }
  }
  console.log(`users found: ${users.length}`);
  console.table(users);
}

doGetAuth0Users();
1 Like