Currently I have a limited number for users, just a couple of hundred, but expect in the coming months for this to rapid increase. As part of my current process, using /api/v2/users endpoint, I can page through, requesting 100 entries each time which all works fine. However with expected increase in mind, I noticed in the api documentation there is a limit of 1000 even with paging! This seems unnecessary, and very restrictive for what I’m sure is a pretty common request. So my question is, using either the SDK or just standard API request, how can I retrieve ALL my users, whether it’s 200, 500, 1000, 5000, etc…?
For context, I’m request all to check for a specific boolean key/value in user_metadata, which at any point can have true|false|undefined. The number of users for any of these possibilities could, as user numbers increase, be easily 1000+
The above suggested solution, ‘export job’, worked, but had it’s own quirks, see below for how I handled this in typescript. It may be useful and help someone else avoid some pain with this. Assumption that you’ve already setup appropriate auth0 variables AUTH0… When I says quirks, I’m specifically referring to questionable way in which the json response is delivered.
import { GetConnectionsStrategyEnum, ManagementClient } from 'auth0';
import { setTimeout } from 'timers/promises';
const management = new ManagementClient({
domain: AUTH0_USER_MANAGEMENT_DOMAIN,
clientId: AUTH0_USER_MANAGEMENT_CLIENT_ID,
clientSecret: AUTH0_USER_MANAGEMENT_CLIENT_SECRET,
});
const doGetAuth0Users = async() => {
// To make a request to export our users, we first need to find the connection id for our 'auth0' account.
const connections = await management.connections.getAll({
strategy: [GetConnectionsStrategyEnum.auth0],
});
const { data } = connections;
// there should just be exactly one entry so 'safe' to use 'find'...
const auth0connection = data.find((d) => d.strategy === GetConnectionsStrategyEnum.auth0 && d.name === 'Username-Password-Authentication');
// now, submit export users job request...
const exportUsersJob = await management.jobs.exportUsers({
connection_id: auth0connection?.id,
format: 'json',
fields: [
{ name: 'user_id', export_as: 'userId' },
{ name: 'email', export_as: 'emailAddress' },
{ name: 'user_metadata.account_id', export_as: 'accountId' },
]
});
// now wait for export users job status completed...
const users: any[] = [];
let jobCompleted = false;
while (!jobCompleted) {
await setTimeout(5000); // wait 5 seconds for job to complete...
const job = await management.jobs.get({ id: exportUsersJob.data.id });
if (job.data.status === 'completed') {
const res = await fetch(job.data.location); // job.data.location is a .gz file on AWS S3
const blob = await res.blob(); // this blob is a compressed gz stream, so we need to decompress it...
const ds = new DecompressionStream('gzip');
const decompressedStream = blob.stream().pipeThrough(ds);
// bizarrely, we don't get a proper 'json' response for this, instead we get X text rows, each of which has to be parsed into an JSON object!
const rows = (await new Response(decompressedStream).text()).split('\n');
for (let i = 0; i < rows.length; i++) {
if (!(rows[i]) || rows[i]?.length === 0 ) { // oh! it also returns an empty line at the bottom of the response!
continue;
}
const user = JSON.parse(rows[i] || '{}' );
users.push(user);
}
jobCompleted = true; // this should end the while loop as job request is completed!
}
}
console.log(`users found: ${users.length}`);
console.table(users);
}
doGetAuth0Users();
