Retrieving more than 1000 users through import-export extension jobs

[nicolas.tudor]

Hello there,

I’m using pagination to retrieve users for my backend at the endpoint /api/v2/users, which is working well. However, I have reached the limit of 1000 users, which is the maximum number of users this method/endpoint can retrieve.

To overcome this limitation, I’ve attempted to use the import-export extension jobs offered by Auth0, but I only get the following response: {“statusCode”:404,“error”:“Not Found”,“message”:“Not Found”}

I’m sharing a code snippet of the curl request that I’m using:

$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => “https://yadayada.eu.auth0.com/api/v2/jobs/user-exports”,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => “”,
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => “POST”,
CURLOPT_POSTFIELDS => “{“format”: “json”, “export_as”: “provider” }”,
CURLOPT_HTTPHEADER => array(
"authorization: Bearer ".$token,
“cache-control: application/json”
),
));

$response = curl_exec($curl);

If anyone has any advice or can offer any help, it would be much appreciated. Thank you!

[gparascandolo]

Hi @nicolas.tudor , it looks like you made a small typo on the Management API endpoint for the request

Your curl request is pointing to /api/v2/jobs/user-exports, instead of the correct /api/v2/jobs/users-exports endpoint

Hope this helps!

[nicolas.tudor]

Hi @gparascandolo that was indeed a silly typo thank you for pointing it out.
Now that the job is exported, I wrote a function that takes the jobId of the previous request. The current response has a location field, which points to an AWS url to download the users from the exported job, I thought I was done when I realised that only 4 users were retrieved out of the 1050, the response looks like json wrapped in a string, and since it’s incomplete I cannot use json_decode() on it.

This is my userImport function:

function userImport($token, $jobId){

	$curl = curl_init();

$status = "pending";

$loops = 0;

while($status != "completed"){

curl_setopt_array($curl, array(
CURLOPT_URL => "https://yadayada/api/v2/jobs/".$jobId."",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
    "authorization: Bearer ".$token,
    "cache-control: no-cache"
),

));

$response = curl_exec($curl);
$err = curl_error($curl);

$data = json_decode($response);

$status = $data->status;

if($status == “completed”){
$location = $data->location;
$compressedUsers = file_get_contents($location);
$decompressedUsers = gzdecode($compressedUsers); //returns 4 users in a string
$jsonUsers = json_decode($decompressedUsers); //returns null because decompressedUsers is most likely incomplete.

return $jsonUsers;

}

if ($err) {
echo “cURL Error #:” . $err;
break;
} else {
echo $response;
}

if($loops > 2){
	break;
}

sleep(20);

$loops++;

}

curl_close($curl);

}

Apologies for asking again, but would you be able to tell me if I’m missing something here?

[gparascandolo]

Based on what I can see in the initial code snippet, I believe the POST fields you are putting in the request might be causing an issue

Try simply passing in {"format": "json"} in the POST fields and try again.

In case you’ve made modifications to the request from your initial post, make sure you are not passing in connection_id or limit params either.

If all else fails, you can try installing and using the User Import / Export Extension in your tenant to export all your users.

I hope this helps!