Help using the Bulk User Export API

Help
1

Hi all,

My company is using Auth0 to store all our users. Until recently, we had about 900 users and were using the https://<company>.auth0.com/api/v2/, but now I have over 1000 users and I can getting the error:

auth0.v3.exceptions.Auth0Error: 400: You can only page through the first 1000 records. See https://auth0.com/docs/users/search/v3/view-search-results-by-page#limitation

I have been reading about Bulk User Expors, and tried switching my config file to use the new API:

{
    "auth0": {
        "domain": "<company>.auth0.com",
        "clientId": "xxxxxxx",
        "clientSecret": "xxxxx-xxxxxx",
        "scope": "read:users update:users create:users delete:users",
        "audience": "https://<company>.auth0.com/api/v2/jobs/users-exports"
    }
}

but now I am getting a new new error:
auth0.v3.exceptions.Auth0Error: 403: Service not enabled within domain: https://bggoplan.auth0.com/api/v2/jobs/users-exports/

Can someone help me. I simply want to export all the users out (about 1100) the same way as before.

Derek

2

Hi Derek,

https://bggoplan.auth0.com/api/v2/jobs/users-exports/ is not a valid audience. The audience should be https://YOUR_TENANT_DOMAIN/api/v2/ because you need a Management API Access Token to create a user export job.

Hope this helps!

2 Likes
3

Thanks for helping on this one Supun!

4

Hi Supan,

Sorry, I am still unclear on your solution. If I use https://YOUR_TENANT_DOMAIN/api/v2/, I have my original https://bggoplan.auth0.com/api/v2/, and hence hit the 1000 limitation. From what I am reading, I need to call the https://bggoplan.auth0.com/api/v2/jobs/users-exports/ API endpoint

5

Hi @derekm,

Yes, you need to call the https://bggoplan.auth0.com/api/v2/jobs/users-exports/ endpoint to create a user export job. To call this endpoint, you need an Access Token for the Management API.

There are two ways you can get an Access Token for the Management API.

  • Get a Token for testing using the Auth0 Dashboard (Applications → APIs → Auth0 Management API → API Explorer)
  • Get a Token from the /oauth/token endpoint using the client credentials grant.
curl --request POST \
  --url 'https://YOUR_DOMAIN/oauth/token' \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data grant_type=client_credentials \
  --data 'client_id=YOUR_CLIENT_ID' \
  --data client_secret=YOUR_CLIENT_SECRET \
  --data 'audience=https://YOUR_DOMAIN/api/v2/'

In there, the audience value should be https://YOUR_DOMAIN/api/v2/ to retrieve an Access Token.
Learn more: Get Management API Access Tokens for Production

3 Likes
7

Hi Supun,

So I understand a bit better now how this works after following the steps at:
https://auth0.com/docs/manage-users/user-migration/bulk-user-exports

There are three steps:

  1. Use a POST call to the https://MY_DOMAIN/oauth/token endpoint to get an auth token (done)
  2. Then take this token and insert it into the next POST call to the endpoint: https://MY_DOMAIN/api/v2/jobs/users-exports
  3. Then take the job_id and insert it into the 3rd GET call to the endpoint: https://bggoplan.auth0.com/api/v2/jobs/MY_JOB_ID

But this just gives me a link to a document that I download. Essentially is the same end result as using the User Import / Export extension.

This is NOT what I want. I want to be able to call an endpoint and have it return a list of all the users (similar to the GET /api/v2/users endpoint. I require it be done this way, so I can write a python script and run it as a cron job. However, since I have over 1000 users, I am getting an error when I call the GET /api/v2/users endpoint.

Can anyone help? Can this be done at all the way I wish it to be?

8

Hi all,

I was able to figure it out. I set up 3 postman REST calls:

  1. POST call to get the Auth Token. https://<company>/oauth/token

  2. The second POST call was to the users-exports endpoint (https://<company>/api/v2/jobs/users-exports) and had the following body:

{"connection_id": "con_xxxxxx", 
"format": "json",
"fields": [
    {"name": "user_id"}, 
    {"name": "name"}, 
    {"name": "email"}, 
    {"name": "app_metadata.apps"}, 
    {"name": "app_metadata.welcomeSent"}
]}
  1. The third was a GET call to retrieve the actual job itself (https://<company>/api/v2/jobs/<job id>)
9

Programmatically (using Python) the code looks like this (code assumes I already
have the Auth Token).

import requests
from auth0.v3.management import Auth0
...

    # Go to 'Authentication -> Database -> Database Connections -> Database Identifier' 
    # for CONNECTION_ID
    messBody = {
        "connection_id": CONNECTION_ID,
        "format": "json",
        "fields": [
            {"name": "user_id"},
            {"name": "name"},
            {"name": "email"},
            {"name": "app_metadata.apps"},
            {"name": "app_metadata.welcomeSent"}
        ]
    }

    try:
        # Retrieve contents of the auth0 object (via a job)
        resp = (auth0.jobs.export_users(body=messBody))
        time.sleep(60) # Sleep for 1 minute to allow the job to finish processing
        locationFile = auth0.jobs.get(resp['id'])['location']

    except:
        print('ERROR: Job ' + resp['id'] + ' failed due to timeout accessing location file')
        exit(1)

    r = requests.get(locationFile) # Create HTTP response object
    time.sleep(5)

    # Send a HTTP request to the server and save
    # the HTTP response in a response object called r
    with open(mydomain.json.gz,'wb') as f:

    # Saving received content as a png file in binary format, and 
    # write the contents of the response (r.content) to a new file 
    # in binary mode.
        f.write(r.content)
        time.sleep(5)
        f.close()

I hope this helps someone.

11

Thanks for sharing that with the rest of community!

1 Like