User export jobs stuck in `pending` status

[rfrankel_matatika]

User export jobs seem to be stuck in pending status over the last 12 hours (as of 2023-10-26T00:00:00Z).

POST https://{{domain}}/api/v2/jobs/users-exports
GET https://{{domain}}/api/v2/jobs/{{job_id}}

{
    "type": "users_export",
    "status": "pending",
    "format": "json",
    "fields": [
        {
            "name": "user_id"
        },
        {
            "name": "email"
        },
        {
            "name": "email_verified"
        },
        {
            "name": "username"
        },
        {
            "name": "phone_number"
        },
        {
            "name": "phone_verified"
        },
        {
            "name": "created_at"
        },
        {
            "name": "updated_at"
        },
        {
            "name": "identities"
        },
        {
            "name": "app_metadata"
        },
        {
            "name": "user_metadata"
        },
        {
            "name": "picture"
        },
        {
            "name": "name"
        },
        {
            "name": "nickname"
        },
        {
            "name": "multifactor"
        },
        {
            "name": "last_ip"
        },
        {
            "name": "last_login"
        },
        {
            "name": "logins_count"
        },
        {
            "name": "blocked"
        },
        {
            "name": "given_name"
        },
        {
            "name": "family_name"
        },
        {
            "name": "last_password_reset"
        },
        {
            "name": "locale"
        }
    ],
    "created_at": "2023-10-26T12:40:21.753Z",
    "id": "job_hxoCJr9lcsuuRcbc"
}

[marcelina.barycka]

Hey there @rfrankel_matatika !

Potentially, this can be related to a need to upgrade the auth0 node runtime version. Another community member reported issue with user import/export, please take a look at the provided solution proposition - Error when using User Import/Export extension

Thanks!

[rfrankel_matatika]

Hi @marcelina.barycka, I updated the runtime to Node 18 but no immediate change. I’ll try a couple more times in case it takes a little while to apply, but I would not have thought I need to change this as I am using the management API endpoint - not the import/export extension.

[rfrankel_matatika]

This looks to have resolved itself now, although I can’t say for sure if it was changing the Node runtime version that fixed it - I went to change back to 12 and the option is no longer there.

[marcelina.barycka]

I’m happy it works now for you. I do not have node 12 on my tenant neither, looks like it’s being slowly deprecated.

[pavol.soltes]

I am facing the same issue. Changing node for extension didn’t work (also using the management api endpoint not extension)

[ja6a-regnosys]

We have been facing user export job errors for the past week. We don’t use the extension - just the regular management API jobs user bulk export.

[marcelina.barycka]

Hi and thanks for letting us know! @pavol.soltes @ja6a-regnosys

Does the issue auto-resolved for you in time as well? Might be a temporary performance issue but still worth notifying engineering on our side.

Would exporting for each connection separately (by providing connection id to the request payload) improve the efficiency for you?

I believe the growing number of users in process of time may impact the performance of exporting them and breaking it down based on a connection may bring some release.

Does calling the GET {your_domain}/api/v2/jobs/{id}/errors gives you any additional insight?

Thanks!

[ja6a-regnosys]

In order to use the management API it is necessary to sent the connection id for the specific user database. We already send this id. Also we don’t keep hold of the job ids unfortunately. We get a 429 error - which isn’t a rate limit error IMO:

{
“statusCode”: 429,
“error”: “Too Many Requests”,
“message”: “There are 10 active export users jobs, please wait until some of them are finished and try again”
}

[marcelina.barycka]

Alright, thanks for your feedback @ja6a-regnosys !

Just on a side note, when exporting users with the Management API, you can omit connection_id property so that all tenant members will be returned. On the other hand, to import users, the connection_id is indeed necessary.

The error message says that there were already 10 hits to the POST https://.auth0.com/api/v2/jobs/users-exports API. Each request to this endpoint is considered a job. At least one of these jobs needs to be completed to run a new one.

[rfrankel_matatika]

FWIW, we have still been running into this issue a couple of times over the last week, but usually a retry after an hour or so solves it (at the time of the original post, it did not work at all over about 24 hours).

[marcelina.barycka]

Thanks @rfrankel_matatika ! -

After 24 hours all job-related data is automatically deleted which clears the line.

The conclusions are -
-The process may take some time, especially when the number of users are bigger.
-Its better the recheck the job status after some time instead of resending multiple times the same POST job request (because this creates each time a new job, the line grows).

[ja6a-regnosys]

@rfrankel_matatika

at the time of the original post, it did not work at all over about 24 hour

Still think it might be a problem on the Auth0 side. More than one customer suddenly started experiencing problems in the same area at the same time out of the blue.

[marcelina.barycka]

@ja6a-regnosys The incident is reported internally and I will post here any additional information once received.

[marcelina.barycka]

Hi @rfrankel_matatika @ja6a-regnosys @pavol.soltes -
Just got internal feedback -
For the period of time, the exports users worker was overloaded by a large number of exports causing delays but at the same time, the engineering team confirms that there are no anomaly observed for the service.

So apologies for the inconvenience caused by delays and thanks for your patience! For the future, If your subscription allows you to submit a support ticket, there is an option to speed the process up.

[rfrankel_matatika]

@marcelina.barycka Thanks for the follow-up!

[ja6a-regnosys]

Happening again this morning.

[ja6a-regnosys]

I am following up with support and have a ticket. I fear it has gone cold though. Just to add I can confirm that my jobs were also being stuck in ‘pending’.