Service down for a week!!

The Search API is down for a week (Which mean my service that uses your service is down for a whole week!!!) and i’ve been setting patience for your promise to have it working by Monday.

We are pass monday and so far -

  1. No one answer my emails
  2. No one answer my tweets on your support page
  3. We are pass Monday - after 7!!! days the service is down, and it is still down
    I am loosing customers any day, and no one from your side who is responsible, even let us know what is going on.

Maybe here you will find the time to update us,
Thank you,
Tomer

I understand your frustration around this situation and I apologize for it. As you might imagine we also wanted to have this resolved as soon as possible (it it’s not good for you it’s also not good for us), however, we needed to do so in a way that really addressed the underlying issue. As per the latest update on the status page incident the migration of free accounts is also now completed.

In summary (and I will be mostly repeating what’s available through the status page in a condensed form) the situation is that production accounts associated with paying tenants identified at the start of the incident had been migrated (before any others as you will sure understand) except for a few that exhibited a metadata usage pattern that warranted individual review and were addressed/contacted individually.

The situation for free tenants is similar from the perspective that tenants have been migrated. However, there were some restrictions on the type of user searches that can be made in order to safeguard against similar issues. In the status page incident there’s more information about what to do if you are affected by those restrictions and you can also directly reference me (at least for the remainder of this week; I’ll be offline on the next ones) in any question posted in this forum related to particular issues that you still experience with user search.

Hi,
Thank you for you response.
I do understand you can prioritize paying customers before free once, but that is something you should have mention before and not after, at least give us a chance to change our status.

Second - Are those restriction like using app_metadata as a part of the search should be back any time soon?

Tomer

It’s important to recognise how this incident has been perceived, in the interest of learning for future incidents.

  1. Lack of communication about what was wrong with no ETA for a resolution

  2. Apparent long delay in resolution (not helped by Labor Day, I’m sure)

  3. “Resolution” by a breaking API change without warning

It feels like nit-picking to point out publishing an email address to enable support that didn’t work ( Searchapi-issues@auth0.com - Auth0 Community ) but that was certainly unfortunate!

I recognise dealing with production problems is stressful. I’m sure we’ve all been there before, so please don’t take this as angry criticism, rather a constructive contribution towards an improved future service for us all.

I use auth0 for a web app used by schools.

The start of the school year (this week) is when teachers will want to do the most user management to set up student accounts.

This was the worst possible problem happening at the worst possible time for me.

I’m sure this is bad for everyone, so I don’t claim my need is the greatest. But I’ve been tearing my hair out over this telling schools to hold off until this is fixed. And I still can’t tell them when that will be.

Hi,
Thank you for you response.
I do understand you can prioritize paying customers before free once, but that is something you should have mention before and not after, at least give us a chance to change our status.

Second - Are those restriction like using app_metadata as a part of the search should be back any time soon?

Tomer

Yes, the fact that first notification sent did not clearly state that the priority would be given to paying production tenants was something that I’ve already seen debated internally. In relation to restriction, at this time, we are getting feedback on use cases and requirements from customer in order to review and see what can be made available. However, there’s no deadline set for removing the current restrictions if the decision is to remove them.

Not taken, don’t worry. As a personal opinion I believe that making mistakes is unavoidable (the opinion may be biased by the fact that I personally can’t avoid making some) so the best thing to do is at learn something from those mistakes.

Do paying customers suffer from the same restrictions as well?

We tried to centralize communication through the status page, but I do understand that one point the frequency of updates could have been greater even if there wasn’t much more to add since the last update. In relation to the breaking change it’s indeed not what we wished, but I believe at some point there needed to be a decision that would not lead to the same outcome. Considering that not doing the decision would still leave customers broken (new changes were not being indexed) the restrictions were put in place.

At this time I don’t have definitive information on how the restrictions will work in terms of scope so in order to not lead you (or anyone else that ends up reading this) in error I would recommend that if you experience the inability to query for attributes outside the ones listed in the status page incident then you should expose your use case using the email address also listed in the incident.

@jmangelo @prashant

Sad to hear, but there is another issue
http://community.auth0.com/questions/9747/api-stopped-querying-by-metadata
to be categorized as above conversation, from the beginning.

more than 4 days meta_data not responding ok, some people started voting for the issue.
Plese, reply. Production stuck. While DB of customers stuck.

@jmangelo @prashant

Sad to hear, but there is another issue
http://community.auth0.com/questions/9747/api-stopped-querying-by-metadata
to be categorized as above conversation, from the beginning.

more than 4 days meta_data not responding ok, some people started voting for the issue.
Plese, reply. Production stuck. While DB of customers stuck.

I replied in the other post so we can use the other one for follow-ups.