Running Unit Tests against the Management API

Help
, , , ,
1

I saw that the current state of Django libraries was pretty limited to auth backends, JWT verification, and social login using Auth0 instead of facebook or twitter. So I’ve been working hard to build a library to provide better end to end integration of Django with Auth0 and I’m trying to do the right thing and test it.

My test harness ensures Auth0 contains no users, by deleting them all one by one, and confirming they are each deleted with a subsequent user get call. then re-confirms the deletions worked by ensuring that fetching the Auth0 users via the API returns no users. It then ensures there are no Django users, then proceeds to create a specified number of users, and after test is done, it will proceed to delete all the Django and Auth0 users to leave the environment clean for the next test.

The current minimum test run is one test with 1 user in auth0, one test with 5 users in auth0 and one test with 10 users in auth0. Each test is a simple function that tries to verify that it finds the correct number of users that should be in Auth0.

Without long pauses after api activity, one test run everything will pass, the next all three will fail, then one will succeed at random, and so on, with I’m stuck trying to balance speed and consistency.

My first two thoughts are:

  1. Are there documented consistency times for API requests that modify the content of an Auth0 account?
    Then I would know how long I should wait, how far apart to space my retry attempts, at the moment each ‘full isolation’ takes at least 15-30 seconds, any better information on lowering this without re-introducing inconsistency would be very helpful.

  2. Is it possible to get a development testing only version of the on premises install, perhaps something moderately crippled by running as a docker container or a set of docker containers instead of a large capable virtual machine? If I wasn’t using the shared public infrastructure that might help speed things up since the system would only ever have to deal with up to a few dozen test users.

Any advice from people that have written similar test harnesses would be quite helpful, so would any sort of response from Auth0 on the matter.

2

Part of what you described consists of testing code outside of your control; in general, automated testing and in particular unit testing should focus on the code you control and mock the external dependencies as what you want to make sure is that your code correctly handles the various possible scenarios documented for the external dependency.

In conclusion, you should mock Auth0 so that you have full control of the responses and in this way test the exact scenarios your want to test within your own code. In relation to your second point having a development version of the appliance would imply you would also have to have a production version of the appliance so I guess the strict answer to your second question is no.