Logstash extension BATCH_SIZE for auth0

Help
1

Hello is there any way to increase the BATCH_SIZE to be more than 300? We receive way more than that many log entries in five minutes. Thank you

2

Hi @bcsmith,

I assume you are using one of the logs-to extensions. If so, then this extension runs as a cron webtask. These can be rescheduled with the Webtask CLI.

Installing the CLI

  1. Navigate to https://manage.auth0.com/#/tenant/webtasks for the tenant you’ve installed the extension
  2. Follow the first two steps to install and setup the profile.
  3. run wt cron ls -p "{your-profile-name}"

Changing the cron frequency

  1. Find the logs-to-* extension to view its current cron schedule.
  2. Determine what you want the new schedule to be and update the webtask using wt cron reschedule command.

Here are the webtask docs you can use to see more options for cron scheduling: https://webtask.io/docs/cron

Changing Batch Size

I don’t know if there is a batch size limit for the extensions. Technically, the API endpoint the extension uses for retrieving logs is 100 records, but the extensions themselves can handle larger batch sizes (I believe). I believe when you request a batch of say 500 the extension will break this into 5 requests of 100 records each. I don’t recall how log extensions with version 1.* works, but version 2.0 should have a free form entry text box allowing entry of an integer.

If interested here is the code:

At any route, you could probably play with both the cron scheduling and batch size. Keep in mind this extension uses Manage API to get logs. So, if you have a batch of 500 this will be 5 calls that would work against your API rate limits with auth0.