Logstash extension BATCH_SIZE for auth0

bcsmith · May 4, 2018, 12:51am

Hello is there any way to increase the BATCH_SIZE to be more than 300? We receive way more than that many log entries in five minutes. Thank you

sgmeyer · May 4, 2018, 11:18am

Hi @bcsmith,

I assume you are using one of the logs-to extensions. If so, then this extension runs as a cron webtask. These can be rescheduled with the Webtask CLI.

Installing the CLI

Navigate to https://manage.auth0.com/#/tenant/webtasks for the tenant you’ve installed the extension
Follow the first two steps to install and setup the profile.
run wt cron ls -p "{your-profile-name}"

Changing the cron frequency

Find the logs-to-* extension to view its current cron schedule.
Determine what you want the new schedule to be and update the webtask using wt cron reschedule command.

Here are the webtask docs you can use to see more options for cron scheduling: https://webtask.io/docs/cron

Changing Batch Size

I don’t know if there is a batch size limit for the extensions. Technically, the API endpoint the extension uses for retrieving logs is 100 records, but the extensions themselves can handle larger batch sizes (I believe). I believe when you request a batch of say 500 the extension will break this into 5 requests of 100 records each. I don’t recall how log extensions with version 1.* works, but version 2.0 should have a free form entry text box allowing entry of an integer.

If interested here is the code:

github.com

auth0-extensions/auth0-log-extension-tools/blob/e626e29ce627b1eae7066ca4a4355e03bf60917b/src/processor.js

const _ = require('lodash');
const tools = require('auth0-extension-tools');

const logTypes = require('./logTypes');
const LogsApiStream = require('./stream');
const StorageProvider = require('./storage');

function LogsProcessor(storageContext, options) {
  if (options === null || options === undefined) {
    throw new tools.ArgumentError('Must provide an options object');
  }

  this.storage = new StorageProvider(storageContext);
  this.options = _.assign({ },
    {
      batchSize: 100,
      maxRetries: 5,
      maxRunTimeSeconds: 20
    },
    options

This file has been truncated. show original

At any route, you could probably play with both the cron scheduling and batch size. Keep in mind this extension uses Manage API to get logs. So, if you have a batch of 500 this will be 5 calls that would work against your API rate limits with auth0.