Cannot export logs to logstash

Help
,
1

Hi all! I have trouble with sending logs from auth0 to my logstash. I have checked the documentation, also have checked the community, but didn’t find any answers on my questions. Here my issue.

  1. I have created pipeline with “HTTP” input
    image
    image1360×540 91.8 KB
  2. Set filter mutate
    image
    image938×340 34.5 KB
  3. And for the testing purpose set the output to “stdout”
  4. Set all credentials and my logstash URL on the settings page of the extension.
    image
    image766×1022 53.9 KB

In my dashboard, I see success exports Screenshot by Lightshot But don’t see any data in my stdout (also I have tried to set the output to elasticsearch, but situation the same, any logs doesn’t appear in my index). I have tried to send HTTP requests from the postman, but in this case all working great Screenshot by Lightshot

Could you help me? Seems my logstash pipeline works, but extension might be has wrong configuration.

Thanks in advance!

2

Good afternoon,

Are you able to open the extension and check for failed logs?

Thanks!

3

Thank you for your answer!

If you mean the first error export it’s my first try where I did forget to specify username and password for logstash, but here failed logs

checkpoint: null
start: 2020-04-30T13:21:38.769Z
end: 2020-04-30T13:21:40.185Z
logsProcessed: 0
error: {"error":{"code":"ETIMEDOUT"},"status":500}

Also I can provide logs from success job

checkpoint: 90020200504111156297000427220690989121255641378705113186
start: 2020-05-04T11:14:43.177Z
end: 2020-05-04T11:14:44.060Z
logsProcessed: 15

Do I understand what logs you need?

4

Good morning,

Extensions use our webtask platform and since they’re limited to running 30 seconds at a time, this can cause timeouts. We realize this can be an issue and it’s part of the reason we developed the new log streaming functionality (Log Streams).

If the uploads take too long the job will be killed and I think that would cause those socket hangups/timeouts.

A possible workaround might be to reduce the batch size and increase the run frequency in the extension’s configuration.

5

Thank you so much, seems it will work, but I don’t see how can I specify basic authentication for custom webhook, logstash doesn’t support bearer authentication

6

Resolved by specifying username and password in format https://<username>:<password>@url_to_logstash

Thank you again!

1 Like
7

No worries! We’re here for you! Glad you have figured it out!

8

Hey Konrad, Karen,

I am facing with the same issue. Cannot export logs to myu logstash with the logstash auth0 extension unfortunately. I understand I need to use the logstash http input plugin in my config file: Http input plugin | Logstash Reference [8.6] | Elastic

But what properties do I add there? Or do I put it empty, like this: ?
input {
http {
}
}

Also, I couldn’t find any documentation if any authentication is required? In a form of an API token that you might include in the http input plugin under some property. I did noticed the Auth0 Authorization extension and even generated one, but It seems to me that they have no concerned with each other.

Please advise.

Thanks in advanced :slight_smile:

9

Don’t use Auth0 extension. Based on answer above, you need to use feature “Log streams”. Unfortunately this feature available only on Developer plan, but it works.

For configuring logstash http plugin here no requirements. You can add basic authentication to your http listener but you can test without it. My advice, start your http listener, confirm that you have access to your listener (just send any request with curl to your logstash endpoint) and if logstash works and save your test request, you can set settings in auth0 using log streams. All examples I did provide above

10

Thanks Kostya :slight_smile:

I am interested to see if I can make it with the Logstash extension. I know it can be done via Log streams but it requires development on my side that I’d rather to avoid. I am waiting to see what will be the answers of the Auth0 team.

Guys, Is the Auth0 Logstash extension cannot work for me? If so, there are any plans from your side to develop a dedicated Auth0 input plugin for Logstash?

11

No necessary additional development from your side doesn’t need. Just set your http plugin in logstash (as you did set up for auth0 extension) and enter information in log streams how can auth0 can make a call to your logstash (if any authentication not require need populate only URL to you listener) and that’s it, auth0 will start send logs to logstash.

12

I’ll try that. Thanks a lot!! :slight_smile:

13

Hi Kostya, I am trying now to use the Log streams with a custom webhook and added my logstash URL in it. Then I have started my logstash, my logstash.conf input is:
input {
http {
}
}

Still no logs showing up from Auth0. Can you share a screenshot of your logstash.conf and also your Custom webhook configuration?

Thanks in advanced :slight_smile:

14

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.