Graylog2 Server Currently Unavailable Port 12900



  • I have Graylog installed but am trying to do something a little more complicated than a typical install, using external ElasticSearch cluster. I'm getting this error of "We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:12900/. Please verify that the server is healthy and working correctly."

    0_1471705482740_Screenshot from 2016-08-20 11-03-50.png

    I've reconfigured to use the private IP address as well, that does not work either, as you can see in the screen shot. I've checked on the box and 12900 is listening and telnet can connect to it. 12900 is the REST API for the system.

    # netstat -tulpn | grep 12900
    tcp6       0      0 127.0.0.1:12900         :::*                    LISTEN      11769/java
    
    telnet localhost 12900
    Trying ::1...
    telnet: connect to address ::1: Connection refused
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    


  • Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!



  • @scottalanmiller SELinux?



  • @JaredBusch said in Graylog2:

    @scottalanmiller SELinux?

    Decent guess but that didn't change it šŸ˜ž



  • from the localhost can you see /system/sessions as accessible?



  • HTTPS vs HTTP?

    Is that port blocked in your FW?



  • @gjacobse said in Graylog2:

    HTTPS vs HTTP?

    Is that port blocked in your FW?

    It's unblocked and tested. It's all HTTP. No proxy in place yet.



  • @DustinB3403 said in Graylog2:

    from the localhost can you see /system/sessions as accessible?

    It doesn't like local...

    curl localhost:9000/system/sessions
    <!DOCTYPE html>
    <html>
      <head>
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="robots" content="noindex, nofollow">
        <meta charset="UTF-8">
        <title>Graylog Web Interface</title>
        <link rel="shortcut icon" href="/favicon.png">
        
      </head>
      <body>
        <script src="/config.js"></script>
        <script src="/vendor.js"></script>
        
        <script src="/polyfill.f2a88026f0d0e7332f17.js"></script>
        
        <script src="/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.124facd49c87779f691b.js"></script>
        
        <script src="/plugin/org.graylog.plugins.map.MapWidgetPlugin/plugin.org.graylog.plugins.map.MapWidgetPlugin.3a6e86e960b80407f74e.js"></script>
        
        <script src="/plugin/org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin/plugin.org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin.7f1e11d52a66d206799c.js"></script>
        
        <script src="/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.74e8dfee2c1d6b11bc87.js"></script>
        
        <script src="/app.f2a88026f0d0e7332f17.js"></script>
        
      </body>
    </html>
    


  • I've totally disabled the firewall, just to be sure that that is not the problem.



  • Just realized that 12900 is exclusively listening on the loopback.



  • But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.



  • @DustinB3403 said in Graylog2:

    But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.

    It errored out on browser detection.



  • Here is the current status of the configuration file:

    grep -v ^# /etc/graylog/server/server.conf 
    
    is_master = true
    node_id_file = /etc/graylog/server/node-id
    password_secret = [redacted]
    root_password_sha2 = [redacted]
    root_email = [redacted]
    root_timezone = UTC
    plugin_dir = /usr/share/graylog-server/plugin
    rest_listen_uri = http://127.0.0.1:12900/
    web_listen_uri = http://0.0.0.0:9000/
    rotation_strategy = count
    elasticsearch_max_docs_per_index = 20000000
    rotation_strategy = count
    elasticsearch_max_docs_per_index = 20000000
    elasticsearch_max_number_of_indices = 20
    retention_strategy = delete
    elasticsearch_max_number_of_indices = 20
    retention_strategy = delete
    elasticsearch_shards = 4
    elasticsearch_replicas = 0
    elasticsearch_index_prefix = graylog
    allow_leading_wildcard_searches = false
    allow_highlighting = false
    elasticsearch_cluster_name = ntg-graylog2
    elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 192.168.1.51:9300, 192.168.1.52:9300, 192.168.1.53:9300, 192.168.1.51:9500
    elasticsearch_node_master = false
    elasticsearch_node_data = false
    elasticsearch_analyzer = standard
    output_batch_size = 500
    output_flush_interval = 1
    output_fault_count_threshold = 5
    output_fault_penalty_seconds = 30
    processbuffer_processors = 5
    outputbuffer_processors = 3
    processor_wait_strategy = blocking
    ring_size = 65536
    inputbuffer_ring_size = 65536
    inputbuffer_processors = 2
    inputbuffer_wait_strategy = blocking
    message_journal_enabled = true
    message_journal_dir = /var/lib/graylog-server/journal
    lb_recognition_period_seconds = 3
    mongodb_uri = mongodb://localhost/graylog
    mongodb_max_connections = 1000
    mongodb_threads_allowed_to_block_multiplier = 5
    content_packs_dir = /usr/share/graylog-server/contentpacks
    




  • Here is the log output on a restart. Getting errors trying to connect to a local service, that I know is running.

    2016-08-20T17:19:14.300Z INFO  [node] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] started
    2016-08-20T17:19:14.547Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:16.789Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
    2016-08-20T17:19:16.791Z INFO  [HttpServer] [HttpServer] Started.
    2016-08-20T17:19:16.791Z INFO  [WebInterfaceService] Started Web Interface at <http://0.0.0.0:9000/>
    2016-08-20T17:19:17.596Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:19.310Z WARN  [IndexerSetupService] Could not connect to Elasticsearch
    2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] If you're using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct.
    2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] See http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html for details.
    2016-08-20T17:19:20.648Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:22.266Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:12900]
    2016-08-20T17:19:22.266Z INFO  [HttpServer] [HttpServer-1] Started.
    2016-08-20T17:19:22.266Z INFO  [RestApiService] Started REST API at <http://0.0.0.0:12900/>
    2016-08-20T17:19:22.267Z INFO  [ServiceManagerListener] Services are healthy
    2016-08-20T17:19:22.268Z INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=196, BufferSynchronizerService [RUNNING]=202, MetricsReporterService [RUNNING]=203, KafkaJournal [RUNNING]=240, InputSetupService [RUNNING]=276, PeriodicalsService [RUNNING]=642, JournalReader [RUNNING]=854, WebInterfaceService [RUNNING]=6864, IndexerSetupService [RUNNING]=9386, RestApiService [RUNNING]=12343}
    2016-08-20T17:19:22.272Z INFO  [ServerBootstrap] Graylog server up and running.
    2016-08-20T17:19:22.277Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from UninitializedĀ [LB:DEAD] to RunningĀ [LB:ALIVE]
    2016-08-20T17:19:23.711Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:25.121Z INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
    2016-08-20T17:19:26.736Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:29.764Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:32.785Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:35.810Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:38.826Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:41.845Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:44.860Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:47.874Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:50.889Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    


  • I made the same change as the person in the GitHub thread and opened port 12900 to the world so that I could test from a browser. And it indeed can be reached for me, the same as him. So the error appears to be wrong.

    0_1471713807470_Screenshot from 2016-08-20 13-23-18.png



  • Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!



  • Don't forget to mark the thread as solved then šŸ˜‰



  • @StrongBad said in Graylog2:

    Don't forget to mark the thread as solved then šŸ˜‰

    Marked


Log in to reply