siem

How to send Kong logs to splunk

techhadoop , , , ,

To send Kong API Gateway logs to Splunk, you can leverage several approaches based on the logging mechanism Kong uses, such as:

  1. HTTP Logging Plugin (sending logs via HTTP to Splunk’s HTTP Event Collector)
  2. Syslog Logging Plugin (sending logs to a syslog server integrated with Splunk)
  3. File-based Logging (sending logs using Splunk Universal Forwarder)

Here’s how you can achieve this:

1. Using the HTTP Logging Plugin (Recommended for Splunk HEC)

You can use Kong’s HTTP Log Plugin to send logs directly to Splunk HTTP Event Collector (HEC) over HTTP(S).

Steps:

1.1. Set Up HTTP Event Collector (HEC) in Splunk

  1. Go to your Splunk Web Interface.
  2. Navigate to Settings > Data Inputs > HTTP Event Collector.
  3. Create a new HEC token.
    • Set a source type (e.g., kong_logs).
    • Note down the token and HEC URL (e.g., http://<SPLUNK_URL&gt;:8088/services/collector).
  4. Ensure that HEC is enabled and configured to accept data.

1.2. Install the HTTP Log Plugin in Kong

The HTTP Log Plugin sends Kong logs to a specified HTTP endpoint (in this case, Splunk HEC).

  • You can configure the plugin at the service, route, or global level.

Example Configuration (using curl):

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=http-log” \

    –data “config.http_endpoint=http://<SPLUNK_HEC_URL>:8088/services/collector” \

    –data “config.method=POST” \

    –data “config.timeout=10000” \

    –data “config.keepalive=10000” \

    –data “config.headers.Splunk”=”<SPLUNK_HEC_TOKEN>”

  • Replace <KONG_ADMIN_URL> with your Kong Admin URL.
  • Replace <SPLUNK_HEC_URL> with your Splunk HEC endpoint.
  • Replace <SPLUNK_HEC_TOKEN> with the HEC token from Splunk.

You can customize the headers, format, and log levels as per your needs.

1.3. Log Format Configuration (Optional)

You can customize the log format that Kong sends to Splunk by configuring the log_format property of the HTTP Log Plugin.

bash

Copy code

–data “config.log_format={‘message’: ‘Kong Log: $request_uri $status’, ‘client_ip’: ‘$remote_addr’}”

Splunk will now start receiving the logs sent by Kong via the HTTP Event Collector.

2. Using the Syslog Logging Plugin

Kong can send logs to a syslog server that can be monitored by Splunk.

Steps:

2.1. Set Up Syslog Logging Plugin in Kong

  1. Install the Syslog Logging Plugin on your Kong instance:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=syslog” \

    –data “config.host=<SYSLOG_SERVER_IP>” \

    –data “config.port=514” \

    –data “config.facility=user” \

    –data “config.log_level=info”

  1. Replace <SYSLOG_SERVER_IP> with your syslog server IP or domain.

2.2. Configure Splunk to Receive Syslog Data

  1. On your Splunk instance, configure a new data input for receiving syslog data:
    • Go to Settings > Data Inputs > UDP.
    • Create a new UDP input on port 514 (or another port if you’re using a different one).
    • Set a source type like syslog or a custom type like kong_logs.
  2. You can also use a dedicated syslog server (like rsyslog or syslog-ng) to forward syslog messages from Kong to Splunk.

3. Using the File Log Plugin and Splunk Universal Forwarder

If you’re using file-based logging in Kong, you can set up the File Log Plugin and use the Splunk Universal Forwarder to monitor and send log files to Splunk.

Steps:

3.1. Set Up the File Log Plugin in Kong

  1. Install the File Log Plugin in Kong and configure it to log to a specific file.

Example configuration:

bash

Copy code

curl -X POST http://<KONG_ADMIN_URL&gt;:8001/services/<service-id>/plugins \

    –data “name=file-log” \

    –data “config.path=/var/log/kong/kong.log”

  • Replace /var/log/kong/kong.log with the path where you want the log files stored.

3.2. Install and Configure Splunk Universal Forwarder

  1. Install the Splunk Universal Forwarder on the server where Kong logs are stored.
  2. Configure the forwarder to monitor the log file:

In the inputs.conf file, specify the log file you want to forward:

bash

Copy code

[monitor:///var/log/kong/kong.log]

index = kong

sourcetype = kong:logs

  1. In the outputs.conf file, configure the forwarder to send logs to your main Splunk indexer:

bash

Copy code

[tcpout]

defaultGroup = indexers

[tcpout:indexers]

server = <SPLUNK_INDEXER_IP>:9997

  1. Start the Splunk Universal Forwarder to begin sending logs.

4. Using AWS Lambda (If Kong is in AWS)

If you’re using Kong on AWS (e.g., on EC2), and your logs are stored in AWS CloudWatch, you can set up an AWS Lambda function to forward logs to Splunk HTTP Event Collector (HEC).

Steps:

  1. Set up CloudWatch Logs to capture Kong logs.
  2. Create an AWS Lambda function to forward logs from CloudWatch to Splunk.
  3. Use the Splunk-provided AWS Lambda blueprint to send logs to Splunk HEC.

Summary of Methods:

  • HTTP Log Plugin: Send logs directly to Splunk’s HTTP Event Collector (HEC). This is the easiest and most efficient method.
  • Syslog Logging Plugin: Send logs to a syslog server that can be ingested by Splunk.
  • File Log Plugin + Splunk Universal Forwarder: Write logs to a file and forward them to Splunk using the Splunk Universal Forwarder.
  • AWS Lambda (CloudWatch Logs): Use Lambda to stream logs from CloudWatch to Splunk HEC (for AWS-hosted Kong).

Choose the method based on your infrastructure and logging requirements