Skip to content

Instantly share code, notes, and snippets.

@schmichri

schmichri/Entry in Grafana Config

Last active March 14, 2025 09:32
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save schmichri/8d43e20140c4931013239cc4a83d9237 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save schmichri/8d43e20140c4931013239cc4a83d9237 to your computer and use it in GitHub Desktop.
Download ZIP
Nginx-Ingress Logs with Fluent-bit and Loki in Grafana | for logging Search Engine Crawlers for https://www.iunera.com and https://www.license-token.com/
Raw
Entry in Grafana Config
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus-server.monitoring.svc.cluster.local
access: proxy
isDefault: false
- name: Loki
type: loki
url: http://loki:3100
access: proxy
isDefault: false
Raw
Example Query in Grafana
# LogQL in Grafana
# Shows all HTTP/200 responses on www.license-token.com group by the remote IP
sum by (remote_ip) (
count_over_time({kubernetes_namespace_name=~"ingress", backend_service="license-token-license-token-website-springboot-4000"} | json | remote_ip != "" and status=~"^(200)$" [5m])
)
Raw
fluent-bit.helm.yml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: fluent-bit
namespace: monitoring
spec:
interval: 5m
chart:
spec:
chart: fluent-bit
version: ">=0.48.9"
sourceRef:
kind: HelmRepository
name: fluent
namespace: monitoring
values:
serviceMonitor:
enabled: false
tolerations:
- operator: "Exists" # This will match all taints
# Ensure pods can be scheduled on any node
affinity: {}
nodeSelector: {}
logLevel: info
hotReload:
enabled: false
config:
service: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
# Exclude unneeded logs
[FILTER]
Name grep
Match kube.*
Exclude kubernetes.pod_name fluent-bit-*
Exclude kubernetes.pod_name weave-net-*
Exclude kubernetes.pod_name coredns-*
# Apply to ALL pods - add Kubernetes metadata
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
# Only applies to ingress-nginx-controller pods - JSON parsing
[FILTER]
Name parser
Match kube.var.log.containers.*ingress-nginx-controller*.log
Key_Name log
Parser json
Reserve_Data On
Preserve_Key True
[FILTER]
Name modify
Match kube.var.log.containers.*ingress-nginx-controller*.log
Copy remote_ip http_remote_ip
Copy remote_user http_remote_user
Copy request_method http_request_method
Copy upstream_name backend_service
Copy status http_response_status
outputs: |
# Output for system components with minimal labels
[OUTPUT]
Name loki
Match kube.var.log.containers.*kube-system*.log
Host loki
Port 3100
Labels job=system
Label_Keys $kubernetes['namespace_name'],$kubernetes['container_name']
Remove_Keys kubernetes, time, time_local
Auto_Kubernetes_Labels Off
Line_Format json
# Output for ingress logs with relevant labels
[OUTPUT]
Name loki
Match kube.var.log.containers.*ingress-nginx-controller*.log
Host loki
Port 3100
Labels job=ingress
Label_Keys $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name'],$http_remote_ip,$http_remote_user,$http_request_method,$http_response_status,$backend_service
Remove_Keys kubernetes, time, time_local
Auto_Kubernetes_Labels Off
Line_Format json
# Default output for all other logs
[OUTPUT]
Name loki
Match kube.*
Host loki
Port 3100
Labels job=apps
Label_Keys $kubernetes['namespace_name'],$kubernetes['container_name']
Remove_Keys kubernetes, time, time_local
Auto_Kubernetes_Labels Off
Line_Format json
Raw
Nginx-ingress Config Snippet
controller:
# Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
allowSnippetAnnotations: "true"
config:
log-format-escape-json: "true"
# we want to have single lines Json Lines. More easy to parse and read
# more info https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/
log-format-upstream: '{"timestamp":"$time_iso8601","time_local":"$time_local","proxy_protocol_addr":"$proxy_protocol_addr","remote_ip":"$remote_addr","remote_user":"$remote_user","request_method":"$request_method","request_uri":"$request_uri","request_protocol":"$server_protocol","status":"$status","bytes_sent":"$bytes_sent","request_time":"$request_time","referer":"$http_referer","user_agent":"$http_user_agent","request_length":"$request_length","request_id":"$req_id","upstream_name":"$proxy_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":"$upstream_response_length","upstream_response_time":"$upstream_response_time","upstream_status":"$upstream_status"}'
# Expanded JSON (same as the complacted above)
# {
# "timestamp": "$time_iso8601",
# "time_local": "$time_local",
# "proxy_protocol_addr": "$proxy_protocol_addr",
# "remote_ip": "$remote_addr",
# "remote_user": "$remote_user",
# "request_method": "$request_method",
# "request_uri": "$request_uri",
# "request_protocol": "$server_protocol",
# "status": "$status",
# "bytes_sent": "$bytes_sent",
# "request_time": "$request_time",
# "referer": "$http_referer",
# "user_agent": "$http_user_agent",
# "request_length": "$request_length",
# "request_id": "$req_id",
# "upstream_name": "$proxy_upstream_name",
# "upstream_addr": "$upstream_addr",
# "upstream_response_length": "$upstream_response_length",
# "upstream_response_time": "$upstream_response_time",
# "upstream_status": "$upstream_status"
#}
Raw
Pretty default Loki Installation with helm in deploymentMode: SingleBinary mode
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: loki
namespace: monitoring
spec:
releaseName: loki
targetNamespace: monitoring
chart:
spec:
chart: loki
version: ">=6.28.0"
sourceRef:
kind: HelmRepository
name: grafana
namespace: monitoring
interval: 1m0s
install:
createNamespace: false
values:
loki:
auth_enabled: false
commonConfig:
replication_factor: 1
schemaConfig:
configs:
- from: "2024-04-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
tracing:
enabled: true
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 2
limits_Config:
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
max_query_series: 100000
deploymentMode: SingleBinary
singleBinary:
replicas: 1
resources:
limits:
cpu: 3
memory: 4Gi
requests:
cpu: 2
memory: 2Gi
extraEnv:
# Keep a little bit lower than memory limits
- name: GOMEMLIMIT
value: 3750MiB
chunksCache:
# default is 500MB, with limited memory keep this smaller
writebackSizeLimit: 10MB
# Enable minio for storage
minio:
enabled: true
# Zero out replica counts of other deployment modes
backend:
replicas: 0
read:
replicas: 0
write:
replicas: 0
ingester:
replicas: 0
querier:
replicas: 0
queryFrontend:
replicas: 0
queryScheduler:
replicas: 0
distributor:
replicas: 0
compactor:
replicas: 0
indexGateway:
replicas: 0
bloomCompactor:
replicas: 0
bloomGateway:
replicas: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment