home-docker/GeoIP-Analytics-README.md

Nginx Ingress GeoIP Analytics

This solution provides geographic visualization of traffic to your Kubernetes nginx ingress controller by extracting IP addresses from access logs and displaying them on a world map in Grafana.

Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Nginx Ingress  │───▶│   Promtail   │───▶│      Loki       │───▶│ GeoIP Enricher  │
│  (Access Logs)  │    │ (Log Scraper)│    │ (Log Storage)   │    │  (IP Analysis)  │
└─────────────────┘    └──────────────┘    └─────────────────┘    └─────────────────┘
                                                                             │
                                                                             ▼
┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐    ┌─────────────────┐
│     Grafana     │◀───│  Prometheus  │◀───│  Pushgateway    │◀───│  Geo Metrics    │
│  (Visualization)│    │  (Metrics)   │    │ (Metrics Proxy) │    │    (Push)       │
└─────────────────┘    └──────────────┘    └─────────────────┘    └─────────────────┘

Components

1. Loki + Promtail (Log Collection)

• Loki: Stores and indexes nginx access logs
• Promtail: Scrapes logs from nginx ingress pods and parses IP addresses

2. GeoIP Enricher (Custom Service)

• Queries Loki for recent access logs
• Extracts unique IP addresses
• Enriches IPs with geolocation data using ip-api.com
• Pushes geographic metrics to Prometheus via Pushgateway

3. Prometheus + Pushgateway (Metrics Storage)

• Stores geographic metrics with labels for country, city, latitude, longitude
• Provides time-series data for visualization

4. Grafana Dashboard (Visualization)

• World map showing request origins
• Pie chart of requests by country
• Time series of request rates
• Detailed geographic breakdown table

Installation

Deploy with Terraform

cd terraform

# Initialize Terraform
terraform init

# Plan the deployment
terraform plan

# Apply the configuration
terraform apply

This will deploy:

• Loki and Promtail for log collection
• Prometheus Pushgateway for metrics
• GeoIP enricher service
• Grafana with pre-configured dashboard
• All necessary Kubernetes resources

Access the Services

After deployment, you can access:

• Grafana: https://grafana.junv.cc (admin/admin123)
• Loki: http://loki.logging.svc.cluster.local:3100
• Pushgateway: http://prometheus-pushgateway.prometheus.svc.cluster.local:9091

Verification

# Check all components are running
kubectl -n logging get pods
kubectl -n prometheus get pods
kubectl -n grafana get pods

# Check GeoIP enricher logs
kubectl -n logging logs -l app=geoip-enricher -f

# Verify metrics are being pushed
kubectl -n prometheus port-forward svc/prometheus-pushgateway 9091:9091
# Visit http://localhost:9091/metrics and search for "nginx_geo"

Configuration

GeoIP Service Rate Limits

The solution uses the free ip-api.com service with these limits:

• 45 requests per minute
• 1000 requests per day

For production use, consider:

• Using MaxMind GeoLite2 database (local lookups)
• Implementing IP caching to reduce API calls
• Using paid geolocation services for higher limits

Log Format

The nginx ingress is configured with a detailed log format:

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

Data Collection Frequency

• GeoIP enricher runs every 5 minutes
• Queries last 5 minutes of logs from Loki
• Processes up to 5000 log entries per cycle

Troubleshooting

No Data in Grafana

1. Check if logs are being collected:

   kubectl -n logging logs -l app.kubernetes.io/name=promtail
   

2. Verify Loki has data:

   kubectl -n logging port-forward svc/loki 3100:3100
   # Visit http://localhost:3100 and query: {job="nginx-ingress"}
   

3. Check GeoIP enricher logs:

   kubectl -n logging logs -l app=geoip-enricher -f
   

4. Verify metrics in Pushgateway:

   kubectl -n prometheus port-forward svc/prometheus-pushgateway 9091:9091
   # Visit http://localhost:9091/metrics and search for "nginx_geo"
   

GeoIP Enricher Not Processing IPs

1. Check if nginx logs are in expected format:

   kubectl -n ingress-nginx logs -l app.kubernetes.io/component=controller
   

2. Verify network connectivity to ip-api.com:

   kubectl -n logging exec -it deployment/geoip-enricher -- curl -s "http://ip-api.com/json/8.8.8.8"
   

High Memory Usage

1. Adjust resource limits in geoip-enricher.yaml:

   resources:
     limits:
       memory: "512Mi"  # Increase if needed
   

2. Reduce log query frequency:

   • Edit the sleep time in enricher.py (default: 300 seconds)

Security Considerations

1. Network Policies: Restrict GeoIP enricher network access
2. Resource Limits: Set appropriate CPU/memory limits
3. RBAC: Create minimal service account permissions
4. Data Retention: Configure Loki retention policies

Monitoring

Monitor the solution with these queries:

# Enricher health
up{job="geoip-enricher"}

# Processing rate
increase(nginx_geo_requests_total[5m])

# Unique countries detected
count by (country) (nginx_geo_requests_by_country_total)

Scaling

For high-traffic deployments:

1. Horizontal scaling: Increase GeoIP enricher replicas
2. Caching: Implement Redis cache for IP lookups
3. Batching: Process IPs in larger batches
4. Local database: Use MaxMind GeoLite2 for offline lookups

Cost Optimization

1. IP filtering: Skip private/internal IP ranges
2. Deduplication: Cache recent IP lookups
3. Sampling: Process only a percentage of requests
4. Regional focus: Limit processing to specific regions

License

This solution is provided as-is for educational and operational use. Please ensure compliance with your organization's security and privacy policies.