home-docker/agents.md

AI Agent Context: Home Docker Infrastructure

Purpose: This document helps AI assistants (GitHub Copilot, Claude, ChatGPT, etc.) understand the repository structure, architecture, and context for better code assistance.

---

📋 Repository Overview

Name: home-docker (wahyd4/home-docker) Purpose: Home media server and personal cloud infrastructure managed via Kubernetes (K3s) and Terraform Architecture: Self-hosted Kubernetes cluster with NGINX Ingress, cert-manager, and various homelab applications Primary Technologies: Kubernetes (K3s), Terraform, Docker, Helm, ArgoCD

---

🏗️ Infrastructure Architecture

Kubernetes Cluster Setup

• Distribution: K3s (lightweight Kubernetes)
• Control Plane: 192.168.1.2 (server-3)
• Container Runtime: containerd 2.0.5-k3s2
• Kubernetes Version: v1.33.3+k3s1
• OS: Ubuntu 24.04.3 LTS
• Network: 192.168.1.0/24 (local network)

Key Infrastructure Components

1. Ingress Controller: NGINX Ingress Controller (ingress-nginx namespace)

   • NodePort on ports 80/443
   • Handles all external traffic
   • Integrated with cert-manager for TLS
   • Domain *.junv.cc

2. Certificate Management: cert-manager

   • Let's Encrypt integration (letsencrypt-prod ClusterIssuer)
   • Automatic TLS certificate provisioning
   • Email: me@junv.cc

3. GitOps: ArgoCD

   • Continuous deployment from Git
   • Web UI: https://argo.junv.cc
   • Uses external Redis (redis-master.db.svc.cluster.local)

4. Security: Multi-layered protection

   CrowdSec (Community threat intelligence):

   • Monitors NGINX Ingress logs
   • SSH monitoring (port 22422)
   • System log analysis
   • Enrolled in CrowdSec Console
   • Storage: /mnt/k8s/crowdsec/ on server-3

   fail2ban (Local intrusion prevention):

   • SSH brute force protection (port 22422)
   • Ban time: 1 hour (3600 seconds)
   • Max retries: 5 attempts in 10 minutes
   • Recidive jail: 24 hour ban for repeat offenders
   • Ignores local network (192.168.1.0/24)

5. Monitoring: Prometheus + Grafana

   • Metrics collection
   • Custom dashboards
   • ServiceMonitor for various services

---

📁 Directory Structure

Root Level Files

• terraform/: Terraform configurations for infrastructure as code

  • main.tf: K3s cluster setup, SSH configuration
  • argocd.tf: ArgoCD, cert-manager, NGINX Ingress deployment
  • crowdsec.tf: CrowdSec security engine setup
  • fail2ban.tf: fail2ban intrusion prevention setup
  • postgres.tf: PostgreSQL installation via SSH
  • redis.tf: Redis installation via SSH

• home-apps/: Home automation and productivity apps

  • ddns.yaml: Dynamic DNS clients (Cloudflare, Aliyun)
  • n8n.yaml: Workflow automation platform
  • searxng.yaml: Privacy-respecting search engine
  • home.yaml: Home dashboard application
  • server-backup.yaml: Backup automation

• apps/: General applications

  • passkey-auth.yaml: Authentication service

• ai/: AI/ML applications

  • crawl4ai.yaml: AI web crawler

• media/: Media server applications

  • alist.yaml: File listing service
  • jackett.yaml: Torrent indexer
  • media.yaml: Media server deployment
  • nextcloud-pv.yaml: Nextcloud persistent volume
  • static-file.yaml: Static file server

• db/: Database services

  • redis-values.yaml: Redis Helm values
  • qdrant.yaml: Vector database for AI

• archive/: Deprecated/archived configurations

  • Legacy configurations no longer in active use
  • Old services like Jellyfin, Plex, Metabase, etc.

Key Scripts

• apply.sh: Manual kubectl apply commands (mostly deprecated in favor of Terraform)
• backup-k8s.sh: Kubernetes cluster backup script
• restore-k8s.sh: Kubernetes cluster restoration
• k3s.sh: K3s installation and management notes
• fix_redis.sh: Redis troubleshooting script

---

🔧 Terraform Configuration

SSH Access Configuration

locals {
  ssh_host        = "192.168.1.2"
  ssh_user        = "junv"
  ssh_port        = 22422  # Custom SSH port
  ssh_private_key = file("~/.ssh/id_ed25519")
}

Terraform Providers

1. ssh (loafoe/ssh v2.6.0): Remote command execution
2. kubernetes (hashicorp/kubernetes v2.27.0): K8s resource management
3. helm (hashicorp/helm v2.12.1): Helm chart deployment
4. kubectl (gavinbunney/kubectl v1.14.0): Raw K8s manifests

Storage Strategy

• Local Path Storage: /mnt/k8s/ on server-3 (192.168.1.2)
• Storage Class: local-path (K3s default)
• Node Affinity: Resources pinned to server-3 hostname
• Examples:
  • CrowdSec: /mnt/k8s/crowdsec/
  • PostgreSQL: /mnt/k8s/postgres/
  • Redis: /mnt/k8s/redis/

---

🔐 Security & Access

SSH Configuration

• Port: 22422 (non-standard for security)
• Key: Ed25519 (~/.ssh/id_ed25519)
• User: junv
• Host: 192.168.1.2

Network Security

• Local Network: 192.168.1.0/24 (trusted)
• CrowdSec: Monitors external threats
• iptables: Previously attempted but removed due to kubectl connectivity issues

CrowdSec Configuration

• Monitored Logs:

  • NGINX Ingress Controller logs (container logs with poll_without_inotify)
  • SSH attempts: /var/log/auth.log
  • System logs: /var/log/kern.log, /var/log/syslog

• Collections Installed:

  • crowdsecurity/nginx
  • crowdsecurity/base-http-scenarios
  • crowdsecurity/linux
  • crowdsecurity/sshd
  • crowdsecurity/iptables
  • crowdsecurity/endlessh

• Enrollment: junv-home-k3s-cluster (enrolled in CrowdSec Console)

---

🚀 Common Operations

Terraform Workflow

cd terraform/
terraform init
terraform plan
terraform apply

Kubernetes Access

# Copy kubeconfig from server
scp -P 22422 junv@192.168.1.2:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i '' 's/127.0.0.1/192.168.1.2/g' ~/.kube/config

# Or use Terraform local
# Defined in main.tf as local.kubeconfig_cmd

CrowdSec Commands

# View metrics
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli metrics

# List decisions (bans)
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli decisions list

# List bouncers
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli bouncers list

# Quick status view
./terraform/view_crowdsec.sh

fail2ban Commands

# View status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status"

# SSH jail status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd"

# List banned IPs
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd | grep 'Banned IP'"

# Unban an IP
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client set sshd unbanip <IP>"

# Quick status view
./terraform/view_fail2ban.sh

Backup/Restore

# Backup cluster
./backup-k8s.sh

# Restore cluster
./restore-k8s.sh

---

📦 Helm Charts Used

Chart	Repository	Version	Namespace	Purpose
crowdsec	https://crowdsecurity.github.io/helm-charts	0.20.0	crowdsec	Security engine
cert-manager	https://charts.jetstack.io	v1.17.2	cert-manager	Certificate management
ingress-nginx	https://kubernetes.github.io/ingress-nginx	4.10.5	ingress-nginx	Ingress controller
argocd	https://argoproj.github.io/argo-helm	latest	argocd	GitOps CD
kube-prometheus-stack	prometheus-community	latest	default	Monitoring
redis	bitnami	latest	db	Cache/queue

---

🗄️ Persistent Storage Locations

All persistent data stored on server-3 at 192.168.1.2:/mnt/k8s/

/mnt/k8s/
├── crowdsec/
│   ├── lapi-config/  (1Gi)
│   └── lapi-data/    (2Gi)
├── postgres/
├── redis/
└── [other services]

---

🌐 Domain Configuration

• Primary Domain: junv.cc
• Subdomains:
  • argo.junv.cc (ArgoCD)
  • [other services configured via Ingress resources]

---

🔍 Important Patterns & Conventions

Namespace Organization

• crowdsec: Security monitoring
• ingress-nginx: Ingress controller
• cert-manager: Certificate management
• argocd: GitOps deployment
• home-apps: Home automation apps
• media: Media server apps
• db: Database services
• default: Legacy prometheus/monitoring

Resource Naming

• PersistentVolumes: {service}-{type}-pv (e.g., crowdsec-lapi-config-pv)
• Deployments: Usually match service name
• Secrets: {service}-credentials or {service}-{purpose}

Storage Classes

• local-path: K3s default local storage (most common)
• nfs-client: NFS-based storage (if NFS provisioner installed)

---

⚠️ Known Issues & Fixes

Issue: iptables blocking kubectl

Problem: CrowdSec iptables rules blocked kubectl access to port 6443 Solution: Disabled iptables integration in crowdsec.tf (commented out ssh_resource.iptables_logging) Status: CrowdSec still works via log analysis without iptables

Issue: Container log symlinks

Problem: CrowdSec warning about inotify polling with symlinked container logs Solution: Added poll_without_inotify: true to NGINX Ingress acquisition config Status: Resolved

Issue: SSH timeout during Terraform apply

Problem: SSH connection timeout when iptables rules were being applied Root Cause: iptables port scan detection was blocking Terraform's own SSH connections Solution: Removed iptables integration entirely

---

🔗 External Dependencies

• NFS Server: 192.168.1.4:/k8s (for NFS storage provisioner)
• External etcd Cluster:
  • 192.168.1.6:2379
  • 192.168.1.10:2379
  • 192.168.1.11:2379
• CrowdSec Console: https://app.crowdsec.net

---

📝 Migration Notes

From Docker Compose to Kubernetes

• legacy-docker-compose/: Contains old Docker Compose configurations
• Most services migrated to Kubernetes YAML manifests
• Some services archived if no longer needed

From Manual kubectl to Terraform

• apply.sh: Contains legacy manual kubectl commands
• Gradually being replaced by Terraform resources
• Reference for secret creation and one-time setup tasks

---

🎯 Development Workflow

Adding a New Service

1. Create YAML manifest in appropriate directory (home-apps/, media/, etc.)
2. Define namespace, deployment, service, ingress
3. If needs persistence: create PV/PVC with /mnt/k8s/{service}/ path
4. If needs TLS: add cert-manager annotations to Ingress
5. If monitoring needed: add ServiceMonitor resource
6. Consider adding to ArgoCD for GitOps deployment

Modifying Terraform Infrastructure

1. Edit appropriate .tf file in terraform/
2. Run terraform plan to preview changes
3. Run terraform apply to apply changes
4. For SSH-based resources: ensure connectivity to 192.168.1.2:22422

Debugging

• Pod logs: kubectl logs -n {namespace} {pod-name}
• Pod shell: kubectl exec -it -n {namespace} {pod-name} -- /bin/bash
• Events: kubectl get events -n {namespace} --sort-by='.lastTimestamp'
• CrowdSec: ./terraform/view_crowdsec.sh

---

🤖 AI Assistant Guidelines

When assisting with this repository:

1. Always consider the namespace - resources are organized by namespace
2. Use Terraform when possible - prefer Terraform over manual kubectl
3. Respect storage patterns - use /mnt/k8s/{service}/ for persistence
4. Check SSH port - always use port 22422, not default 22
5. Local network trust - 192.168.1.0/24 is trusted, don't block it
6. CrowdSec integration - avoid iptables, use log-based monitoring only
7. Node affinity - resources should be pinned to "server-3" hostname
8. Poll without inotify - use for container logs (symlinks)
9. Ingress patterns - use cert-manager annotations for TLS
10. Secret management - reference existing secrets in apply.sh for creation patterns

---

📚 Reference Documentation

• K3s Docs: https://docs.k3s.io/
• CrowdSec Docs: https://docs.crowdsec.net/
• NGINX Ingress: https://kubernetes.github.io/ingress-nginx/
• cert-manager: https://cert-manager.io/
• ArgoCD: https://argo-cd.readthedocs.io/
• Terraform Kubernetes Provider: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs

---

🏷️ Tags & Keywords

kubernetes k3s terraform homelab nginx-ingress crowdsec argocd cert-manager prometheus grafana self-hosted infrastructure-as-code gitops security monitoring media-server home-automation

---

Last Updated: October 5, 2025 Maintainer: junv (wahyd4) Repository: https://github.com/wahyd4/home-docker