home-docker/agents.md

AI Agent Context: Home Docker Infrastructure

Purpose: This document helps AI assistants (GitHub Copilot, Claude, ChatGPT, etc.) understand the repository structure, architecture, and context for better code assistance.

DO NOT DELETE ANY LOCAL OR REMOTE FILES WITHOUT ASKING ME TO CONFIRM!

📋 Repository Overview

Name: home-docker (wahyd4/home-docker) Purpose: Home media server and personal cloud infrastructure managed via Kubernetes (K3s) and Terraform Architecture: Self-hosted Kubernetes cluster with NGINX Ingress, cert-manager, and various homelab applications Primary Technologies: Kubernetes (K3s), Terraform, Docker, Helm, ArgoCD

---

🏗️ Infrastructure Architecture

Kubernetes Cluster Setup

• Distribution: K3s (lightweight Kubernetes)
• Control Plane: 192.168.1.2 (server-3)
• Container Runtime: containerd 2.0.5-k3s2
• Kubernetes Version: v1.33.3+k3s1
• OS: Ubuntu 24.04.3 LTS
• Network: 192.168.1.0/24 (local network)

Key Infrastructure Components

1. Ingress Controller: NGINX Ingress Controller (ingress-nginx namespace)

   • NodePort on ports 80/443
   • Handles all external traffic
   • Integrated with cert-manager for TLS
   • Domain *.junv.cc

2. Certificate Management: cert-manager

   • Let's Encrypt integration (letsencrypt-prod ClusterIssuer)
   • Automatic TLS certificate provisioning
   • Email: me@junv.cc

3. GitOps: ArgoCD

   • Continuous deployment from Git
   • Web UI: https://argo.junv.cc
   • Uses external Redis (redis-master.db.svc.cluster.local)

4. Security: Multi-layered protection

   CrowdSec (Community threat intelligence):

   • Monitors NGINX Ingress logs
   • SSH monitoring (port 22422)
   • System log analysis
   • Enrolled in CrowdSec Console
   • Storage: /mnt/k8s/crowdsec/ on server-3

   fail2ban (Local intrusion prevention):

   • SSH brute force protection (port 22422)
   • Ban time: 1 hour (3600 seconds)
   • Max retries: 5 attempts in 10 minutes
   • Recidive jail: 24 hour ban for repeat offenders
   • Ignores local network (192.168.1.0/24)

5. Monitoring: Prometheus + Grafana

   • Metrics collection
   • Custom dashboards
   • ServiceMonitor for various services

---

📁 Directory Structure

Root Level Files

• terraform/: Terraform configurations for infrastructure as code

  • main.tf: K3s cluster setup, SSH configuration
  • argocd.tf: ArgoCD, cert-manager, NGINX Ingress deployment
  • crowdsec.tf: CrowdSec security engine setup
  • fail2ban.tf: fail2ban intrusion prevention setup
  • postgres.tf: PostgreSQL installation via SSH
  • redis.tf: Redis installation via SSH

• home-apps/: Home automation and productivity apps

  • ddns.yaml: Dynamic DNS clients (Cloudflare, Aliyun)
  • n8n.yaml: Workflow automation platform
  • searxng.yaml: Privacy-respecting search engine
  • home.yaml: Home dashboard application
  • server-backup.yaml: Backup automation

• apps/: General applications

  • passkey-auth.yaml: Authentication service

• ai/: AI/ML applications

  • crawl4ai.yaml: AI web crawler

• media/: Media server applications

  • alist.yaml: File listing service
  • jackett.yaml: Torrent indexer
  • media.yaml: Media server deployment
  • nextcloud-pv.yaml: Nextcloud persistent volume
  • static-file.yaml: Static file server

• db/: Database services

  • redis-values.yaml: Redis Helm values
  • qdrant.yaml: Vector database for AI

• archive/: Deprecated/archived configurations

  • Legacy configurations no longer in active use
  • Old services like Jellyfin, Plex, Metabase, etc.

Key Scripts

• apply.sh: Manual kubectl apply commands (mostly deprecated in favor of Terraform)
• backup-k8s.sh: Kubernetes cluster backup script
• restore-k8s.sh: Kubernetes cluster restoration
• k3s.sh: K3s installation and management notes
• fix_redis.sh: Redis troubleshooting script

---

🔧 Terraform Configuration

SSH Access Configuration

locals {
  ssh_host        = "192.168.1.2"
  ssh_user        = "junv"
  ssh_port        = 22422  # Custom SSH port
  ssh_private_key = file("~/.ssh/id_ed25519")
}

Terraform Providers

1. ssh (loafoe/ssh v2.6.0): Remote command execution
2. kubernetes (hashicorp/kubernetes v2.27.0): K8s resource management
3. helm (hashicorp/helm v2.12.1): Helm chart deployment
4. kubectl (gavinbunney/kubectl v1.14.0): Raw K8s manifests

Storage Strategy

• Local Path Storage: /mnt/k8s/ on server-3 (192.168.1.2)
• Storage Class: local-path (K3s default)
• Node Affinity: Resources pinned to server-3 hostname
• Examples:
  • CrowdSec: /mnt/k8s/crowdsec/
  • PostgreSQL: /mnt/k8s/postgres/
  • Redis: /mnt/k8s/redis/

---

🔐 Security & Access

SSH Configuration

• Port: 22422 (non-standard for security)
• Key: Ed25519 (~/.ssh/id_ed25519)
• User: junv
• Host: 192.168.1.2

Network Security

• Local Network: 192.168.1.0/24 (trusted)
• CrowdSec: Monitors external threats
• iptables: Previously attempted but removed due to kubectl connectivity issues

CrowdSec Configuration

• Monitored Logs:

  • NGINX Ingress Controller logs (container logs with poll_without_inotify)
  • SSH attempts: /var/log/auth.log
  • System logs: /var/log/kern.log, /var/log/syslog

• Collections Installed:

  • crowdsecurity/nginx
  • crowdsecurity/base-http-scenarios
  • crowdsecurity/linux
  • crowdsecurity/sshd
  • crowdsecurity/iptables
  • crowdsecurity/endlessh

• Enrollment: junv-home-k3s-cluster (enrolled in CrowdSec Console)

---

🚀 Common Operations

Terraform Workflow

cd terraform/
terraform init
terraform plan
terraform apply

Kubernetes Access

# Copy kubeconfig from server
scp -P 22422 junv@192.168.1.2:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i '' 's/127.0.0.1/192.168.1.2/g' ~/.kube/config

# Or use Terraform local
# Defined in main.tf as local.kubeconfig_cmd

CrowdSec Commands

# View metrics
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli metrics

# List decisions (bans)
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli decisions list

# List bouncers
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli bouncers list

# Quick status view
./terraform/view_crowdsec.sh

fail2ban Commands

# View status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status"

# SSH jail status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd"

# List banned IPs
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd | grep 'Banned IP'"

# Unban an IP
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client set sshd unbanip <IP>"

# Quick status view
./terraform/view_fail2ban.sh

Backup/Restore

# Backup cluster
./backup-k8s.sh

# Restore cluster
./restore-k8s.sh

---

📦 Helm Charts Used

Chart	Repository	Version	Namespace	Purpose
crowdsec	https://crowdsecurity.github.io/helm-charts	0.20.0	crowdsec	Security engine
cert-manager	https://charts.jetstack.io	v1.17.2	cert-manager	Certificate management
ingress-nginx	https://kubernetes.github.io/ingress-nginx	4.10.5	ingress-nginx	Ingress controller
argocd	https://argoproj.github.io/argo-helm	latest	argocd	GitOps CD
kube-prometheus-stack	prometheus-community	latest	default	Monitoring
redis	bitnami	latest	db	Cache/queue

---

🗄️ Persistent Storage Locations

All persistent data stored on server-3 at 192.168.1.2:/mnt/k8s/

/mnt/k8s/
├── crowdsec/
│   ├── lapi-config/  (1Gi)
│   └── lapi-data/    (2Gi)
├── postgres/
├── redis/
└── [other services]

---

🌐 Domain Configuration

• Primary Domain: junv.cc
• Subdomains:
  • argo.junv.cc (ArgoCD)
  • [other services configured via Ingress resources]

---

🔍 Important Patterns & Conventions

Namespace Organization

• crowdsec: Security monitoring
• ingress-nginx: Ingress controller
• cert-manager: Certificate management
• argocd: GitOps deployment
• home-apps: Home automation apps
• media: Media server apps
• db: Database services
• default: Legacy prometheus/monitoring

Resource Naming

• PersistentVolumes: {service}-{type}-pv (e.g., crowdsec-lapi-config-pv)
• Deployments: Usually match service name
• Secrets: {service}-credentials or {service}-{purpose}

Storage Classes

• local-path: K3s default local storage (most common)
• nfs-client: NFS-based storage (if NFS provisioner installed)

---

⚠️ Known Issues & Fixes

Issue: iptables blocking kubectl

Problem: CrowdSec iptables rules blocked kubectl access to port 6443 Solution: Disabled iptables integration in crowdsec.tf (commented out ssh_resource.iptables_logging) Status: CrowdSec still works via log analysis without iptables

Issue: Container log symlinks

Problem: CrowdSec warning about inotify polling with symlinked container logs Solution: Added poll_without_inotify: true to NGINX Ingress acquisition config Status: Resolved

Issue: SSH timeout during Terraform apply

Problem: SSH connection timeout when iptables rules were being applied Root Cause: iptables port scan detection was blocking Terraform's own SSH connections Solution: Removed iptables integration entirely

---

🔗 External Dependencies

• NFS Server: 192.168.1.5:/fs/1000/nfs/k8s (for NFS storage provisioner)
• External etcd Cluster:
  • 192.168.1.6:2379
  • 192.168.1.10:2379
  • 192.168.1.11:2379
• CrowdSec Console: https://app.crowdsec.net

---

📝 Migration Notes

From Docker Compose to Kubernetes

• legacy-docker-compose/: Contains old Docker Compose configurations
• Most services migrated to Kubernetes YAML manifests
• Some services archived if no longer needed

From Manual kubectl to Terraform

• apply.sh: Contains legacy manual kubectl commands
• Gradually being replaced by Terraform resources
• Reference for secret creation and one-time setup tasks

---

🎯 Development Workflow

Adding a New Service

1. Create YAML manifest in appropriate directory (home-apps/, media/, etc.)
2. Define namespace, deployment, service, ingress
3. If needs persistence: create PV/PVC with /mnt/k8s/{service}/ path
4. If needs TLS: add cert-manager annotations to Ingress
5. If monitoring needed: add ServiceMonitor resource
6. Consider adding to ArgoCD for GitOps deployment

Modifying Terraform Infrastructure

1. Edit appropriate .tf file in terraform/
2. Run terraform plan to preview changes
3. Run terraform apply to apply changes
4. For SSH-based resources: ensure connectivity to 192.168.1.2:22422

Debugging

• Pod logs: kubectl logs -n {namespace} {pod-name}
• Pod shell: kubectl exec -it -n {namespace} {pod-name} -- /bin/bash
• Events: kubectl get events -n {namespace} --sort-by='.lastTimestamp'
• CrowdSec: ./terraform/view_crowdsec.sh

---

🤖 AI Assistant Guidelines

When assisting with this repository:

1. Always consider the namespace - resources are organized by namespace
2. Use Terraform when possible - prefer Terraform over manual kubectl
3. Respect storage patterns - use /mnt/k8s/{service}/ for persistence
4. Check SSH port - always use port 22422, not default 22
5. Local network trust - 192.168.1.0/24 is trusted, don't block it
6. CrowdSec integration - avoid iptables, use log-based monitoring only
7. Node affinity - resources should be pinned to "server-3" hostname
8. Poll without inotify - use for container logs (symlinks)
9. Ingress patterns - use cert-manager annotations for TLS
10. Secret management - reference existing secrets in apply.sh for creation patterns

---

📚 Reference Documentation

• K3s Docs: https://docs.k3s.io/
• CrowdSec Docs: https://docs.crowdsec.net/
• NGINX Ingress: https://kubernetes.github.io/ingress-nginx/
• cert-manager: https://cert-manager.io/
• ArgoCD: https://argo-cd.readthedocs.io/
• Terraform Kubernetes Provider: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs

---

🏷️ Tags & Keywords

kubernetes k3s terraform homelab nginx-ingress crowdsec argocd cert-manager prometheus grafana self-hosted infrastructure-as-code gitops security monitoring media-server home-automation

---

Last Updated: October 5, 2025 Maintainer: junv (wahyd4) Repository: https://github.com/wahyd4/home-docker