From fa46a10aceefc1b4f5873757820e53a438698d78 Mon Sep 17 00:00:00 2001 From: Junwei Zhao Date: Sun, 5 Oct 2025 09:47:41 +1100 Subject: [PATCH] Add agents.md --- agents.md | 400 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 400 insertions(+) create mode 100644 agents.md diff --git a/agents.md b/agents.md new file mode 100644 index 0000000..4b40bf3 --- /dev/null +++ b/agents.md @@ -0,0 +1,400 @@ +# AI Agent Context: Home Docker Infrastructure + +> **Purpose**: This document helps AI assistants (GitHub Copilot, Claude, ChatGPT, etc.) understand the repository structure, architecture, and context for better code assistance. + +--- + +## 📋 Repository Overview + +**Name**: `home-docker` (wahyd4/home-docker) +**Purpose**: Home media server and personal cloud infrastructure managed via Kubernetes (K3s) and Terraform +**Architecture**: Self-hosted Kubernetes cluster with NGINX Ingress, cert-manager, and various homelab applications +**Primary Technologies**: Kubernetes (K3s), Terraform, Docker, Helm, ArgoCD + +--- + +## 🏗️ Infrastructure Architecture + +### Kubernetes Cluster Setup + +- **Distribution**: K3s (lightweight Kubernetes) +- **Control Plane**: 192.168.1.2 (server-3) +- **Container Runtime**: containerd 2.0.5-k3s2 +- **Kubernetes Version**: v1.33.3+k3s1 +- **OS**: Ubuntu 24.04.3 LTS +- **Network**: 192.168.1.0/24 (local network) + +### Key Infrastructure Components + +1. **Ingress Controller**: NGINX Ingress Controller (ingress-nginx namespace) + - NodePort on ports 80/443 + - Handles all external traffic + - Integrated with cert-manager for TLS + - Domain *.junv.cc + +2. **Certificate Management**: cert-manager + - Let's Encrypt integration (letsencrypt-prod ClusterIssuer) + - Automatic TLS certificate provisioning + - Email: me@junv.cc + +3. **GitOps**: ArgoCD + - Continuous deployment from Git + - Web UI: https://argo.junv.cc + - Uses external Redis (redis-master.db.svc.cluster.local) + +4. **Security**: CrowdSec + - Monitors NGINX Ingress logs + - SSH monitoring (port 22422) + - System log analysis + - Enrolled in CrowdSec Console + - Storage: /mnt/k8s/crowdsec/ on server-3 + +5. **Monitoring**: Prometheus + Grafana + - Metrics collection + - Custom dashboards + - ServiceMonitor for various services + +--- + +## 📁 Directory Structure + +### Root Level Files + +- **terraform/**: Terraform configurations for infrastructure as code + - `main.tf`: K3s cluster setup, SSH configuration + - `argocd.tf`: ArgoCD, cert-manager, NGINX Ingress deployment + - `crowdsec.tf`: CrowdSec security engine setup + - `postgres.tf`: PostgreSQL installation via SSH + - `redis.tf`: Redis installation via SSH + +- **home-apps/**: Home automation and productivity apps + - `ddns.yaml`: Dynamic DNS clients (Cloudflare, Aliyun) + - `n8n.yaml`: Workflow automation platform + - `searxng.yaml`: Privacy-respecting search engine + - `home.yaml`: Home dashboard application + - `server-backup.yaml`: Backup automation + +- **apps/**: General applications + - `passkey-auth.yaml`: Authentication service + +- **ai/**: AI/ML applications + - `crawl4ai.yaml`: AI web crawler + +- **media/**: Media server applications + - `alist.yaml`: File listing service + - `jackett.yaml`: Torrent indexer + - `media.yaml`: Media server deployment + - `nextcloud-pv.yaml`: Nextcloud persistent volume + - `static-file.yaml`: Static file server + +- **db/**: Database services + - `redis-values.yaml`: Redis Helm values + - `qdrant.yaml`: Vector database for AI + +- **archive/**: Deprecated/archived configurations + - Legacy configurations no longer in active use + - Old services like Jellyfin, Plex, Metabase, etc. + +### Key Scripts + +- `apply.sh`: Manual kubectl apply commands (mostly deprecated in favor of Terraform) +- `backup-k8s.sh`: Kubernetes cluster backup script +- `restore-k8s.sh`: Kubernetes cluster restoration +- `k3s.sh`: K3s installation and management notes +- `fix_redis.sh`: Redis troubleshooting script + +--- + +## 🔧 Terraform Configuration + +### SSH Access Configuration + +```hcl +locals { + ssh_host = "192.168.1.2" + ssh_user = "junv" + ssh_port = 22422 # Custom SSH port + ssh_private_key = file("~/.ssh/id_ed25519") +} +``` + +### Terraform Providers + +1. **ssh** (loafoe/ssh v2.6.0): Remote command execution +2. **kubernetes** (hashicorp/kubernetes v2.27.0): K8s resource management +3. **helm** (hashicorp/helm v2.12.1): Helm chart deployment +4. **kubectl** (gavinbunney/kubectl v1.14.0): Raw K8s manifests + +### Storage Strategy + +- **Local Path Storage**: `/mnt/k8s/` on server-3 (192.168.1.2) +- **Storage Class**: `local-path` (K3s default) +- **Node Affinity**: Resources pinned to `server-3` hostname +- **Examples**: + - CrowdSec: `/mnt/k8s/crowdsec/` + - PostgreSQL: `/mnt/k8s/postgres/` + - Redis: `/mnt/k8s/redis/` + +--- + +## 🔐 Security & Access + +### SSH Configuration + +- **Port**: 22422 (non-standard for security) +- **Key**: Ed25519 (~/.ssh/id_ed25519) +- **User**: junv +- **Host**: 192.168.1.2 + +### Network Security + +- **Local Network**: 192.168.1.0/24 (trusted) +- **CrowdSec**: Monitors external threats +- **iptables**: Previously attempted but removed due to kubectl connectivity issues + +### CrowdSec Configuration + +- **Monitored Logs**: + - NGINX Ingress Controller logs (container logs with poll_without_inotify) + - SSH attempts: /var/log/auth.log + - System logs: /var/log/kern.log, /var/log/syslog + +- **Collections Installed**: + - crowdsecurity/nginx + - crowdsecurity/base-http-scenarios + - crowdsecurity/linux + - crowdsecurity/sshd + - crowdsecurity/iptables + - crowdsecurity/endlessh + +- **Enrollment**: junv-home-k3s-cluster (enrolled in CrowdSec Console) + +--- + +## 🚀 Common Operations + +### Terraform Workflow + +```bash +cd terraform/ +terraform init +terraform plan +terraform apply +``` + +### Kubernetes Access + +```bash +# Copy kubeconfig from server +scp -P 22422 junv@192.168.1.2:/etc/rancher/k3s/k3s.yaml ~/.kube/config +sed -i '' 's/127.0.0.1/192.168.1.2/g' ~/.kube/config + +# Or use Terraform local +# Defined in main.tf as local.kubeconfig_cmd +``` + +### CrowdSec Commands + +```bash +# View metrics +kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli metrics + +# List decisions (bans) +kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli decisions list + +# List bouncers +kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli bouncers list + +# Quick status view +./terraform/view_crowdsec.sh +``` + +### Backup/Restore + +```bash +# Backup cluster +./backup-k8s.sh + +# Restore cluster +./restore-k8s.sh +``` + +--- + +## 📦 Helm Charts Used + +| Chart | Repository | Version | Namespace | Purpose | +|-------|-----------|---------|-----------|---------| +| crowdsec | https://crowdsecurity.github.io/helm-charts | 0.20.0 | crowdsec | Security engine | +| cert-manager | https://charts.jetstack.io | v1.17.2 | cert-manager | Certificate management | +| ingress-nginx | https://kubernetes.github.io/ingress-nginx | 4.10.5 | ingress-nginx | Ingress controller | +| argocd | https://argoproj.github.io/argo-helm | latest | argocd | GitOps CD | +| kube-prometheus-stack | prometheus-community | latest | default | Monitoring | +| redis | bitnami | latest | db | Cache/queue | + +--- + +## 🗄️ Persistent Storage Locations + +All persistent data stored on **server-3** at `192.168.1.2:/mnt/k8s/` + +``` +/mnt/k8s/ +├── crowdsec/ +│ ├── lapi-config/ (1Gi) +│ └── lapi-data/ (2Gi) +├── postgres/ +├── redis/ +└── [other services] +``` + +--- + +## 🌐 Domain Configuration + +- **Primary Domain**: junv.cc +- **Subdomains**: + - argo.junv.cc (ArgoCD) + - [other services configured via Ingress resources] + +--- + +## 🔍 Important Patterns & Conventions + +### Namespace Organization + +- `crowdsec`: Security monitoring +- `ingress-nginx`: Ingress controller +- `cert-manager`: Certificate management +- `argocd`: GitOps deployment +- `home-apps`: Home automation apps +- `media`: Media server apps +- `db`: Database services +- `default`: Legacy prometheus/monitoring + +### Resource Naming + +- PersistentVolumes: `{service}-{type}-pv` (e.g., `crowdsec-lapi-config-pv`) +- Deployments: Usually match service name +- Secrets: `{service}-credentials` or `{service}-{purpose}` + +### Storage Classes + +- `local-path`: K3s default local storage (most common) +- `nfs-client`: NFS-based storage (if NFS provisioner installed) + +--- + +## ⚠️ Known Issues & Fixes + +### Issue: iptables blocking kubectl + +**Problem**: CrowdSec iptables rules blocked kubectl access to port 6443 +**Solution**: Disabled iptables integration in `crowdsec.tf` (commented out `ssh_resource.iptables_logging`) +**Status**: CrowdSec still works via log analysis without iptables + +### Issue: Container log symlinks + +**Problem**: CrowdSec warning about inotify polling with symlinked container logs +**Solution**: Added `poll_without_inotify: true` to NGINX Ingress acquisition config +**Status**: Resolved + +### Issue: SSH timeout during Terraform apply + +**Problem**: SSH connection timeout when iptables rules were being applied +**Root Cause**: iptables port scan detection was blocking Terraform's own SSH connections +**Solution**: Removed iptables integration entirely + +--- + +## 🔗 External Dependencies + +- **NFS Server**: 192.168.1.4:/k8s (for NFS storage provisioner) +- **External etcd Cluster**: + - 192.168.1.6:2379 + - 192.168.1.10:2379 + - 192.168.1.11:2379 +- **CrowdSec Console**: https://app.crowdsec.net + +--- + +## 📝 Migration Notes + +### From Docker Compose to Kubernetes + +- **legacy-docker-compose/**: Contains old Docker Compose configurations +- Most services migrated to Kubernetes YAML manifests +- Some services archived if no longer needed + +### From Manual kubectl to Terraform + +- **apply.sh**: Contains legacy manual kubectl commands +- Gradually being replaced by Terraform resources +- Reference for secret creation and one-time setup tasks + +--- + +## 🎯 Development Workflow + +### Adding a New Service + +1. Create YAML manifest in appropriate directory (home-apps/, media/, etc.) +2. Define namespace, deployment, service, ingress +3. If needs persistence: create PV/PVC with `/mnt/k8s/{service}/` path +4. If needs TLS: add cert-manager annotations to Ingress +5. If monitoring needed: add ServiceMonitor resource +6. Consider adding to ArgoCD for GitOps deployment + +### Modifying Terraform Infrastructure + +1. Edit appropriate `.tf` file in `terraform/` +2. Run `terraform plan` to preview changes +3. Run `terraform apply` to apply changes +4. For SSH-based resources: ensure connectivity to 192.168.1.2:22422 + +### Debugging + +- **Pod logs**: `kubectl logs -n {namespace} {pod-name}` +- **Pod shell**: `kubectl exec -it -n {namespace} {pod-name} -- /bin/bash` +- **Events**: `kubectl get events -n {namespace} --sort-by='.lastTimestamp'` +- **CrowdSec**: `./terraform/view_crowdsec.sh` + +--- + +## 🤖 AI Assistant Guidelines + +When assisting with this repository: + +1. **Always consider the namespace** - resources are organized by namespace +2. **Use Terraform when possible** - prefer Terraform over manual kubectl +3. **Respect storage patterns** - use `/mnt/k8s/{service}/` for persistence +4. **Check SSH port** - always use port 22422, not default 22 +5. **Local network trust** - 192.168.1.0/24 is trusted, don't block it +6. **CrowdSec integration** - avoid iptables, use log-based monitoring only +7. **Node affinity** - resources should be pinned to "server-3" hostname +8. **Poll without inotify** - use for container logs (symlinks) +9. **Ingress patterns** - use cert-manager annotations for TLS +10. **Secret management** - reference existing secrets in apply.sh for creation patterns + +--- + +## 📚 Reference Documentation + +- **K3s Docs**: https://docs.k3s.io/ +- **CrowdSec Docs**: https://docs.crowdsec.net/ +- **NGINX Ingress**: https://kubernetes.github.io/ingress-nginx/ +- **cert-manager**: https://cert-manager.io/ +- **ArgoCD**: https://argo-cd.readthedocs.io/ +- **Terraform Kubernetes Provider**: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs + +--- + +## 🏷️ Tags & Keywords + +`kubernetes` `k3s` `terraform` `homelab` `nginx-ingress` `crowdsec` `argocd` `cert-manager` `prometheus` `grafana` `self-hosted` `infrastructure-as-code` `gitops` `security` `monitoring` `media-server` `home-automation` + +--- + +**Last Updated**: October 5, 2025 +**Maintainer**: junv (wahyd4) +**Repository**: https://github.com/wahyd4/home-docker