home-docker/agents.md

# AI Agent Context: Home Docker Infrastructure

> **Purpose**: This document helps AI assistants (GitHub Copilot, Claude, ChatGPT, etc.) understand the repository structure, architecture, and context for better code assistance.

DO NOT DELETE ANY LOCAL OR REMOTE FILES WITHOUT ASKING ME TO CONFIRM!
---

## 📋 Repository Overview

**Name**: `home-docker` (wahyd4/home-docker)
**Purpose**: Home media server and personal cloud infrastructure managed via Kubernetes (K3s) and Terraform
**Architecture**: Self-hosted Kubernetes cluster with NGINX Ingress, cert-manager, and various homelab applications
**Primary Technologies**: Kubernetes (K3s), Terraform, Docker, Helm, ArgoCD

---

## 🏗️ Infrastructure Architecture

### Kubernetes Cluster Setup

- **Distribution**: K3s (lightweight Kubernetes)
- **Control Plane**: 192.168.1.2 (server-3)
- **Container Runtime**: containerd 2.0.5-k3s2
- **Kubernetes Version**: v1.33.3+k3s1
- **OS**: Ubuntu 24.04.3 LTS
- **Network**: 192.168.1.0/24 (local network)

### Key Infrastructure Components

1. **Ingress Controller**: NGINX Ingress Controller (ingress-nginx namespace)
   - NodePort on ports 80/443
   - Handles all external traffic
   - Integrated with cert-manager for TLS
   - Domain *.junv.cc

2. **Certificate Management**: cert-manager
   - Let's Encrypt integration (letsencrypt-prod ClusterIssuer)
   - Automatic TLS certificate provisioning
   - Email: me@junv.cc

3. **GitOps**: ArgoCD
   - Continuous deployment from Git
   - Web UI: https://argo.junv.cc
   - Uses external Redis (redis-master.db.svc.cluster.local)

4. **Security**: Multi-layered protection

   **CrowdSec** (Community threat intelligence):
   - Monitors NGINX Ingress logs
   - SSH monitoring (port 22422)
   - System log analysis
   - Enrolled in CrowdSec Console
   - Storage: /mnt/k8s/crowdsec/ on server-3

   **fail2ban** (Local intrusion prevention):
   - SSH brute force protection (port 22422)
   - Ban time: 1 hour (3600 seconds)
   - Max retries: 5 attempts in 10 minutes
   - Recidive jail: 24 hour ban for repeat offenders
   - Ignores local network (192.168.1.0/24)

5. **Monitoring**: Prometheus + Grafana
   - Metrics collection
   - Custom dashboards
   - ServiceMonitor for various services

---

## 📁 Directory Structure

### Root Level Files

- **terraform/**: Terraform configurations for infrastructure as code
  - `main.tf`: K3s cluster setup, SSH configuration
  - `argocd.tf`: ArgoCD, cert-manager, NGINX Ingress deployment
  - `crowdsec.tf`: CrowdSec security engine setup
  - `fail2ban.tf`: fail2ban intrusion prevention setup
  - `postgres.tf`: PostgreSQL installation via SSH
  - `redis.tf`: Redis installation via SSH

- **home-apps/**: Home automation and productivity apps
  - `ddns.yaml`: Dynamic DNS clients (Cloudflare, Aliyun)
  - `n8n.yaml`: Workflow automation platform
  - `searxng.yaml`: Privacy-respecting search engine
  - `home.yaml`: Home dashboard application
  - `server-backup.yaml`: Backup automation

- **apps/**: General applications
  - `passkey-auth.yaml`: Authentication service

- **ai/**: AI/ML applications
  - `crawl4ai.yaml`: AI web crawler

- **media/**: Media server applications
  - `alist.yaml`: File listing service
  - `jackett.yaml`: Torrent indexer
  - `media.yaml`: Media server deployment
  - `nextcloud-pv.yaml`: Nextcloud persistent volume
  - `static-file.yaml`: Static file server

- **db/**: Database services
  - `redis-values.yaml`: Redis Helm values
  - `qdrant.yaml`: Vector database for AI

- **archive/**: Deprecated/archived configurations
  - Legacy configurations no longer in active use
  - Old services like Jellyfin, Plex, Metabase, etc.

### Key Scripts

- `apply.sh`: Manual kubectl apply commands (mostly deprecated in favor of Terraform)
- `backup-k8s.sh`: Kubernetes cluster backup script
- `restore-k8s.sh`: Kubernetes cluster restoration
- `k3s.sh`: K3s installation and management notes
- `fix_redis.sh`: Redis troubleshooting script

---

## 🔧 Terraform Configuration

### SSH Access Configuration

```hcl
locals {
  ssh_host        = "192.168.1.2"
  ssh_user        = "junv"
  ssh_port        = 22422  # Custom SSH port
  ssh_private_key = file("~/.ssh/id_ed25519")
}
```

### Terraform Providers

1. **ssh** (loafoe/ssh v2.6.0): Remote command execution
2. **kubernetes** (hashicorp/kubernetes v2.27.0): K8s resource management
3. **helm** (hashicorp/helm v2.12.1): Helm chart deployment
4. **kubectl** (gavinbunney/kubectl v1.14.0): Raw K8s manifests

### Storage Strategy

- **Local Path Storage**: `/mnt/k8s/` on server-3 (192.168.1.2)
- **Storage Class**: `local-path` (K3s default)
- **Node Affinity**: Resources pinned to `server-3` hostname
- **Examples**:
  - CrowdSec: `/mnt/k8s/crowdsec/`
  - PostgreSQL: `/mnt/k8s/postgres/`
  - Redis: `/mnt/k8s/redis/`

---

## 🔐 Security & Access

### SSH Configuration

- **Port**: 22422 (non-standard for security)
- **Key**: Ed25519 (~/.ssh/id_ed25519)
- **User**: junv
- **Host**: 192.168.1.2

### Network Security

- **Local Network**: 192.168.1.0/24 (trusted)
- **CrowdSec**: Monitors external threats
- **iptables**: Previously attempted but removed due to kubectl connectivity issues

### CrowdSec Configuration

- **Monitored Logs**:
  - NGINX Ingress Controller logs (container logs with poll_without_inotify)
  - SSH attempts: /var/log/auth.log
  - System logs: /var/log/kern.log, /var/log/syslog

- **Collections Installed**:
  - crowdsecurity/nginx
  - crowdsecurity/base-http-scenarios
  - crowdsecurity/linux
  - crowdsecurity/sshd
  - crowdsecurity/iptables
  - crowdsecurity/endlessh

- **Enrollment**: junv-home-k3s-cluster (enrolled in CrowdSec Console)

---

## 🚀 Common Operations

### Terraform Workflow

```bash
cd terraform/
terraform init
terraform plan
terraform apply
```

### Kubernetes Access

```bash
# Copy kubeconfig from server
scp -P 22422 junv@192.168.1.2:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i '' 's/127.0.0.1/192.168.1.2/g' ~/.kube/config

# Or use Terraform local
# Defined in main.tf as local.kubeconfig_cmd
```

### CrowdSec Commands

```bash
# View metrics
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli metrics

# List decisions (bans)
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli decisions list

# List bouncers
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli bouncers list

# Quick status view
./terraform/view_crowdsec.sh
```

### fail2ban Commands

```bash
# View status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status"

# SSH jail status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd"

# List banned IPs
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd | grep 'Banned IP'"

# Unban an IP
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client set sshd unbanip <IP>"

# Quick status view
./terraform/view_fail2ban.sh
```

### Backup/Restore

```bash
# Backup cluster
./backup-k8s.sh

# Restore cluster
./restore-k8s.sh
```

---

## 📦 Helm Charts Used

| Chart | Repository | Version | Namespace | Purpose |
|-------|-----------|---------|-----------|---------|
| crowdsec | https://crowdsecurity.github.io/helm-charts | 0.20.0 | crowdsec | Security engine |
| cert-manager | https://charts.jetstack.io | v1.17.2 | cert-manager | Certificate management |
| ingress-nginx | https://kubernetes.github.io/ingress-nginx | 4.10.5 | ingress-nginx | Ingress controller |
| argocd | https://argoproj.github.io/argo-helm | latest | argocd | GitOps CD |
| kube-prometheus-stack | prometheus-community | latest | default | Monitoring |
| redis | bitnami | latest | db | Cache/queue |

---

## 🗄️ Persistent Storage Locations

All persistent data stored on **server-3** at `192.168.1.2:/mnt/k8s/`

```
/mnt/k8s/
├── crowdsec/
│   ├── lapi-config/  (1Gi)
│   └── lapi-data/    (2Gi)
├── postgres/
├── redis/
└── [other services]
```

---

## 🌐 Domain Configuration

- **Primary Domain**: junv.cc
- **Subdomains**:
  - argo.junv.cc (ArgoCD)
  - [other services configured via Ingress resources]

---

## 🔍 Important Patterns & Conventions

### Namespace Organization

- `crowdsec`: Security monitoring
- `ingress-nginx`: Ingress controller
- `cert-manager`: Certificate management
- `argocd`: GitOps deployment
- `home-apps`: Home automation apps
- `media`: Media server apps
- `db`: Database services
- `default`: Legacy prometheus/monitoring

### Resource Naming

- PersistentVolumes: `{service}-{type}-pv` (e.g., `crowdsec-lapi-config-pv`)
- Deployments: Usually match service name
- Secrets: `{service}-credentials` or `{service}-{purpose}`

### Storage Classes

- `local-path`: K3s default local storage (most common)
- `nfs-client`: NFS-based storage (if NFS provisioner installed)

---

## ⚠️ Known Issues & Fixes

### Issue: iptables blocking kubectl

**Problem**: CrowdSec iptables rules blocked kubectl access to port 6443
**Solution**: Disabled iptables integration in `crowdsec.tf` (commented out `ssh_resource.iptables_logging`)
**Status**: CrowdSec still works via log analysis without iptables

### Issue: Container log symlinks

**Problem**: CrowdSec warning about inotify polling with symlinked container logs
**Solution**: Added `poll_without_inotify: true` to NGINX Ingress acquisition config
**Status**: Resolved

### Issue: SSH timeout during Terraform apply

**Problem**: SSH connection timeout when iptables rules were being applied
**Root Cause**: iptables port scan detection was blocking Terraform's own SSH connections
**Solution**: Removed iptables integration entirely

---

## 🔗 External Dependencies

- **NFS Server**: 192.168.1.5:/fs/1000/nfs/k8s (for NFS storage provisioner)
- **External etcd Cluster**:
  - 192.168.1.6:2379
  - 192.168.1.10:2379
  - 192.168.1.11:2379
- **CrowdSec Console**: https://app.crowdsec.net

---

## 📝 Migration Notes

### From Docker Compose to Kubernetes

- **legacy-docker-compose/**: Contains old Docker Compose configurations
- Most services migrated to Kubernetes YAML manifests
- Some services archived if no longer needed

### From Manual kubectl to Terraform

- **apply.sh**: Contains legacy manual kubectl commands
- Gradually being replaced by Terraform resources
- Reference for secret creation and one-time setup tasks

---

## 🎯 Development Workflow

### Adding a New Service

1. Create YAML manifest in appropriate directory (home-apps/, media/, etc.)
2. Define namespace, deployment, service, ingress
3. If needs persistence: create PV/PVC with `/mnt/k8s/{service}/` path
4. If needs TLS: add cert-manager annotations to Ingress
5. If monitoring needed: add ServiceMonitor resource
6. Consider adding to ArgoCD for GitOps deployment

### Modifying Terraform Infrastructure

1. Edit appropriate `.tf` file in `terraform/`
2. Run `terraform plan` to preview changes
3. Run `terraform apply` to apply changes
4. For SSH-based resources: ensure connectivity to 192.168.1.2:22422

### Debugging

- **Pod logs**: `kubectl logs -n {namespace} {pod-name}`
- **Pod shell**: `kubectl exec -it -n {namespace} {pod-name} -- /bin/bash`
- **Events**: `kubectl get events -n {namespace} --sort-by='.lastTimestamp'`
- **CrowdSec**: `./terraform/view_crowdsec.sh`

---

## 🤖 AI Assistant Guidelines

When assisting with this repository:

1. **Always consider the namespace** - resources are organized by namespace
2. **Use Terraform when possible** - prefer Terraform over manual kubectl
3. **Respect storage patterns** - use `/mnt/k8s/{service}/` for persistence
4. **Check SSH port** - always use port 22422, not default 22
5. **Local network trust** - 192.168.1.0/24 is trusted, don't block it
6. **CrowdSec integration** - avoid iptables, use log-based monitoring only
7. **Node affinity** - resources should be pinned to "server-3" hostname
8. **Poll without inotify** - use for container logs (symlinks)
9. **Ingress patterns** - use cert-manager annotations for TLS
10. **Secret management** - reference existing secrets in apply.sh for creation patterns

---

## 📚 Reference Documentation

- **K3s Docs**: https://docs.k3s.io/
- **CrowdSec Docs**: https://docs.crowdsec.net/
- **NGINX Ingress**: https://kubernetes.github.io/ingress-nginx/
- **cert-manager**: https://cert-manager.io/
- **ArgoCD**: https://argo-cd.readthedocs.io/
- **Terraform Kubernetes Provider**: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs

---

## 🏷️ Tags & Keywords

`kubernetes` `k3s` `terraform` `homelab` `nginx-ingress` `crowdsec` `argocd` `cert-manager` `prometheus` `grafana` `self-hosted` `infrastructure-as-code` `gitops` `security` `monitoring` `media-server` `home-automation`

---

**Last Updated**: October 5, 2025
**Maintainer**: junv (wahyd4)
**Repository**: https://github.com/wahyd4/home-docker