home-docker/agents.md

# AI Agent Context: Home Docker Infrastructure

> **Purpose**: This document helps AI assistants (GitHub Copilot, Claude, ChatGPT, etc.) understand the repository structure, architecture, and context for better code assistance.

---

## 📋 Repository Overview

**Name**: `home-docker` (wahyd4/home-docker)
**Purpose**: Home media server and personal cloud infrastructure managed via Kubernetes (K3s) and Terraform
**Architecture**: Self-hosted Kubernetes cluster with NGINX Ingress, cert-manager, and various homelab applications
**Primary Technologies**: Kubernetes (K3s), Terraform, Docker, Helm, ArgoCD

---

## 🏗️ Infrastructure Architecture

### Kubernetes Cluster Setup

- **Distribution**: K3s (lightweight Kubernetes)
- **Control Plane**: 192.168.1.2 (server-3)
- **Container Runtime**: containerd 2.0.5-k3s2
- **Kubernetes Version**: v1.33.3+k3s1
- **OS**: Ubuntu 24.04.3 LTS
- **Network**: 192.168.1.0/24 (local network)

### Key Infrastructure Components

1. **Ingress Controller**: NGINX Ingress Controller (ingress-nginx namespace)
   - NodePort on ports 80/443
   - Handles all external traffic
   - Integrated with cert-manager for TLS
   - Domain *.junv.cc

2. **Certificate Management**: cert-manager
   - Let's Encrypt integration (letsencrypt-prod ClusterIssuer)
   - Automatic TLS certificate provisioning
   - Email: me@junv.cc

3. **GitOps**: ArgoCD
   - Continuous deployment from Git
   - Web UI: https://argo.junv.cc
   - Uses external Redis (redis-master.db.svc.cluster.local)

4. **Security**: Multi-layered protection

   **CrowdSec** (Community threat intelligence):
   - Monitors NGINX Ingress logs
   - SSH monitoring (port 22422)
   - System log analysis
   - Enrolled in CrowdSec Console
   - Storage: /mnt/k8s/crowdsec/ on server-3

   **fail2ban** (Local intrusion prevention):
   - SSH brute force protection (port 22422)
   - Ban time: 1 hour (3600 seconds)
   - Max retries: 5 attempts in 10 minutes
   - Recidive jail: 24 hour ban for repeat offenders
   - Ignores local network (192.168.1.0/24)

5. **Monitoring**: Prometheus + Grafana
   - Metrics collection
   - Custom dashboards
   - ServiceMonitor for various services

---

## 📁 Directory Structure

### Root Level Files

- **terraform/**: Terraform configurations for infrastructure as code
  - `main.tf`: K3s cluster setup, SSH configuration
  - `argocd.tf`: ArgoCD, cert-manager, NGINX Ingress deployment
  - `crowdsec.tf`: CrowdSec security engine setup
  - `fail2ban.tf`: fail2ban intrusion prevention setup
  - `postgres.tf`: PostgreSQL installation via SSH
  - `redis.tf`: Redis installation via SSH

- **home-apps/**: Home automation and productivity apps
  - `ddns.yaml`: Dynamic DNS clients (Cloudflare, Aliyun)
  - `n8n.yaml`: Workflow automation platform
  - `searxng.yaml`: Privacy-respecting search engine
  - `home.yaml`: Home dashboard application
  - `server-backup.yaml`: Backup automation

- **apps/**: General applications
  - `passkey-auth.yaml`: Authentication service

- **ai/**: AI/ML applications
  - `crawl4ai.yaml`: AI web crawler

- **media/**: Media server applications
  - `alist.yaml`: File listing service
  - `jackett.yaml`: Torrent indexer
  - `media.yaml`: Media server deployment
  - `nextcloud-pv.yaml`: Nextcloud persistent volume
  - `static-file.yaml`: Static file server

- **db/**: Database services
  - `redis-values.yaml`: Redis Helm values
  - `qdrant.yaml`: Vector database for AI

- **archive/**: Deprecated/archived configurations
  - Legacy configurations no longer in active use
  - Old services like Jellyfin, Plex, Metabase, etc.

### Key Scripts

- `apply.sh`: Manual kubectl apply commands (mostly deprecated in favor of Terraform)
- `backup-k8s.sh`: Kubernetes cluster backup script
- `restore-k8s.sh`: Kubernetes cluster restoration
- `k3s.sh`: K3s installation and management notes
- `fix_redis.sh`: Redis troubleshooting script

---

## 🔧 Terraform Configuration

### SSH Access Configuration

```hcl
locals {
  ssh_host        = "192.168.1.2"
  ssh_user        = "junv"
  ssh_port        = 22422  # Custom SSH port
  ssh_private_key = file("~/.ssh/id_ed25519")
}
```

### Terraform Providers

1. **ssh** (loafoe/ssh v2.6.0): Remote command execution
2. **kubernetes** (hashicorp/kubernetes v2.27.0): K8s resource management
3. **helm** (hashicorp/helm v2.12.1): Helm chart deployment
4. **kubectl** (gavinbunney/kubectl v1.14.0): Raw K8s manifests

### Storage Strategy

- **Local Path Storage**: `/mnt/k8s/` on server-3 (192.168.1.2)
- **Storage Class**: `local-path` (K3s default)
- **Node Affinity**: Resources pinned to `server-3` hostname
- **Examples**:
  - CrowdSec: `/mnt/k8s/crowdsec/`
  - PostgreSQL: `/mnt/k8s/postgres/`
  - Redis: `/mnt/k8s/redis/`

---

## 🔐 Security & Access

### SSH Configuration

- **Port**: 22422 (non-standard for security)
- **Key**: Ed25519 (~/.ssh/id_ed25519)
- **User**: junv
- **Host**: 192.168.1.2

### Network Security

- **Local Network**: 192.168.1.0/24 (trusted)
- **CrowdSec**: Monitors external threats
- **iptables**: Previously attempted but removed due to kubectl connectivity issues

### CrowdSec Configuration

- **Monitored Logs**:
  - NGINX Ingress Controller logs (container logs with poll_without_inotify)
  - SSH attempts: /var/log/auth.log
  - System logs: /var/log/kern.log, /var/log/syslog

- **Collections Installed**:
  - crowdsecurity/nginx
  - crowdsecurity/base-http-scenarios
  - crowdsecurity/linux
  - crowdsecurity/sshd
  - crowdsecurity/iptables
  - crowdsecurity/endlessh

- **Enrollment**: junv-home-k3s-cluster (enrolled in CrowdSec Console)

---

## 🚀 Common Operations

### Terraform Workflow

```bash
cd terraform/
terraform init
terraform plan
terraform apply
```

### Kubernetes Access

```bash
# Copy kubeconfig from server
scp -P 22422 junv@192.168.1.2:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i '' 's/127.0.0.1/192.168.1.2/g' ~/.kube/config

# Or use Terraform local
# Defined in main.tf as local.kubeconfig_cmd
```

### CrowdSec Commands

```bash
# View metrics
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli metrics

# List decisions (bans)
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli decisions list

# List bouncers
kubectl exec -n crowdsec deployment/crowdsec-lapi -- cscli bouncers list

# Quick status view
./terraform/view_crowdsec.sh
```

### fail2ban Commands

```bash
# View status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status"

# SSH jail status
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd"

# List banned IPs
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client status sshd | grep 'Banned IP'"

# Unban an IP
ssh -p 22422 junv@192.168.1.2 "sudo fail2ban-client set sshd unbanip <IP>"

# Quick status view
./terraform/view_fail2ban.sh
```

### Backup/Restore

```bash
# Backup cluster
./backup-k8s.sh

# Restore cluster
./restore-k8s.sh
```

---

## 📦 Helm Charts Used

| Chart | Repository | Version | Namespace | Purpose |
|-------|-----------|---------|-----------|---------|
| crowdsec | https://crowdsecurity.github.io/helm-charts | 0.20.0 | crowdsec | Security engine |
| cert-manager | https://charts.jetstack.io | v1.17.2 | cert-manager | Certificate management |
| ingress-nginx | https://kubernetes.github.io/ingress-nginx | 4.10.5 | ingress-nginx | Ingress controller |
| argocd | https://argoproj.github.io/argo-helm | latest | argocd | GitOps CD |
| kube-prometheus-stack | prometheus-community | latest | default | Monitoring |
| redis | bitnami | latest | db | Cache/queue |

---

## 🗄️ Persistent Storage Locations

All persistent data stored on **server-3** at `192.168.1.2:/mnt/k8s/`

```
/mnt/k8s/
├── crowdsec/
│   ├── lapi-config/  (1Gi)
│   └── lapi-data/    (2Gi)
├── postgres/
├── redis/
└── [other services]
```

---

## 🌐 Domain Configuration

- **Primary Domain**: junv.cc
- **Subdomains**:
  - argo.junv.cc (ArgoCD)
  - [other services configured via Ingress resources]

---

## 🔍 Important Patterns & Conventions

### Namespace Organization

- `crowdsec`: Security monitoring
- `ingress-nginx`: Ingress controller
- `cert-manager`: Certificate management
- `argocd`: GitOps deployment
- `home-apps`: Home automation apps
- `media`: Media server apps
- `db`: Database services
- `default`: Legacy prometheus/monitoring

### Resource Naming

- PersistentVolumes: `{service}-{type}-pv` (e.g., `crowdsec-lapi-config-pv`)
- Deployments: Usually match service name
- Secrets: `{service}-credentials` or `{service}-{purpose}`

### Storage Classes

- `local-path`: K3s default local storage (most common)
- `nfs-client`: NFS-based storage (if NFS provisioner installed)

---

## ⚠️ Known Issues & Fixes

### Issue: iptables blocking kubectl

**Problem**: CrowdSec iptables rules blocked kubectl access to port 6443
**Solution**: Disabled iptables integration in `crowdsec.tf` (commented out `ssh_resource.iptables_logging`)
**Status**: CrowdSec still works via log analysis without iptables

### Issue: Container log symlinks

**Problem**: CrowdSec warning about inotify polling with symlinked container logs
**Solution**: Added `poll_without_inotify: true` to NGINX Ingress acquisition config
**Status**: Resolved

### Issue: SSH timeout during Terraform apply

**Problem**: SSH connection timeout when iptables rules were being applied
**Root Cause**: iptables port scan detection was blocking Terraform's own SSH connections
**Solution**: Removed iptables integration entirely

---

## 🔗 External Dependencies

- **NFS Server**: 192.168.1.5:/fs/1000/nfs/k8s (for NFS storage provisioner)
- **External etcd Cluster**:
  - 192.168.1.6:2379
  - 192.168.1.10:2379
  - 192.168.1.11:2379
- **CrowdSec Console**: https://app.crowdsec.net

---

## 📝 Migration Notes

### From Docker Compose to Kubernetes

- **legacy-docker-compose/**: Contains old Docker Compose configurations
- Most services migrated to Kubernetes YAML manifests
- Some services archived if no longer needed

### From Manual kubectl to Terraform

- **apply.sh**: Contains legacy manual kubectl commands
- Gradually being replaced by Terraform resources
- Reference for secret creation and one-time setup tasks

---

## 🎯 Development Workflow

### Adding a New Service

1. Create YAML manifest in appropriate directory (home-apps/, media/, etc.)
2. Define namespace, deployment, service, ingress
3. If needs persistence: create PV/PVC with `/mnt/k8s/{service}/` path
4. If needs TLS: add cert-manager annotations to Ingress
5. If monitoring needed: add ServiceMonitor resource
6. Consider adding to ArgoCD for GitOps deployment

### Modifying Terraform Infrastructure

1. Edit appropriate `.tf` file in `terraform/`
2. Run `terraform plan` to preview changes
3. Run `terraform apply` to apply changes
4. For SSH-based resources: ensure connectivity to 192.168.1.2:22422

### Debugging

- **Pod logs**: `kubectl logs -n {namespace} {pod-name}`
- **Pod shell**: `kubectl exec -it -n {namespace} {pod-name} -- /bin/bash`
- **Events**: `kubectl get events -n {namespace} --sort-by='.lastTimestamp'`
- **CrowdSec**: `./terraform/view_crowdsec.sh`

---

## 🤖 AI Assistant Guidelines

When assisting with this repository:

1. **Always consider the namespace** - resources are organized by namespace
2. **Use Terraform when possible** - prefer Terraform over manual kubectl
3. **Respect storage patterns** - use `/mnt/k8s/{service}/` for persistence
4. **Check SSH port** - always use port 22422, not default 22
5. **Local network trust** - 192.168.1.0/24 is trusted, don't block it
6. **CrowdSec integration** - avoid iptables, use log-based monitoring only
7. **Node affinity** - resources should be pinned to "server-3" hostname
8. **Poll without inotify** - use for container logs (symlinks)
9. **Ingress patterns** - use cert-manager annotations for TLS
10. **Secret management** - reference existing secrets in apply.sh for creation patterns

---

## 📚 Reference Documentation

- **K3s Docs**: https://docs.k3s.io/
- **CrowdSec Docs**: https://docs.crowdsec.net/
- **NGINX Ingress**: https://kubernetes.github.io/ingress-nginx/
- **cert-manager**: https://cert-manager.io/
- **ArgoCD**: https://argo-cd.readthedocs.io/
- **Terraform Kubernetes Provider**: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs

---

## 🏷️ Tags & Keywords

`kubernetes` `k3s` `terraform` `homelab` `nginx-ingress` `crowdsec` `argocd` `cert-manager` `prometheus` `grafana` `self-hosted` `infrastructure-as-code` `gitops` `security` `monitoring` `media-server` `home-automation`

---

**Last Updated**: October 5, 2025
**Maintainer**: junv (wahyd4)
**Repository**: https://github.com/wahyd4/home-docker