links/CELERY_TO_APSCHEDULER_MIGRATION.md

# Migration from Celery to APScheduler

## Summary

This project has been migrated from using Celery (with Redis broker) to APScheduler for background task processing. This simplifies the deployment architecture by running all background tasks in-process with the Django application, eliminating the need for separate worker containers and Redis infrastructure.

## Key Changes

### 1. Dependencies (`pyproject.toml`)
- **Removed**: `celery>=5.3.0`, `redis>=4.5.0`, `flower>=2.0.0`
- **Added**: `apscheduler>=3.10.0`

### 2. New Files Created
- `core/scheduler.py`: APScheduler configuration and initialization
- `core/apps.py`: Django app configuration to start scheduler on app ready
- `core/__init__.py`: App configuration reference

### 3. Modified Files

#### `links/tasks.py`
- Removed `@shared_task` decorators
- Changed function signatures to accept `retry_count` parameter
- Replaced Celery's `.delay()` and `.retry()` with:
  - Threading for immediate background execution
  - APScheduler for delayed/scheduled retries
- Updated retry logic to use APScheduler's job scheduling

#### `links/page_views.py`
- Added `from threading import Thread` import
- Replaced all `task.delay()` calls with thread-based execution
- Updated screenshot creation to use threading

#### `core/settings.py`
- Removed Celery configuration variables:
  - `CELERY_BROKER_URL`
  - `CELERY_RESULT_BACKEND`
  - `CELERY_ACCEPT_CONTENT`
  - `CELERY_TASK_SERIALIZER`
  - `CELERY_RESULT_SERIALIZER`
  - `CELERY_TIMEZONE`
  - `CELERYBEAT_SCHEDULE_FILENAME`
  - `CELERY_BEAT_SCHEDULE`
- Added `core.apps.CoreConfig` to `INSTALLED_APPS`

#### `docker-compose.yml`
- Removed services:
  - `celery_worker`
  - `celery_beat`
  - `celery_flower`
  - `redis`
- Removed Celery environment variables from `web` service
- Removed volume mount for `/app/data/celery`

#### `k8s/manifest.yaml`
- Removed environment variables:
  - `CELERY_BROKER_URL`
  - `CELERY_RESULT_BACKEND`
- Commented out worker deployment (lines 271-362)

#### `core/celery.py`
- **Deleted**: No longer needed

### 4. Documentation Updates (`agents.md`)
- Updated architecture diagram
- Replaced Celery sections with APScheduler documentation
- Updated troubleshooting guide
- Updated command reference
- Updated environment variables documentation

## How It Works Now

### Task Execution

#### Immediate Background Execution
Tasks that need to run immediately use Python threading:
```python
from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()
```

#### Delayed/Scheduled Execution
Tasks that need to be retried or scheduled use APScheduler:
```python
from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)
```

#### Periodic Tasks
Registered in `core/apps.py` when Django starts:
```python
scheduler.add_job(
    schedule_pending_pages,
    'interval',
    seconds=120,
    id='schedule_pending_pages',
    replace_existing=True
)
```

### Scheduler Configuration
- **Executor**: ThreadPoolExecutor with 20 threads
- **Max Instances**: 3 per job
- **Misfire Grace Time**: 300 seconds
- **Timezone**: Inherited from Django's `TIME_ZONE` setting

## Benefits

1. **Simplified Deployment**: No need for separate worker containers
2. **Reduced Infrastructure**: No Redis dependency for task queue
3. **Easier Development**: Everything runs in one process
4. **Lower Resource Usage**: Fewer containers to manage
5. **Simpler Debugging**: All logs in one place

## Migration Checklist

- [x] Update dependencies in `pyproject.toml`
- [x] Create APScheduler configuration
- [x] Update task functions
- [x] Update task callers
- [x] Remove Celery configuration from settings
- [x] Update Docker Compose
- [x] Update Kubernetes manifests
- [x] Update documentation
- [x] Test local development
- [ ] Test Docker build
- [ ] Test Kubernetes deployment

## Testing

### Local Development
```bash
source .venv/bin/activate
uv sync
python manage.py migrate
python manage.py runserver
```

The scheduler will start automatically and you should see log messages indicating:
- "APScheduler started"
- "Scheduled periodic task: schedule_pending_pages"

### Docker Development
```bash
docker-compose up web
```

### Production (Kubernetes)
```bash
kubectl apply -f k8s/manifest.yaml
```

Monitor logs to ensure scheduler starts correctly.

## Rollback Plan

If needed to rollback:
1. Revert changes to `pyproject.toml` (restore Celery dependencies)
2. Restore `core/celery.py`
3. Revert changes to `links/tasks.py` (restore `@shared_task` decorators)
4. Revert changes to `links/page_views.py` (restore `.delay()` calls)
5. Restore Celery configuration in `core/settings.py`
6. Restore worker services in `docker-compose.yml` and `k8s/manifest.yaml`
7. Run `uv sync` to install Celery dependencies

## Notes

- APScheduler stores job state in memory, so scheduled jobs are lost on restart
- For production with multiple replicas, consider using a persistent job store if needed
- Thread pool size (20) can be adjusted in `core/scheduler.py` based on workload
- The periodic task runs every 120 seconds to check for pending pages