links/CELERY_TO_APSCHEDULER_MIGRATION.md

Migration from Celery to APScheduler

Summary

This project has been migrated from using Celery (with Redis broker) to APScheduler for background task processing. This simplifies the deployment architecture by running all background tasks in-process with the Django application, eliminating the need for separate worker containers and Redis infrastructure.

Key Changes

1. Dependencies (pyproject.toml)

• Removed: celery>=5.3.0, redis>=4.5.0, flower>=2.0.0
• Added: apscheduler>=3.10.0

2. New Files Created

• core/scheduler.py: APScheduler configuration and initialization
• core/apps.py: Django app configuration to start scheduler on app ready
• core/__init__.py: App configuration reference

3. Modified Files

links/tasks.py

• Removed @shared_task decorators
• Changed function signatures to accept retry_count parameter
• Replaced Celery's .delay() and .retry() with:
  • Threading for immediate background execution
  • APScheduler for delayed/scheduled retries
• Updated retry logic to use APScheduler's job scheduling

links/page_views.py

• Added from threading import Thread import
• Replaced all task.delay() calls with thread-based execution
• Updated screenshot creation to use threading

core/settings.py

• Removed Celery configuration variables:
  • CELERY_BROKER_URL
  • CELERY_RESULT_BACKEND
  • CELERY_ACCEPT_CONTENT
  • CELERY_TASK_SERIALIZER
  • CELERY_RESULT_SERIALIZER
  • CELERY_TIMEZONE
  • CELERYBEAT_SCHEDULE_FILENAME
  • CELERY_BEAT_SCHEDULE
• Added core.apps.CoreConfig to INSTALLED_APPS

docker-compose.yml

• Removed services:
  • celery_worker
  • celery_beat
  • celery_flower
  • redis
• Removed Celery environment variables from web service
• Removed volume mount for /app/data/celery

k8s/manifest.yaml

• Removed environment variables:
  • CELERY_BROKER_URL
  • CELERY_RESULT_BACKEND
• Commented out worker deployment (lines 271-362)

core/celery.py

• Deleted: No longer needed

4. Documentation Updates (agents.md)

• Updated architecture diagram
• Replaced Celery sections with APScheduler documentation
• Updated troubleshooting guide
• Updated command reference
• Updated environment variables documentation

How It Works Now

Task Execution

Immediate Background Execution

Tasks that need to run immediately use Python threading:

from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()

Delayed/Scheduled Execution

Tasks that need to be retried or scheduled use APScheduler:

from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)

Periodic Tasks

Registered in core/apps.py when Django starts:

scheduler.add_job(
    schedule_pending_pages,
    'interval',
    seconds=120,
    id='schedule_pending_pages',
    replace_existing=True
)

Scheduler Configuration

• Executor: ThreadPoolExecutor with 20 threads
• Max Instances: 3 per job
• Misfire Grace Time: 300 seconds
• Timezone: Inherited from Django's TIME_ZONE setting

Benefits

1. Simplified Deployment: No need for separate worker containers
2. Reduced Infrastructure: No Redis dependency for task queue
3. Easier Development: Everything runs in one process
4. Lower Resource Usage: Fewer containers to manage
5. Simpler Debugging: All logs in one place

Migration Checklist

• Update dependencies in pyproject.toml
• Create APScheduler configuration
• Update task functions
• Update task callers
• Remove Celery configuration from settings
• Update Docker Compose
• Update Kubernetes manifests
• Update documentation
• Test local development
• Test Docker build
• Test Kubernetes deployment

Testing

Local Development

source .venv/bin/activate
uv sync
python manage.py migrate
python manage.py runserver

The scheduler will start automatically and you should see log messages indicating:

• "APScheduler started"
• "Scheduled periodic task: schedule_pending_pages"

Docker Development

docker-compose up web

Production (Kubernetes)

kubectl apply -f k8s/manifest.yaml

Monitor logs to ensure scheduler starts correctly.

Rollback Plan

If needed to rollback:

1. Revert changes to pyproject.toml (restore Celery dependencies)
2. Restore core/celery.py
3. Revert changes to links/tasks.py (restore @shared_task decorators)
4. Revert changes to links/page_views.py (restore .delay() calls)
5. Restore Celery configuration in core/settings.py
6. Restore worker services in docker-compose.yml and k8s/manifest.yaml
7. Run uv sync to install Celery dependencies

Notes

• APScheduler stores job state in memory, so scheduled jobs are lost on restart
• For production with multiple replicas, consider using a persistent job store if needed
• Thread pool size (20) can be adjusted in core/scheduler.py based on workload
• The periodic task runs every 120 seconds to check for pending pages