links/agents.md

Agents Guide - URL Manager

This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.

Project Overview

URL Manager (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.

Tech Stack

• Backend: Django 5.1, Django REST Framework, APScheduler
• Frontend: Tailwind CSS, AlpineJS (minimal JS)
• Database: SQLite (default, configurable)
• Task Queue: APScheduler (in-process background scheduler)
• Storage: Local filesystem or AWS S3/Cloudflare R2
• Web Automation: Selenium with Chromium (for screenshots and scraping)
• Mobile: SwiftUI iOS app with Core Data
• Deployment: Kubernetes (k8s manifests included)

Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                      Web Interface                          │
│         (Django Templates + Tailwind CSS)                   │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Django Application                        │
│  ┌──────────────┬──────────────┬──────────────────────┐    │
│  │ Links Module │ Pages Module │ Collections Module   │    │
│  │              │              │ (Images, Posts)      │    │
│  └──────────────┴──────────────┴──────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │           REST API (DRF ViewSets)                  │    │
│  └────────────────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│       Background Tasks (APScheduler + Threading)            │
│  - Screenshot capture (background threads)                  │
│  - Page metadata extraction (background threads)            │
│  - Periodic task scheduling (APScheduler)                   │
└─────────────────────────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Storage Layer                             │
│  - Local filesystem (default)                               │
│  - Cloudflare R2 / AWS S3 (optional)                        │
└─────────────────────────────────────────────────────────────┘

Data Models

Core Models (links/models.py)

Link

The central model for URL management:

• alias: Unique slug identifier for the short link
• original_url: Target URL (supports templates with {param,default=value})
• text: Markdown content (for custom pages)
• link_type: LINK (regular URL) or CUSTOM (markdown content)
• click_count: Usage analytics
• tags: Many-to-many relationship with Tag model
• description: Optional description

Template URL Feature: URLs can contain parameters like https://example.com/{query,default=test} that are resolved at access time.

Page

Bookmarked pages with auto-extracted metadata:

• url: Original page URL
• title: Auto-extracted or manual
• summary: Auto-extracted description
• content: Full page content (optional)
• screenshot: Associated Screenshot model
• tags: Many-to-many with Tag
• processing_status: PENDING, PROCESSING, COMPLETED, FAILED
• priority: For async processing queue

Tag

Hierarchical tagging system:

• name: Tag name
• slug: URL-friendly slug
• description: Optional description
• icon: Optional icon
• parent: Self-referential for hierarchy
• color: UI color code

ImageCollection & Image

Image gallery management:

• Collections group related images
• Support for Cloudflare R2 / S3 storage
• Automatic thumbnail generation
• Bulk upload capabilities

Post

Blog-like content management:

• Markdown content support
• Tag categorization
• Publication status tracking

mini_apps

A collection of mini apps which can be located at mini_apps_views.py

Analytics Models

• ClickLog: Individual click tracking with timestamps
• LinkChangeLog: Audit trail for modifications

API Architecture

REST API (Django REST Framework)

Located in various *_views.py files with corresponding *_urls.py:

Image API (api_views.py)

• ImageCollectionViewSet: CRUD for image collections
  • Custom action: upload_images - Bulk image upload with R2/S3 integration
• ImageViewSet: Image management with descriptions
• MusicViewSet: Music file management

Page API (page_views.py)

• PageViewSet: Full CRUD for bookmarked pages
  • screenshot action: Trigger screenshot capture
  • extract_metadata action: Re-extract page metadata

Post API (post_views.py)

• PostViewSet: Blog post management with markdown rendering

API Documentation

• OpenAPI 3.1 spec available at /static/openapi.yaml
• Documents all endpoints, request/response schemas, and authentication

Background Task System (APScheduler)

The application uses APScheduler for background task scheduling, running in-process with the Django application. This eliminates the need for separate worker processes and Redis broker.

Tasks (links/tasks.py)

process_page(page_id, retry_count=0)

Asynchronously fetches page metadata:

1. Downloads page HTML
2. Extracts title, description, meta tags
3. Stores content in Page model
4. Updates processing status
5. Triggers screenshot capture

Features:

• Retry logic with exponential backoff (Fibonacci sequence)
• Timeout protection (20s)
• BeautifulSoup for HTML parsing
• Multiple metadata extraction strategies (title, h1, og:tags)
• Runs in background thread for non-blocking execution

capture_screenshot(page_id, screenshot_id, retry_count=0)

Captures website screenshots:

1. Launches headless Chromium via Selenium
2. Configures viewport and options
3. Takes screenshot
4. Uploads to storage (R2/S3 or local)
5. Updates Screenshot model

Configuration:

• Headless mode
• Custom user agent
• 60s timeout
• Supports custom viewport sizes
• Error handling and retry logic via APScheduler
• Runs in background thread

schedule_pending_pages()

Periodic maintenance task:

• Runs every 120 seconds
• Checks for pages with PENDING status
• Schedules processing for pending pages
• Respects retry limits and backoff

APScheduler Configuration

# core/scheduler.py
scheduler = BackgroundScheduler(
    executors={'default': ThreadPoolExecutor(20)},
    job_defaults={
        'coalesce': False,
        'max_instances': 3,
        'misfire_grace_time': 300
    },
    timezone=settings.TIME_ZONE
)

Initialization in core/apps.py:

• Scheduler starts automatically when Django application starts
• Periodic jobs registered in CoreConfig.ready() method
• No separate worker process needed

Task Execution

Tasks are executed using Python threading for immediate background execution:

from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()

For scheduled/delayed tasks, APScheduler is used:

from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)

Key Features & Implementation

1. Template URL Processing

File: links/models.py - Link.get_template_parameters()

Extracts and processes URL parameters:

# URL: https://search.com/{query,default=test}&lang={lang,default=en}
# Renders to: https://search.com/python&lang=en (with query="python")

2. Asynchronous Page Processing

Files: links/tasks.py, links/page_views.py

When a page is bookmarked:

1. Page record created with processing_status=PENDING
2. Background thread started for metadata extraction
3. Background worker fetches and parses page
4. Title, description, content extracted
5. Screenshot capture triggered in separate thread
6. Status updated to COMPLETED

3. Search System

File: links/search_views.py

Multi-field search across:

• Link aliases, URLs, descriptions
• Page titles, content, summaries
• Tag names
• Full-text search with filtering
• Tag-based filtering
• Type-based filtering (links, pages, posts)

4. Storage Abstraction

File: links/storage.py - R2Storage

Unified interface for:

• Local filesystem storage
• Cloudflare R2 (S3-compatible)
• AWS S3

Environment variables:

• R2_ENDPOINT_URL
• R2_ACCESS_KEY_ID
• R2_SECRET_ACCESS_KEY
• R2_BUCKET_NAME

5. i18n Support

Locales: English (en), Simplified Chinese (zh_Hans)

Translation files in locale/ directory. Uses Django's i18n framework.

Development Guide for AI Agents

Common Tasks

Adding a New Model Field

1. Update model in links/models.py
2. Create migration: python manage.py makemigrations
3. Apply migration: python manage.py migrate
4. Update serializer in links/serializers.py (if API exposed)
5. Update forms in links/forms.py (if form-based)
6. Update templates in links/templates/

Adding a New API Endpoint

1. Add serializer in links/serializers.py
2. Create ViewSet in appropriate *_views.py file
3. Register route in *_urls.py
4. Update OpenAPI spec in static/openapi.yaml
5. Add tests

Adding a Background Task

1. Define task function in links/tasks.py (no decorator needed)
2. For immediate execution, use threading:

   from threading import Thread
   thread = Thread(target=task_function, args=(arg1,))
   thread.daemon = True
   thread.start()
   

3. For delayed/scheduled execution, use APScheduler:

   from core.scheduler import scheduler
   scheduler.add_job(task_function, 'date', run_date=run_time, args=[arg1])
   

4. For periodic tasks, add to core/apps.py in CoreConfig.ready():

   scheduler.add_job(
       task_function,
       'interval',
       seconds=120,
       id='task_id',
       replace_existing=True
   )
   

5. Add logging for debugging
6. Implement retry logic if needed

Adding a New Template View

1. Create view function in links/views.py or create new view file
2. Add URL pattern to links/urls.py or appropriate urls file
3. Create template in links/templates/
4. Add i18n translation strings
5. Update navigation if needed

Code Style Guidelines

• Python: Follow PEP 8, use Black formatter (line length: 100)
• Imports: Use isort with Black profile
• Type hints: Encouraged for new code
• Docstrings: Use for complex functions and classes
• Logging: Use Django's logging framework

import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message")
logger.error("Error message", exc_info=True)

Testing

Framework: pytest with pytest-django

Run tests:

pytest
pytest links/tests/test_models.py
pytest -k "test_link_creation"

Database

Default: SQLite at data/db.sqlite3

Migrations managed with Django migrations. Always create migrations for model changes:

python manage.py makemigrations
python manage.py migrate

Static Assets

Tailwind CSS: Compiled via django-tailwind

Development:

python manage.py tailwind start  # Watch mode

Production:

python manage.py tailwind build  # Minified build
python manage.py collectstatic   # Collect to staticfiles/

Docker Development

docker-compose.yml Services

• web: Django development server (port 8000) with APScheduler running in-process
• node: Tailwind CSS compiler

Note: Redis, Celery worker, and Celery beat services have been removed as they are no longer needed.

Helper Script: docker.sh

./docker.sh build      # Build images
./docker.sh start      # Start all services
./docker.sh stop       # Stop all services
./docker.sh logs       # View logs
./docker.sh migrate    # Run migrations
./docker.sh shell      # Django shell

Environment Variables (.env)

Create .env file:

DJANGO_SETTINGS_MODULE=core.settings
SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=*

# Optional: R2/S3 Storage
R2_ENDPOINT_URL=https://...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=...

iOS App (Heygo)

Architecture

• SwiftUI for declarative UI
• Core Data for local persistence
• MVVM pattern with Combine
• Charts framework for analytics

Syncing Strategy

Currently standalone (no backend sync). Future enhancement could:

1. Use Django REST API
2. Implement OAuth authentication
3. Sync via background refresh
4. Conflict resolution strategy

Key Files

• mobile/Heygo/HeygoApp.swift: App entry point
• mobile/Heygo/Views/: SwiftUI views
• mobile/Heygo/ViewModels/LinkViewModel.swift: Business logic
• mobile/Heygo/Models/: Core Data models

Deployment

Production Dockerfile

Multi-stage build:

1. Builder stage: Install dependencies, compile static assets, build translations
2. Production stage: Slim image with only runtime dependencies

Runtime requirements:

• Python 3.12
• Chromium + ChromeDriver (for screenshots)

Kubernetes

Manifests in k8s/manifest.yaml:

• Deployment for web service (includes APScheduler)
• Service for load balancing
• ConfigMap for configuration
• PersistentVolumeClaim for data

Note: Worker deployment is no longer needed as tasks run in-process.

Environment Setup

Production checklist:

• Set DEBUG=False
• Configure SECRET_KEY (strong random value)
• Set ALLOWED_HOSTS to actual domains
• Configure database (PostgreSQL recommended for production)
• Configure R2/S3 for media storage
• Set up SSL/TLS termination
• Configure backup strategy
• Set up monitoring and logging

Common Troubleshooting

Screenshots Not Generating

1. Check Chromium installation: which chromium
2. Check Django application logs for task errors
3. Verify APScheduler is running (check startup logs)
4. Review application logs for screenshot task errors
5. Verify storage configuration (R2/S3 credentials)

Metadata Extraction Failing

1. Check website accessibility (some sites block bots)
2. Verify timeout settings (increase if needed)
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
4. Review error logs in application logs

Migration Issues

1. Check for unapplied migrations: python manage.py showmigrations
2. Look for migration conflicts
3. Use python manage.py migrate --fake-initial cautiously
4. For complex issues, may need to squash migrations

Performance Issues

1. Add database indexes for frequently queried fields
2. Implement caching (Django cache framework)
3. Tune APScheduler thread pool size if needed (default: 20 threads)
4. Use database connection pooling
5. Enable query optimization (select_related, prefetch_related)

File Structure Guide

Core Django App Structure

core/
├── settings.py       # Django settings, installed apps, middleware
├── urls.py          # Main URL routing
├── scheduler.py     # APScheduler configuration
├── apps.py          # App configuration and scheduler initialization
└── middleware.py    # Custom middleware (locale, etc.)

links/
├── models.py        # Data models (Link, Page, Tag, etc.)
├── views.py         # Main template views
├── api_views.py     # REST API viewsets (Image, Music)
├── page_views.py    # Page-specific views and API
├── post_views.py    # Post/blog views and API
├── collection_views.py  # Collection management
├── search_views.py  # Search functionality
├── tag_views.py     # Tag management
├── forms.py         # Django forms
├── serializers.py   # DRF serializers
├── tasks.py         # Background task functions (APScheduler)
├── storage.py       # Storage abstraction (R2/S3)
├── urls.py          # Links app URL routing
├── *_urls.py        # Feature-specific URL routing
└── templates/       # Django templates

new_theme/
├── static/          # Tailwind compiled CSS
├── static_src/      # Tailwind source files
└── templates/       # Theme-specific templates

templates/
├── base.html        # Base template with navigation
└── base_blank.html  # Minimal base template

Key Configuration Files

• pyproject.toml: Python dependencies (managed by uv)
• uv.lock: Locked dependency versions
• package.json: Node.js dependencies (Tailwind)
• tailwind.config.js: Tailwind configuration
• docker-compose.yml: Development environment
• Dockerfile: Production image
• Dockerfile.local: Development image

API Authentication (Currently no auth is enabled, don't need to consider)

Currently, the API may not have authentication enabled. To add:

1. Add Django REST framework token authentication:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ]
}

2. Generate tokens for users:

from rest_framework.authtoken.models import Token
token = Token.objects.create(user=user)

Performance Optimization Tips

Database Queries

# Use select_related for foreign keys
pages = Page.objects.select_related('screenshot').all()

# Use prefetch_related for many-to-many
links = Link.objects.prefetch_related('tags').all()

# Add indexes to models
class Page(models.Model):
    url = models.URLField(db_index=True)  # Add index

Caching

from django.core.cache import cache

# Cache expensive operations
result = cache.get('key')
if result is None:
    result = expensive_operation()
    cache.set('key', result, timeout=3600)

APScheduler Task Optimization

# Configure thread pool size in core/scheduler.py
executors = {
    'default': ThreadPoolExecutor(20),  # Adjust based on workload
}

# Add job with proper configuration
scheduler.add_job(
    my_task,
    'interval',
    seconds=60,
    max_instances=3,  # Limit concurrent instances
    id='unique_job_id',
    replace_existing=True
)

# For one-time delayed tasks
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=300)
scheduler.add_job(my_task, 'date', run_date=run_date, args=[arg1])

Security Considerations

1. CSRF Protection: Enabled by default, ensure templates use {% csrf_token %}
2. SQL Injection: Use Django ORM (parameterized queries)
3. XSS: Django auto-escapes templates, use |safe filter cautiously
4. File Upload: Validate file types, scan for malware, limit sizes
5. API Rate Limiting: Implement throttling with DRF throttle classes
6. Secrets Management: Use environment variables, never commit secrets

Monitoring and Logging

Application Logging

# Configure in settings.py
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/app/logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}

APScheduler Monitoring

• Check scheduler status: scheduler.running returns True/False
• List all jobs: scheduler.get_jobs()
• Monitor task execution via Django logs
• Track task execution times in application logs

Health Checks

Implement health check endpoint:

# views.py
def health_check(request):
    from core.scheduler import scheduler
    # Check database
    # Check APScheduler status
    scheduler_running = scheduler.running
    return JsonResponse({
        'status': 'healthy',
        'scheduler_running': scheduler_running
    })

Contributing Guidelines

When contributing code:

1. Follow existing code patterns and structure
2. Add tests for new features
3. Update documentation (README, this agents.md, docstrings)
4. Run formatters: black . and isort .
5. Ensure migrations are included for model changes
6. Update OpenAPI spec for API changes
7. Add i18n translation strings for user-facing text
8. Test in Docker environment before submitting

Useful Commands Reference

Django Management

Make sure enable enable .venv before running any commands through source .venv/bin/activate

python manage.py runserver              # Development server (APScheduler starts automatically)
python manage.py shell                  # Django shell
python manage.py dbshell               # Database shell
python manage.py createsuperuser       # Create admin user
python manage.py test                  # Run tests
python manage.py collectstatic         # Collect static files
python manage.py compilemessages       # Compile translations
python manage.py makemessages -l zh_Hans  # Extract translation strings

APScheduler (In-Process)

APScheduler starts automatically with Django. To interact with it:

# In Django shell
from core.scheduler import scheduler
scheduler.get_jobs()                    # List all scheduled jobs
scheduler.print_jobs()                  # Print job details
scheduler.running                       # Check if scheduler is running

UV Package Manager

uv pip install package_name            # Install package
uv pip install -r requirements.txt     # Install from requirements
uv pip freeze > requirements.txt       # Export requirements
uv sync                                # Sync from pyproject.toml

Resources

• Django Documentation: https://docs.djangoproject.com/
• Django REST Framework: https://www.django-rest-framework.org/
• Celery Documentation: https://docs.celeryq.dev/
• Tailwind CSS: https://tailwindcss.com/docs
• SwiftUI: https://developer.apple.com/documentation/swiftui/

Note:

• No need to generate extra summary guide or docs, unless I ask you to.