links/agents.md

Agents Guide - URL Manager

This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.

Project Overview

URL Manager (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.

Tech Stack

• Backend: Django 5.1, Django REST Framework, Celery
• Frontend: Tailwind CSS, AlpineJS (minimal JS)
• Database: SQLite (default, configurable)
• Task Queue: Celery with Redis broker
• Storage: Local filesystem or AWS S3/Cloudflare R2
• Web Automation: Selenium with Chromium (for screenshots and scraping)
• Mobile: SwiftUI iOS app with Core Data
• Deployment: Kubernetes (k8s manifests included)

Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                      Web Interface                          │
│         (Django Templates + Tailwind CSS)                   │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Django Application                        │
│  ┌──────────────┬──────────────┬──────────────────────┐    │
│  │ Links Module │ Pages Module │ Collections Module   │    │
│  │              │              │ (Images, IPTV, Posts)│    │
│  └──────────────┴──────────────┴──────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │           REST API (DRF ViewSets)                  │    │
│  └────────────────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│               Background Tasks (Celery)                     │
│  - Screenshot capture                                       │
│  - Page metadata extraction                                 │
│  - Periodic cleanup tasks                                   │
└─────────────────────────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Storage Layer                             │
│  - Local filesystem (default)                               │
│  - Cloudflare R2 / AWS S3 (optional)                        │
└─────────────────────────────────────────────────────────────┘

Data Models

Core Models (links/models.py)

Link

The central model for URL management:

• alias: Unique slug identifier for the short link
• original_url: Target URL (supports templates with {param,default=value})
• text: Markdown content (for custom pages)
• link_type: LINK (regular URL) or CUSTOM (markdown content)
• click_count: Usage analytics
• tags: Many-to-many relationship with Tag model
• description: Optional description

Template URL Feature: URLs can contain parameters like https://example.com/{query,default=test} that are resolved at access time.

Page

Bookmarked pages with auto-extracted metadata:

• url: Original page URL
• title: Auto-extracted or manual
• summary: Auto-extracted description
• content: Full page content (optional)
• screenshot: Associated Screenshot model
• tags: Many-to-many with Tag
• processing_status: PENDING, PROCESSING, COMPLETED, FAILED
• priority: For async processing queue

Tag

Hierarchical tagging system:

• name: Tag name
• slug: URL-friendly slug
• description: Optional description
• icon: Optional icon
• parent: Self-referential for hierarchy
• color: UI color code

ImageCollection & Image

Image gallery management:

• Collections group related images
• Support for Cloudflare R2 / S3 storage
• Automatic thumbnail generation
• Bulk upload capabilities

Post

Blog-like content management:

• Markdown content support
• Tag categorization
• Publication status tracking

mini_apps

A collection of mini apps which can be located at mini_apps_views.py

Analytics Models

• ClickLog: Individual click tracking with timestamps
• LinkChangeLog: Audit trail for modifications

API Architecture

REST API (Django REST Framework)

Located in various *_views.py files with corresponding *_urls.py:

Image API (api_views.py)

• ImageCollectionViewSet: CRUD for image collections
  • Custom action: upload_images - Bulk image upload with R2/S3 integration
• ImageViewSet: Image management with descriptions
• MusicViewSet: Music file management

Page API (page_views.py)

• PageViewSet: Full CRUD for bookmarked pages
  • screenshot action: Trigger screenshot capture
  • extract_metadata action: Re-extract page metadata

Post API (post_views.py)

• PostViewSet: Blog post management with markdown rendering

API Documentation

• OpenAPI 3.1 spec available at /static/openapi.yaml
• Documents all endpoints, request/response schemas, and authentication

Background Task System (Celery)

Tasks (links/tasks.py)

@shared_task: fetch_page_async

Asynchronously fetches page metadata:

1. Downloads page HTML
2. Extracts title, description, meta tags
3. Stores content in Page model
4. Updates processing status

Features:

• Retry logic with exponential backoff (Fibonacci sequence)
• Timeout protection (20s)
• BeautifulSoup for HTML parsing
• Multiple metadata extraction strategies (title, h1, og:tags)

@shared_task: capture_screenshot_async

Captures website screenshots:

1. Launches headless Chromium via Selenium
2. Configures viewport and options
3. Takes screenshot
4. Uploads to storage (R2/S3 or local)
5. Creates Screenshot model

Configuration:

• Headless mode
• Custom user agent
• 60s timeout
• Supports custom viewport sizes
• Error handling and retry logic

@shared_task: cleanup_old_screenshots

Periodic maintenance task:

• Removes screenshots older than configured retention period
• Cleans up orphaned files
• Maintains storage efficiency

Celery Configuration

# core/celery.py
CELERY_BROKER_URL = redis://redis:6379/0
CELERY_RESULT_BACKEND = redis://redis:6379/0

Beat schedule in settings.py for periodic tasks.

Key Features & Implementation

1. Template URL Processing

File: links/models.py - Link.get_template_parameters()

Extracts and processes URL parameters:

# URL: https://search.com/{query,default=test}&lang={lang,default=en}
# Renders to: https://search.com/python&lang=en (with query="python")

2. Asynchronous Page Processing

Files: links/tasks.py, links/views.py

When a page is bookmarked:

1. Page record created with processing_status=PENDING
2. Celery task dispatched for metadata extraction
3. Background worker fetches and parses page
4. Title, description, content extracted
5. Optional screenshot capture triggered
6. Status updated to COMPLETED

3. Search System

File: links/search_views.py

Multi-field search across:

• Link aliases, URLs, descriptions
• Page titles, content, summaries
• Tag names
• Full-text search with filtering
• Tag-based filtering
• Type-based filtering (links, pages, posts)

4. Storage Abstraction

File: links/storage.py - R2Storage

Unified interface for:

• Local filesystem storage
• Cloudflare R2 (S3-compatible)
• AWS S3

Environment variables:

• R2_ENDPOINT_URL
• R2_ACCESS_KEY_ID
• R2_SECRET_ACCESS_KEY
• R2_BUCKET_NAME

5. i18n Support

Locales: English (en), Simplified Chinese (zh_Hans)

Translation files in locale/ directory. Uses Django's i18n framework.

Development Guide for AI Agents

Common Tasks

Adding a New Model Field

1. Update model in links/models.py
2. Create migration: python manage.py makemigrations
3. Apply migration: python manage.py migrate
4. Update serializer in links/serializers.py (if API exposed)
5. Update forms in links/forms.py (if form-based)
6. Update templates in links/templates/

Adding a New API Endpoint

1. Add serializer in links/serializers.py
2. Create ViewSet in appropriate *_views.py file
3. Register route in *_urls.py
4. Update OpenAPI spec in static/openapi.yaml
5. Add tests

Adding a Background Task

1. Define task in links/tasks.py with @shared_task decorator
2. Call task with .delay() or .apply_async()
3. Configure retry logic and timeouts
4. Add logging for debugging
5. For periodic tasks, add to Celery beat schedule

Adding a New Template View

1. Create view function in links/views.py or create new view file
2. Add URL pattern to links/urls.py or appropriate urls file
3. Create template in links/templates/
4. Add i18n translation strings
5. Update navigation if needed

Code Style Guidelines

• Python: Follow PEP 8, use Black formatter (line length: 100)
• Imports: Use isort with Black profile
• Type hints: Encouraged for new code
• Docstrings: Use for complex functions and classes
• Logging: Use Django's logging framework

import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message")
logger.error("Error message", exc_info=True)

Testing

Framework: pytest with pytest-django

Run tests:

pytest
pytest links/tests/test_models.py
pytest -k "test_link_creation"

Database

Default: SQLite at data/db.sqlite3

Migrations managed with Django migrations. Always create migrations for model changes:

python manage.py makemigrations
python manage.py migrate

Static Assets

Tailwind CSS: Compiled via django-tailwind

Development:

python manage.py tailwind start  # Watch mode

Production:

python manage.py tailwind build  # Minified build
python manage.py collectstatic   # Collect to staticfiles/

Docker Development

docker-compose.yml Services

• web: Django development server (port 8000)
• node: Tailwind CSS compiler
• celery_worker: Background task processor
• celery_beat: Periodic task scheduler
• redis: Message broker and cache

Helper Script: docker.sh

./docker.sh build      # Build images
./docker.sh start      # Start all services
./docker.sh stop       # Stop all services
./docker.sh logs       # View logs
./docker.sh migrate    # Run migrations
./docker.sh shell      # Django shell

Environment Variables (.env)

Create .env file:

DJANGO_SETTINGS_MODULE=core.settings
SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=*

# Celery
CELERY_BROKER_URL=redis://redis:6379/0
CELERY_RESULT_BACKEND=redis://redis:6379/0

# Optional: R2/S3 Storage
R2_ENDPOINT_URL=https://...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=...

iOS App (Heygo)

Architecture

• SwiftUI for declarative UI
• Core Data for local persistence
• MVVM pattern with Combine
• Charts framework for analytics

Syncing Strategy

Currently standalone (no backend sync). Future enhancement could:

1. Use Django REST API
2. Implement OAuth authentication
3. Sync via background refresh
4. Conflict resolution strategy

Key Files

• mobile/Heygo/HeygoApp.swift: App entry point
• mobile/Heygo/Views/: SwiftUI views
• mobile/Heygo/ViewModels/LinkViewModel.swift: Business logic
• mobile/Heygo/Models/: Core Data models

Deployment

Production Dockerfile

Multi-stage build:

1. Builder stage: Install dependencies, compile static assets, build translations
2. Production stage: Slim image with only runtime dependencies

Runtime requirements:

• Python 3.12
• Chromium + ChromeDriver (for screenshots)
• Redis (external service)

Kubernetes

Manifests in k8s/manifest.yaml:

• Deployment for web service
• Service for load balancing
• ConfigMap for configuration
• Optional: Redis StatefulSet, PersistentVolumeClaim for data

Environment Setup

Production checklist:

• Set DEBUG=False
• Configure SECRET_KEY (strong random value)
• Set ALLOWED_HOSTS to actual domains
• Configure database (PostgreSQL recommended for production)
• Set up Redis instance
• Configure R2/S3 for media storage
• Set up SSL/TLS termination
• Configure backup strategy
• Set up monitoring and logging

Common Troubleshooting

Screenshots Not Generating

1. Check Chromium installation: which chromium
2. Verify Celery worker is running: docker ps or check logs
3. Check Redis connection
4. Review Celery worker logs for errors
5. Verify storage configuration (R2/S3 credentials)

Metadata Extraction Failing

1. Check website accessibility (some sites block bots)
2. Verify timeout settings (increase if needed)
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
4. Review error logs in Celery worker

Migration Issues

1. Check for unapplied migrations: python manage.py showmigrations
2. Look for migration conflicts
3. Use python manage.py migrate --fake-initial cautiously
4. For complex issues, may need to squash migrations

Performance Issues

1. Add database indexes for frequently queried fields
2. Implement caching (Django cache framework + Redis)
3. Optimize Celery task execution (tune concurrency)
4. Use database connection pooling
5. Enable query optimization (select_related, prefetch_related)

File Structure Guide

Core Django App Structure

core/
├── settings.py       # Django settings, installed apps, middleware
├── urls.py          # Main URL routing
├── celery.py        # Celery configuration
└── middleware.py    # Custom middleware (locale, etc.)

links/
├── models.py        # Data models (Link, Page, Tag, etc.)
├── views.py         # Main template views
├── api_views.py     # REST API viewsets (Image, Music)
├── page_views.py    # Page-specific views and API
├── post_views.py    # Post/blog views and API
├── iptv_views.py    # IPTV functionality
├── collection_views.py  # Collection management
├── search_views.py  # Search functionality
├── tag_views.py     # Tag management
├── forms.py         # Django forms
├── serializers.py   # DRF serializers
├── tasks.py         # Celery background tasks
├── storage.py       # Storage abstraction (R2/S3)
├── urls.py          # Links app URL routing
├── *_urls.py        # Feature-specific URL routing
└── templates/       # Django templates

new_theme/
├── static/          # Tailwind compiled CSS
├── static_src/      # Tailwind source files
└── templates/       # Theme-specific templates

templates/
├── base.html        # Base template with navigation
└── base_blank.html  # Minimal base template

Key Configuration Files

• pyproject.toml: Python dependencies (managed by uv)
• uv.lock: Locked dependency versions
• package.json: Node.js dependencies (Tailwind)
• tailwind.config.js: Tailwind configuration
• docker-compose.yml: Development environment
• Dockerfile: Production image
• Dockerfile.local: Development image

API Authentication (Currently no auth is enabled, don't need to consider)

Currently, the API may not have authentication enabled. To add:

1. Add Django REST framework token authentication:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ]
}

2. Generate tokens for users:

from rest_framework.authtoken.models import Token
token = Token.objects.create(user=user)

Performance Optimization Tips

Database Queries

# Use select_related for foreign keys
pages = Page.objects.select_related('screenshot').all()

# Use prefetch_related for many-to-many
links = Link.objects.prefetch_related('tags').all()

# Add indexes to models
class Page(models.Model):
    url = models.URLField(db_index=True)  # Add index

Caching

from django.core.cache import cache

# Cache expensive operations
result = cache.get('key')
if result is None:
    result = expensive_operation()
    cache.set('key', result, timeout=3600)

Celery Task Optimization

@shared_task(bind=True, max_retries=3)
def my_task(self):
    # Use task routing for different queues
    # Set time limits
    # Implement idempotency

Security Considerations

1. CSRF Protection: Enabled by default, ensure templates use {% csrf_token %}
2. SQL Injection: Use Django ORM (parameterized queries)
3. XSS: Django auto-escapes templates, use |safe filter cautiously
4. File Upload: Validate file types, scan for malware, limit sizes
5. API Rate Limiting: Implement throttling with DRF throttle classes
6. Secrets Management: Use environment variables, never commit secrets

Monitoring and Logging

Application Logging

# Configure in settings.py
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/app/logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}

Celery Monitoring

• Use Flower: celery -A core flower (port 5555)
• Monitor task success/failure rates
• Track task execution times

Health Checks

Implement health check endpoint:

# views.py
def health_check(request):
    # Check database
    # Check Redis
    # Check Celery workers
    return JsonResponse({'status': 'healthy'})

Contributing Guidelines

When contributing code:

1. Follow existing code patterns and structure
2. Add tests for new features
3. Update documentation (README, this agents.md, docstrings)
4. Run formatters: black . and isort .
5. Ensure migrations are included for model changes
6. Update OpenAPI spec for API changes
7. Add i18n translation strings for user-facing text
8. Test in Docker environment before submitting

Useful Commands Reference

Django Management

python manage.py runserver              # Development server
python manage.py shell                  # Django shell
python manage.py dbshell               # Database shell
python manage.py createsuperuser       # Create admin user
python manage.py test                  # Run tests
python manage.py collectstatic         # Collect static files
python manage.py compilemessages       # Compile translations
python manage.py makemessages -l zh_Hans  # Extract translation strings

Celery

celery -A core worker -l info          # Start worker
celery -A core beat -l info            # Start beat scheduler
celery -A core worker --purge          # Clear all tasks
celery -A core inspect active          # Show active tasks
celery -A core inspect stats           # Worker statistics

UV Package Manager

uv pip install package_name            # Install package
uv pip install -r requirements.txt     # Install from requirements
uv pip freeze > requirements.txt       # Export requirements
uv sync                                # Sync from pyproject.toml

Resources

• Django Documentation: https://docs.djangoproject.com/
• Django REST Framework: https://www.django-rest-framework.org/
• Celery Documentation: https://docs.celeryq.dev/
• Tailwind CSS: https://tailwindcss.com/docs
• SwiftUI: https://developer.apple.com/documentation/swiftui/

---

Last Updated: October 2025 Project Version: 0.1.0 Maintainer: wahyd4

For questions or issues, please refer to the GitHub repository or create an issue.