# Agents Guide - URL Manager

This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.

## Project Overview

**URL Manager** (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.

### Tech Stack

- **Backend**: Django 5.1, Django REST Framework, APScheduler
- **Frontend**: Tailwind CSS, AlpineJS (minimal JS)
- **Database**: SQLite (default, configurable)
- **Task Queue**: APScheduler (in-process background scheduler)
- **Storage**: Local filesystem or AWS S3/Cloudflare R2
- **Web Automation**: Selenium with Chromium (for screenshots and scraping)
- **Mobile**: SwiftUI iOS app with Core Data
- **Deployment**: Kubernetes (k8s manifests included)

## Architecture

### Core Components

```
┌─────────────────────────────────────────────────────────────┐
│                      Web Interface                          │
│         (Django Templates + Tailwind CSS)                   │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Django Application                        │
│  ┌──────────────┬──────────────┬──────────────────────┐    │
│  │ Links Module │ Pages Module │ Collections Module   │    │
│  │              │              │ (Images, Posts)      │    │
│  └──────────────┴──────────────┴──────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │           REST API (DRF ViewSets)                  │    │
│  └────────────────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│       Background Tasks (APScheduler + Threading)            │
│  - Screenshot capture (background threads)                  │
│  - Page metadata extraction (background threads)            │
│  - Periodic task scheduling (APScheduler)                   │
└─────────────────────────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Storage Layer                             │
│  - Local filesystem (default)                               │
│  - Cloudflare R2 / AWS S3 (optional)                        │
└─────────────────────────────────────────────────────────────┘
```

## Data Models

### Core Models (links/models.py)

#### Link
The central model for URL management:
- **alias**: Unique slug identifier for the short link
- **original_url**: Target URL (supports templates with `{param,default=value}`)
- **text**: Markdown content (for custom pages)
- **link_type**: LINK (regular URL) or CUSTOM (markdown content)
- **click_count**: Usage analytics
- **tags**: Many-to-many relationship with Tag model
- **description**: Optional description

**Template URL Feature**: URLs can contain parameters like `https://example.com/{query,default=test}` that are resolved at access time.

#### Page
Bookmarked pages with auto-extracted metadata:
- **url**: Original page URL
- **title**: Auto-extracted or manual
- **summary**: Auto-extracted description
- **content**: Full page content (optional)
- **screenshot**: Associated Screenshot model
- **tags**: Many-to-many with Tag
- **processing_status**: PENDING, PROCESSING, COMPLETED, FAILED
- **priority**: For async processing queue

#### Tag
Hierarchical tagging system:
- **name**: Tag name
- **slug**: URL-friendly slug
- **description**: Optional description
- **icon**: Optional icon
- **parent**: Self-referential for hierarchy
- **color**: UI color code

#### ImageCollection & Image
Image gallery management:
- Collections group related images
- Support for Cloudflare R2 / S3 storage
- Automatic thumbnail generation
- Bulk upload capabilities

#### Post
Blog-like content management:
- Markdown content support
- Tag categorization
- Publication status tracking

#### mini_apps
A collection of mini apps which can be located at mini_apps_views.py

#### Analytics Models
- **ClickLog**: Individual click tracking with timestamps
- **LinkChangeLog**: Audit trail for modifications

## API Architecture

### REST API (Django REST Framework)

Located in various `*_views.py` files with corresponding `*_urls.py`:

#### Image API (`api_views.py`)
- `ImageCollectionViewSet`: CRUD for image collections
  - Custom action: `upload_images` - Bulk image upload with R2/S3 integration
- `ImageViewSet`: Image management with descriptions
- `MusicViewSet`: Music file management

#### Page API (`page_views.py`)
- `PageViewSet`: Full CRUD for bookmarked pages
  - `screenshot` action: Trigger screenshot capture
  - `extract_metadata` action: Re-extract page metadata

#### Post API (`post_views.py`)
- `PostViewSet`: Blog post management with markdown rendering

### API Documentation
- OpenAPI 3.1 spec available at `/static/openapi.yaml`
- Documents all endpoints, request/response schemas, and authentication

## Background Task System (APScheduler)

The application uses APScheduler for background task scheduling, running in-process with the Django application. This eliminates the need for separate worker processes and Redis broker.

### Tasks (links/tasks.py)

#### process_page(page_id, retry_count=0)
Asynchronously fetches page metadata:
1. Downloads page HTML
2. Extracts title, description, meta tags
3. Stores content in Page model
4. Updates processing status
5. Triggers screenshot capture

**Features**:
- Retry logic with exponential backoff (Fibonacci sequence)
- Timeout protection (20s)
- BeautifulSoup for HTML parsing
- Multiple metadata extraction strategies (title, h1, og:tags)
- Runs in background thread for non-blocking execution

#### capture_screenshot(page_id, screenshot_id, retry_count=0)
Captures website screenshots:
1. Launches headless Chromium via Selenium
2. Configures viewport and options
3. Takes screenshot
4. Uploads to storage (R2/S3 or local)
5. Updates Screenshot model

**Configuration**:
- Headless mode
- Custom user agent
- 60s timeout
- Supports custom viewport sizes
- Error handling and retry logic via APScheduler
- Runs in background thread

#### schedule_pending_pages()
Periodic maintenance task:
- Runs every 120 seconds
- Checks for pages with PENDING status
- Schedules processing for pending pages
- Respects retry limits and backoff

### APScheduler Configuration

```python
# core/scheduler.py
scheduler = BackgroundScheduler(
    executors={'default': ThreadPoolExecutor(20)},
    job_defaults={
        'coalesce': False,
        'max_instances': 3,
        'misfire_grace_time': 300
    },
    timezone=settings.TIME_ZONE
)
```

**Initialization** in `core/apps.py`:
- Scheduler starts automatically when Django application starts
- Periodic jobs registered in `CoreConfig.ready()` method
- No separate worker process needed

### Task Execution

Tasks are executed using Python threading for immediate background execution:
```python
from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()
```

For scheduled/delayed tasks, APScheduler is used:
```python
from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)
```

## Key Features & Implementation

### 1. Template URL Processing
**File**: `links/models.py` - `Link.get_template_parameters()`

Extracts and processes URL parameters:
```python
# URL: https://search.com/{query,default=test}&lang={lang,default=en}
# Renders to: https://search.com/python&lang=en (with query="python")
```

### 2. Asynchronous Page Processing
**Files**: `links/tasks.py`, `links/page_views.py`

When a page is bookmarked:
1. Page record created with `processing_status=PENDING`
2. Background thread started for metadata extraction
3. Background worker fetches and parses page
4. Title, description, content extracted
5. Screenshot capture triggered in separate thread
6. Status updated to COMPLETED

### 3. Search System
**File**: `links/search_views.py`

Multi-field search across:
- Link aliases, URLs, descriptions
- Page titles, content, summaries
- Tag names
- Full-text search with filtering
- Tag-based filtering
- Type-based filtering (links, pages, posts)

### 4. Storage Abstraction
**File**: `links/storage.py` - `R2Storage`

Unified interface for:
- Local filesystem storage
- Cloudflare R2 (S3-compatible)
- AWS S3

Environment variables:
- `R2_ENDPOINT_URL`
- `R2_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET_NAME`

### 5. i18n Support
**Locales**: English (`en`), Simplified Chinese (`zh_Hans`)

Translation files in `locale/` directory. Uses Django's i18n framework.

## Development Guide for AI Agents

### Common Tasks

#### Adding a New Model Field
1. Update model in `links/models.py`
2. Create migration: `python manage.py makemigrations`
3. Apply migration: `python manage.py migrate`
4. Update serializer in `links/serializers.py` (if API exposed)
5. Update forms in `links/forms.py` (if form-based)
6. Update templates in `links/templates/`

#### Adding a New API Endpoint
1. Add serializer in `links/serializers.py`
2. Create ViewSet in appropriate `*_views.py` file
3. Register route in `*_urls.py`
4. Update OpenAPI spec in `static/openapi.yaml`
5. Add tests

#### Adding a Background Task
1. Define task function in `links/tasks.py` (no decorator needed)
2. For immediate execution, use threading:
   ```python
   from threading import Thread
   thread = Thread(target=task_function, args=(arg1,))
   thread.daemon = True
   thread.start()
   ```
3. For delayed/scheduled execution, use APScheduler:
   ```python
   from core.scheduler import scheduler
   scheduler.add_job(task_function, 'date', run_date=run_time, args=[arg1])
   ```
4. For periodic tasks, add to `core/apps.py` in `CoreConfig.ready()`:
   ```python
   scheduler.add_job(
       task_function,
       'interval',
       seconds=120,
       id='task_id',
       replace_existing=True
   )
   ```
5. Add logging for debugging
6. Implement retry logic if needed

#### Adding a New Template View
1. Create view function in `links/views.py` or create new view file
2. Add URL pattern to `links/urls.py` or appropriate urls file
3. Create template in `links/templates/`
4. Add i18n translation strings
5. Update navigation if needed

### Code Style Guidelines

- **Python**: Follow PEP 8, use Black formatter (line length: 100)
- **Imports**: Use isort with Black profile
- **Type hints**: Encouraged for new code
- **Docstrings**: Use for complex functions and classes
- **Logging**: Use Django's logging framework

```python
import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message")
logger.error("Error message", exc_info=True)
```

### Testing

**Framework**: pytest with pytest-django

Run tests:
```bash
pytest
pytest links/tests/test_models.py
pytest -k "test_link_creation"
```

### Database

**Default**: SQLite at `data/db.sqlite3`

Migrations managed with Django migrations. Always create migrations for model changes:
```bash
python manage.py makemigrations
python manage.py migrate
```

### Static Assets

**Tailwind CSS**: Compiled via django-tailwind

Development:
```bash
python manage.py tailwind start  # Watch mode
```

Production:
```bash
python manage.py tailwind build  # Minified build
python manage.py collectstatic   # Collect to staticfiles/
```

## Docker Development

### docker-compose.yml Services

- **web**: Django development server (port 8000) with APScheduler running in-process
- **node**: Tailwind CSS compiler

Note: Redis, Celery worker, and Celery beat services have been removed as they are no longer needed.

### Helper Script: docker.sh

```bash
./docker.sh build      # Build images
./docker.sh start      # Start all services
./docker.sh stop       # Stop all services
./docker.sh logs       # View logs
./docker.sh migrate    # Run migrations
./docker.sh shell      # Django shell
```

### Environment Variables (.env)

Create `.env` file:
```env
DJANGO_SETTINGS_MODULE=core.settings
SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=*

# Optional: R2/S3 Storage
R2_ENDPOINT_URL=https://...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=...
```

## iOS App (Heygo)

### Architecture
- **SwiftUI** for declarative UI
- **Core Data** for local persistence
- **MVVM** pattern with Combine
- **Charts** framework for analytics

### Syncing Strategy
Currently standalone (no backend sync). Future enhancement could:
1. Use Django REST API
2. Implement OAuth authentication
3. Sync via background refresh
4. Conflict resolution strategy

### Key Files
- `mobile/Heygo/HeygoApp.swift`: App entry point
- `mobile/Heygo/Views/`: SwiftUI views
- `mobile/Heygo/ViewModels/LinkViewModel.swift`: Business logic
- `mobile/Heygo/Models/`: Core Data models

## Deployment

### Production Dockerfile

Multi-stage build:
1. **Builder stage**: Install dependencies, compile static assets, build translations
2. **Production stage**: Slim image with only runtime dependencies

**Runtime requirements**:
- Python 3.12
- Chromium + ChromeDriver (for screenshots)

### Kubernetes

Manifests in `k8s/manifest.yaml`:
- Deployment for web service (includes APScheduler)
- Service for load balancing
- ConfigMap for configuration
- PersistentVolumeClaim for data

Note: Worker deployment is no longer needed as tasks run in-process.

### Environment Setup

Production checklist:
- [ ] Set `DEBUG=False`
- [ ] Configure `SECRET_KEY` (strong random value)
- [ ] Set `ALLOWED_HOSTS` to actual domains
- [ ] Configure database (PostgreSQL recommended for production)
- [ ] Configure R2/S3 for media storage
- [ ] Set up SSL/TLS termination
- [ ] Configure backup strategy
- [ ] Set up monitoring and logging

## Common Troubleshooting

### Screenshots Not Generating
1. Check Chromium installation: `which chromium`
2. Check Django application logs for task errors
3. Verify APScheduler is running (check startup logs)
4. Review application logs for screenshot task errors
5. Verify storage configuration (R2/S3 credentials)

### Metadata Extraction Failing
1. Check website accessibility (some sites block bots)
2. Verify timeout settings (increase if needed)
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
4. Review error logs in application logs

### Migration Issues
1. Check for unapplied migrations: `python manage.py showmigrations`
2. Look for migration conflicts
3. Use `python manage.py migrate --fake-initial` cautiously
4. For complex issues, may need to squash migrations

### Performance Issues
1. Add database indexes for frequently queried fields
2. Implement caching (Django cache framework)
3. Tune APScheduler thread pool size if needed (default: 20 threads)
4. Use database connection pooling
5. Enable query optimization (select_related, prefetch_related)

## File Structure Guide

### Core Django App Structure

```
core/
├── settings.py       # Django settings, installed apps, middleware
├── urls.py          # Main URL routing
├── scheduler.py     # APScheduler configuration
├── apps.py          # App configuration and scheduler initialization
└── middleware.py    # Custom middleware (locale, etc.)

links/
├── models.py        # Data models (Link, Page, Tag, etc.)
├── views.py         # Main template views
├── api_views.py     # REST API viewsets (Image, Music)
├── page_views.py    # Page-specific views and API
├── post_views.py    # Post/blog views and API
├── collection_views.py  # Collection management
├── search_views.py  # Search functionality
├── tag_views.py     # Tag management
├── forms.py         # Django forms
├── serializers.py   # DRF serializers
├── tasks.py         # Background task functions (APScheduler)
├── storage.py       # Storage abstraction (R2/S3)
├── urls.py          # Links app URL routing
├── *_urls.py        # Feature-specific URL routing
└── templates/       # Django templates

new_theme/
├── static/          # Tailwind compiled CSS
├── static_src/      # Tailwind source files
└── templates/       # Theme-specific templates

templates/
├── base.html        # Base template with navigation
└── base_blank.html  # Minimal base template
```

### Key Configuration Files

- `pyproject.toml`: Python dependencies (managed by uv)
- `uv.lock`: Locked dependency versions
- `package.json`: Node.js dependencies (Tailwind)
- `tailwind.config.js`: Tailwind configuration
- `docker-compose.yml`: Development environment
- `Dockerfile`: Production image
- `Dockerfile.local`: Development image

## API Authentication (Currently no auth is enabled, don't need to consider)

Currently, the API may not have authentication enabled. To add:

1. Add Django REST framework token authentication:
```python
# settings.py
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ]
}
```

2. Generate tokens for users:
```python
from rest_framework.authtoken.models import Token
token = Token.objects.create(user=user)
```

## Performance Optimization Tips

### Database Queries
```python
# Use select_related for foreign keys
pages = Page.objects.select_related('screenshot').all()

# Use prefetch_related for many-to-many
links = Link.objects.prefetch_related('tags').all()

# Add indexes to models
class Page(models.Model):
    url = models.URLField(db_index=True)  # Add index
```

### Caching
```python
from django.core.cache import cache

# Cache expensive operations
result = cache.get('key')
if result is None:
    result = expensive_operation()
    cache.set('key', result, timeout=3600)
```

### APScheduler Task Optimization
```python
# Configure thread pool size in core/scheduler.py
executors = {
    'default': ThreadPoolExecutor(20),  # Adjust based on workload
}

# Add job with proper configuration
scheduler.add_job(
    my_task,
    'interval',
    seconds=60,
    max_instances=3,  # Limit concurrent instances
    id='unique_job_id',
    replace_existing=True
)

# For one-time delayed tasks
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=300)
scheduler.add_job(my_task, 'date', run_date=run_date, args=[arg1])
```

## Security Considerations

1. **CSRF Protection**: Enabled by default, ensure templates use `{% csrf_token %}`
2. **SQL Injection**: Use Django ORM (parameterized queries)
3. **XSS**: Django auto-escapes templates, use `|safe` filter cautiously
4. **File Upload**: Validate file types, scan for malware, limit sizes
5. **API Rate Limiting**: Implement throttling with DRF throttle classes
6. **Secrets Management**: Use environment variables, never commit secrets

## Monitoring and Logging

### Application Logging
```python
# Configure in settings.py
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/app/logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}
```

### APScheduler Monitoring
- Check scheduler status: `scheduler.running` returns True/False
- List all jobs: `scheduler.get_jobs()`
- Monitor task execution via Django logs
- Track task execution times in application logs

### Health Checks
Implement health check endpoint:
```python
# views.py
def health_check(request):
    from core.scheduler import scheduler
    # Check database
    # Check APScheduler status
    scheduler_running = scheduler.running
    return JsonResponse({
        'status': 'healthy',
        'scheduler_running': scheduler_running
    })
```

## Contributing Guidelines

When contributing code:
1. Follow existing code patterns and structure
2. Add tests for new features
3. Update documentation (README, this agents.md, docstrings)
4. Run formatters: `black .` and `isort .`
5. Ensure migrations are included for model changes
6. Update OpenAPI spec for API changes
7. Add i18n translation strings for user-facing text
8. Test in Docker environment before submitting

## Useful Commands Reference

### just (Command Runner)
The project uses [`just`](https://github.com/casey/just) as its command runner. Run `just` (no args) to list all recipes.

```bash
# Local development
just dev              # Run Django + Tailwind together
just tailwind         # Run Tailwind CSS watcher (standalone)
just migrate          # Apply pending migrations
just makemigrations   # Create new migrations
just shell            # Django shell
just dbshell          # Raw DB shell
just superuser        # Create a superuser
just build-css        # Build minified Tailwind CSS
just collectstatic    # Collect static files
just test             # Run tests
just compilemessages  # Compile i18n translations
just makemessages     # Extract translatable strings (zh_Hans)

# Docker
just docker-build           # Build images
just docker-start           # Start services (detached)
just docker-stop            # Stop services
just docker-logs            # Tail all service logs
just docker-logs web        # Tail a specific service's logs
just docker-shell           # Shell into web container
just docker-migrate         # Run migrations in Docker
just docker-static          # Collect static in Docker
just docker-rebuild web     # Rebuild + restart a service
```

### Direct Django Management
When you need to run `manage.py` directly, activate the venv first:
```bash
source .venv/bin/activate
uv run manage.py <command>
```

### APScheduler (In-Process)
APScheduler starts automatically with Django. To interact with it:
```python
# In Django shell (just shell)
from core.scheduler import scheduler
scheduler.get_jobs()     # List all scheduled jobs
scheduler.print_jobs()   # Print job details
scheduler.running        # Check if scheduler is running
```

### UV Package Manager
```bash
uv add package_name     # Add a new dependency
uv sync                 # Sync environment from pyproject.toml / uv.lock
```

## Resources

- **Django Documentation**: https://docs.djangoproject.com/
- **Django REST Framework**: https://www.django-rest-framework.org/
- **APScheduler Documentation**: https://apscheduler.readthedocs.io/
- **Tailwind CSS**: https://tailwindcss.com/docs
- **SwiftUI**: https://developer.apple.com/documentation/swiftui/
- **just**: https://just.systems/man/en/

Note:
- No need to generate extra summary guide or docs, unless I ask you to.