links/agents.md

# Agents Guide - URL Manager

This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.

## Project Overview

**URL Manager** (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.

### Tech Stack

- **Backend**: Django 5.1, Django REST Framework, APScheduler
- **Frontend**: Tailwind CSS, AlpineJS (minimal JS)
- **Database**: SQLite (default, configurable)
- **Task Queue**: APScheduler (in-process background scheduler)
- **Storage**: Local filesystem or AWS S3/Cloudflare R2
- **Web Automation**: Selenium with Chromium (for screenshots and scraping)
- **Mobile**: SwiftUI iOS app with Core Data
- **Deployment**: Kubernetes (k8s manifests included)

## Architecture

### Core Components

```
┌─────────────────────────────────────────────────────────────┐
│                      Web Interface                          │
│         (Django Templates + Tailwind CSS)                   │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Django Application                        │
│  ┌──────────────┬──────────────┬──────────────────────┐    │
│  │ Links Module │ Pages Module │ Collections Module   │    │
│  │              │              │ (Images, Posts)      │    │
│  └──────────────┴──────────────┴──────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │           REST API (DRF ViewSets)                  │    │
│  └────────────────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│       Background Tasks (APScheduler + Threading)            │
│  - Screenshot capture (background threads)                  │
│  - Page metadata extraction (background threads)            │
│  - Periodic task scheduling (APScheduler)                   │
└─────────────────────────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Storage Layer                             │
│  - Local filesystem (default)                               │
│  - Cloudflare R2 / AWS S3 (optional)                        │
└─────────────────────────────────────────────────────────────┘
```

## Data Models

### Core Models (links/models.py)

#### Link
The central model for URL management:
- **alias**: Unique slug identifier for the short link
- **original_url**: Target URL (supports templates with `{param,default=value}`)
- **text**: Markdown content (for custom pages)
- **link_type**: LINK (regular URL) or CUSTOM (markdown content)
- **click_count**: Usage analytics
- **tags**: Many-to-many relationship with Tag model
- **description**: Optional description

**Template URL Feature**: URLs can contain parameters like `https://example.com/{query,default=test}` that are resolved at access time.

#### Page
Bookmarked pages with auto-extracted metadata:
- **url**: Original page URL
- **title**: Auto-extracted or manual
- **summary**: Auto-extracted description
- **content**: Full page content (optional)
- **screenshot**: Associated Screenshot model
- **tags**: Many-to-many with Tag
- **processing_status**: PENDING, PROCESSING, COMPLETED, FAILED
- **priority**: For async processing queue

#### Tag
Hierarchical tagging system:

#### Bookmark
Archived social-media posts (originally x.com / Twitter bookmarks), each saved with full preview data so a viewer page renders offline without further x.com API calls:
- **tweet_id**: Source post id (unique, indexed)
- **url**: Canonical x.com post URL
- **author_screen_name / author_name / author_profile_image_url**: Denormalised author
- **text**: Full post text (searchable)
- **summary**: Optional human/agent-written summary (writable via API `PATCH`)
- **tweet_created_at / bookmark_created_at**: Parsed source dates (indexed)
- **view_count / favorite_count / retweet_count / reply_count / bookmark_count / quote_count**: Engagement counters
- **has_video / has_photo / has_card / is_quote / is_retweet**: Indexed boolean flags for fast filter pages
- **media**: JSON list of media objects (photo URL variants; video/animated_gif with `poster`, `best_mp4`, `duration_ms`, `variants[]`)
- **card_data**: JSON link-preview card (title/description/site/image/url)
- **quoted_data**: JSON nested quoted post (recursive shape)
- **data**: The complete original normalised export object (kept for round-tripping / the offline viewer)
- UI: `/ui/bookmarks/` (list, `BookmarkListView`) + `/ui/bookmarks/<pk>/` (detail). Top-menu item "Bookmarks" in `templates/base.html`.
- Import: `python manage.py import_x_bookmarks <dir|file.json> [--clear]` (loads `x-bookmarks-exporter` JSON) or `POST /api/bookmarks/import`.
- **No `<video>` streaming** in the viewer — videos render as poster thumbnails linking to x.com (performance + ToS).
- **Search**: bookmarks are indexed in the Whoosh full-text index (`search_backend.index_bookmark`, rebuilt via `python manage.py rebuild_search_index`), wired into signals (`post_save`/`post_delete` on `Bookmark` auto-updates the index — e.g. when an agent `PATCH`es a summary). They appear in `/ui/search/` (React app, type filter "Bookmarks"), `/search/api/v2/?type=bookmark`, and the simple `/search/api/?q=` lookup.
- **name**: Tag name
- **slug**: URL-friendly slug
- **description**: Optional description
- **icon**: Optional icon
- **parent**: Self-referential for hierarchy
- **color**: UI color code

#### ImageCollection & Image
Image gallery management:
- Collections group related images
- Support for Cloudflare R2 / S3 storage
- Automatic thumbnail generation
- Bulk upload capabilities

#### Post
Blog-like content management:
- Markdown content support
- Tag categorization
- Publication status tracking

#### mini_apps
A collection of mini apps which can be located at mini_apps_views.py

#### Analytics Models
- **ClickLog**: Individual click tracking with timestamps
- **LinkChangeLog**: Audit trail for modifications

## API Architecture

### REST API (Django REST Framework)

Located in various `*_views.py` files with corresponding `*_urls.py`:

#### Image API (`api_views.py`)
- `ImageCollectionViewSet`: CRUD for image collections
  - Custom action: `upload_images` - Bulk image upload with R2/S3 integration
- `ImageViewSet`: Image management with descriptions
- `MusicViewSet`: Music file management

#### Page API (`page_views.py`)
- `PageViewSet`: Full CRUD for bookmarked pages
  - `screenshot` action: Trigger screenshot capture
  - `extract_metadata` action: Re-extract page metadata

#### Bookmark API (`bookmark_views.py` → `BookmarkViewSet`, mounted at `/api/bookmarks`)
- `BookmarkViewSet`: Full CRUD for archived social-media bookmarks
  - `GET /api/bookmarks`: list with filters `?q=`, `?has_video=true`, `?has_photo=`, `?has_card=`, `?is_quote=`, `?is_retweet=`, `?author=`, `?ordering=`, `?page_size=` (default 24, max 200)
  - `POST /api/bookmarks`: create one (flat `Bookmark`-shaped body)
  - `GET/PATCH/DELETE /api/bookmarks/{id}`: read / partial-update (e.g. write an AI `summary`) / delete
  - `POST /api/bookmarks/import`: bulk upsert from `{"bookmarks":[ {export obj}, ... ]}` — accepts the full `x-bookmarks-exporter` normalised object (nested author/media/card/quoted/retweet + x.com-style `created_at` date); matches by `tweet_id`. Returns `{created, updated, skipped, total}`
  - `PUT|POST /api/bookmarks/bulk`: bulk upsert from a bare JSON array of flat `Bookmark`-shaped objects
  - `GET /api/bookmarks/stats`: `{total, has_video, has_photo, has_card, is_quote, is_retweet}` (used by viewer filter chips)

#### Post API (`post_views.py`)
- `PostViewSet`: Blog post management with markdown rendering

### API Documentation
- OpenAPI 3.0.3 spec available at `/static/openapi.yaml` (browse at [http://localhost:8000/ui/api-docs/](http://localhost:8000/ui/api-docs/) via ReDoc)
- Documents all endpoints, request/response schemas, and authentication
- **The spec is the contract.** Any task that touches an API must keep it in sync (see the Agent Workflow below)

#### File API (`file_views.py` → `FileUploadViewSet`, mounted at `/api/files`)
- `FileUploadViewSet`: Full CRUD for generic file uploads
  - `create`: Accepts `multipart/form-data` with a `files` field (one or more files). Returns `201` with an array of `FileUpload` objects
  - `destroy`: Deletes the record and removes the file from disk
  - `download` action: Streams file content. Supports HTTP `Range` requests — replies with `206 Partial Content` + `Content-Range`/`Accept-Ranges` headers so `<video>`/`<audio>` can seek. Inline disposition for images, video, and audio; `attachment` for other types
  - `toggle-public` action: Flips `is_public` and returns `{is_public, public_url}`
  - `set-expiry` action: Sets or clears an ISO-8601 expiry datetime on the public link
- Public access lives at `/public/files/{id}-{filename}` (filename is cosmetic; lookup is by UUID only). Internal-network requests serve any file; external requests only serve `is_public=True` files that haven't expired. Range support applies here too.

## Background Task System (APScheduler)

The application uses APScheduler for background task scheduling, running in-process with the Django application. This eliminates the need for separate worker processes and Redis broker.

### Tasks (links/tasks.py)

#### process_page(page_id, retry_count=0)
Asynchronously fetches page metadata:
1. Downloads page HTML
2. Extracts title, description, meta tags
3. Stores content in Page model
4. Updates processing status
5. Triggers screenshot capture

**Features**:
- Retry logic with exponential backoff (Fibonacci sequence)
- Timeout protection (20s)
- BeautifulSoup for HTML parsing
- Multiple metadata extraction strategies (title, h1, og:tags)
- Runs in background thread for non-blocking execution

#### capture_screenshot(page_id, screenshot_id, retry_count=0)
Captures website screenshots:
1. Launches headless Chromium via Selenium
2. Configures viewport and options
3. Takes screenshot
4. Uploads to storage (R2/S3 or local)
5. Updates Screenshot model

**Configuration**:
- Headless mode
- Custom user agent
- 60s timeout
- Supports custom viewport sizes
- Error handling and retry logic via APScheduler
- Runs in background thread

#### schedule_pending_pages()
Periodic maintenance task:
- Runs every 120 seconds
- Checks for pages with PENDING status
- Schedules processing for pending pages
- Respects retry limits and backoff

### APScheduler Configuration

```python
# core/scheduler.py
scheduler = BackgroundScheduler(
    executors={'default': ThreadPoolExecutor(20)},
    job_defaults={
        'coalesce': False,
        'max_instances': 3,
        'misfire_grace_time': 300
    },
    timezone=settings.TIME_ZONE
)
```

**Initialization** in `core/apps.py`:
- Scheduler starts automatically when Django application starts
- Periodic jobs registered in `CoreConfig.ready()` method
- No separate worker process needed

### Task Execution

Tasks are executed using Python threading for immediate background execution:
```python
from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()
```

For scheduled/delayed tasks, APScheduler is used:
```python
from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)
```

## Key Features & Implementation

### 1. Template URL Processing
**File**: `links/models.py` - `Link.get_template_parameters()`

Extracts and processes URL parameters:
```python
# URL: https://search.com/{query,default=test}&lang={lang,default=en}
# Renders to: https://search.com/python&lang=en (with query="python")
```

### 2. Asynchronous Page Processing
**Files**: `links/tasks.py`, `links/page_views.py`

When a page is bookmarked:
1. Page record created with `processing_status=PENDING`
2. Background thread started for metadata extraction
3. Background worker fetches and parses page
4. Title, description, content extracted
5. Screenshot capture triggered in separate thread
6. Status updated to COMPLETED

### 3. Search System
**File**: `links/search_views.py`

Multi-field search across:
- Link aliases, URLs, descriptions
- Page titles, content, summaries
- Tag names
- Full-text search with filtering
- Tag-based filtering
- Type-based filtering (links, pages, posts)

### 4. Storage Abstraction
**File**: `links/storage.py` - `R2Storage`

Unified interface for:
- Local filesystem storage
- Cloudflare R2 (S3-compatible)
- AWS S3

Environment variables:
- `R2_ENDPOINT_URL`
- `R2_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET_NAME`

### 5. i18n Support
**Locales**: English (`en`), Simplified Chinese (`zh_Hans`)

Translation files in `locale/` directory. Uses Django's i18n framework.

## Development Guide for AI Agents

### Agent Workflow (always follow before coding)

Before writing or modifying any feature, **check the existing contract first** to avoid breaking it:

1. **Read the OpenAPI spec.** Open `static/openapi.yaml` (or browse [/ui/api-docs/](http://localhost:8000/ui/api-docs/)) and find the paths/schemas the task touches. The spec is the source of truth for endpoint shapes, status codes, and fields. Contrast it with what the request asks for — if they differ, ask the user whether the spec or the request is the intended behaviour.
2. **Verify the implementation against the spec.** Inspect the relevant `*_views.py`, `serializers.py`, and model so the code you produce matches (or deliberately updates) the documented contract. Common drift: renamed fields, missing status codes (`206`, `416` for media ranges), wrong `Content-Disposition`, stale path templates (e.g. `{token}/` vs `{id}-{filename}`).
3. **Keep the spec in sync.** If the code change alters any API behaviour, update `static/openapi.yaml` in the same change. Validate with `python -c "import yaml; yaml.safe_load(open('static/openapi.yaml'))"`. Regenerate or sanity-check any derived docs that reference the spec.
4. **Rebuild Tailwind when you touch templates.** New utility classes are only included after `npm run build:css` (or `just tailwind` in watch mode). Serving stale CSS is a frequent source of "works in one viewport, broken in another" bugs (e.g. duplicate listings, hidden sections).
5. **Run the tests and linters.** Use `just test` (or `pytest`) for affected modules. Prefer an integration test that asserts the documented request → response shape, including status code and headers. Confirm `black`/`isort` pass if available.
6. **Don't leave duplicates.** Whether it's duplicate DOM rendering (desktop/mobile forks that rely on responsive CSS), duplicate file records, or duplicate submission — each piece of state should have exactly one source of truth. Prefer a single render driven by JS data over copy-pasted server-rendered blocks.

### Engineering Best Practices

- **Single source of truth for state.** Render lists from one data array; never duplicate the same dataset in desktop and mobile markup expecting CSS to hide one.
- **APIs must return everything the client needs.** If the UI updates dynamically after a mutation (no full reload), the response must include the fields the client renders (e.g. `formatted_size`, `mime_type`, `download_url`, `created_at`).
- **Media must be seekable.** Any endpoint serving `<video>`/`<audio>` content must honour HTTP `Range` requests and return `Accept-Ranges: bytes`, `206 Partial Content`, and `Content-Range`. Reuse the `_stream_file()` helper in `links/file_views.py` rather than raw `FileResponse(open(...))`.
- **Prefer AJAX over full-page reloads** for incremental updates (uploads, deletes, toggles). It avoids re-rendering the whole page and prevents re-submission on refresh.
- **Responsive design without forking markup.** Use flex/grid + `hidden`/`block` toggles on breakpoints instead of rendering a whole second table/card tree.
- **Accessibility.** Buttons/links need visible focus rings, tap targets ≥ 44 px on mobile, ARIA labels for icon-only controls, and `sr-only` labels for visually-hidden text.
- **Tests are mandatory for API changes.** Add or update a test that pins the contract (status code, body, headers). Run `pytest` before declaring done.

### UI Style Guidelines

**The home page is the canonical style reference.** Every UI page in this app must align with the Apple-inspired design system defined in `links/templates/links/link_list.html`. Do not introduce ad-hoc Tailwind color palettes (e.g. `red-600`, `gray-200` buttons) or generic card styles on new pages — reuse the home page's tokens and component classes so the whole app shares one visual language.

#### Canonical reference
- **Home page template**: `links/templates/links/link_list.html` — the `{% block extra_css %}` `<style>` block defines the design system. Copy its `:root` tokens and component classes into new pages rather than inventing new ones.
- **Files page** (`links/templates/links/files/list.html`) is an example of a non-home page that was restyled to match — use it as a secondary reference for list/card/button patterns outside the home page.

#### Design tokens (CSS custom properties)
Always declare these in the page's `{% block extra_css %}` `<style>` block:
```css
:root {
  --apple-bg: #f5f5f7;            /* page canvas */
  --apple-text: #1d1d1f;          /* primary text */
  --apple-gray: #6e6e73;          /* secondary text */
  --apple-blue: #0071e3;          /* primary accent / links */
  --apple-blue-hover: #0077ed;
  --apple-red: #ff3b30;           /* destructive */
  --apple-green: #1e7b34;         /* success / public */
  --apple-separator: rgba(0, 0, 0, 0.06);
  --apple-ease: cubic-bezier(0.28, 0.11, 0.32, 1);
  --apple-radius-lg: 28px;        /* cards */
  --apple-radius-md: 22px;        /* modals / inner cards */
  --apple-shadow-card: 0 4px 24px rgba(0, 0, 0, 0.05);
  --apple-shadow-lift: 0 18px 44px rgba(0, 0, 0, 0.10);
}
body {
  background-color: var(--apple-bg) !important;
  font-family: -apple-system, BlinkMacSystemFont, "SF Pro Display", "SF Pro Text",
    "Helvetica Neue", "Segoe UI", Roboto, Arial, sans-serif;
  color: var(--apple-text);
  -webkit-font-smoothing: antialiased;
}
```

#### Reusable component classes (defined on the home page — copy, don't reinvent)
- **Layout**: `.apple-wrap` (max-width 72rem, centered)
- **Cards**: `.apple-card` (white, 28px radius, soft shadow)
- **Section headers**: `.section-title` (clamp 1.45–1.8rem, 700 weight, -0.022em tracking), `.library-header`, `.library-count`
- **Buttons**: `.apple-btn` base + `.apple-btn-primary` (blue pill), `.apple-btn-secondary` (gray pill), `.apple-btn-danger` (outline red). All pills: `border-radius: 980px`, `min-height: 44px`.
- **Icon buttons**: `.icon-btn` (40px circular, hover lift) with `.blue` / `.green` / `.red` / `.amber` modifiers
- **Badges**: `.badge` pill + `.badge-public` (green) / `.badge-private` (gray) / `.badge-link` / `.badge-template` / `.badge-custom`
- **Links**: `.apple-link` (blue, inline-flex with chevron), `.alias-link`
- **Tables**: `.apple-table` with uppercase tracked headers, separator borders, hover row tint
- **Inputs**: frosted-glass pill inputs (see `.hero-search input` / `.library-filter input`) — `border-radius: 980px`, focus ring `0 0 0 4px rgba(0,113,227,0.14)`
- **Modals**: `.apple-modal-bg` (blurred dark backdrop) + `.apple-modal` (22px radius white card)
- **Empty state**: `.empty-state` (centered, gray, 2.6rem icon)
- **Accessibility**: `a:focus-visible, button:focus-visible { outline: 2px solid var(--apple-blue); outline-offset: 3px; }`

#### Rules
1. **No raw Tailwind color utilities for primary UI.** Use the `--apple-*` tokens via the component classes above. Tailwind utilities are fine for layout/spacing (`flex`, `gap-4`, `hidden sm:block`), not for the brand palette.
2. **Pill buttons, circular icon buttons, 28px-radius cards** — these shapes are the app's signature. Don't substitute `rounded-md`/`rounded-lg` for primary components.
3. **Frosted glass for overlays.** Modals, dropdowns, and the upload HUD use `backdrop-filter: blur(20-24px) saturate(180%)` over a semi-transparent white/dark background.
4. **Soft shadows, never harsh.** Cards use `--apple-shadow-card`; lifted/hover states use `--apple-shadow-lift`.
5. ** SF system font stack** on `body` — never hardcoded `font-family` on individual elements.
6. **Respect `prefers-reduced-motion`.** Disable transitions/animations under reduced motion.
7. **When restyling an existing page**, verify computed styles match the home page's (background `rgb(245,245,247)`, card radius `28px`, primary button `rgb(0,113,227)` / radius `980px`). The Files page restyle is the reference for this check.

### Common Tasks

#### Adding a New Model Field
1. Update model in `links/models.py`
2. Create migration: `python manage.py makemigrations`
3. Apply migration: `python manage.py migrate`
4. Update serializer in `links/serializers.py` (if API exposed)
5. Update forms in `links/forms.py` (if form-based)
6. Update templates in `links/templates/`

#### Adding a New API Endpoint
1. Add or update serializer in `links/serializers.py`
2. Create ViewSet (or function view) in the appropriate `*_views.py` file
3. Register the route in the matching `*_urls.py`
4. **Update `static/openapi.yaml`** — path, parameters, request schema, every response status code and headers, and any new component schema. Validate the YAML parses.
5. **Write an integration test** (`tests/`) that asserts the documented request → response shape (status code, body fields, key headers). Add range-media coverage when serving media.
6. Verify a fresh `just dev` server still returns `200` from `/ui/api-docs/` and the live endpoint matches the spec

#### Adding a Background Task
1. Define task function in `links/tasks.py` (no decorator needed)
2. For immediate execution, use threading:
   ```python
   from threading import Thread
   thread = Thread(target=task_function, args=(arg1,))
   thread.daemon = True
   thread.start()
   ```
3. For delayed/scheduled execution, use APScheduler:
   ```python
   from core.scheduler import scheduler
   scheduler.add_job(task_function, 'date', run_date=run_time, args=[arg1])
   ```
4. For periodic tasks, add to `core/apps.py` in `CoreConfig.ready()`:
   ```python
   scheduler.add_job(
       task_function,
       'interval',
       seconds=120,
       id='task_id',
       replace_existing=True
   )
   ```
5. Add logging for debugging
6. Implement retry logic if needed

#### Adding a New Template View
1. Create view function in `links/views.py` or create new view file
2. Add URL pattern to `links/urls.py` or appropriate urls file
3. Create template in `links/templates/`
4. **Follow the UI Style Guidelines above** — reuse the home page's `--apple-*` tokens and component classes; don't invent new Tailwind palettes or generic card styles.
5. Add i18n translation strings
6. Update navigation if needed
7. Rebuild Tailwind CSS (`npm run build:css` or `just tailwind`) if new utility classes were introduced

### Code Style Guidelines

- **Python**: Follow PEP 8, use Black formatter (line length: 100)
- **Imports**: Use isort with Black profile
- **Type hints**: Encouraged for new code
- **Docstrings**: Use for complex functions and classes
- **Logging**: Use Django's logging framework

```python
import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message")
logger.error("Error message", exc_info=True)
```

### Testing

**Framework**: pytest with pytest-django

Run tests:
```bash
pytest
pytest links/tests/test_models.py
pytest -k "test_link_creation"
```

### Database

**Default**: SQLite at `data/db.sqlite3`

Migrations managed with Django migrations. Always create migrations for model changes:
```bash
python manage.py makemigrations
python manage.py migrate
```

### Static Assets

**Tailwind CSS**: Compiled via django-tailwind

Development:
```bash
python manage.py tailwind start  # Watch mode
```

Production:
```bash
python manage.py tailwind build  # Minified build
python manage.py collectstatic   # Collect to staticfiles/
```

## Docker Development

### docker-compose.yml Services

- **web**: Django development server (port 8000) with APScheduler running in-process
- **node**: Tailwind CSS compiler

Note: Redis, Celery worker, and Celery beat services have been removed as they are no longer needed.

### Helper Script: docker.sh

```bash
./docker.sh build      # Build images
./docker.sh start      # Start all services
./docker.sh stop       # Stop all services
./docker.sh logs       # View logs
./docker.sh migrate    # Run migrations
./docker.sh shell      # Django shell
```

### Environment Variables (.env)

Create `.env` file:
```env
DJANGO_SETTINGS_MODULE=core.settings
SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=*

# Optional: R2/S3 Storage
R2_ENDPOINT_URL=https://...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=...
```

## iOS App (Heygo)

### Architecture
- **SwiftUI** for declarative UI
- **Core Data** for local persistence
- **MVVM** pattern with Combine
- **Charts** framework for analytics

### Syncing Strategy
Currently standalone (no backend sync). Future enhancement could:
1. Use Django REST API
2. Implement OAuth authentication
3. Sync via background refresh
4. Conflict resolution strategy

### Key Files
- `mobile/Heygo/HeygoApp.swift`: App entry point
- `mobile/Heygo/Views/`: SwiftUI views
- `mobile/Heygo/ViewModels/LinkViewModel.swift`: Business logic
- `mobile/Heygo/Models/`: Core Data models

## Deployment

### Production Dockerfile

Multi-stage build:
1. **Builder stage**: Install dependencies, compile static assets, build translations
2. **Production stage**: Slim image with only runtime dependencies

**Runtime requirements**:
- Python 3.12
- Chromium + ChromeDriver (for screenshots)

### Kubernetes

Manifests in `k8s/manifest.yaml`:
- Deployment for web service (includes APScheduler)
- Service for load balancing
- ConfigMap for configuration
- PersistentVolumeClaim for data

Note: Worker deployment is no longer needed as tasks run in-process.

### Environment Setup

Production checklist:
- [ ] Set `DEBUG=False`
- [ ] Configure `SECRET_KEY` (strong random value)
- [ ] Set `ALLOWED_HOSTS` to actual domains
- [ ] Configure database (PostgreSQL recommended for production)
- [ ] Configure R2/S3 for media storage
- [ ] Set up SSL/TLS termination
- [ ] Configure backup strategy
- [ ] Set up monitoring and logging

## Common Troubleshooting

### Screenshots Not Generating
1. Check Chromium installation: `which chromium`
2. Check Django application logs for task errors
3. Verify APScheduler is running (check startup logs)
4. Review application logs for screenshot task errors
5. Verify storage configuration (R2/S3 credentials)

### Metadata Extraction Failing
1. Check website accessibility (some sites block bots)
2. Verify timeout settings (increase if needed)
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
4. Review error logs in application logs

### Migration Issues
1. Check for unapplied migrations: `python manage.py showmigrations`
2. Look for migration conflicts
3. Use `python manage.py migrate --fake-initial` cautiously
4. For complex issues, may need to squash migrations

### Performance Issues
1. Add database indexes for frequently queried fields
2. Implement caching (Django cache framework)
3. Tune APScheduler thread pool size if needed (default: 20 threads)
4. Use database connection pooling
5. Enable query optimization (select_related, prefetch_related)

## File Structure Guide

### Core Django App Structure

```
core/
├── settings.py       # Django settings, installed apps, middleware
├── urls.py          # Main URL routing
├── scheduler.py     # APScheduler configuration
├── apps.py          # App configuration and scheduler initialization
└── middleware.py    # Custom middleware (locale, etc.)

links/
├── models.py        # Data models (Link, Page, Tag, etc.)
├── views.py         # Main template views
├── api_views.py     # REST API viewsets (Image, Music)
├── page_views.py    # Page-specific views and API
├── post_views.py    # Post/blog views and API
├── collection_views.py  # Collection management
├── search_views.py  # Search functionality
├── tag_views.py     # Tag management
├── forms.py         # Django forms
├── serializers.py   # DRF serializers
├── tasks.py         # Background task functions (APScheduler)
├── storage.py       # Storage abstraction (R2/S3)
├── urls.py          # Links app URL routing
├── *_urls.py        # Feature-specific URL routing
└── templates/       # Django templates

new_theme/
├── static/          # Tailwind compiled CSS
├── static_src/      # Tailwind source files
└── templates/       # Theme-specific templates

templates/
├── base.html        # Base template with navigation
└── base_blank.html  # Minimal base template
```

### Key Configuration Files

- `pyproject.toml`: Python dependencies (managed by uv)
- `uv.lock`: Locked dependency versions
- `package.json`: Node.js dependencies (Tailwind)
- `tailwind.config.js`: Tailwind configuration
- `docker-compose.yml`: Development environment
- `Dockerfile`: Production image
- `Dockerfile.local`: Development image

## API Authentication (Currently no auth is enabled, don't need to consider)

Currently, the API may not have authentication enabled. To add:

1. Add Django REST framework token authentication:
```python
# settings.py
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ]
}
```

2. Generate tokens for users:
```python
from rest_framework.authtoken.models import Token
token = Token.objects.create(user=user)
```

## Performance Optimization Tips

### Database Queries
```python
# Use select_related for foreign keys
pages = Page.objects.select_related('screenshot').all()

# Use prefetch_related for many-to-many
links = Link.objects.prefetch_related('tags').all()

# Add indexes to models
class Page(models.Model):
    url = models.URLField(db_index=True)  # Add index
```

### Caching
```python
from django.core.cache import cache

# Cache expensive operations
result = cache.get('key')
if result is None:
    result = expensive_operation()
    cache.set('key', result, timeout=3600)
```

### APScheduler Task Optimization
```python
# Configure thread pool size in core/scheduler.py
executors = {
    'default': ThreadPoolExecutor(20),  # Adjust based on workload
}

# Add job with proper configuration
scheduler.add_job(
    my_task,
    'interval',
    seconds=60,
    max_instances=3,  # Limit concurrent instances
    id='unique_job_id',
    replace_existing=True
)

# For one-time delayed tasks
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=300)
scheduler.add_job(my_task, 'date', run_date=run_date, args=[arg1])
```

## Security Considerations

1. **CSRF Protection**: Enabled by default, ensure templates use `{% csrf_token %}`
2. **SQL Injection**: Use Django ORM (parameterized queries)
3. **XSS**: Django auto-escapes templates, use `|safe` filter cautiously
4. **File Upload**: Validate file types, scan for malware, limit sizes
5. **API Rate Limiting**: Implement throttling with DRF throttle classes
6. **Secrets Management**: Use environment variables, never commit secrets

## Monitoring and Logging

### Application Logging
```python
# Configure in settings.py
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/app/logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}
```

### APScheduler Monitoring
- Check scheduler status: `scheduler.running` returns True/False
- List all jobs: `scheduler.get_jobs()`
- Monitor task execution via Django logs
- Track task execution times in application logs

### Health Checks
Implement health check endpoint:
```python
# views.py
def health_check(request):
    from core.scheduler import scheduler
    # Check database
    # Check APScheduler status
    scheduler_running = scheduler.running
    return JsonResponse({
        'status': 'healthy',
        'scheduler_running': scheduler_running
    })
```

## Useful Commands Reference

### just (Command Runner)
The project uses [`just`](https://github.com/casey/just) as its command runner. Run `just` (no args) to list all recipes.

```bash
# Local development
just dev              # Run Django + Tailwind together
just tailwind         # Run Tailwind CSS watcher (standalone)
just migrate          # Apply pending migrations
just makemigrations   # Create new migrations
just shell            # Django shell
just dbshell          # Raw DB shell
just superuser        # Create a superuser
just build-css        # Build minified Tailwind CSS
just collectstatic    # Collect static files
just test             # Run tests
just compilemessages  # Compile i18n translations
just makemessages     # Extract translatable strings (zh_Hans)

# Docker
just docker-build           # Build images
just docker-start           # Start services (detached)
just docker-stop            # Stop services
just docker-logs            # Tail all service logs
just docker-logs web        # Tail a specific service's logs
just docker-shell           # Shell into web container
just docker-migrate         # Run migrations in Docker
just docker-static          # Collect static in Docker
just docker-rebuild web     # Rebuild + restart a service
```

### Direct Django Management
When you need to run `manage.py` directly, activate the venv first:
```bash
source .venv/bin/activate
uv run manage.py <command>
```

### APScheduler (In-Process)
APScheduler starts automatically with Django. To interact with it:
```python
# In Django shell (just shell)
from core.scheduler import scheduler
scheduler.get_jobs()     # List all scheduled jobs
scheduler.print_jobs()   # Print job details
scheduler.running        # Check if scheduler is running
```

### UV Package Manager
```bash
uv add package_name     # Add a new dependency
uv sync                 # Sync environment from pyproject.toml / uv.lock
```

## Resources

- **Django Documentation**: https://docs.djangoproject.com/
- **Django REST Framework**: https://www.django-rest-framework.org/
- **APScheduler Documentation**: https://apscheduler.readthedocs.io/
- **Tailwind CSS**: https://tailwindcss.com/docs
- **SwiftUI**: https://developer.apple.com/documentation/swiftui/
- **just**: https://just.systems/man/en/

Note:
- No need to generate extra summary guide or docs, unless I ask you to.