links/agents.md

Agents Guide - URL Manager

This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.

Project Overview

URL Manager (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.

Tech Stack

• Backend: Django 5.1, Django REST Framework, APScheduler
• Frontend: Tailwind CSS, AlpineJS (minimal JS)
• Database: SQLite (default, configurable)
• Task Queue: APScheduler (in-process background scheduler)
• Storage: Local filesystem or AWS S3/Cloudflare R2
• Web Automation: Selenium with Chromium (for screenshots and scraping)
• Mobile: SwiftUI iOS app with Core Data
• Deployment: Kubernetes (k8s manifests included)

Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                      Web Interface                          │
│         (Django Templates + Tailwind CSS)                   │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Django Application                        │
│  ┌──────────────┬──────────────┬──────────────────────┐    │
│  │ Links Module │ Pages Module │ Collections Module   │    │
│  │              │              │ (Images, Posts)      │    │
│  └──────────────┴──────────────┴──────────────────────┘    │
│  ┌────────────────────────────────────────────────────┐    │
│  │           REST API (DRF ViewSets)                  │    │
│  └────────────────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│       Background Tasks (APScheduler + Threading)            │
│  - Screenshot capture (background threads)                  │
│  - Page metadata extraction (background threads)            │
│  - Periodic task scheduling (APScheduler)                   │
└─────────────────────────────────────────────────────────────┘
                   │
┌──────────────────┴──────────────────────────────────────────┐
│                   Storage Layer                             │
│  - Local filesystem (default)                               │
│  - Cloudflare R2 / AWS S3 (optional)                        │
└─────────────────────────────────────────────────────────────┘

Data Models

Core Models (links/models.py)

Link

The central model for URL management:

• alias: Unique slug identifier for the short link
• original_url: Target URL (supports templates with {param,default=value})
• text: Markdown content (for custom pages)
• link_type: LINK (regular URL) or CUSTOM (markdown content)
• click_count: Usage analytics
• tags: Many-to-many relationship with Tag model
• description: Optional description

Template URL Feature: URLs can contain parameters like https://example.com/{query,default=test} that are resolved at access time.

Page

Bookmarked pages with auto-extracted metadata:

• url: Original page URL
• title: Auto-extracted or manual
• summary: Auto-extracted description
• content: Full page content (optional)
• screenshot: Associated Screenshot model
• tags: Many-to-many with Tag
• processing_status: PENDING, PROCESSING, COMPLETED, FAILED
• priority: For async processing queue

Tag

Hierarchical tagging system:

Bookmark

Archived social-media posts (originally x.com / Twitter bookmarks), each saved with full preview data so a viewer page renders offline without further x.com API calls:

• tweet_id: Source post id (unique, indexed)
• url: Canonical x.com post URL
• author_screen_name / author_name / author_profile_image_url: Denormalised author
• text: Full post text (searchable)
• summary: Optional human/agent-written summary (writable via API PATCH)
• tweet_created_at / bookmark_created_at: Parsed source dates (indexed)
• view_count / favorite_count / retweet_count / reply_count / bookmark_count / quote_count: Engagement counters
• has_video / has_photo / has_card / is_quote / is_retweet: Indexed boolean flags for fast filter pages
• media: JSON list of media objects (photo URL variants; video/animated_gif with poster, best_mp4, duration_ms, variants[])
• card_data: JSON link-preview card (title/description/site/image/url)
• quoted_data: JSON nested quoted post (recursive shape)
• data: The complete original normalised export object (kept for round-tripping / the offline viewer)
• UI: /ui/bookmarks/ (list, BookmarkListView) + /ui/bookmarks/<pk>/ (detail). Top-menu item "Bookmarks" in templates/base.html.
• Import: python manage.py import_x_bookmarks <dir|file.json> [--clear] (loads x-bookmarks-exporter JSON) or POST /api/bookmarks/import.
• No <video> streaming in the viewer — videos render as poster thumbnails linking to x.com (performance + ToS).
• Search: bookmarks are indexed in the Whoosh full-text index (search_backend.index_bookmark, rebuilt via python manage.py rebuild_search_index), wired into signals (post_save/post_delete on Bookmark auto-updates the index — e.g. when an agent PATCHes a summary). They appear in /ui/search/ (React app, type filter "Bookmarks"), /search/api/v2/?type=bookmark, and the simple /search/api/?q= lookup.
• name: Tag name
• slug: URL-friendly slug
• description: Optional description
• icon: Optional icon
• parent: Self-referential for hierarchy
• color: UI color code

ImageCollection & Image

Image gallery management:

• Collections group related images
• Support for Cloudflare R2 / S3 storage
• Automatic thumbnail generation
• Bulk upload capabilities

Post

Blog-like content management:

• Markdown content support
• Tag categorization
• Publication status tracking

mini_apps

A collection of mini apps which can be located at mini_apps_views.py

Analytics Models

• ClickLog: Individual click tracking with timestamps
• LinkChangeLog: Audit trail for modifications

API Architecture

REST API (Django REST Framework)

Located in various *_views.py files with corresponding *_urls.py:

Image API (api_views.py)

• ImageCollectionViewSet: CRUD for image collections
  • Custom action: upload_images - Bulk image upload with R2/S3 integration
• ImageViewSet: Image management with descriptions
• MusicViewSet: Music file management

Page API (page_views.py)

• PageViewSet: Full CRUD for bookmarked pages
  • screenshot action: Trigger screenshot capture
  • extract_metadata action: Re-extract page metadata

Bookmark API (bookmark_views.py → BookmarkViewSet, mounted at /api/bookmarks)

• BookmarkViewSet: Full CRUD for archived social-media bookmarks
  • GET /api/bookmarks: list with filters ?q=, ?has_video=true, ?has_photo=, ?has_card=, ?is_quote=, ?is_retweet=, ?author=, ?ordering=, ?page_size= (default 24, max 200)
  • POST /api/bookmarks: create one (flat Bookmark-shaped body)
  • GET/PATCH/DELETE /api/bookmarks/{id}: read / partial-update (e.g. write an AI summary) / delete
  • POST /api/bookmarks/import: bulk upsert from {"bookmarks":[ {export obj}, ... ]} — accepts the full x-bookmarks-exporter normalised object (nested author/media/card/quoted/retweet + x.com-style created_at date); matches by tweet_id. Returns {created, updated, skipped, total}
  • PUT|POST /api/bookmarks/bulk: bulk upsert from a bare JSON array of flat Bookmark-shaped objects
  • GET /api/bookmarks/stats: {total, has_video, has_photo, has_card, is_quote, is_retweet} (used by viewer filter chips)

Post API (post_views.py)

• PostViewSet: Blog post management with markdown rendering

API Documentation

• OpenAPI 3.0.3 spec available at /static/openapi.yaml (browse at http://localhost:8000/ui/api-docs/ via ReDoc)
• Documents all endpoints, request/response schemas, and authentication
• The spec is the contract. Any task that touches an API must keep it in sync (see the Agent Workflow below)

File API (file_views.py → FileUploadViewSet, mounted at /api/files)

• FileUploadViewSet: Full CRUD for generic file uploads
  • create: Accepts multipart/form-data with a files field (one or more files). Returns 201 with an array of FileUpload objects
  • destroy: Deletes the record and removes the file from disk
  • download action: Streams file content. Supports HTTP Range requests — replies with 206 Partial Content + Content-Range/Accept-Ranges headers so <video>/<audio> can seek. Inline disposition for images, video, and audio; attachment for other types
  • toggle-public action: Flips is_public and returns {is_public, public_url}
  • set-expiry action: Sets or clears an ISO-8601 expiry datetime on the public link
• Public access lives at /public/files/{id}-{filename} (filename is cosmetic; lookup is by UUID only). Internal-network requests serve any file; external requests only serve is_public=True files that haven't expired. Range support applies here too.

Background Task System (APScheduler)

The application uses APScheduler for background task scheduling, running in-process with the Django application. This eliminates the need for separate worker processes and Redis broker.

Tasks (links/tasks.py)

process_page(page_id, retry_count=0)

Asynchronously fetches page metadata:

1. Downloads page HTML
2. Extracts title, description, meta tags
3. Stores content in Page model
4. Updates processing status
5. Triggers screenshot capture

Features:

• Retry logic with exponential backoff (Fibonacci sequence)
• Timeout protection (20s)
• BeautifulSoup for HTML parsing
• Multiple metadata extraction strategies (title, h1, og:tags)
• Runs in background thread for non-blocking execution

capture_screenshot(page_id, screenshot_id, retry_count=0)

Captures website screenshots:

1. Launches headless Chromium via Selenium
2. Configures viewport and options
3. Takes screenshot
4. Uploads to storage (R2/S3 or local)
5. Updates Screenshot model

Configuration:

• Headless mode
• Custom user agent
• 60s timeout
• Supports custom viewport sizes
• Error handling and retry logic via APScheduler
• Runs in background thread

schedule_pending_pages()

Periodic maintenance task:

• Runs every 120 seconds
• Checks for pages with PENDING status
• Schedules processing for pending pages
• Respects retry limits and backoff

APScheduler Configuration

# core/scheduler.py
scheduler = BackgroundScheduler(
    executors={'default': ThreadPoolExecutor(20)},
    job_defaults={
        'coalesce': False,
        'max_instances': 3,
        'misfire_grace_time': 300
    },
    timezone=settings.TIME_ZONE
)

Initialization in core/apps.py:

• Scheduler starts automatically when Django application starts
• Periodic jobs registered in CoreConfig.ready() method
• No separate worker process needed

Task Execution

Tasks are executed using Python threading for immediate background execution:

from threading import Thread
thread = Thread(target=process_page, args=(page_id,))
thread.daemon = True
thread.start()

For scheduled/delayed tasks, APScheduler is used:

from core.scheduler import scheduler
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=delay)
scheduler.add_job(
    capture_screenshot,
    'date',
    run_date=run_date,
    args=[page_id, screenshot_id, retry_count + 1]
)

Key Features & Implementation

1. Template URL Processing

File: links/models.py - Link.get_template_parameters()

Extracts and processes URL parameters:

# URL: https://search.com/{query,default=test}&lang={lang,default=en}
# Renders to: https://search.com/python&lang=en (with query="python")

2. Asynchronous Page Processing

Files: links/tasks.py, links/page_views.py

When a page is bookmarked:

1. Page record created with processing_status=PENDING
2. Background thread started for metadata extraction
3. Background worker fetches and parses page
4. Title, description, content extracted
5. Screenshot capture triggered in separate thread
6. Status updated to COMPLETED

3. Search System

File: links/search_views.py

Multi-field search across:

• Link aliases, URLs, descriptions
• Page titles, content, summaries
• Tag names
• Full-text search with filtering
• Tag-based filtering
• Type-based filtering (links, pages, posts)

4. Storage Abstraction

File: links/storage.py - R2Storage

Unified interface for:

• Local filesystem storage
• Cloudflare R2 (S3-compatible)
• AWS S3

Environment variables:

• R2_ENDPOINT_URL
• R2_ACCESS_KEY_ID
• R2_SECRET_ACCESS_KEY
• R2_BUCKET_NAME

5. i18n Support

Locales: English (en), Simplified Chinese (zh_Hans)

Translation files in locale/ directory. Uses Django's i18n framework.

Development Guide for AI Agents

Agent Workflow (always follow before coding)

Before writing or modifying any feature, check the existing contract first to avoid breaking it:

1. Read the OpenAPI spec. Open static/openapi.yaml (or browse /ui/api-docs/) and find the paths/schemas the task touches. The spec is the source of truth for endpoint shapes, status codes, and fields. Contrast it with what the request asks for — if they differ, ask the user whether the spec or the request is the intended behaviour.
2. Verify the implementation against the spec. Inspect the relevant *_views.py, serializers.py, and model so the code you produce matches (or deliberately updates) the documented contract. Common drift: renamed fields, missing status codes (206, 416 for media ranges), wrong Content-Disposition, stale path templates (e.g. {token}/ vs {id}-{filename}).
3. Keep the spec in sync. If the code change alters any API behaviour, update static/openapi.yaml in the same change. Validate with python -c "import yaml; yaml.safe_load(open('static/openapi.yaml'))". Regenerate or sanity-check any derived docs that reference the spec.
4. Rebuild Tailwind when you touch templates. New utility classes are only included after npm run build:css (or just tailwind in watch mode). Serving stale CSS is a frequent source of "works in one viewport, broken in another" bugs (e.g. duplicate listings, hidden sections).
5. Run the tests and linters. Use just test (or pytest) for affected modules. Prefer an integration test that asserts the documented request → response shape, including status code and headers. Confirm black/isort pass if available.
6. Don't leave duplicates. Whether it's duplicate DOM rendering (desktop/mobile forks that rely on responsive CSS), duplicate file records, or duplicate submission — each piece of state should have exactly one source of truth. Prefer a single render driven by JS data over copy-pasted server-rendered blocks.

Engineering Best Practices

• Single source of truth for state. Render lists from one data array; never duplicate the same dataset in desktop and mobile markup expecting CSS to hide one.
• APIs must return everything the client needs. If the UI updates dynamically after a mutation (no full reload), the response must include the fields the client renders (e.g. formatted_size, mime_type, download_url, created_at).
• Media must be seekable. Any endpoint serving <video>/<audio> content must honour HTTP Range requests and return Accept-Ranges: bytes, 206 Partial Content, and Content-Range. Reuse the _stream_file() helper in links/file_views.py rather than raw FileResponse(open(...)).
• Prefer AJAX over full-page reloads for incremental updates (uploads, deletes, toggles). It avoids re-rendering the whole page and prevents re-submission on refresh.
• Responsive design without forking markup. Use flex/grid + hidden/block toggles on breakpoints instead of rendering a whole second table/card tree.
• Accessibility. Buttons/links need visible focus rings, tap targets ≥ 44 px on mobile, ARIA labels for icon-only controls, and sr-only labels for visually-hidden text.
• Tests are mandatory for API changes. Add or update a test that pins the contract (status code, body, headers). Run pytest before declaring done.

UI Style Guidelines

The home page is the canonical style reference. Every UI page in this app must align with the Apple-inspired design system defined in links/templates/links/link_list.html. Do not introduce ad-hoc Tailwind color palettes (e.g. red-600, gray-200 buttons) or generic card styles on new pages — reuse the home page's tokens and component classes so the whole app shares one visual language.

Canonical reference

• Home page template: links/templates/links/link_list.html — the {% block extra_css %} <style> block defines the design system. Copy its :root tokens and component classes into new pages rather than inventing new ones.
• Files page (links/templates/links/files/list.html) is an example of a non-home page that was restyled to match — use it as a secondary reference for list/card/button patterns outside the home page.

Design tokens (CSS custom properties)

Always declare these in the page's {% block extra_css %} <style> block:

:root {
  --apple-bg: #f5f5f7;            /* page canvas */
  --apple-text: #1d1d1f;          /* primary text */
  --apple-gray: #6e6e73;          /* secondary text */
  --apple-blue: #0071e3;          /* primary accent / links */
  --apple-blue-hover: #0077ed;
  --apple-red: #ff3b30;           /* destructive */
  --apple-green: #1e7b34;         /* success / public */
  --apple-separator: rgba(0, 0, 0, 0.06);
  --apple-ease: cubic-bezier(0.28, 0.11, 0.32, 1);
  --apple-radius-lg: 28px;        /* cards */
  --apple-radius-md: 22px;        /* modals / inner cards */
  --apple-shadow-card: 0 4px 24px rgba(0, 0, 0, 0.05);
  --apple-shadow-lift: 0 18px 44px rgba(0, 0, 0, 0.10);
}
body {
  background-color: var(--apple-bg) !important;
  font-family: -apple-system, BlinkMacSystemFont, "SF Pro Display", "SF Pro Text",
    "Helvetica Neue", "Segoe UI", Roboto, Arial, sans-serif;
  color: var(--apple-text);
  -webkit-font-smoothing: antialiased;
}

Reusable component classes (defined on the home page — copy, don't reinvent)

• Layout: .apple-wrap (max-width 72rem, centered)
• Cards: .apple-card (white, 28px radius, soft shadow)
• Section headers: .section-title (clamp 1.45–1.8rem, 700 weight, -0.022em tracking), .library-header, .library-count
• Buttons: .apple-btn base + .apple-btn-primary (blue pill), .apple-btn-secondary (gray pill), .apple-btn-danger (outline red). All pills: border-radius: 980px, min-height: 44px.
• Icon buttons: .icon-btn (40px circular, hover lift) with .blue / .green / .red / .amber modifiers
• Badges: .badge pill + .badge-public (green) / .badge-private (gray) / .badge-link / .badge-template / .badge-custom
• Links: .apple-link (blue, inline-flex with chevron), .alias-link
• Tables: .apple-table with uppercase tracked headers, separator borders, hover row tint
• Inputs: frosted-glass pill inputs (see .hero-search input / .library-filter input) — border-radius: 980px, focus ring 0 0 0 4px rgba(0,113,227,0.14)
• Modals: .apple-modal-bg (blurred dark backdrop) + .apple-modal (22px radius white card)
• Empty state: .empty-state (centered, gray, 2.6rem icon)
• Accessibility: a:focus-visible, button:focus-visible { outline: 2px solid var(--apple-blue); outline-offset: 3px; }

Rules

1. No raw Tailwind color utilities for primary UI. Use the --apple-* tokens via the component classes above. Tailwind utilities are fine for layout/spacing (flex, gap-4, hidden sm:block), not for the brand palette.
2. Pill buttons, circular icon buttons, 28px-radius cards — these shapes are the app's signature. Don't substitute rounded-md/rounded-lg for primary components.
3. Frosted glass for overlays. Modals, dropdowns, and the upload HUD use backdrop-filter: blur(20-24px) saturate(180%) over a semi-transparent white/dark background.
4. Soft shadows, never harsh. Cards use --apple-shadow-card; lifted/hover states use --apple-shadow-lift.
5. ** SF system font stack** on body — never hardcoded font-family on individual elements.
6. Respect prefers-reduced-motion. Disable transitions/animations under reduced motion.
7. When restyling an existing page, verify computed styles match the home page's (background rgb(245,245,247), card radius 28px, primary button rgb(0,113,227) / radius 980px). The Files page restyle is the reference for this check.

Common Tasks

Adding a New Model Field

1. Update model in links/models.py
2. Create migration: python manage.py makemigrations
3. Apply migration: python manage.py migrate
4. Update serializer in links/serializers.py (if API exposed)
5. Update forms in links/forms.py (if form-based)
6. Update templates in links/templates/

Adding a New API Endpoint

1. Add or update serializer in links/serializers.py
2. Create ViewSet (or function view) in the appropriate *_views.py file
3. Register the route in the matching *_urls.py
4. Update static/openapi.yaml — path, parameters, request schema, every response status code and headers, and any new component schema. Validate the YAML parses.
5. Write an integration test (tests/) that asserts the documented request → response shape (status code, body fields, key headers). Add range-media coverage when serving media.
6. Verify a fresh just dev server still returns 200 from /ui/api-docs/ and the live endpoint matches the spec

Adding a Background Task

1. Define task function in links/tasks.py (no decorator needed)
2. For immediate execution, use threading:

   from threading import Thread
   thread = Thread(target=task_function, args=(arg1,))
   thread.daemon = True
   thread.start()
   

3. For delayed/scheduled execution, use APScheduler:

   from core.scheduler import scheduler
   scheduler.add_job(task_function, 'date', run_date=run_time, args=[arg1])
   

4. For periodic tasks, add to core/apps.py in CoreConfig.ready():

   scheduler.add_job(
       task_function,
       'interval',
       seconds=120,
       id='task_id',
       replace_existing=True
   )
   

5. Add logging for debugging
6. Implement retry logic if needed

Adding a New Template View

1. Create view function in links/views.py or create new view file
2. Add URL pattern to links/urls.py or appropriate urls file
3. Create template in links/templates/
4. Follow the UI Style Guidelines above — reuse the home page's --apple-* tokens and component classes; don't invent new Tailwind palettes or generic card styles.
5. Add i18n translation strings
6. Update navigation if needed
7. Rebuild Tailwind CSS (npm run build:css or just tailwind) if new utility classes were introduced

Code Style Guidelines

• Python: Follow PEP 8, use Black formatter (line length: 100)
• Imports: Use isort with Black profile
• Type hints: Encouraged for new code
• Docstrings: Use for complex functions and classes
• Logging: Use Django's logging framework

import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message")
logger.error("Error message", exc_info=True)

Testing

Framework: pytest with pytest-django

Run tests:

pytest
pytest links/tests/test_models.py
pytest -k "test_link_creation"

Database

Default: SQLite at data/db.sqlite3

Migrations managed with Django migrations. Always create migrations for model changes:

python manage.py makemigrations
python manage.py migrate

Static Assets

Tailwind CSS: Compiled via django-tailwind

Development:

python manage.py tailwind start  # Watch mode

Production:

python manage.py tailwind build  # Minified build
python manage.py collectstatic   # Collect to staticfiles/

Docker Development

docker-compose.yml Services

• web: Django development server (port 8000) with APScheduler running in-process
• node: Tailwind CSS compiler

Note: Redis, Celery worker, and Celery beat services have been removed as they are no longer needed.

Helper Script: docker.sh

./docker.sh build      # Build images
./docker.sh start      # Start all services
./docker.sh stop       # Stop all services
./docker.sh logs       # View logs
./docker.sh migrate    # Run migrations
./docker.sh shell      # Django shell

Environment Variables (.env)

Create .env file:

DJANGO_SETTINGS_MODULE=core.settings
SECRET_KEY=your-secret-key
DEBUG=True
ALLOWED_HOSTS=*

# Optional: R2/S3 Storage
R2_ENDPOINT_URL=https://...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=...

iOS App (Heygo)

Architecture

• SwiftUI for declarative UI
• Core Data for local persistence
• MVVM pattern with Combine
• Charts framework for analytics

Syncing Strategy

Currently standalone (no backend sync). Future enhancement could:

1. Use Django REST API
2. Implement OAuth authentication
3. Sync via background refresh
4. Conflict resolution strategy

Key Files

• mobile/Heygo/HeygoApp.swift: App entry point
• mobile/Heygo/Views/: SwiftUI views
• mobile/Heygo/ViewModels/LinkViewModel.swift: Business logic
• mobile/Heygo/Models/: Core Data models

Deployment

Production Dockerfile

Multi-stage build:

1. Builder stage: Install dependencies, compile static assets, build translations
2. Production stage: Slim image with only runtime dependencies

Runtime requirements:

• Python 3.12
• Chromium + ChromeDriver (for screenshots)

Kubernetes

Manifests in k8s/manifest.yaml:

• Deployment for web service (includes APScheduler)
• Service for load balancing
• ConfigMap for configuration
• PersistentVolumeClaim for data

Note: Worker deployment is no longer needed as tasks run in-process.

Environment Setup

Production checklist:

• Set DEBUG=False
• Configure SECRET_KEY (strong random value)
• Set ALLOWED_HOSTS to actual domains
• Configure database (PostgreSQL recommended for production)
• Configure R2/S3 for media storage
• Set up SSL/TLS termination
• Configure backup strategy
• Set up monitoring and logging

Common Troubleshooting

Screenshots Not Generating

1. Check Chromium installation: which chromium
2. Check Django application logs for task errors
3. Verify APScheduler is running (check startup logs)
4. Review application logs for screenshot task errors
5. Verify storage configuration (R2/S3 credentials)

Metadata Extraction Failing

1. Check website accessibility (some sites block bots)
2. Verify timeout settings (increase if needed)
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
4. Review error logs in application logs

Migration Issues

1. Check for unapplied migrations: python manage.py showmigrations
2. Look for migration conflicts
3. Use python manage.py migrate --fake-initial cautiously
4. For complex issues, may need to squash migrations

Performance Issues

1. Add database indexes for frequently queried fields
2. Implement caching (Django cache framework)
3. Tune APScheduler thread pool size if needed (default: 20 threads)
4. Use database connection pooling
5. Enable query optimization (select_related, prefetch_related)

File Structure Guide

Core Django App Structure

core/
├── settings.py       # Django settings, installed apps, middleware
├── urls.py          # Main URL routing
├── scheduler.py     # APScheduler configuration
├── apps.py          # App configuration and scheduler initialization
└── middleware.py    # Custom middleware (locale, etc.)

links/
├── models.py        # Data models (Link, Page, Tag, etc.)
├── views.py         # Main template views
├── api_views.py     # REST API viewsets (Image, Music)
├── page_views.py    # Page-specific views and API
├── post_views.py    # Post/blog views and API
├── collection_views.py  # Collection management
├── search_views.py  # Search functionality
├── tag_views.py     # Tag management
├── forms.py         # Django forms
├── serializers.py   # DRF serializers
├── tasks.py         # Background task functions (APScheduler)
├── storage.py       # Storage abstraction (R2/S3)
├── urls.py          # Links app URL routing
├── *_urls.py        # Feature-specific URL routing
└── templates/       # Django templates

new_theme/
├── static/          # Tailwind compiled CSS
├── static_src/      # Tailwind source files
└── templates/       # Theme-specific templates

templates/
├── base.html        # Base template with navigation
└── base_blank.html  # Minimal base template

Key Configuration Files

• pyproject.toml: Python dependencies (managed by uv)
• uv.lock: Locked dependency versions
• package.json: Node.js dependencies (Tailwind)
• tailwind.config.js: Tailwind configuration
• docker-compose.yml: Development environment
• Dockerfile: Production image
• Dockerfile.local: Development image

API Authentication (Currently no auth is enabled, don't need to consider)

Currently, the API may not have authentication enabled. To add:

1. Add Django REST framework token authentication:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ]
}

2. Generate tokens for users:

from rest_framework.authtoken.models import Token
token = Token.objects.create(user=user)

Performance Optimization Tips

Database Queries

# Use select_related for foreign keys
pages = Page.objects.select_related('screenshot').all()

# Use prefetch_related for many-to-many
links = Link.objects.prefetch_related('tags').all()

# Add indexes to models
class Page(models.Model):
    url = models.URLField(db_index=True)  # Add index

Caching

from django.core.cache import cache

# Cache expensive operations
result = cache.get('key')
if result is None:
    result = expensive_operation()
    cache.set('key', result, timeout=3600)

APScheduler Task Optimization

# Configure thread pool size in core/scheduler.py
executors = {
    'default': ThreadPoolExecutor(20),  # Adjust based on workload
}

# Add job with proper configuration
scheduler.add_job(
    my_task,
    'interval',
    seconds=60,
    max_instances=3,  # Limit concurrent instances
    id='unique_job_id',
    replace_existing=True
)

# For one-time delayed tasks
from datetime import datetime, timedelta
run_date = datetime.now() + timedelta(seconds=300)
scheduler.add_job(my_task, 'date', run_date=run_date, args=[arg1])

Security Considerations

1. CSRF Protection: Enabled by default, ensure templates use {% csrf_token %}
2. SQL Injection: Use Django ORM (parameterized queries)
3. XSS: Django auto-escapes templates, use |safe filter cautiously
4. File Upload: Validate file types, scan for malware, limit sizes
5. API Rate Limiting: Implement throttling with DRF throttle classes
6. Secrets Management: Use environment variables, never commit secrets

Monitoring and Logging

Application Logging

# Configure in settings.py
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': '/app/logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    },
}

APScheduler Monitoring

• Check scheduler status: scheduler.running returns True/False
• List all jobs: scheduler.get_jobs()
• Monitor task execution via Django logs
• Track task execution times in application logs

Health Checks

Implement health check endpoint:

# views.py
def health_check(request):
    from core.scheduler import scheduler
    # Check database
    # Check APScheduler status
    scheduler_running = scheduler.running
    return JsonResponse({
        'status': 'healthy',
        'scheduler_running': scheduler_running
    })

Useful Commands Reference

just (Command Runner)

The project uses just as its command runner. Run just (no args) to list all recipes.

# Local development
just dev              # Run Django + Tailwind together
just tailwind         # Run Tailwind CSS watcher (standalone)
just migrate          # Apply pending migrations
just makemigrations   # Create new migrations
just shell            # Django shell
just dbshell          # Raw DB shell
just superuser        # Create a superuser
just build-css        # Build minified Tailwind CSS
just collectstatic    # Collect static files
just test             # Run tests
just compilemessages  # Compile i18n translations
just makemessages     # Extract translatable strings (zh_Hans)

# Docker
just docker-build           # Build images
just docker-start           # Start services (detached)
just docker-stop            # Stop services
just docker-logs            # Tail all service logs
just docker-logs web        # Tail a specific service's logs
just docker-shell           # Shell into web container
just docker-migrate         # Run migrations in Docker
just docker-static          # Collect static in Docker
just docker-rebuild web     # Rebuild + restart a service

Direct Django Management

When you need to run manage.py directly, activate the venv first:

source .venv/bin/activate
uv run manage.py <command>

APScheduler (In-Process)

APScheduler starts automatically with Django. To interact with it:

# In Django shell (just shell)
from core.scheduler import scheduler
scheduler.get_jobs()     # List all scheduled jobs
scheduler.print_jobs()   # Print job details
scheduler.running        # Check if scheduler is running

UV Package Manager

uv add package_name     # Add a new dependency
uv sync                 # Sync environment from pyproject.toml / uv.lock

Resources

• Django Documentation: https://docs.djangoproject.com/
• Django REST Framework: https://www.django-rest-framework.org/
• APScheduler Documentation: https://apscheduler.readthedocs.io/
• Tailwind CSS: https://tailwindcss.com/docs
• SwiftUI: https://developer.apple.com/documentation/swiftui/
• just: https://just.systems/man/en/

Note:

• No need to generate extra summary guide or docs, unless I ask you to.