mirror of
https://github.com/wahyd4/links.git
synced 2026-08-09 05:06:16 +10:00
769 lines
29 KiB
Markdown
769 lines
29 KiB
Markdown
# Agents Guide - URL Manager
|
|
|
|
This guide is designed to help AI coding agents understand and work with the URL Manager project effectively.
|
|
|
|
## Project Overview
|
|
|
|
**URL Manager** (also known as "Heygo" / 黑狗) is a comprehensive link management system built with Django and available as both a web application and native iOS app. The system provides short link creation, bookmark management with automatic metadata extraction, template URLs with parameters, collections, and advanced search capabilities.
|
|
|
|
### Tech Stack
|
|
|
|
- **Backend**: Django 5.1, Django REST Framework, APScheduler
|
|
- **Frontend**: Tailwind CSS, AlpineJS (minimal JS)
|
|
- **Database**: SQLite (default, configurable)
|
|
- **Task Queue**: APScheduler (in-process background scheduler)
|
|
- **Storage**: Local filesystem or AWS S3/Cloudflare R2
|
|
- **Web Automation**: Selenium with Chromium (for screenshots and scraping)
|
|
- **Mobile**: SwiftUI iOS app with Core Data
|
|
- **Deployment**: Kubernetes (k8s manifests included)
|
|
|
|
## Architecture
|
|
|
|
### Core Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Web Interface │
|
|
│ (Django Templates + Tailwind CSS) │
|
|
└──────────────────┬──────────────────────────────────────────┘
|
|
│
|
|
┌──────────────────┴──────────────────────────────────────────┐
|
|
│ Django Application │
|
|
│ ┌──────────────┬──────────────┬──────────────────────┐ │
|
|
│ │ Links Module │ Pages Module │ Collections Module │ │
|
|
│ │ │ │ (Images, Posts) │ │
|
|
│ └──────────────┴──────────────┴──────────────────────┘ │
|
|
│ ┌────────────────────────────────────────────────────┐ │
|
|
│ │ REST API (DRF ViewSets) │ │
|
|
│ └────────────────────────────────────────────────────┘ │
|
|
└──────────────────┬──────────────────────────────────────────┘
|
|
│
|
|
┌──────────────────┴──────────────────────────────────────────┐
|
|
│ Background Tasks (APScheduler + Threading) │
|
|
│ - Screenshot capture (background threads) │
|
|
│ - Page metadata extraction (background threads) │
|
|
│ - Periodic task scheduling (APScheduler) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│
|
|
┌──────────────────┴──────────────────────────────────────────┐
|
|
│ Storage Layer │
|
|
│ - Local filesystem (default) │
|
|
│ - Cloudflare R2 / AWS S3 (optional) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Data Models
|
|
|
|
### Core Models (links/models.py)
|
|
|
|
#### Link
|
|
The central model for URL management:
|
|
- **alias**: Unique slug identifier for the short link
|
|
- **original_url**: Target URL (supports templates with `{param,default=value}`)
|
|
- **text**: Markdown content (for custom pages)
|
|
- **link_type**: LINK (regular URL) or CUSTOM (markdown content)
|
|
- **click_count**: Usage analytics
|
|
- **tags**: Many-to-many relationship with Tag model
|
|
- **description**: Optional description
|
|
|
|
**Template URL Feature**: URLs can contain parameters like `https://example.com/{query,default=test}` that are resolved at access time.
|
|
|
|
#### Page
|
|
Bookmarked pages with auto-extracted metadata:
|
|
- **url**: Original page URL
|
|
- **title**: Auto-extracted or manual
|
|
- **summary**: Auto-extracted description
|
|
- **content**: Full page content (optional)
|
|
- **screenshot**: Associated Screenshot model
|
|
- **tags**: Many-to-many with Tag
|
|
- **processing_status**: PENDING, PROCESSING, COMPLETED, FAILED
|
|
- **priority**: For async processing queue
|
|
|
|
#### Tag
|
|
Hierarchical tagging system:
|
|
- **name**: Tag name
|
|
- **slug**: URL-friendly slug
|
|
- **description**: Optional description
|
|
- **icon**: Optional icon
|
|
- **parent**: Self-referential for hierarchy
|
|
- **color**: UI color code
|
|
|
|
#### ImageCollection & Image
|
|
Image gallery management:
|
|
- Collections group related images
|
|
- Support for Cloudflare R2 / S3 storage
|
|
- Automatic thumbnail generation
|
|
- Bulk upload capabilities
|
|
|
|
#### Post
|
|
Blog-like content management:
|
|
- Markdown content support
|
|
- Tag categorization
|
|
- Publication status tracking
|
|
|
|
#### mini_apps
|
|
A collection of mini apps which can be located at mini_apps_views.py
|
|
|
|
#### Analytics Models
|
|
- **ClickLog**: Individual click tracking with timestamps
|
|
- **LinkChangeLog**: Audit trail for modifications
|
|
|
|
## API Architecture
|
|
|
|
### REST API (Django REST Framework)
|
|
|
|
Located in various `*_views.py` files with corresponding `*_urls.py`:
|
|
|
|
#### Image API (`api_views.py`)
|
|
- `ImageCollectionViewSet`: CRUD for image collections
|
|
- Custom action: `upload_images` - Bulk image upload with R2/S3 integration
|
|
- `ImageViewSet`: Image management with descriptions
|
|
- `MusicViewSet`: Music file management
|
|
|
|
#### Page API (`page_views.py`)
|
|
- `PageViewSet`: Full CRUD for bookmarked pages
|
|
- `screenshot` action: Trigger screenshot capture
|
|
- `extract_metadata` action: Re-extract page metadata
|
|
|
|
#### Post API (`post_views.py`)
|
|
- `PostViewSet`: Blog post management with markdown rendering
|
|
|
|
### API Documentation
|
|
- OpenAPI 3.0.3 spec available at `/static/openapi.yaml` (browse at [http://localhost:8000/ui/api-docs/](http://localhost:8000/ui/api-docs/) via ReDoc)
|
|
- Documents all endpoints, request/response schemas, and authentication
|
|
- **The spec is the contract.** Any task that touches an API must keep it in sync (see the Agent Workflow below)
|
|
|
|
#### File API (`file_views.py` → `FileUploadViewSet`, mounted at `/api/files`)
|
|
- `FileUploadViewSet`: Full CRUD for generic file uploads
|
|
- `create`: Accepts `multipart/form-data` with a `files` field (one or more files). Returns `201` with an array of `FileUpload` objects
|
|
- `destroy`: Deletes the record and removes the file from disk
|
|
- `download` action: Streams file content. Supports HTTP `Range` requests — replies with `206 Partial Content` + `Content-Range`/`Accept-Ranges` headers so `<video>`/`<audio>` can seek. Inline disposition for images, video, and audio; `attachment` for other types
|
|
- `toggle-public` action: Flips `is_public` and returns `{is_public, public_url}`
|
|
- `set-expiry` action: Sets or clears an ISO-8601 expiry datetime on the public link
|
|
- Public access lives at `/public/files/{id}-{filename}` (filename is cosmetic; lookup is by UUID only). Internal-network requests serve any file; external requests only serve `is_public=True` files that haven't expired. Range support applies here too.
|
|
|
|
## Background Task System (APScheduler)
|
|
|
|
The application uses APScheduler for background task scheduling, running in-process with the Django application. This eliminates the need for separate worker processes and Redis broker.
|
|
|
|
### Tasks (links/tasks.py)
|
|
|
|
#### process_page(page_id, retry_count=0)
|
|
Asynchronously fetches page metadata:
|
|
1. Downloads page HTML
|
|
2. Extracts title, description, meta tags
|
|
3. Stores content in Page model
|
|
4. Updates processing status
|
|
5. Triggers screenshot capture
|
|
|
|
**Features**:
|
|
- Retry logic with exponential backoff (Fibonacci sequence)
|
|
- Timeout protection (20s)
|
|
- BeautifulSoup for HTML parsing
|
|
- Multiple metadata extraction strategies (title, h1, og:tags)
|
|
- Runs in background thread for non-blocking execution
|
|
|
|
#### capture_screenshot(page_id, screenshot_id, retry_count=0)
|
|
Captures website screenshots:
|
|
1. Launches headless Chromium via Selenium
|
|
2. Configures viewport and options
|
|
3. Takes screenshot
|
|
4. Uploads to storage (R2/S3 or local)
|
|
5. Updates Screenshot model
|
|
|
|
**Configuration**:
|
|
- Headless mode
|
|
- Custom user agent
|
|
- 60s timeout
|
|
- Supports custom viewport sizes
|
|
- Error handling and retry logic via APScheduler
|
|
- Runs in background thread
|
|
|
|
#### schedule_pending_pages()
|
|
Periodic maintenance task:
|
|
- Runs every 120 seconds
|
|
- Checks for pages with PENDING status
|
|
- Schedules processing for pending pages
|
|
- Respects retry limits and backoff
|
|
|
|
### APScheduler Configuration
|
|
|
|
```python
|
|
# core/scheduler.py
|
|
scheduler = BackgroundScheduler(
|
|
executors={'default': ThreadPoolExecutor(20)},
|
|
job_defaults={
|
|
'coalesce': False,
|
|
'max_instances': 3,
|
|
'misfire_grace_time': 300
|
|
},
|
|
timezone=settings.TIME_ZONE
|
|
)
|
|
```
|
|
|
|
**Initialization** in `core/apps.py`:
|
|
- Scheduler starts automatically when Django application starts
|
|
- Periodic jobs registered in `CoreConfig.ready()` method
|
|
- No separate worker process needed
|
|
|
|
### Task Execution
|
|
|
|
Tasks are executed using Python threading for immediate background execution:
|
|
```python
|
|
from threading import Thread
|
|
thread = Thread(target=process_page, args=(page_id,))
|
|
thread.daemon = True
|
|
thread.start()
|
|
```
|
|
|
|
For scheduled/delayed tasks, APScheduler is used:
|
|
```python
|
|
from core.scheduler import scheduler
|
|
from datetime import datetime, timedelta
|
|
run_date = datetime.now() + timedelta(seconds=delay)
|
|
scheduler.add_job(
|
|
capture_screenshot,
|
|
'date',
|
|
run_date=run_date,
|
|
args=[page_id, screenshot_id, retry_count + 1]
|
|
)
|
|
```
|
|
|
|
## Key Features & Implementation
|
|
|
|
### 1. Template URL Processing
|
|
**File**: `links/models.py` - `Link.get_template_parameters()`
|
|
|
|
Extracts and processes URL parameters:
|
|
```python
|
|
# URL: https://search.com/{query,default=test}&lang={lang,default=en}
|
|
# Renders to: https://search.com/python&lang=en (with query="python")
|
|
```
|
|
|
|
### 2. Asynchronous Page Processing
|
|
**Files**: `links/tasks.py`, `links/page_views.py`
|
|
|
|
When a page is bookmarked:
|
|
1. Page record created with `processing_status=PENDING`
|
|
2. Background thread started for metadata extraction
|
|
3. Background worker fetches and parses page
|
|
4. Title, description, content extracted
|
|
5. Screenshot capture triggered in separate thread
|
|
6. Status updated to COMPLETED
|
|
|
|
### 3. Search System
|
|
**File**: `links/search_views.py`
|
|
|
|
Multi-field search across:
|
|
- Link aliases, URLs, descriptions
|
|
- Page titles, content, summaries
|
|
- Tag names
|
|
- Full-text search with filtering
|
|
- Tag-based filtering
|
|
- Type-based filtering (links, pages, posts)
|
|
|
|
### 4. Storage Abstraction
|
|
**File**: `links/storage.py` - `R2Storage`
|
|
|
|
Unified interface for:
|
|
- Local filesystem storage
|
|
- Cloudflare R2 (S3-compatible)
|
|
- AWS S3
|
|
|
|
Environment variables:
|
|
- `R2_ENDPOINT_URL`
|
|
- `R2_ACCESS_KEY_ID`
|
|
- `R2_SECRET_ACCESS_KEY`
|
|
- `R2_BUCKET_NAME`
|
|
|
|
### 5. i18n Support
|
|
**Locales**: English (`en`), Simplified Chinese (`zh_Hans`)
|
|
|
|
Translation files in `locale/` directory. Uses Django's i18n framework.
|
|
|
|
## Development Guide for AI Agents
|
|
|
|
### Agent Workflow (always follow before coding)
|
|
|
|
Before writing or modifying any feature, **check the existing contract first** to avoid breaking it:
|
|
|
|
1. **Read the OpenAPI spec.** Open `static/openapi.yaml` (or browse [/ui/api-docs/](http://localhost:8000/ui/api-docs/)) and find the paths/schemas the task touches. The spec is the source of truth for endpoint shapes, status codes, and fields. Contrast it with what the request asks for — if they differ, ask the user whether the spec or the request is the intended behaviour.
|
|
2. **Verify the implementation against the spec.** Inspect the relevant `*_views.py`, `serializers.py`, and model so the code you produce matches (or deliberately updates) the documented contract. Common drift: renamed fields, missing status codes (`206`, `416` for media ranges), wrong `Content-Disposition`, stale path templates (e.g. `{token}/` vs `{id}-{filename}`).
|
|
3. **Keep the spec in sync.** If the code change alters any API behaviour, update `static/openapi.yaml` in the same change. Validate with `python -c "import yaml; yaml.safe_load(open('static/openapi.yaml'))"`. Regenerate or sanity-check any derived docs that reference the spec.
|
|
4. **Rebuild Tailwind when you touch templates.** New utility classes are only included after `npm run build:css` (or `just tailwind` in watch mode). Serving stale CSS is a frequent source of "works in one viewport, broken in another" bugs (e.g. duplicate listings, hidden sections).
|
|
5. **Run the tests and linters.** Use `just test` (or `pytest`) for affected modules. Prefer an integration test that asserts the documented request → response shape, including status code and headers. Confirm `black`/`isort` pass if available.
|
|
6. **Don't leave duplicates.** Whether it's duplicate DOM rendering (desktop/mobile forks that rely on responsive CSS), duplicate file records, or duplicate submission — each piece of state should have exactly one source of truth. Prefer a single render driven by JS data over copy-pasted server-rendered blocks.
|
|
|
|
### Engineering Best Practices
|
|
|
|
- **Single source of truth for state.** Render lists from one data array; never duplicate the same dataset in desktop and mobile markup expecting CSS to hide one.
|
|
- **APIs must return everything the client needs.** If the UI updates dynamically after a mutation (no full reload), the response must include the fields the client renders (e.g. `formatted_size`, `mime_type`, `download_url`, `created_at`).
|
|
- **Media must be seekable.** Any endpoint serving `<video>`/`<audio>` content must honour HTTP `Range` requests and return `Accept-Ranges: bytes`, `206 Partial Content`, and `Content-Range`. Reuse the `_stream_file()` helper in `links/file_views.py` rather than raw `FileResponse(open(...))`.
|
|
- **Prefer AJAX over full-page reloads** for incremental updates (uploads, deletes, toggles). It avoids re-rendering the whole page and prevents re-submission on refresh.
|
|
- **Responsive design without forking markup.** Use flex/grid + `hidden`/`block` toggles on breakpoints instead of rendering a whole second table/card tree.
|
|
- **Accessibility.** Buttons/links need visible focus rings, tap targets ≥ 44 px on mobile, ARIA labels for icon-only controls, and `sr-only` labels for visually-hidden text.
|
|
- **Tests are mandatory for API changes.** Add or update a test that pins the contract (status code, body, headers). Run `pytest` before declaring done.
|
|
|
|
### Common Tasks
|
|
|
|
#### Adding a New Model Field
|
|
1. Update model in `links/models.py`
|
|
2. Create migration: `python manage.py makemigrations`
|
|
3. Apply migration: `python manage.py migrate`
|
|
4. Update serializer in `links/serializers.py` (if API exposed)
|
|
5. Update forms in `links/forms.py` (if form-based)
|
|
6. Update templates in `links/templates/`
|
|
|
|
#### Adding a New API Endpoint
|
|
1. Add or update serializer in `links/serializers.py`
|
|
2. Create ViewSet (or function view) in the appropriate `*_views.py` file
|
|
3. Register the route in the matching `*_urls.py`
|
|
4. **Update `static/openapi.yaml`** — path, parameters, request schema, every response status code and headers, and any new component schema. Validate the YAML parses.
|
|
5. **Write an integration test** (`tests/`) that asserts the documented request → response shape (status code, body fields, key headers). Add range-media coverage when serving media.
|
|
6. Verify a fresh `just dev` server still returns `200` from `/ui/api-docs/` and the live endpoint matches the spec
|
|
|
|
#### Adding a Background Task
|
|
1. Define task function in `links/tasks.py` (no decorator needed)
|
|
2. For immediate execution, use threading:
|
|
```python
|
|
from threading import Thread
|
|
thread = Thread(target=task_function, args=(arg1,))
|
|
thread.daemon = True
|
|
thread.start()
|
|
```
|
|
3. For delayed/scheduled execution, use APScheduler:
|
|
```python
|
|
from core.scheduler import scheduler
|
|
scheduler.add_job(task_function, 'date', run_date=run_time, args=[arg1])
|
|
```
|
|
4. For periodic tasks, add to `core/apps.py` in `CoreConfig.ready()`:
|
|
```python
|
|
scheduler.add_job(
|
|
task_function,
|
|
'interval',
|
|
seconds=120,
|
|
id='task_id',
|
|
replace_existing=True
|
|
)
|
|
```
|
|
5. Add logging for debugging
|
|
6. Implement retry logic if needed
|
|
|
|
#### Adding a New Template View
|
|
1. Create view function in `links/views.py` or create new view file
|
|
2. Add URL pattern to `links/urls.py` or appropriate urls file
|
|
3. Create template in `links/templates/`
|
|
4. Add i18n translation strings
|
|
5. Update navigation if needed
|
|
|
|
### Code Style Guidelines
|
|
|
|
- **Python**: Follow PEP 8, use Black formatter (line length: 100)
|
|
- **Imports**: Use isort with Black profile
|
|
- **Type hints**: Encouraged for new code
|
|
- **Docstrings**: Use for complex functions and classes
|
|
- **Logging**: Use Django's logging framework
|
|
|
|
```python
|
|
import logging
|
|
logger = logging.getLogger(__name__)
|
|
logger.debug("Debug message")
|
|
logger.error("Error message", exc_info=True)
|
|
```
|
|
|
|
### Testing
|
|
|
|
**Framework**: pytest with pytest-django
|
|
|
|
Run tests:
|
|
```bash
|
|
pytest
|
|
pytest links/tests/test_models.py
|
|
pytest -k "test_link_creation"
|
|
```
|
|
|
|
### Database
|
|
|
|
**Default**: SQLite at `data/db.sqlite3`
|
|
|
|
Migrations managed with Django migrations. Always create migrations for model changes:
|
|
```bash
|
|
python manage.py makemigrations
|
|
python manage.py migrate
|
|
```
|
|
|
|
### Static Assets
|
|
|
|
**Tailwind CSS**: Compiled via django-tailwind
|
|
|
|
Development:
|
|
```bash
|
|
python manage.py tailwind start # Watch mode
|
|
```
|
|
|
|
Production:
|
|
```bash
|
|
python manage.py tailwind build # Minified build
|
|
python manage.py collectstatic # Collect to staticfiles/
|
|
```
|
|
|
|
## Docker Development
|
|
|
|
### docker-compose.yml Services
|
|
|
|
- **web**: Django development server (port 8000) with APScheduler running in-process
|
|
- **node**: Tailwind CSS compiler
|
|
|
|
Note: Redis, Celery worker, and Celery beat services have been removed as they are no longer needed.
|
|
|
|
### Helper Script: docker.sh
|
|
|
|
```bash
|
|
./docker.sh build # Build images
|
|
./docker.sh start # Start all services
|
|
./docker.sh stop # Stop all services
|
|
./docker.sh logs # View logs
|
|
./docker.sh migrate # Run migrations
|
|
./docker.sh shell # Django shell
|
|
```
|
|
|
|
### Environment Variables (.env)
|
|
|
|
Create `.env` file:
|
|
```env
|
|
DJANGO_SETTINGS_MODULE=core.settings
|
|
SECRET_KEY=your-secret-key
|
|
DEBUG=True
|
|
ALLOWED_HOSTS=*
|
|
|
|
# Optional: R2/S3 Storage
|
|
R2_ENDPOINT_URL=https://...
|
|
R2_ACCESS_KEY_ID=...
|
|
R2_SECRET_ACCESS_KEY=...
|
|
R2_BUCKET_NAME=...
|
|
```
|
|
|
|
## iOS App (Heygo)
|
|
|
|
### Architecture
|
|
- **SwiftUI** for declarative UI
|
|
- **Core Data** for local persistence
|
|
- **MVVM** pattern with Combine
|
|
- **Charts** framework for analytics
|
|
|
|
### Syncing Strategy
|
|
Currently standalone (no backend sync). Future enhancement could:
|
|
1. Use Django REST API
|
|
2. Implement OAuth authentication
|
|
3. Sync via background refresh
|
|
4. Conflict resolution strategy
|
|
|
|
### Key Files
|
|
- `mobile/Heygo/HeygoApp.swift`: App entry point
|
|
- `mobile/Heygo/Views/`: SwiftUI views
|
|
- `mobile/Heygo/ViewModels/LinkViewModel.swift`: Business logic
|
|
- `mobile/Heygo/Models/`: Core Data models
|
|
|
|
## Deployment
|
|
|
|
### Production Dockerfile
|
|
|
|
Multi-stage build:
|
|
1. **Builder stage**: Install dependencies, compile static assets, build translations
|
|
2. **Production stage**: Slim image with only runtime dependencies
|
|
|
|
**Runtime requirements**:
|
|
- Python 3.12
|
|
- Chromium + ChromeDriver (for screenshots)
|
|
|
|
### Kubernetes
|
|
|
|
Manifests in `k8s/manifest.yaml`:
|
|
- Deployment for web service (includes APScheduler)
|
|
- Service for load balancing
|
|
- ConfigMap for configuration
|
|
- PersistentVolumeClaim for data
|
|
|
|
Note: Worker deployment is no longer needed as tasks run in-process.
|
|
|
|
### Environment Setup
|
|
|
|
Production checklist:
|
|
- [ ] Set `DEBUG=False`
|
|
- [ ] Configure `SECRET_KEY` (strong random value)
|
|
- [ ] Set `ALLOWED_HOSTS` to actual domains
|
|
- [ ] Configure database (PostgreSQL recommended for production)
|
|
- [ ] Configure R2/S3 for media storage
|
|
- [ ] Set up SSL/TLS termination
|
|
- [ ] Configure backup strategy
|
|
- [ ] Set up monitoring and logging
|
|
|
|
## Common Troubleshooting
|
|
|
|
### Screenshots Not Generating
|
|
1. Check Chromium installation: `which chromium`
|
|
2. Check Django application logs for task errors
|
|
3. Verify APScheduler is running (check startup logs)
|
|
4. Review application logs for screenshot task errors
|
|
5. Verify storage configuration (R2/S3 credentials)
|
|
|
|
### Metadata Extraction Failing
|
|
1. Check website accessibility (some sites block bots)
|
|
2. Verify timeout settings (increase if needed)
|
|
3. Check for JavaScript-heavy sites (may need Selenium instead of requests)
|
|
4. Review error logs in application logs
|
|
|
|
### Migration Issues
|
|
1. Check for unapplied migrations: `python manage.py showmigrations`
|
|
2. Look for migration conflicts
|
|
3. Use `python manage.py migrate --fake-initial` cautiously
|
|
4. For complex issues, may need to squash migrations
|
|
|
|
### Performance Issues
|
|
1. Add database indexes for frequently queried fields
|
|
2. Implement caching (Django cache framework)
|
|
3. Tune APScheduler thread pool size if needed (default: 20 threads)
|
|
4. Use database connection pooling
|
|
5. Enable query optimization (select_related, prefetch_related)
|
|
|
|
## File Structure Guide
|
|
|
|
### Core Django App Structure
|
|
|
|
```
|
|
core/
|
|
├── settings.py # Django settings, installed apps, middleware
|
|
├── urls.py # Main URL routing
|
|
├── scheduler.py # APScheduler configuration
|
|
├── apps.py # App configuration and scheduler initialization
|
|
└── middleware.py # Custom middleware (locale, etc.)
|
|
|
|
links/
|
|
├── models.py # Data models (Link, Page, Tag, etc.)
|
|
├── views.py # Main template views
|
|
├── api_views.py # REST API viewsets (Image, Music)
|
|
├── page_views.py # Page-specific views and API
|
|
├── post_views.py # Post/blog views and API
|
|
├── collection_views.py # Collection management
|
|
├── search_views.py # Search functionality
|
|
├── tag_views.py # Tag management
|
|
├── forms.py # Django forms
|
|
├── serializers.py # DRF serializers
|
|
├── tasks.py # Background task functions (APScheduler)
|
|
├── storage.py # Storage abstraction (R2/S3)
|
|
├── urls.py # Links app URL routing
|
|
├── *_urls.py # Feature-specific URL routing
|
|
└── templates/ # Django templates
|
|
|
|
new_theme/
|
|
├── static/ # Tailwind compiled CSS
|
|
├── static_src/ # Tailwind source files
|
|
└── templates/ # Theme-specific templates
|
|
|
|
templates/
|
|
├── base.html # Base template with navigation
|
|
└── base_blank.html # Minimal base template
|
|
```
|
|
|
|
### Key Configuration Files
|
|
|
|
- `pyproject.toml`: Python dependencies (managed by uv)
|
|
- `uv.lock`: Locked dependency versions
|
|
- `package.json`: Node.js dependencies (Tailwind)
|
|
- `tailwind.config.js`: Tailwind configuration
|
|
- `docker-compose.yml`: Development environment
|
|
- `Dockerfile`: Production image
|
|
- `Dockerfile.local`: Development image
|
|
|
|
## API Authentication (Currently no auth is enabled, don't need to consider)
|
|
|
|
Currently, the API may not have authentication enabled. To add:
|
|
|
|
1. Add Django REST framework token authentication:
|
|
```python
|
|
# settings.py
|
|
REST_FRAMEWORK = {
|
|
'DEFAULT_AUTHENTICATION_CLASSES': [
|
|
'rest_framework.authentication.TokenAuthentication',
|
|
],
|
|
'DEFAULT_PERMISSION_CLASSES': [
|
|
'rest_framework.permissions.IsAuthenticated',
|
|
]
|
|
}
|
|
```
|
|
|
|
2. Generate tokens for users:
|
|
```python
|
|
from rest_framework.authtoken.models import Token
|
|
token = Token.objects.create(user=user)
|
|
```
|
|
|
|
## Performance Optimization Tips
|
|
|
|
### Database Queries
|
|
```python
|
|
# Use select_related for foreign keys
|
|
pages = Page.objects.select_related('screenshot').all()
|
|
|
|
# Use prefetch_related for many-to-many
|
|
links = Link.objects.prefetch_related('tags').all()
|
|
|
|
# Add indexes to models
|
|
class Page(models.Model):
|
|
url = models.URLField(db_index=True) # Add index
|
|
```
|
|
|
|
### Caching
|
|
```python
|
|
from django.core.cache import cache
|
|
|
|
# Cache expensive operations
|
|
result = cache.get('key')
|
|
if result is None:
|
|
result = expensive_operation()
|
|
cache.set('key', result, timeout=3600)
|
|
```
|
|
|
|
### APScheduler Task Optimization
|
|
```python
|
|
# Configure thread pool size in core/scheduler.py
|
|
executors = {
|
|
'default': ThreadPoolExecutor(20), # Adjust based on workload
|
|
}
|
|
|
|
# Add job with proper configuration
|
|
scheduler.add_job(
|
|
my_task,
|
|
'interval',
|
|
seconds=60,
|
|
max_instances=3, # Limit concurrent instances
|
|
id='unique_job_id',
|
|
replace_existing=True
|
|
)
|
|
|
|
# For one-time delayed tasks
|
|
from datetime import datetime, timedelta
|
|
run_date = datetime.now() + timedelta(seconds=300)
|
|
scheduler.add_job(my_task, 'date', run_date=run_date, args=[arg1])
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
1. **CSRF Protection**: Enabled by default, ensure templates use `{% csrf_token %}`
|
|
2. **SQL Injection**: Use Django ORM (parameterized queries)
|
|
3. **XSS**: Django auto-escapes templates, use `|safe` filter cautiously
|
|
4. **File Upload**: Validate file types, scan for malware, limit sizes
|
|
5. **API Rate Limiting**: Implement throttling with DRF throttle classes
|
|
6. **Secrets Management**: Use environment variables, never commit secrets
|
|
|
|
## Monitoring and Logging
|
|
|
|
### Application Logging
|
|
```python
|
|
# Configure in settings.py
|
|
LOGGING = {
|
|
'version': 1,
|
|
'handlers': {
|
|
'file': {
|
|
'level': 'INFO',
|
|
'class': 'logging.FileHandler',
|
|
'filename': '/app/logs/django.log',
|
|
},
|
|
},
|
|
'loggers': {
|
|
'django': {
|
|
'handlers': ['file'],
|
|
'level': 'INFO',
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
### APScheduler Monitoring
|
|
- Check scheduler status: `scheduler.running` returns True/False
|
|
- List all jobs: `scheduler.get_jobs()`
|
|
- Monitor task execution via Django logs
|
|
- Track task execution times in application logs
|
|
|
|
### Health Checks
|
|
Implement health check endpoint:
|
|
```python
|
|
# views.py
|
|
def health_check(request):
|
|
from core.scheduler import scheduler
|
|
# Check database
|
|
# Check APScheduler status
|
|
scheduler_running = scheduler.running
|
|
return JsonResponse({
|
|
'status': 'healthy',
|
|
'scheduler_running': scheduler_running
|
|
})
|
|
```
|
|
|
|
## Useful Commands Reference
|
|
|
|
### just (Command Runner)
|
|
The project uses [`just`](https://github.com/casey/just) as its command runner. Run `just` (no args) to list all recipes.
|
|
|
|
```bash
|
|
# Local development
|
|
just dev # Run Django + Tailwind together
|
|
just tailwind # Run Tailwind CSS watcher (standalone)
|
|
just migrate # Apply pending migrations
|
|
just makemigrations # Create new migrations
|
|
just shell # Django shell
|
|
just dbshell # Raw DB shell
|
|
just superuser # Create a superuser
|
|
just build-css # Build minified Tailwind CSS
|
|
just collectstatic # Collect static files
|
|
just test # Run tests
|
|
just compilemessages # Compile i18n translations
|
|
just makemessages # Extract translatable strings (zh_Hans)
|
|
|
|
# Docker
|
|
just docker-build # Build images
|
|
just docker-start # Start services (detached)
|
|
just docker-stop # Stop services
|
|
just docker-logs # Tail all service logs
|
|
just docker-logs web # Tail a specific service's logs
|
|
just docker-shell # Shell into web container
|
|
just docker-migrate # Run migrations in Docker
|
|
just docker-static # Collect static in Docker
|
|
just docker-rebuild web # Rebuild + restart a service
|
|
```
|
|
|
|
### Direct Django Management
|
|
When you need to run `manage.py` directly, activate the venv first:
|
|
```bash
|
|
source .venv/bin/activate
|
|
uv run manage.py <command>
|
|
```
|
|
|
|
### APScheduler (In-Process)
|
|
APScheduler starts automatically with Django. To interact with it:
|
|
```python
|
|
# In Django shell (just shell)
|
|
from core.scheduler import scheduler
|
|
scheduler.get_jobs() # List all scheduled jobs
|
|
scheduler.print_jobs() # Print job details
|
|
scheduler.running # Check if scheduler is running
|
|
```
|
|
|
|
### UV Package Manager
|
|
```bash
|
|
uv add package_name # Add a new dependency
|
|
uv sync # Sync environment from pyproject.toml / uv.lock
|
|
```
|
|
|
|
## Resources
|
|
|
|
- **Django Documentation**: https://docs.djangoproject.com/
|
|
- **Django REST Framework**: https://www.django-rest-framework.org/
|
|
- **APScheduler Documentation**: https://apscheduler.readthedocs.io/
|
|
- **Tailwind CSS**: https://tailwindcss.com/docs
|
|
- **SwiftUI**: https://developer.apple.com/documentation/swiftui/
|
|
- **just**: https://just.systems/man/en/
|
|
|
|
Note:
|
|
- No need to generate extra summary guide or docs, unless I ask you to.
|