Merge pull request #5 from wahyd4/feat/published-date

Add format=llm support which get rid of extra content and only focus on title, url, snippet

FEATURES.md

@@ -86,12 +86,23 @@
 
 - Full REST API for search, autocomplete, engine management, bookmarks, history, stats, and settings
 - `/api/search` accepts both **GET and POST** requests with query string parameters
+- **`format=llm`** — minimal LLM-optimised response: only `query`, `results` (title, url, snippet, date), and `total_results`; no engine noise
+- **`max_results`** — hard limit on returned results (1–100); `numResults` is a supported alias
 - Compatibility parameters: `pageNumber` (alias for `page`), `numResults`, `format`, `imageProxy`, `safesearch`
 - Every result includes `result_id`, `rank`, `engine`, and `published_date`
 - `has_next` and `total_results` fields for cursor-aware pagination
 - `X-Response-Time-Ms` response header on all endpoints
 - OpenAPI specification with interactive docs via [Swagger UI](https://swagger.io/tools/swagger-ui/) (`/docs`) and [Redoc](https://github.com/Redocly/redoc) (`/redoc`)
 
+## MCP Tool Server
+
+- **`/api/mcp`** — [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) endpoint; exposes HeySearch as a native tool for LLMs
+- Compatible with Claude Desktop, Cursor, Continue, VS Code Copilot, and any MCP-capable client
+- **Transport**: Streamable HTTP (JSON-RPC 2.0 POST)
+- **Available tools**: `search` (web + image search) and `autocomplete`
+- `search` tool supports `query`, `category`, `num_results` (1–20), `engines`, `sort`, and `date_filter` arguments
+- Add to Claude Desktop by pointing `url` at `http://your-host/api/mcp` with `"transport": "http"`
+
 ## Reliability
 
 - **Retry mechanism** — failed upstream requests are retried with exponential backoff (via [tenacity](https://github.com/jd/tenacity), 2 attempts)

README.md

@@ -20,7 +20,7 @@ Both are open-source, self-hosted, privacy-respecting metasearch engines. Here's
 |---|---|---|
 | **Setup** | `docker compose up -d` — one command, zero config | Requires YAML config, engine tuning, sometimes breaks |
 | **UI** | Modern, clean React UI with dark mode, background images, image lightbox | Functional but dated — not mobile friendly |
-| **AI agent friendly** | Clean JSON REST API, OpenAPI docs at `/docs`, designed to be queried programmatically | API exists but less documented; HTML-heavy responses |
+| **AI agent friendly** | MCP tool server at `/api/mcp` (Claude Desktop, Cursor, Continue), `format=llm` for minimal responses, clean JSON REST API, OpenAPI docs at `/docs` | API exists but less documented; HTML-heavy responses |
 | **Bookmarks** | Built-in bookmark manager for results | ❌ |
 | **Search history** | Full search history with timestamps, re-run any past query in one click | ❌ |
 | **Usage stats** | Built-in analytics dashboard — top queries, click-through rates, engine usage | ❌ |
@@ -90,6 +90,7 @@ The `/app/data` volume stores the SQLite database (engine settings, excluded dom
 | GET        | `/api/settings`             | Get app settings (cache TTL)|
 | PUT        | `/api/settings`             | Update settings             |
 | DELETE     | `/api/cache`                | Flush search cache          |
+| POST       | `/api/mcp`                  | MCP tool server (for LLMs)  |
 
 ### Search endpoint parameters
 
@@ -99,37 +100,84 @@ The `/app/data` volume stores the SQLite database (engine settings, excluded dom
 | `category`   | `web`    | `web` or `images`                                    |
 | `page`       | `1`      | Page number (1–50)                                   |
 | `pageNumber` | —        | Alias for `page` (takes precedence when provided)    |
-| `numResults` | —        | Requested result count hint (informational)          |
+| `max_results`| —        | Hard limit on results returned (1–100)               |
-| `format`     | —        | Response format hint (e.g. `json`)                   |
+| `numResults` | —        | Alias for `max_results`                              |
+| `format`     | —        | `llm` for minimal LLM-friendly response (see below)  |
 | `imageProxy` | —        | Client image-proxy preference flag (informational)   |
 | `safesearch` | —        | Safe search level: `0` off, `1` moderate, `2` strict |
 | `engines`    | —        | Comma-separated engine names to restrict (e.g. `google,bing`) |
 | `image_size` | —        | `large`, `medium`, or `small` (images only)          |
 | `sort`       | `default`| `default`, `date_asc`, or `date_desc`                |
+| `date_filter`| —        | `day`, `week`, `month`, or `year`                    |
 
-## Using with AI Agents / curl
+## Using with AI Agents / LLMs
 
-The `/api/search` endpoint returns clean JSON — ideal for LLMs and AI agents to consume directly.
+HeySearch is designed to be used by LLMs and AI agents. There are two integration methods:
 
-```bash
+### 1. MCP Tool Server (recommended)
-# Web search
-curl "http://localhost:8000/api/search?q=python+async&format=json" | jq
 
-# Restrict to specific engines
+[Model Context Protocol](https://modelcontextprotocol.io/) (MCP) is the standard for LLM tool use. Add HeySearch to any MCP-compatible client:
-curl "http://localhost:8000/api/search?q=rust+programming&engines=brave,google" | jq
 
-# Image search
+**Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
-curl "http://localhost:8000/api/search?q=mountain+landscape&category=images&image_size=large" | jq
+```json
-
+{
-# Paginate results
+  "mcpServers": {
-curl "http://localhost:8000/api/search?q=machine+learning&page=2" | jq
+    "heysearch": {
-
+      "url": "http://localhost:8000/api/mcp",
-# Extract just titles and URLs from web results
+      "transport": "http"
-curl "http://localhost:8000/api/search?q=openai" | \
+    }
-  jq '[.results[] | {title, url, snippet}]'
+  }
+}
 ```
 
+**Cursor / Continue / VS Code Copilot** — add `http://localhost:8000/api/mcp` as an MCP server URL in the tool settings.
 
+**Manual test:**
+```bash
+# List available tools
+curl -X POST http://localhost:8000/api/mcp \
+  -H 'Content-Type: application/json' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
+
+# Call the search tool
+curl -X POST http://localhost:8000/api/mcp \
+  -H 'Content-Type: application/json' \
+  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"python async","num_results":3}}}'
+```
+
+**Available MCP tools:** `search`, `autocomplete`
+
+### 2. REST API with `format=llm`
+
+For direct API calls from LLM agents, use `format=llm` to get a minimal, token-efficient response:
+
+```bash
+# LLM-optimised response — only title, url, snippet, date. No engine noise.
+curl "http://localhost:8000/api/search?q=python+async&format=llm&max_results=5" | jq
+```
+
+Response shape:
+```json
+{
+  "query": "python async",
+  "category": "web",
+  "results": [
+    { "title": "...", "url": "https://...", "snippet": "...", "date": "2024-01-15" }
+  ],
+  "total_results": 5
+}
+```
+
+```bash
+# Restrict to specific engines
+curl "http://localhost:8000/api/search?q=rust+programming&engines=brave,google&format=llm" | jq
+
+# Image search with size filter
+curl "http://localhost:8000/api/search?q=mountain+landscape&category=images&image_size=large" | jq
+
+# Limit results (hard limit, not a hint)
+curl "http://localhost:8000/api/search?q=openai&max_results=3&format=llm" | jq
+```
 
 > Interactive API docs (Swagger UI) are available at `http://localhost:8000/docs`.
 

backend/app/api/routes.py

@@ -7,7 +7,7 @@ from fastapi import APIRouter, Query, Request
 from fastapi.responses import JSONResponse
 from pydantic import BaseModel, Field
 
-from app.models import SearchResponse, EngineInfo, APIError
+from app.models import SearchResponse, EngineInfo, APIError, LLMWebResult, LLMImageResult, LLMSearchResponse
 from app.search import search, get_autocomplete
 from app.engines import registry
 from app.excluded import get_excluded_domains, add_excluded_domain, remove_excluded_domain
@@ -35,6 +35,12 @@ Supports both GET and POST methods with query parameters.
 curl '$BASE_URL/api/search?q=hello+world&category=web&page=1'
 ```
 
+**LLM / AI-agent optimised response (`format=llm`):**
+```bash
+curl '$BASE_URL/api/search?q=hello+world&format=llm&max_results=5'
+```
+Returns a minimal JSON response with only `query`, `results` (title, url, snippet, date), and `total_results` — ideal for RAG pipelines and tool-calling.
+
 **Example response (truncated):**
 ```json
 {
@@ -75,8 +81,9 @@ async def api_search(
     category: Literal["web", "images"] = Query("web", description="Search category"),
     page: int = Query(1, ge=1, le=50, description="Page number"),
     pageNumber: int | None = Query(None, ge=1, le=50, description="Alias for page (1-based page number)"),
-    numResults: int | None = Query(None, ge=1, le=100, description="Number of results requested (informational)"),
+    numResults: int | None = Query(None, ge=1, le=100, description="Number of results to return (applies as hard limit)"),
-    format: str | None = Query(None, description="Response format hint (e.g. 'json')"),
+    max_results: int | None = Query(None, ge=1, le=100, description="Maximum number of results to return"),
+    format: str | None = Query(None, description="Response format: 'llm' for a minimal LLM-friendly response, omit for full JSON"),
     imageProxy: bool | None = Query(None, description="Whether the client wants image proxying"),
     safesearch: str | None = Query(None, description="Safe search level (0=off, 1=moderate, 2=strict)"),
     engines: str | None = Query(None, description="Comma-separated engine names to use (e.g. 'google,bing')"),
@@ -86,7 +93,9 @@ async def api_search(
 ):
     effective_page = pageNumber if pageNumber is not None else page
     engine_list = [e.strip() for e in engines.split(",")] if engines else None
-    result = await search(q, category=category, page=effective_page, engines=engine_list, image_size=image_size, sort=sort, date_filter=date_filter)
+    # max_results takes precedence; numResults is a supported alias
+    effective_max = max_results if max_results is not None else numResults
+    result = await search(q, category=category, page=effective_page, engines=engine_list, image_size=image_size, sort=sort, date_filter=date_filter, max_results=effective_max)
     origin_ip = request.client.host if request.client else ""
     user_agent = request.headers.get("user-agent", "")
     _stats.record_search(
@@ -97,6 +106,33 @@ async def api_search(
         result_count=result.total_results,
         cached=result.cached,
     )
+
+    if format == "llm":
+        llm_results: list[LLMWebResult | LLMImageResult] = []
+        for r in result.results:
+            if category == "images":
+                llm_results.append(LLMImageResult(
+                    title=r.title,
+                    url=r.url,
+                    img_src=getattr(r, "img_src", ""),
+                    date=r.published_date or "",
+                ))
+            else:
+                llm_results.append(LLMWebResult(
+                    title=r.title,
+                    url=r.url,
+                    snippet=getattr(r, "content", ""),
+                    date=r.published_date or "",
+                ))
+        # Return JSONResponse directly to bypass response_model=SearchResponse
+        # coercion, which would otherwise strip LLM-only fields (snippet, date).
+        return JSONResponse(content=LLMSearchResponse(
+            query=result.query,
+            category=result.category,
+            results=llm_results,
+            total_results=len(llm_results),
+        ).model_dump())
+
     return result
 
 

backend/app/main.py

@@ -13,6 +13,7 @@ from fastapi.responses import JSONResponse
 from fastapi.middleware.cors import CORSMiddleware
 
 from app.api.routes import router
+from app.mcp_server import router as mcp_router
 from app.engines import registry
 from app.excluded import init_db
 from app.settings import init_settings_table, set_setting
@@ -131,6 +132,7 @@ async def global_exception_handler(request: Request, exc: Exception):
 
 
 app.include_router(router, prefix="/api")
+app.include_router(mcp_router, prefix="/api")
 
 # Alias /search → /api/search for compatibility with external clients
 @app.api_route("/search", methods=["GET", "POST"], include_in_schema=False)

backend/app/mcp_server.py

@@ -0,0 +1,313 @@
+"""MCP (Model Context Protocol) server for HeySearch.
+
+Exposes HeySearch as an MCP tool server so LLMs (Claude, Cursor, Continue,
+VS Code Copilot, etc.) can invoke search directly via the standard MCP
+JSON-RPC 2.0 protocol.
+
+Transport: Streamable HTTP — clients POST JSON-RPC messages to /mcp.
+
+Supported methods:
+  initialize       — capability negotiation
+  tools/list       — enumerate available tools
+  tools/call       — invoke a tool (search, autocomplete)
+  ping             — liveness check
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any
+
+from fastapi import APIRouter, Request
+from fastapi.responses import JSONResponse, Response
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(tags=["MCP"])
+
+MCP_PROTOCOL_VERSION = "2024-11-05"
+SERVER_INFO = {"name": "HeySearch", "version": "1.4.0"}
+
+# ---------------------------------------------------------------------------
+# Tool definitions (JSON Schema)
+# ---------------------------------------------------------------------------
+
+_SEARCH_TOOL: dict[str, Any] = {
+    "name": "search",
+    "description": (
+        "Search the web or images using the HeySearch privacy-respecting "
+        "metasearch engine. Results are aggregated from Brave, DuckDuckGo, "
+        "Google, and Bing and deduplicated. Returns titles, URLs, snippets, "
+        "and publication dates."
+    ),
+    "inputSchema": {
+        "type": "object",
+        "properties": {
+            "query": {
+                "type": "string",
+                "description": "The search query string.",
+            },
+            "category": {
+                "type": "string",
+                "enum": ["web", "images"],
+                "default": "web",
+                "description": "Search category: 'web' for text results, 'images' for image results.",
+            },
+            "num_results": {
+                "type": "integer",
+                "minimum": 1,
+                "maximum": 20,
+                "default": 5,
+                "description": "Maximum number of results to return (1–20).",
+            },
+            "engines": {
+                "type": "string",
+                "description": (
+                    "Comma-separated engine names to restrict the search "
+                    "(e.g. 'google,bing'). Omit to use all enabled engines."
+                ),
+            },
+            "sort": {
+                "type": "string",
+                "enum": ["default", "date_asc", "date_desc"],
+                "default": "default",
+                "description": "Sort order: default (relevance), date_asc, or date_desc.",
+            },
+            "date_filter": {
+                "type": "string",
+                "enum": ["", "day", "week", "month", "year"],
+                "default": "",
+                "description": "Filter results by recency: day (24 h), week, month, or year.",
+            },
+        },
+        "required": ["query"],
+    },
+}
+
+_AUTOCOMPLETE_TOOL: dict[str, Any] = {
+    "name": "autocomplete",
+    "description": (
+        "Get search query autocomplete suggestions from HeySearch. "
+        "Useful for expanding or refining a partial query."
+    ),
+    "inputSchema": {
+        "type": "object",
+        "properties": {
+            "query": {
+                "type": "string",
+                "description": "Partial search query to get suggestions for.",
+            },
+        },
+        "required": ["query"],
+    },
+}
+
+_ALL_TOOLS = [_SEARCH_TOOL, _AUTOCOMPLETE_TOOL]
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _ok(req_id: Any, result: Any) -> dict:
+    return {"jsonrpc": "2.0", "id": req_id, "result": result}
+
+
+def _err(req_id: Any, code: int, message: str) -> dict:
+    return {"jsonrpc": "2.0", "id": req_id, "error": {"code": code, "message": message}}
+
+
+def _tool_result(text: str, is_error: bool = False) -> dict:
+    return {"content": [{"type": "text", "text": text}], "isError": is_error}
+
+
+# ---------------------------------------------------------------------------
+# Tool handlers
+# ---------------------------------------------------------------------------
+
+
+async def _handle_search(arguments: dict) -> str:
+    from app.search import search  # local import to avoid circular deps
+
+    query: str = arguments.get("query", "").strip()
+    if not query:
+        return "Error: 'query' argument is required."
+
+    category: str = arguments.get("category", "web")
+    num_results: int = min(int(arguments.get("num_results", 5)), 20)
+    engines_str: str | None = arguments.get("engines")
+    sort: str = arguments.get("sort", "default")
+    date_filter: str = arguments.get("date_filter", "")
+
+    engine_list = [e.strip() for e in engines_str.split(",")] if engines_str else None
+
+    result = await search(
+        query,
+        category=category,
+        page=1,
+        engines=engine_list,
+        sort=sort,
+        date_filter=date_filter,
+        max_results=num_results,
+    )
+
+    if not result.results:
+        return f"No results found for: {query}"
+
+    lines: list[str] = [f"Search results for: {query}\n"]
+    for i, r in enumerate(result.results, 1):
+        lines.append(f"{i}. {r.title}")
+        lines.append(f"   URL: {r.url}")
+        snippet = getattr(r, "content", "") or getattr(r, "img_src", "")
+        if snippet:
+            lines.append(f"   {snippet}")
+        if r.published_date:
+            lines.append(f"   Date: {r.published_date}")
+        lines.append("")
+
+    return "\n".join(lines)
+
+
+async def _handle_autocomplete(arguments: dict) -> str:
+    from app.search import get_autocomplete  # local import
+
+    query: str = arguments.get("query", "").strip()
+    if not query:
+        return "Error: 'query' argument is required."
+
+    suggestions = await get_autocomplete(query)
+    if not suggestions:
+        return f"No suggestions found for: {query}"
+
+    return "Suggestions:\n" + "\n".join(f"- {s}" for s in suggestions)
+
+
+# ---------------------------------------------------------------------------
+# Request dispatcher
+# ---------------------------------------------------------------------------
+
+
+async def _dispatch(req: dict) -> dict | None:
+    """Handle one JSON-RPC request object. Returns None for notifications."""
+    method: str = req.get("method", "")
+    req_id = req.get("id")
+    params: dict = req.get("params") or {}
+
+    # Notifications (no id) — acknowledge silently
+    if req_id is None:
+        return None
+
+    if method == "initialize":
+        return _ok(req_id, {
+            "protocolVersion": MCP_PROTOCOL_VERSION,
+            "capabilities": {"tools": {}},
+            "serverInfo": SERVER_INFO,
+        })
+
+    if method == "ping":
+        return _ok(req_id, {})
+
+    if method == "tools/list":
+        cursor = params.get("cursor")  # pagination cursor (unused — all tools fit in one page)
+        return _ok(req_id, {"tools": _ALL_TOOLS})
+
+    if method == "tools/call":
+        tool_name: str = params.get("name", "")
+        arguments: dict = params.get("arguments") or {}
+
+        if tool_name == "search":
+            try:
+                text = await _handle_search(arguments)
+                return _ok(req_id, _tool_result(text))
+            except Exception as exc:
+                logger.exception("MCP search tool error")
+                return _ok(req_id, _tool_result(f"Search failed: {exc}", is_error=True))
+
+        if tool_name == "autocomplete":
+            try:
+                text = await _handle_autocomplete(arguments)
+                return _ok(req_id, _tool_result(text))
+            except Exception as exc:
+                logger.exception("MCP autocomplete tool error")
+                return _ok(req_id, _tool_result(f"Autocomplete failed: {exc}", is_error=True))
+
+        return _err(req_id, -32601, f"Unknown tool: {tool_name}")
+
+    return _err(req_id, -32601, f"Method not found: {method}")
+
+
+# ---------------------------------------------------------------------------
+# FastAPI endpoint
+# ---------------------------------------------------------------------------
+
+
+@router.post(
+    "/mcp",
+    summary="MCP tool server",
+    description="""[Model Context Protocol](https://modelcontextprotocol.io/) (MCP) endpoint.
+
+Exposes HeySearch as an MCP tool server. LLMs and AI coding assistants
+(Claude Desktop, Cursor, Continue, VS Code Copilot, etc.) can add this
+server to their MCP configuration to invoke search directly.
+
+**Transport:** Streamable HTTP — POST JSON-RPC 2.0 messages to this endpoint.
+
+**Available tools:** `search`, `autocomplete`
+
+**Quick config example (Claude Desktop / `claude_desktop_config.json`):**
+```json
+{
+  "mcpServers": {
+    "heysearch": {
+      "url": "$BASE_URL/api/mcp",
+      "transport": "http"
+    }
+  }
+}
+```
+
+**Manual test:**
+```bash
+# List available tools
+curl -X POST $BASE_URL/api/mcp \\
+  -H 'Content-Type: application/json' \\
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
+
+# Invoke the search tool
+curl -X POST $BASE_URL/api/mcp \\
+  -H 'Content-Type: application/json' \\
+  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"python async","num_results":3}}}'
+```
+""",
+    include_in_schema=True,
+)
+async def mcp_endpoint(request: Request):
+    """Handle MCP JSON-RPC 2.0 requests (single or batch)."""
+    try:
+        body = await request.json()
+    except Exception:
+        return JSONResponse(
+            status_code=400,
+            content=_err(None, -32700, "Parse error: request body must be valid JSON"),
+        )
+
+    # Batch request
+    if isinstance(body, list):
+        responses = [await _dispatch(req) for req in body if isinstance(req, dict)]
+        responses = [r for r in responses if r is not None]
+        if not responses:
+            return Response(status_code=204)
+        return JSONResponse(content=responses)
+
+    # Single request
+    if not isinstance(body, dict):
+        return JSONResponse(
+            status_code=400,
+            content=_err(None, -32600, "Invalid request: expected a JSON object or array"),
+        )
+
+    result = await _dispatch(body)
+    if result is None:
+        # Notification — no response body
+        return Response(status_code=204)
+    return JSONResponse(content=result)

backend/app/models.py

@@ -84,3 +84,33 @@ class APIError(BaseModel):
     message: str
     details: str = ""
     retry_hint: str = ""
+
+
+# --- LLM-optimised response models ---
+
+class LLMWebResult(BaseModel):
+    """A single web result stripped to the fields LLMs need."""
+    title: str
+    url: str
+    snippet: str = ""
+    date: str = ""
+
+
+class LLMImageResult(BaseModel):
+    """A single image result stripped to the fields LLMs need."""
+    title: str
+    url: str
+    img_src: str
+    date: str = ""
+
+
+class LLMSearchResponse(BaseModel):
+    """Minimal search response for LLM / AI-agent consumption.
+
+    Contains only the fields needed for RAG and tool-calling workflows.
+    Omits engine metadata, error details, and other browser-UI noise.
+    """
+    query: str
+    category: str = "web"
+    results: list[LLMWebResult | LLMImageResult] = Field(default_factory=list)
+    total_results: int = 0

backend/app/search.py

@@ -112,6 +112,7 @@ async def search(
     image_size: str = "",
     sort: SortOrder = "default",
     date_filter: DateFilter = "",
+    max_results: int | None = None,
 ) -> SearchResponse:
     """Search across all enabled engines concurrently, with optional Redis caching."""
     engines_key = ",".join(sorted(engines)) if engines else ""
@@ -124,6 +125,8 @@ async def search(
             resp = SearchResponse(**cached_data)
             resp.cached = True
             _apply_sort(resp.results, sort)
+            if max_results is not None:
+                resp.results = resp.results[:max_results]
             resp.total_results = len(resp.results)
             return resp
 
@@ -202,6 +205,8 @@ async def search(
     # Apply date filter, then sort
     _apply_date_filter(response.results, date_filter)
     _apply_sort(response.results, sort)
+    if max_results is not None:
+        response.results = response.results[:max_results]
     response.total_results = len(response.results)
 
     return response