First Commit

README.md

@@ -0,0 +1,232 @@
+# MailRelay
+
+A self-hosted email forwarding tool that monitors a Proton Mail account on a set interval, exports new emails using the official Proton Mail Export CLI, and forwards them to an iCloud Mail account.
+
+---
+
+## Requirements
+
+| Requirement | Notes |
+|---|---|
+| Python 3.11+ | [python.org](https://www.python.org/downloads/) |
+| `wget` | Pre-installed on most Linux distros; on macOS: `brew install wget` |
+| Linux x86\_64 **or** macOS (for testing) | The Proton Export CLI download is the Linux x86\_64 build |
+| iCloud app-specific password | Generate at [appleid.apple.com](https://appleid.apple.com) → Sign-In and Security → App-Specific Passwords |
+| Proton Mail TOTP secret | The **base32 secret key** shown when you first enabled 2FA in Proton Mail settings (not a one-time code) |
+
+---
+
+## Project layout
+
+```
+mailrelay/
+├── main.py               Entry point — CLI flags, scheduler startup
+├── config.py             Encrypted config read/write (age + TOML)
+├── exporter.py           pexpect automation of proton-mail-export-cli
+├── processor.py          EML + metadata merging, dedup check
+├── forwarder.py          IMAP push to iCloud (Mode 1)
+├── packager.py           MBOX generation and local download server (Mode 2)
+├── scheduler.py          APScheduler interval logic
+├── database.py           SQLite dedup tracking
+├── otp.py                pyotp TOTP generation
+├── logger.py             Rotating log setup
+├── tools.py              Proton Export CLI download + install manager
+├── requirements.txt      Python dependencies
+├── setup.sh              One-shot venv + dependency installer
+└── data/
+    ├── config.age        Encrypted config (created on first run)
+    ├── mailrelay.db      SQLite database (created on first run)
+    ├── mailrelay.log     Rotating log file (created on first run)
+    ├── exports/          Temporary working directory for CLI exports
+    └── downloads/        Temporary directory for generated MBOX files
+```
+
+---
+
+## Setup
+
+### 1. Clone or download the project
+
+```bash
+git clone <repo-url> mailrelay
+cd mailrelay
+```
+
+### 2. Create the virtual environment and install dependencies
+
+Run the included setup script:
+
+```bash
+bash setup.sh
+```
+
+This will:
+- Verify Python 3.11+ is available
+- Create a `.venv` virtual environment in the project directory
+- Upgrade pip
+- Install all required packages from `requirements.txt`
+
+To activate the environment manually in future sessions:
+
+```bash
+source .venv/bin/activate
+```
+
+### 3. Run first-time setup
+
+```bash
+python main.py --setup
+```
+
+The setup wizard will:
+
+1. Check for the **Proton Mail Export CLI** in `tools/proton-export/`. If it is not present, you will be prompted to download it automatically via `wget`. The Linux x86\_64 build is downloaded, extracted, and made executable — no manual steps required.
+
+2. Prompt for your credentials and preferences:
+
+   | Prompt | What to enter |
+   |---|---|
+   | Proton Mail email | Your full `@proton.me` address |
+   | Proton Mail password | Your Proton account password |
+   | Proton Mail mailbox password | Your mailbox password if set, or leave blank |
+   | TOTP secret key | The base32 secret from your Proton 2FA setup |
+   | iCloud email | Your full `@icloud.com` address |
+   | iCloud app-specific password | The password generated at appleid.apple.com |
+   | Delivery mode | `1` = automatic IMAP push (default), `2` = manual MBOX download |
+   | Polling interval | Choose a preset or enter a custom number of minutes (minimum 15) |
+   | Master password | A password you choose to encrypt the config file |
+
+All credentials are stored in `data/config.age`, encrypted with [age](https://age-encryption.org/) using your master password. The plaintext is never written to disk.
+
+### 4. Start MailRelay
+
+```bash
+python main.py
+```
+
+You will be prompted for your master password once. MailRelay then runs in the foreground, syncing Proton Mail on your chosen interval.
+
+> **Headless / server use:** Pass the master password via environment variable to avoid the interactive prompt:
+> ```bash
+> MAILRELAY_MASTER_PASSWORD="your-master-password" python main.py
+> ```
+
+---
+
+## CLI reference
+
+| Flag | Description |
+|---|---|
+| `--setup` | Run the first-time setup wizard |
+| `--run-now` | Trigger an immediate sync, then continue running on schedule |
+| `--status` | Print a summary of the last run, next scheduled run, and any pending MBOX downloads |
+| `--logs` | Print the last 100 lines of the log file |
+| `--config` | Change a single config value without redoing full setup |
+
+### Examples
+
+```bash
+# First-time setup
+python main.py --setup
+
+# Start the service with an immediate sync
+python main.py --run-now
+
+# Check status while the service is running in another terminal
+python main.py --status
+
+# Tail the log
+python main.py --logs
+
+# Change the delivery mode
+python main.py --config
+# > Setting to change: preferences.delivery_mode
+# > New value: mbox
+```
+
+---
+
+## Delivery modes
+
+### Mode 1 — Automatic IMAP push (default)
+
+MailRelay connects to iCloud Mail over IMAP (port 993, SSL) and appends each new email directly to your inbox using your app-specific password. No mail client interaction needed.
+
+If the IMAP push fails for any reason, MailRelay automatically falls back to Mode 2 for that sync cycle and logs the reason.
+
+### Mode 2 — Manual MBOX download
+
+New emails are bundled into a timestamped `.mbox` file and served by a local HTTP server at:
+
+```
+http://127.0.0.1:8765/download/<filename>.mbox
+```
+
+The download URL is shown in `--status` output and written to the log. Import the `.mbox` into iCloud Mail via **File → Import Mailboxes** in the macOS Mail app.
+
+Once the file is downloaded, MailRelay marks the messages as delivered and deletes the file. If the file is not downloaded before the next sync cycle, MailRelay deletes it and reprocesses the messages on the next run.
+
+---
+
+## How a sync cycle works
+
+1. APScheduler fires at the configured interval (or `--run-now` is called)
+2. Any stale un-downloaded MBOX files from the previous cycle are cleaned up; their message IDs are cleared from the database so they will be re-processed
+3. A fresh TOTP code is generated from the stored secret
+4. `proton-mail-export-cli` is launched via pexpect; credentials and the TOTP code are injected automatically
+5. Exported `.eml` and `.metadata.json` pairs are scanned; each message ID is checked against SQLite
+6. New messages only: Proton metadata fields are merged into the EML headers
+7. Delivery proceeds in the configured mode (IMAP push or MBOX bundle)
+8. Delivered message IDs are recorded in SQLite; the export directory is wiped
+
+---
+
+## iCloud app-specific password
+
+A standard Apple ID password will not work for IMAP access. Generate an app-specific password:
+
+1. Go to [appleid.apple.com](https://appleid.apple.com)
+2. Sign In and Security → App-Specific Passwords
+3. Click **+** and give it a label (e.g. `MailRelay`)
+4. Copy the generated password — it will only be shown once
+
+---
+
+## Proton TOTP secret
+
+MailRelay needs the **base32 secret key** that backs your Proton 2FA, not a one-time code. You set this up when you first enabled two-factor authentication in Proton Mail:
+
+- **If you saved the secret at setup time:** use that string directly
+- **If you did not save it:** disable and re-enable 2FA in Proton Mail settings — the QR code setup screen displays the raw secret as text
+
+---
+
+## Logs
+
+Logs are written to `data/mailrelay.log` (rotating, max 5 MB, 3 backups). View the tail with:
+
+```bash
+python main.py --logs
+```
+
+Or follow live:
+
+```bash
+tail -f data/mailrelay.log
+```
+
+---
+
+## Dependencies
+
+| Package | Purpose |
+|---|---|
+| `typer` | CLI flags and interface |
+| `pyrage` | age encryption for the config file |
+| `toml` | Config file format |
+| `pyotp` | TOTP code generation |
+| `pexpect` | Driving the Proton Export CLI interactively |
+| `APScheduler` | Background polling interval |
+| `fastapi` + `uvicorn` | Local MBOX download server |
+
+All stdlib modules used (`imaplib`, `sqlite3`, `email`, `mailbox`, `logging`, `tarfile`) require no installation.

main.py

@@ -0,0 +1,370 @@
+"""MailRelay — entry point and CLI.
+
+Usage examples:
+  python main.py --setup          First-time setup wizard
+  python main.py                  Start the background service (prompts for master password)
+  python main.py --run-now        Immediate sync then keep running
+  python main.py --status         Print last-run summary and exit
+  python main.py --logs           Tail the log file and exit
+  python main.py --config         Interactively change a single config value
+"""
+
+import getpass
+import os
+import signal
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+import typer
+
+from modules import config as cfg
+from modules import database
+from modules import exporter
+from modules import forwarder
+from modules import otp
+from modules import packager
+from modules import processor
+from modules import scheduler
+from modules import tools
+from modules.logger import get_logger, tail_log
+
+log = get_logger(__name__)
+app = typer.Typer(add_completion=False, pretty_exceptions_show_locals=False)
+
+# In-memory master password for the lifetime of the process
+_master_password: Optional[str] = None
+
+# Simple in-memory last-run summary (persisted to log; this is for --status)
+_last_run: dict = {}
+
+
+# ---------------------------------------------------------------------------
+# CLI commands
+# ---------------------------------------------------------------------------
+
+@app.command()
+def main(
+    setup: bool = typer.Option(False, "--setup", help="Run first-time setup wizard."),
+    run_now: bool = typer.Option(False, "--run-now", help="Trigger an immediate sync."),
+    status: bool = typer.Option(False, "--status", help="Print status summary and exit."),
+    logs: bool = typer.Option(False, "--logs", help="Tail the log file and exit."),
+    change_config: bool = typer.Option(False, "--config", help="Change a config value."),
+) -> None:
+    global _master_password
+
+    if setup:
+        _run_setup()
+        return
+
+    if logs:
+        print(tail_log(lines=100))
+        return
+
+    # All other commands need the config to be decrypted
+    _master_password = _prompt_master_password()
+
+    if status:
+        _print_status()
+        return
+
+    if change_config:
+        _run_config_change()
+        return
+
+    # Default: start the service
+    _start_service(run_now=run_now)
+
+
+# ---------------------------------------------------------------------------
+# Setup
+# ---------------------------------------------------------------------------
+
+def _run_setup() -> None:
+    if cfg.config_exists():
+        overwrite = typer.confirm(
+            "A config already exists. Overwrite it?", default=False
+        )
+        if not overwrite:
+            raise typer.Abort()
+
+    # Ensure the export CLI is present before finishing setup
+    try:
+        tools.ensure_export_cli()
+    except tools.ToolSetupError as exc:
+        log.error("Export CLI setup failed: %s", exc)
+        raise typer.Exit(code=1)
+
+    config_data, master_pw = cfg.build_config_interactively()
+    cfg.save_config(config_data, master_pw)
+    database.init_db()
+    print("\nSetup complete. Run 'python main.py' to start MailRelay.")
+
+
+# ---------------------------------------------------------------------------
+# Service startup
+# ---------------------------------------------------------------------------
+
+def _start_service(run_now: bool = False) -> None:
+    global _master_password
+
+    # Verify the export CLI is present (offers download if missing)
+    try:
+        tools.ensure_export_cli()
+    except tools.ToolSetupError as exc:
+        log.error("Export CLI unavailable: %s", exc)
+        raise typer.Exit(code=1)
+
+    conf = _load_config()
+    database.init_db()
+    packager.start_server()
+
+    interval = conf["preferences"]["poll_interval_min"]
+
+    def sync():
+        _sync_cycle(conf)
+
+    scheduler.start(sync, interval)
+
+    if run_now:
+        scheduler.run_now(sync)
+
+    log.info("MailRelay running. Press Ctrl+C to stop.")
+
+    # Keep the main thread alive; handle Ctrl+C / SIGTERM gracefully
+    def _shutdown(sig, frame):
+        log.info("Shutdown signal received.")
+        scheduler.stop()
+        packager.stop_server()
+        sys.exit(0)
+
+    signal.signal(signal.SIGINT, _shutdown)
+    signal.signal(signal.SIGTERM, _shutdown)
+
+    try:
+        while True:
+            time.sleep(1)
+    except SystemExit:
+        pass
+
+
+# ---------------------------------------------------------------------------
+# Core sync cycle
+# ---------------------------------------------------------------------------
+
+def _sync_cycle(conf: dict) -> None:
+    global _last_run
+    start_time = datetime.now(timezone.utc)
+    log.info("=== Sync cycle started at %s ===", start_time.isoformat())
+
+    summary = {
+        "started_at": start_time.isoformat(),
+        "emails_found": 0,
+        "emails_new": 0,
+        "delivered": 0,
+        "failed": 0,
+        "fallback": False,
+        "download_url": None,
+        "error": None,
+    }
+
+    try:
+        # 1. Clean up any stale MBOX files from the previous cycle
+        stale = packager.cleanup_stale()
+        if stale:
+            log.info("%d stale pending ID(s) cleared for re-processing.", len(stale))
+
+        # 2. Generate TOTP code
+        totp_code = otp.generate_totp(conf["proton"]["totp_secret"])
+
+        # 3. Run the export CLI
+        export_dir = exporter.run_export(
+            email=conf["proton"]["email"],
+            password=conf["proton"]["password"],
+            totp_code=totp_code,
+            mailbox_password=conf["proton"].get("mailbox_password", ""),
+        )
+
+        # 4. Scan, pair, deduplicate
+        new_emails = processor.scan_and_filter(export_dir)
+        summary["emails_found"] = len(list(export_dir.glob("*.eml")))
+        summary["emails_new"] = len(new_emails)
+
+        if not new_emails:
+            log.info("No new emails to deliver.")
+        else:
+            delivery_mode = conf["preferences"]["delivery_mode"]
+            _deliver(conf, new_emails, delivery_mode, summary)
+
+        # 5. Clean up export directory
+        _clean_export_dir(export_dir)
+
+    except exporter.ExportError as exc:
+        log.error("Export failed: %s", exc)
+        summary["error"] = str(exc)
+    except Exception as exc:
+        log.error("Unexpected error in sync cycle: %s", exc, exc_info=True)
+        summary["error"] = str(exc)
+    finally:
+        summary["finished_at"] = datetime.now(timezone.utc).isoformat()
+        _last_run = summary
+        log.info(
+            "=== Sync complete — new: %d, delivered: %d, failed: %d%s ===",
+            summary["emails_new"],
+            summary["delivered"],
+            summary["failed"],
+            " [FALLBACK to MBOX]" if summary["fallback"] else "",
+        )
+
+
+def _deliver(conf: dict, emails: list, mode: str, summary: dict) -> None:
+    if mode == "imap":
+        succeeded, failed = forwarder.push_emails(
+            emails,
+            icloud_email=conf["icloud"]["email"],
+            icloud_password=conf["icloud"]["password"],
+        )
+        summary["delivered"] = len(succeeded)
+        summary["failed"] = len(failed)
+
+        if failed:
+            log.warning(
+                "%d message(s) failed IMAP push — falling back to MBOX for this cycle.",
+                len(failed),
+            )
+            summary["fallback"] = True
+            failed_emails = [e for e in emails if e.message_id in failed]
+            _deliver_mbox(failed_emails, summary)
+    else:
+        _deliver_mbox(emails, summary)
+
+
+def _deliver_mbox(emails: list, summary: dict) -> None:
+    url = packager.bundle_emails(emails)
+    if url:
+        summary["download_url"] = url
+        summary["delivered"] += len(emails)
+        log.info("MBOX download available at: %s", url)
+
+
+# ---------------------------------------------------------------------------
+# Status
+# ---------------------------------------------------------------------------
+
+def _print_status() -> None:
+    conf = _load_config()
+    mode = conf["preferences"]["delivery_mode"]
+    interval = conf["preferences"]["poll_interval_min"]
+    next_run = scheduler.next_run_time()
+    pending = database.get_pending_mboxes()
+
+    print("\n=== MailRelay Status ===\n")
+    print(f"  Delivery mode   : {mode.upper()}")
+    print(f"  Poll interval   : {interval} minutes")
+
+    if next_run:
+        print(f"  Next sync       : {next_run.strftime('%Y-%m-%d %H:%M:%S UTC')}")
+    else:
+        print("  Next sync       : (scheduler not running)")
+
+    if _last_run:
+        print(f"\n  Last run        : {_last_run.get('started_at', 'n/a')}")
+        print(f"  New emails      : {_last_run.get('emails_new', 0)}")
+        print(f"  Delivered       : {_last_run.get('delivered', 0)}")
+        print(f"  Failed          : {_last_run.get('failed', 0)}")
+        if _last_run.get("fallback"):
+            print("  Fallback used   : YES (IMAP failed → MBOX)")
+        if _last_run.get("error"):
+            print(f"  Last error      : {_last_run['error']}")
+    else:
+        print("\n  Last run        : (no run yet this session)")
+
+    if pending:
+        print(f"\n  Pending MBOX downloads ({len(pending)}):")
+        for entry in pending:
+            path = Path(entry["mbox_path"])
+            count = len(entry["message_ids"])
+            url = f"http://{packager.SERVER_HOST}:{packager.SERVER_PORT}/download/{path.name}"
+            print(f"    {path.name}  ({count} message(s))  →  {url}")
+    else:
+        print("\n  Pending downloads: none")
+
+    print()
+
+
+# ---------------------------------------------------------------------------
+# Config change
+# ---------------------------------------------------------------------------
+
+def _run_config_change() -> None:
+    print("\nAvailable settings:")
+    print("  proton.email | proton.password | proton.mailbox_password | proton.totp_secret")
+    print("  icloud.email | icloud.password")
+    print("  preferences.delivery_mode | preferences.poll_interval_min")
+    print()
+
+    key_path = input("Setting to change (e.g. preferences.delivery_mode): ").strip()
+    if "." not in key_path:
+        print("Please use section.key format.")
+        return
+
+    section, key = key_path.split(".", 1)
+    new_value_raw = getpass.getpass(f"New value for {key_path}: ") \
+        if "password" in key.lower() or "secret" in key.lower() \
+        else input(f"New value for {key_path}: ").strip()
+
+    # Coerce poll_interval_min to int
+    if key == "poll_interval_min":
+        try:
+            new_value = int(new_value_raw)
+        except ValueError:
+            print("poll_interval_min must be an integer.")
+            return
+    else:
+        new_value = new_value_raw
+
+    cfg.update_config(_master_password, section, key, new_value)
+    print(f"\nUpdated {key_path} successfully.")
+
+    if key == "poll_interval_min" and scheduler._scheduler and scheduler._scheduler.running:
+        scheduler.update_interval(int(new_value))
+        print(f"Live polling interval updated to {new_value} minutes.")
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _prompt_master_password() -> str:
+    # Allow headless/server use via environment variable
+    env_pw = os.environ.get("MAILRELAY_MASTER_PASSWORD")
+    if env_pw:
+        return env_pw
+    return getpass.getpass("Master password: ")
+
+
+def _load_config() -> dict:
+    try:
+        return cfg.load_config(_master_password)
+    except cfg.ConfigError as exc:
+        log.error("%s", exc)
+        raise typer.Exit(code=1)
+
+
+def _clean_export_dir(export_dir: Path) -> None:
+    """Delete all files in the export directory after a sync cycle."""
+    for f in export_dir.iterdir():
+        try:
+            f.unlink()
+        except OSError as exc:
+            log.warning("Could not delete export file %s: %s", f.name, exc)
+
+
+# ---------------------------------------------------------------------------
+# Entry point
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    app()

modules/config.py

@@ -0,0 +1,206 @@
+"""Encrypted config management using age (via pyrage) and TOML.
+
+The config file is stored as an age-encrypted TOML blob at data/config.age.
+The encryption key is a scrypt-derived passphrase key from the master password.
+
+Config schema (as TOML):
+    [proton]
+    email           = "user@proton.me"
+    password        = "..."
+    mailbox_password = "..."   # optional, "" if not set
+    totp_secret     = "..."    # base32 TOTP secret
+
+    [icloud]
+    email    = "user@icloud.com"
+    password = "..."           # app-specific password
+
+    [preferences]
+    delivery_mode     = "imap"   # "imap" | "mbox"
+    poll_interval_min = 60       # integer minutes
+"""
+
+import io
+from pathlib import Path
+from typing import Any
+
+import pyrage
+import toml
+
+CONFIG_PATH = Path(__file__).parent.parent / "data" / "config.age"
+
+# Keys we expose to the rest of the app
+REQUIRED_KEYS = {
+    "proton": ["email", "password", "totp_secret"],
+    "icloud": ["email", "password"],
+    "preferences": ["delivery_mode", "poll_interval_min"],
+}
+
+DELIVERY_MODES = ("imap", "mbox")
+MIN_INTERVAL_MIN = 15
+
+
+class ConfigError(Exception):
+    pass
+
+
+# ---------------------------------------------------------------------------
+# Low-level encrypt / decrypt helpers
+# ---------------------------------------------------------------------------
+
+def _encrypt(plaintext: str, passphrase: str) -> bytes:
+    identity = pyrage.passphrase.Recipient(passphrase)
+    return pyrage.encrypt(plaintext.encode(), [identity])
+
+
+def _decrypt(ciphertext: bytes, passphrase: str) -> str:
+    identity = pyrage.passphrase.Identity(passphrase)
+    return pyrage.decrypt(ciphertext, [identity]).decode()
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def config_exists() -> bool:
+    return CONFIG_PATH.exists()
+
+
+def save_config(data: dict[str, Any], passphrase: str) -> None:
+    """Serialise *data* to TOML, encrypt with *passphrase*, write to disk."""
+    _validate(data)
+    plaintext = toml.dumps(data)
+    ciphertext = _encrypt(plaintext, passphrase)
+    CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
+    CONFIG_PATH.write_bytes(ciphertext)
+
+
+def load_config(passphrase: str) -> dict[str, Any]:
+    """Decrypt the config file and return it as a dict."""
+    if not CONFIG_PATH.exists():
+        raise ConfigError("No config file found. Run --setup first.")
+    try:
+        ciphertext = CONFIG_PATH.read_bytes()
+        plaintext = _decrypt(ciphertext, passphrase)
+    except Exception as exc:
+        raise ConfigError(f"Failed to decrypt config (wrong master password?): {exc}") from exc
+    data = toml.loads(plaintext)
+    _validate(data)
+    return data
+
+
+def update_config(passphrase: str, section: str, key: str, value: Any) -> None:
+    """Load config, change one value, and re-save."""
+    data = load_config(passphrase)
+    if section not in data:
+        data[section] = {}
+    data[section][key] = value
+    save_config(data, passphrase)
+
+
+# ---------------------------------------------------------------------------
+# Validation
+# ---------------------------------------------------------------------------
+
+def _validate(data: dict[str, Any]) -> None:
+    for section, keys in REQUIRED_KEYS.items():
+        if section not in data:
+            raise ConfigError(f"Config missing section [{section}]")
+        for key in keys:
+            if key not in data[section]:
+                raise ConfigError(f"Config missing [{section}].{key}")
+
+    mode = data["preferences"]["delivery_mode"]
+    if mode not in DELIVERY_MODES:
+        raise ConfigError(
+            f"Invalid delivery_mode '{mode}'. Must be one of {DELIVERY_MODES}"
+        )
+
+    interval = data["preferences"]["poll_interval_min"]
+    if not isinstance(interval, int) or interval < MIN_INTERVAL_MIN:
+        raise ConfigError(
+            f"poll_interval_min must be an integer >= {MIN_INTERVAL_MIN}"
+        )
+
+
+# ---------------------------------------------------------------------------
+# Interactive setup wizard helpers
+# ---------------------------------------------------------------------------
+
+INTERVAL_PRESETS = {
+    "1": 15,
+    "2": 30,
+    "3": 60,
+    "4": 360,
+    "5": 1440,
+}
+
+
+def build_config_interactively() -> tuple[dict[str, Any], str]:
+    """Prompt the user for all settings and a master password.
+
+    Returns (config_dict, master_password).
+    """
+    import getpass
+
+    print("\n=== MailRelay First-Time Setup ===\n")
+
+    proton_email = input("Proton Mail email address: ").strip()
+    proton_password = getpass.getpass("Proton Mail password: ")
+    proton_mailbox_pw = getpass.getpass(
+        "Proton Mail mailbox password (leave blank if none): "
+    )
+    totp_secret = input("TOTP secret key (base32, from your 2FA setup): ").strip()
+
+    icloud_email = input("\niCloud email address: ").strip()
+    icloud_password = getpass.getpass("iCloud app-specific password: ")
+
+    print("\nDelivery mode:")
+    print("  1) Automatic IMAP push to iCloud (default)")
+    print("  2) Manual MBOX download")
+    mode_choice = input("Choose [1/2, default 1]: ").strip() or "1"
+    delivery_mode = "imap" if mode_choice != "2" else "mbox"
+
+    print("\nPolling interval:")
+    print("  1) 15 minutes")
+    print("  2) 30 minutes")
+    print("  3) 1 hour (default)")
+    print("  4) 6 hours")
+    print("  5) 24 hours")
+    print("  6) Custom")
+    interval_choice = input("Choose [1-6, default 3]: ").strip() or "3"
+    if interval_choice in INTERVAL_PRESETS:
+        poll_interval = INTERVAL_PRESETS[interval_choice]
+    else:
+        while True:
+            raw = input(f"Enter interval in minutes (min {MIN_INTERVAL_MIN}): ").strip()
+            if raw.isdigit() and int(raw) >= MIN_INTERVAL_MIN:
+                poll_interval = int(raw)
+                break
+            print(f"Please enter a whole number >= {MIN_INTERVAL_MIN}.")
+
+    print("\nSet a master password to encrypt your config.")
+    while True:
+        master_pw = getpass.getpass("Master password: ")
+        confirm = getpass.getpass("Confirm master password: ")
+        if master_pw == confirm:
+            break
+        print("Passwords do not match. Try again.")
+
+    config = {
+        "proton": {
+            "email": proton_email,
+            "password": proton_password,
+            "mailbox_password": proton_mailbox_pw,
+            "totp_secret": totp_secret,
+        },
+        "icloud": {
+            "email": icloud_email,
+            "password": icloud_password,
+        },
+        "preferences": {
+            "delivery_mode": delivery_mode,
+            "poll_interval_min": poll_interval,
+        },
+    }
+
+    return config, master_pw

modules/database.py

@@ -0,0 +1,145 @@
+"""SQLite-backed deduplication and delivery-state tracking."""
+
+import sqlite3
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Generator, Iterable
+
+DB_PATH = Path(__file__).parent.parent / "data" / "mailrelay.db"
+
+# Delivery states
+STATE_PENDING = "pending"       # MBOX generated, not yet downloaded
+STATE_DELIVERED = "delivered"   # IMAP pushed or MBOX confirmed downloaded
+
+
+def _connect() -> sqlite3.Connection:
+    DB_PATH.parent.mkdir(parents=True, exist_ok=True)
+    conn = sqlite3.connect(DB_PATH)
+    conn.row_factory = sqlite3.Row
+    conn.execute("PRAGMA journal_mode=WAL")
+    return conn
+
+
+@contextmanager
+def _db() -> Generator[sqlite3.Connection, None, None]:
+    conn = _connect()
+    try:
+        yield conn
+        conn.commit()
+    except Exception:
+        conn.rollback()
+        raise
+    finally:
+        conn.close()
+
+
+def init_db() -> None:
+    """Create tables if they don't exist."""
+    with _db() as conn:
+        conn.execute("""
+            CREATE TABLE IF NOT EXISTS messages (
+                message_id  TEXT PRIMARY KEY,
+                state       TEXT NOT NULL DEFAULT 'delivered',
+                mbox_path   TEXT,
+                created_at  DATETIME DEFAULT (datetime('now')),
+                updated_at  DATETIME DEFAULT (datetime('now'))
+            )
+        """)
+        conn.execute("""
+            CREATE INDEX IF NOT EXISTS idx_state ON messages(state)
+        """)
+
+
+def is_known(message_id: str) -> bool:
+    """Return True if a message ID is already tracked (any state)."""
+    with _db() as conn:
+        row = conn.execute(
+            "SELECT 1 FROM messages WHERE message_id = ?", (message_id,)
+        ).fetchone()
+        return row is not None
+
+
+def filter_new(message_ids: Iterable[str]) -> list[str]:
+    """Return only the IDs not yet in the database."""
+    ids = list(message_ids)
+    if not ids:
+        return []
+    with _db() as conn:
+        placeholders = ",".join("?" * len(ids))
+        known = {
+            row[0]
+            for row in conn.execute(
+                f"SELECT message_id FROM messages WHERE message_id IN ({placeholders})",
+                ids,
+            )
+        }
+    return [mid for mid in ids if mid not in known]
+
+
+def mark_pending(message_ids: Iterable[str], mbox_path: str) -> None:
+    """Record message IDs as pending (MBOX created, not yet downloaded)."""
+    with _db() as conn:
+        conn.executemany(
+            """
+            INSERT INTO messages (message_id, state, mbox_path)
+            VALUES (?, ?, ?)
+            ON CONFLICT(message_id) DO UPDATE SET
+                state = excluded.state,
+                mbox_path = excluded.mbox_path,
+                updated_at = datetime('now')
+            """,
+            [(mid, STATE_PENDING, mbox_path) for mid in message_ids],
+        )
+
+
+def mark_delivered(message_ids: Iterable[str]) -> None:
+    """Mark message IDs as fully delivered."""
+    ids = list(message_ids)
+    if not ids:
+        return
+    with _db() as conn:
+        conn.executemany(
+            """
+            INSERT INTO messages (message_id, state)
+            VALUES (?, ?)
+            ON CONFLICT(message_id) DO UPDATE SET
+                state = excluded.state,
+                mbox_path = NULL,
+                updated_at = datetime('now')
+            """,
+            [(mid, STATE_DELIVERED) for mid in ids],
+        )
+
+
+def get_pending_mboxes() -> list[dict]:
+    """Return all distinct pending MBOX paths with their message IDs."""
+    with _db() as conn:
+        rows = conn.execute(
+            "SELECT message_id, mbox_path FROM messages WHERE state = ?",
+            (STATE_PENDING,),
+        ).fetchall()
+
+    by_path: dict[str, list[str]] = {}
+    for row in rows:
+        path = row["mbox_path"]
+        by_path.setdefault(path, []).append(row["message_id"])
+
+    return [{"mbox_path": path, "message_ids": ids} for path, ids in by_path.items()]
+
+
+def clear_pending_for_mbox(mbox_path: str) -> list[str]:
+    """Remove pending state for a given MBOX (used on cleanup/re-process).
+
+    Returns the list of message IDs that were pending for that MBOX.
+    """
+    with _db() as conn:
+        rows = conn.execute(
+            "SELECT message_id FROM messages WHERE state = ? AND mbox_path = ?",
+            (STATE_PENDING, mbox_path),
+        ).fetchall()
+        message_ids = [row["message_id"] for row in rows]
+        conn.execute(
+            "DELETE FROM messages WHERE state = ? AND mbox_path = ?",
+            (STATE_PENDING, mbox_path),
+        )
+    return message_ids

modules/exporter.py

@@ -0,0 +1,188 @@
+"""Automate proton-mail-export-cli via pexpect.
+
+The Proton Export CLI interactive prompt sequence (observed order):
+  1. Email address
+  2. Password
+  3. Two-factor authentication code   (only if 2FA is enabled)
+  4. Mailbox password                  (only if separate mailbox password is set)
+
+After authentication the CLI exports all mail to the export directory.
+Each message produces two files:
+  {messageID}.eml
+  {messageID}.metadata.json
+
+Because the exact prompt strings can vary between CLI versions and account
+configurations, each pattern is defined as a regex so minor wording differences
+are tolerated.  If the CLI changes its prompt wording, update PROMPTS below.
+"""
+
+from pathlib import Path
+from typing import Optional
+
+import pexpect
+
+from .logger import get_logger
+from .tools import BINARY_PATH as CLI_BINARY_PATH, ensure_export_cli
+
+log = get_logger(__name__)
+
+# ---------------------------------------------------------------------------
+# Configurable paths
+# ---------------------------------------------------------------------------
+
+EXPORT_DIR = Path(__file__).parent.parent / "data" / "exports"
+
+# ---------------------------------------------------------------------------
+# Prompt patterns (case-insensitive regex matched against CLI output)
+# ---------------------------------------------------------------------------
+
+PROMPTS = {
+    "email":            r"[Ee]mail\s*(address)?[\s:>]+",
+    "password":         r"[Pp]assword[\s:>]+",
+    "totp":             r"[Tt]wo.factor|[Oo]ne.time|[Tt][Oo][Tt][Pp]|[Aa]uth.*code",
+    "mailbox_password": r"[Mm]ailbox\s*[Pp]assword[\s:>]+",
+    "done":             r"[Ee]xport\s*(complete|finished|done)|[Ss]uccessfully\s*export",
+    "error":            r"[Ee]rror|[Ff]ailed|[Ii]nvalid",
+}
+
+# Maximum time to wait for each prompt (seconds)
+PROMPT_TIMEOUT = 120
+# Total export timeout — large mailboxes can take a while
+EXPORT_TIMEOUT = 3600
+
+
+class ExportError(Exception):
+    pass
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def run_export(
+    email: str,
+    password: str,
+    totp_code: str,
+    mailbox_password: str = "",
+    export_dir: Optional[Path] = None,
+) -> Path:
+    """Drive proton-mail-export-cli and return the export directory path.
+
+    Raises ExportError on authentication failure or unexpected CLI output.
+    """
+    out_dir = export_dir or EXPORT_DIR
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    try:
+        binary = ensure_export_cli()
+    except Exception as exc:
+        raise ExportError(str(exc)) from exc
+
+    cmd = [str(binary), "--export-dir", str(out_dir)]
+    log.info("Starting Proton export CLI: %s", " ".join(cmd))
+
+    child = pexpect.spawn(
+        cmd[0], cmd[1:], timeout=PROMPT_TIMEOUT, encoding="utf-8"
+    )
+    child.logfile_read = _PexpectLogger(log)
+
+    try:
+        _drive_cli(child, email, password, totp_code, mailbox_password)
+    except pexpect.TIMEOUT as exc:
+        child.close(force=True)
+        raise ExportError("Timed out waiting for CLI prompt.") from exc
+    except pexpect.EOF as exc:
+        output = child.before or ""
+        child.close()
+        if child.exitstatus and child.exitstatus != 0:
+            raise ExportError(
+                f"CLI exited with code {child.exitstatus}. Output: {output.strip()}"
+            ) from exc
+        # EOF after a successful export is normal
+        log.info("CLI process finished (EOF).")
+
+    log.info("Export complete. Files in: %s", out_dir)
+    return out_dir
+
+
+def _drive_cli(
+    child: pexpect.spawn,
+    email: str,
+    password: str,
+    totp_code: str,
+    mailbox_password: str,
+) -> None:
+    """Respond to each interactive prompt in sequence."""
+    patterns = [
+        pexpect.TIMEOUT,
+        pexpect.EOF,
+        PROMPTS["email"],
+        PROMPTS["password"],
+        PROMPTS["totp"],
+        PROMPTS["mailbox_password"],
+        PROMPTS["done"],
+        PROMPTS["error"],
+    ]
+
+    totp_sent = False
+    mailbox_sent = False
+
+    while True:
+        idx = child.expect(patterns, timeout=PROMPT_TIMEOUT)
+
+        if idx == 0:  # TIMEOUT
+            raise pexpect.TIMEOUT("No prompt received within timeout.")
+
+        if idx == 1:  # EOF — process exited
+            return
+
+        if idx == 2:  # email prompt
+            log.debug("CLI requested email.")
+            child.sendline(email)
+
+        elif idx == 3:  # password prompt
+            # The CLI may show a password prompt for both the account password
+            # and the mailbox password.  We track which we've already sent.
+            if not mailbox_sent and mailbox_password and totp_sent:
+                log.debug("CLI requested mailbox password.")
+                child.sendline(mailbox_password)
+                mailbox_sent = True
+            else:
+                log.debug("CLI requested account password.")
+                child.sendline(password)
+
+        elif idx == 4:  # TOTP prompt
+            log.debug("CLI requested TOTP code.")
+            child.sendline(totp_code)
+            totp_sent = True
+
+        elif idx == 5:  # explicit mailbox password prompt
+            log.debug("CLI requested mailbox password (explicit prompt).")
+            child.sendline(mailbox_password or "")
+            mailbox_sent = True
+
+        elif idx == 6:  # export done
+            log.info("CLI reported export complete.")
+            return
+
+        elif idx == 7:  # error line
+            snippet = (child.before or "").strip().splitlines()[-1]
+            raise ExportError(f"CLI reported an error: {snippet}")
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+class _PexpectLogger:
+    """Thin adapter so pexpect writes its read data to our logger at DEBUG."""
+
+    def __init__(self, logger):
+        self._log = logger
+
+    def write(self, s: str) -> None:
+        if s.strip():
+            self._log.debug("[cli] %s", s.rstrip())
+
+    def flush(self) -> None:
+        pass

modules/forwarder.py

@@ -0,0 +1,120 @@
+"""Push emails to iCloud Mail via IMAP APPEND.
+
+iCloud IMAP settings:
+  Host : imap.mail.me.com
+  Port : 993 (SSL)
+  Auth : email + app-specific password
+
+The APPEND command places a message directly into a mailbox without going
+through SMTP, so no "sent" copy is created and delivery is instant.
+"""
+
+import imaplib
+import socket
+from typing import Optional
+
+from .database import mark_delivered
+from .logger import get_logger
+from .processor import RichEmail
+
+log = get_logger(__name__)
+
+ICLOUD_IMAP_HOST = "imap.mail.me.com"
+ICLOUD_IMAP_PORT = 993
+DEFAULT_MAILBOX = "INBOX"
+CONNECT_TIMEOUT = 30  # seconds
+
+
+class ForwarderError(Exception):
+    pass
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def push_emails(
+    emails: list[RichEmail],
+    icloud_email: str,
+    icloud_password: str,
+    mailbox: str = DEFAULT_MAILBOX,
+) -> tuple[list[str], list[str]]:
+    """APPEND each email to the iCloud mailbox.
+
+    Returns (succeeded_ids, failed_ids).
+    Records succeeded IDs as delivered in SQLite.
+    """
+    if not emails:
+        return [], []
+
+    succeeded: list[str] = []
+    failed: list[str] = []
+
+    try:
+        conn = _connect(icloud_email, icloud_password)
+    except ForwarderError as exc:
+        log.error("IMAP connection failed: %s", exc)
+        return [], [e.message_id for e in emails]
+
+    try:
+        for rich in emails:
+            try:
+                _append(conn, rich, mailbox)
+                succeeded.append(rich.message_id)
+                log.info("IMAP pushed: %s", rich.message_id)
+            except Exception as exc:
+                log.error("Failed to push %s: %s", rich.message_id, exc)
+                failed.append(rich.message_id)
+    finally:
+        _logout(conn)
+
+    if succeeded:
+        mark_delivered(succeeded)
+        log.info("Marked %d message(s) as delivered.", len(succeeded))
+
+    return succeeded, failed
+
+
+# ---------------------------------------------------------------------------
+# Internals
+# ---------------------------------------------------------------------------
+
+def _connect(email_addr: str, password: str) -> imaplib.IMAP4_SSL:
+    log.debug("Connecting to %s:%d …", ICLOUD_IMAP_HOST, ICLOUD_IMAP_PORT)
+    try:
+        conn = imaplib.IMAP4_SSL(
+            ICLOUD_IMAP_HOST,
+            ICLOUD_IMAP_PORT,
+        )
+    except (OSError, socket.gaierror) as exc:
+        raise ForwarderError(f"Cannot reach {ICLOUD_IMAP_HOST}: {exc}") from exc
+
+    try:
+        conn.login(email_addr, password)
+        log.debug("IMAP login successful for %s", email_addr)
+    except imaplib.IMAP4.error as exc:
+        raise ForwarderError(f"IMAP authentication failed: {exc}") from exc
+
+    return conn
+
+
+def _append(
+    conn: imaplib.IMAP4_SSL,
+    rich: RichEmail,
+    mailbox: str,
+) -> None:
+    """APPEND a single message to *mailbox*."""
+    # imaplib.IMAP4.append expects: mailbox, flags, date_time, message
+    # We pass None for flags and date_time to let the server set defaults.
+    status, data = conn.append(mailbox, None, None, rich.raw_bytes)
+    if status != "OK":
+        raise ForwarderError(
+            f"APPEND returned {status} for {rich.message_id}: {data}"
+        )
+
+
+def _logout(conn: imaplib.IMAP4_SSL) -> None:
+    try:
+        conn.logout()
+    except Exception:
+        pass

modules/logger.py

@@ -0,0 +1,47 @@
+"""Rotating log file setup for MailRelay."""
+
+import logging
+from logging.handlers import RotatingFileHandler
+from pathlib import Path
+
+LOG_PATH = Path(__file__).parent.parent / "data" / "mailrelay.log"
+LOG_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
+MAX_BYTES = 5 * 1024 * 1024  # 5 MB
+BACKUP_COUNT = 3
+
+
+def get_logger(name: str) -> logging.Logger:
+    """Return a named logger wired to the shared rotating file + stderr."""
+    logger = logging.getLogger(name)
+
+    if logger.handlers:
+        return logger  # already configured
+
+    logger.setLevel(logging.DEBUG)
+
+    # Rotating file handler
+    LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
+    file_handler = RotatingFileHandler(
+        LOG_PATH, maxBytes=MAX_BYTES, backupCount=BACKUP_COUNT, encoding="utf-8"
+    )
+    file_handler.setLevel(logging.DEBUG)
+    file_handler.setFormatter(logging.Formatter(LOG_FORMAT))
+
+    # Console handler (INFO and above)
+    console_handler = logging.StreamHandler()
+    console_handler.setLevel(logging.INFO)
+    console_handler.setFormatter(logging.Formatter(LOG_FORMAT))
+
+    logger.addHandler(file_handler)
+    logger.addHandler(console_handler)
+
+    return logger
+
+
+def tail_log(lines: int = 50) -> str:
+    """Return the last N lines of the log file as a string."""
+    if not LOG_PATH.exists():
+        return "(no log file yet)"
+    text = LOG_PATH.read_text(encoding="utf-8")
+    all_lines = text.splitlines()
+    return "\n".join(all_lines[-lines:])

modules/otp.py

@@ -0,0 +1,15 @@
+"""TOTP code generation using pyotp."""
+
+import pyotp
+
+
+def generate_totp(secret: str) -> str:
+    """Generate the current TOTP code from a base32 secret."""
+    totp = pyotp.TOTP(secret)
+    return totp.now()
+
+
+def verify_totp(secret: str, code: str) -> bool:
+    """Verify a TOTP code against a secret (useful for debugging setup)."""
+    totp = pyotp.TOTP(secret)
+    return totp.verify(code)

modules/packager.py

@@ -0,0 +1,207 @@
+"""MBOX packaging and local HTTP download server (Mode 2 / fallback).
+
+Flow:
+  1. bundle_emails()   — write a timestamped .mbox into data/downloads/
+                         mark message IDs as "pending" in SQLite
+                         return the local download URL
+  2. start_server()    — launch a FastAPI/uvicorn server in a background thread
+                         serving data/downloads/
+  3. On GET /download/{filename} the server streams the file, then marks all
+     IDs for that MBOX as delivered and deletes the file.
+  4. cleanup_stale()   — called at the start of each sync cycle; deletes any
+                         MBOX files that were never collected and removes their
+                         pending DB entries so they can be re-processed.
+"""
+
+import mailbox
+import os
+import threading
+import time
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import uvicorn
+from fastapi import FastAPI, HTTPException
+from fastapi.responses import FileResponse
+
+from . import database
+from .logger import get_logger
+from .processor import RichEmail
+
+log = get_logger(__name__)
+
+DOWNLOADS_DIR = Path(__file__).parent.parent / "data" / "downloads"
+SERVER_HOST = "127.0.0.1"
+SERVER_PORT = 8765
+
+_server_thread: Optional[threading.Thread] = None
+_uvicorn_server: Optional[uvicorn.Server] = None
+
+app = FastAPI(title="MailRelay Download Server", docs_url=None, redoc_url=None)
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def bundle_emails(emails: list[RichEmail]) -> Optional[str]:
+    """Write emails to a new .mbox file and return the download URL.
+
+    Returns None if emails list is empty.
+    Marks message IDs as 'pending' in SQLite.
+    """
+    if not emails:
+        return None
+
+    DOWNLOADS_DIR.mkdir(parents=True, exist_ok=True)
+    timestamp = datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")
+    mbox_filename = f"mailrelay_{timestamp}.mbox"
+    mbox_path = DOWNLOADS_DIR / mbox_filename
+
+    mbox = mailbox.mbox(str(mbox_path))
+    mbox.lock()
+    try:
+        for rich in emails:
+            mbox.add(mailbox.mboxMessage(rich.raw_bytes))
+    finally:
+        mbox.flush()
+        mbox.unlock()
+        mbox.close()
+
+    message_ids = [e.message_id for e in emails]
+    database.mark_pending(message_ids, str(mbox_path))
+
+    url = f"http://{SERVER_HOST}:{SERVER_PORT}/download/{mbox_filename}"
+    log.info(
+        "MBOX bundle created: %s (%d message(s)). Download: %s",
+        mbox_filename,
+        len(emails),
+        url,
+    )
+    return url
+
+
+def cleanup_stale() -> list[str]:
+    """Remove MBOX files from previous cycles that were never downloaded.
+
+    Returns the list of message IDs that were cleared (will be re-processed
+    on the next sync since they are removed from the DB).
+    """
+    pending = database.get_pending_mboxes()
+    cleared_ids: list[str] = []
+
+    for entry in pending:
+        mbox_path = Path(entry["mbox_path"])
+        ids = entry["message_ids"]
+
+        log.warning(
+            "Stale MBOX detected (never downloaded): %s — clearing %d pending ID(s) for re-processing.",
+            mbox_path.name,
+            len(ids),
+        )
+        cleared = database.clear_pending_for_mbox(str(mbox_path))
+        cleared_ids.extend(cleared)
+
+        if mbox_path.exists():
+            try:
+                mbox_path.unlink()
+                log.info("Deleted stale MBOX: %s", mbox_path.name)
+            except OSError as exc:
+                log.error("Could not delete %s: %s", mbox_path.name, exc)
+
+    return cleared_ids
+
+
+def start_server() -> None:
+    """Start the FastAPI download server in a daemon thread (idempotent)."""
+    global _server_thread, _uvicorn_server
+
+    if _server_thread and _server_thread.is_alive():
+        return  # already running
+
+    config = uvicorn.Config(
+        app,
+        host=SERVER_HOST,
+        port=SERVER_PORT,
+        log_level="warning",
+        access_log=False,
+    )
+    _uvicorn_server = uvicorn.Server(config)
+
+    _server_thread = threading.Thread(
+        target=_uvicorn_server.run, daemon=True, name="mailrelay-download-server"
+    )
+    _server_thread.start()
+    log.info("Download server started at http://%s:%d", SERVER_HOST, SERVER_PORT)
+
+
+def stop_server() -> None:
+    """Gracefully stop the download server."""
+    global _uvicorn_server
+    if _uvicorn_server:
+        _uvicorn_server.should_exit = True
+        log.info("Download server stopped.")
+
+
+# ---------------------------------------------------------------------------
+# FastAPI routes
+# ---------------------------------------------------------------------------
+
+@app.get("/download/{filename}")
+async def download_mbox(filename: str):
+    """Serve a .mbox file, mark it delivered, then delete it."""
+    # Basic path safety — no traversal
+    if "/" in filename or "\\" in filename or filename.startswith("."):
+        raise HTTPException(status_code=400, detail="Invalid filename.")
+
+    mbox_path = DOWNLOADS_DIR / filename
+    if not mbox_path.exists():
+        raise HTTPException(status_code=404, detail="File not found or already downloaded.")
+
+    # We need to deliver after the response is sent.
+    # Use a background task via starlette.
+    from starlette.background import BackgroundTask
+
+    task = BackgroundTask(_on_download_complete, str(mbox_path))
+    log.info("Serving MBOX download: %s", filename)
+    return FileResponse(
+        path=str(mbox_path),
+        media_type="application/mbox",
+        filename=filename,
+        background=task,
+    )
+
+
+@app.get("/status")
+async def server_status():
+    pending = database.get_pending_mboxes()
+    return {
+        "pending_mboxes": [
+            {"file": Path(e["mbox_path"]).name, "message_count": len(e["message_ids"])}
+            for e in pending
+        ]
+    }
+
+
+# ---------------------------------------------------------------------------
+# Internals
+# ---------------------------------------------------------------------------
+
+def _on_download_complete(mbox_path_str: str) -> None:
+    """Called after a successful file download: mark delivered + delete file."""
+    mbox_path = Path(mbox_path_str)
+    ids = database.clear_pending_for_mbox(mbox_path_str)
+    if ids:
+        database.mark_delivered(ids)
+        log.info(
+            "Download confirmed for %s — marked %d message(s) as delivered.",
+            mbox_path.name,
+            len(ids),
+        )
+    if mbox_path.exists():
+        try:
+            mbox_path.unlink()
+            log.info("Deleted downloaded MBOX: %s", mbox_path.name)
+        except OSError as exc:
+            log.error("Could not delete %s: %s", mbox_path.name, exc)

modules/processor.py

@@ -0,0 +1,184 @@
+"""Scan an export directory, pair EML + metadata, check dedup, return new emails.
+
+Proton exports produce two files per message:
+  {messageID}.eml
+  {messageID}.metadata.json
+
+Metadata fields of interest (Proton-specific):
+  Subject, SenderAddress, SenderName, ToList, CCList, BCCList,
+  Time (Unix timestamp), Unread, LabelIDs, ExternalID, NumAttachments
+
+These are mapped to standard RFC 5322 headers when they are missing from the
+raw EML (Proton sometimes omits headers in the raw export).
+"""
+
+import email
+import email.policy
+import json
+from dataclasses import dataclass, field
+from email.message import EmailMessage
+from pathlib import Path
+from typing import Optional
+
+from .database import filter_new
+from .logger import get_logger
+
+log = get_logger(__name__)
+
+
+@dataclass
+class RichEmail:
+    """An enriched email ready for delivery."""
+    message_id: str
+    message: EmailMessage          # the (possibly augmented) email object
+    raw_bytes: bytes               # final RFC 2822 bytes
+    metadata: dict = field(default_factory=dict)
+
+
+class ProcessorError(Exception):
+    pass
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def scan_and_filter(export_dir: Path) -> list[RichEmail]:
+    """Scan *export_dir*, pair EML+metadata, filter already-seen IDs.
+
+    Returns a list of RichEmail objects for new messages only.
+    """
+    pairs = _find_pairs(export_dir)
+    log.info("Found %d exported message(s) in %s", len(pairs), export_dir)
+
+    ids = list(pairs.keys())
+    new_ids = filter_new(ids)
+    log.info("%d new message(s) after deduplication.", len(new_ids))
+
+    results: list[RichEmail] = []
+    for mid in new_ids:
+        eml_path, meta_path = pairs[mid]
+        try:
+            rich = _build_rich_email(mid, eml_path, meta_path)
+            results.append(rich)
+        except Exception as exc:
+            log.warning("Skipping %s — could not process: %s", mid, exc)
+
+    return results
+
+
+# ---------------------------------------------------------------------------
+# Internals
+# ---------------------------------------------------------------------------
+
+def _find_pairs(export_dir: Path) -> dict[str, tuple[Path, Optional[Path]]]:
+    """Return {messageID: (eml_path, meta_path_or_None)} for every .eml found."""
+    pairs: dict[str, tuple[Path, Optional[Path]]] = {}
+
+    for eml_path in export_dir.glob("*.eml"):
+        mid = eml_path.stem
+        meta_path = eml_path.with_suffix(".metadata.json")
+        if not meta_path.exists():
+            meta_path = None
+            log.debug("No metadata file for %s", mid)
+        pairs[mid] = (eml_path, meta_path)
+
+    return pairs
+
+
+def _build_rich_email(
+    message_id: str,
+    eml_path: Path,
+    meta_path: Optional[Path],
+) -> RichEmail:
+    raw = eml_path.read_bytes()
+    msg: EmailMessage = email.message_from_bytes(
+        raw, policy=email.policy.default
+    )  # type: ignore[assignment]
+
+    metadata: dict = {}
+    if meta_path:
+        try:
+            metadata = json.loads(meta_path.read_text(encoding="utf-8"))
+        except Exception as exc:
+            log.warning("Could not parse metadata for %s: %s", message_id, exc)
+
+    # Augment missing headers from metadata
+    _merge_metadata(msg, metadata)
+
+    final_bytes = msg.as_bytes(policy=email.policy.SMTP)
+    return RichEmail(
+        message_id=message_id,
+        message=msg,
+        raw_bytes=final_bytes,
+        metadata=metadata,
+    )
+
+
+def _merge_metadata(msg: EmailMessage, meta: dict) -> None:
+    """Back-fill standard headers from Proton metadata where missing."""
+    if not meta:
+        return
+
+    # Subject
+    if not msg.get("Subject") and meta.get("Subject"):
+        msg["Subject"] = meta["Subject"]
+
+    # From
+    if not msg.get("From"):
+        sender_addr = meta.get("SenderAddress", "")
+        sender_name = meta.get("SenderName", "")
+        if sender_addr:
+            from_value = (
+                f'"{sender_name}" <{sender_addr}>'
+                if sender_name
+                else sender_addr
+            )
+            msg["From"] = from_value
+
+    # To
+    if not msg.get("To"):
+        to_list = meta.get("ToList", [])
+        if to_list:
+            msg["To"] = _format_address_list(to_list)
+
+    # CC
+    if not msg.get("Cc"):
+        cc_list = meta.get("CCList", [])
+        if cc_list:
+            msg["Cc"] = _format_address_list(cc_list)
+
+    # BCC
+    if not msg.get("Bcc"):
+        bcc_list = meta.get("BCCList", [])
+        if bcc_list:
+            msg["Bcc"] = _format_address_list(bcc_list)
+
+    # Date — Proton uses Unix timestamp in "Time"
+    if not msg.get("Date") and meta.get("Time"):
+        import email.utils
+        msg["Date"] = email.utils.formatdate(meta["Time"], localtime=False)
+
+    # Message-ID — prefer ExternalID if the EML header is missing
+    if not msg.get("Message-ID") and meta.get("ExternalID"):
+        msg["Message-ID"] = f"<{meta['ExternalID']}>"
+
+    # X-Proton-* passthrough headers for labels and read status
+    if meta.get("LabelIDs"):
+        msg["X-Proton-LabelIDs"] = ",".join(str(l) for l in meta["LabelIDs"])
+    if "Unread" in meta:
+        msg["X-Proton-Unread"] = str(meta["Unread"])
+
+
+def _format_address_list(entries: list) -> str:
+    """Convert Proton address list entries to RFC 5322 address string."""
+    parts = []
+    for entry in entries:
+        if isinstance(entry, dict):
+            name = entry.get("Name", "")
+            addr = entry.get("Address", "")
+            if addr:
+                parts.append(f'"{name}" <{addr}>' if name else addr)
+        elif isinstance(entry, str):
+            parts.append(entry)
+    return ", ".join(parts)

modules/scheduler.py

@@ -0,0 +1,101 @@
+"""APScheduler wrapper for MailRelay's polling interval.
+
+The scheduler runs a single persistent background job that fires the sync
+function at the user-configured interval.  It also exposes helpers so main.py
+can trigger an immediate run or print the next scheduled time.
+"""
+
+from datetime import datetime, timezone
+from typing import Callable, Optional
+
+from apscheduler.schedulers.background import BackgroundScheduler
+from apscheduler.triggers.interval import IntervalTrigger
+
+from .logger import get_logger
+
+log = get_logger(__name__)
+
+JOB_ID = "mailrelay_sync"
+
+_scheduler: Optional[BackgroundScheduler] = None
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def start(sync_fn: Callable, interval_minutes: int) -> None:
+    """Initialise and start the scheduler with *interval_minutes* between runs.
+
+    *sync_fn* is called with no arguments each time the interval fires.
+    """
+    global _scheduler
+
+    if _scheduler and _scheduler.running:
+        log.warning("Scheduler already running — ignoring start() call.")
+        return
+
+    _scheduler = BackgroundScheduler(timezone="UTC")
+    _scheduler.add_job(
+        _guarded(sync_fn),
+        trigger=IntervalTrigger(minutes=interval_minutes),
+        id=JOB_ID,
+        name="MailRelay sync",
+        replace_existing=True,
+        max_instances=1,      # prevent overlapping runs
+        coalesce=True,        # skip missed fires rather than catching up
+    )
+    _scheduler.start()
+    log.info(
+        "Scheduler started. Sync will run every %d minute(s).", interval_minutes
+    )
+
+
+def stop() -> None:
+    """Gracefully shut down the scheduler."""
+    global _scheduler
+    if _scheduler and _scheduler.running:
+        _scheduler.shutdown(wait=False)
+        log.info("Scheduler stopped.")
+    _scheduler = None
+
+
+def run_now(sync_fn: Callable) -> None:
+    """Trigger an immediate sync outside the normal schedule."""
+    log.info("Manual run triggered.")
+    _guarded(sync_fn)()
+
+
+def next_run_time() -> Optional[datetime]:
+    """Return the next scheduled run time (UTC), or None if not scheduled."""
+    if not _scheduler or not _scheduler.running:
+        return None
+    job = _scheduler.get_job(JOB_ID)
+    if job and job.next_run_time:
+        return job.next_run_time
+    return None
+
+
+def update_interval(interval_minutes: int) -> None:
+    """Change the polling interval without restarting the scheduler."""
+    if not _scheduler or not _scheduler.running:
+        raise RuntimeError("Scheduler is not running.")
+    _scheduler.reschedule_job(
+        JOB_ID,
+        trigger=IntervalTrigger(minutes=interval_minutes),
+    )
+    log.info("Polling interval updated to %d minute(s).", interval_minutes)
+
+
+# ---------------------------------------------------------------------------
+# Internals
+# ---------------------------------------------------------------------------
+
+def _guarded(fn: Callable) -> Callable:
+    """Wrap *fn* so unhandled exceptions are logged but don't kill the scheduler."""
+    def wrapper(*args, **kwargs):
+        try:
+            fn(*args, **kwargs)
+        except Exception as exc:
+            log.error("Unhandled exception in sync function: %s", exc, exc_info=True)
+    return wrapper

modules/tools.py

@@ -0,0 +1,147 @@
+"""Manage the bundled Proton Mail Export CLI binary.
+
+The binary lives in  mailrelay/tools/proton-export/proton-mail-export-cli
+and is downloaded on first use (with user consent).
+
+Public API
+----------
+ensure_export_cli() -> Path
+    Return the path to the binary, downloading it first if needed.
+    Raises ToolSetupError if the user declines or the download fails.
+
+BINARY_PATH : Path
+    Absolute path to where the binary is expected.
+"""
+
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+from pathlib import Path
+
+from .logger import get_logger
+
+log = get_logger(__name__)
+
+# ---------------------------------------------------------------------------
+# Paths
+# ---------------------------------------------------------------------------
+
+TOOLS_DIR = Path(__file__).parent.parent / "tools" / "proton-export"
+BINARY_NAME = "proton-mail-export-cli"
+BINARY_PATH = TOOLS_DIR / BINARY_NAME
+
+DOWNLOAD_URL = (
+    "https://proton.me/download/export-tool/proton-mail-export-cli-linux_x86_64.tar.gz"
+)
+ARCHIVE_NAME = "proton-mail-export-cli-linux_x86_64.tar.gz"
+
+
+class ToolSetupError(Exception):
+    pass
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def ensure_export_cli() -> Path:
+    """Return the path to proton-mail-export-cli, downloading if necessary.
+
+    Raises ToolSetupError if the binary is unavailable and the user declines
+    to download it, or if the download / extraction fails.
+    """
+    if BINARY_PATH.exists():
+        log.debug("Export CLI found at %s", BINARY_PATH)
+        return BINARY_PATH
+
+    log.info("Export CLI not found at %s", BINARY_PATH)
+    _prompt_and_download()
+    return BINARY_PATH
+
+
+# ---------------------------------------------------------------------------
+# Internals
+# ---------------------------------------------------------------------------
+
+def _prompt_and_download() -> None:
+    """Ask the user whether to download the CLI, then do it."""
+    print(
+        "\nThe Proton Mail Export CLI is required but was not found.\n"
+        f"It will be downloaded from:\n  {DOWNLOAD_URL}\n"
+        f"and installed to:\n  {BINARY_PATH}\n"
+    )
+
+    answer = input("Download now? [Y/n]: ").strip().lower()
+    if answer and answer not in ("y", "yes"):
+        raise ToolSetupError(
+            "Download declined. Re-run and choose Y, or place the binary at:\n"
+            f"  {BINARY_PATH}"
+        )
+
+    TOOLS_DIR.mkdir(parents=True, exist_ok=True)
+    archive_path = TOOLS_DIR / ARCHIVE_NAME
+
+    _download(DOWNLOAD_URL, archive_path)
+    _extract(archive_path, TOOLS_DIR)
+
+    if not BINARY_PATH.exists():
+        raise ToolSetupError(
+            f"Extraction completed but '{BINARY_NAME}' not found in {TOOLS_DIR}.\n"
+            "The archive layout may have changed — check the contents manually."
+        )
+
+    # Ensure the binary is executable
+    BINARY_PATH.chmod(BINARY_PATH.stat().st_mode | 0o755)
+    log.info("Export CLI ready at %s", BINARY_PATH)
+
+    # Remove the archive to keep the tools directory tidy
+    try:
+        archive_path.unlink()
+    except OSError:
+        pass
+
+
+def _download(url: str, dest: Path) -> None:
+    """Download *url* to *dest* using wget (with progress output)."""
+    if not shutil.which("wget"):
+        raise ToolSetupError(
+            "'wget' is required to download the export tool but was not found on PATH."
+        )
+
+    log.info("Downloading %s …", url)
+    result = subprocess.run(
+        ["wget", "--show-progress", "-O", str(dest), url],
+        check=False,
+    )
+    if result.returncode != 0:
+        # Clean up partial download
+        if dest.exists():
+            dest.unlink()
+        raise ToolSetupError(
+            f"wget exited with code {result.returncode}. Check your network connection."
+        )
+    log.info("Download complete: %s", dest.name)
+
+
+def _extract(archive_path: Path, dest_dir: Path) -> None:
+    """Extract a .tar.gz archive into *dest_dir*."""
+    log.info("Extracting %s …", archive_path.name)
+    try:
+        with tarfile.open(archive_path, "r:gz") as tar:
+            # Safety: skip any members with absolute paths or path traversal
+            safe_members = [
+                m for m in tar.getmembers()
+                if not os.path.isabs(m.name) and ".." not in m.name
+            ]
+            tar.extractall(path=dest_dir, members=safe_members)
+    except tarfile.TarError as exc:
+        raise ToolSetupError(f"Failed to extract archive: {exc}") from exc
+
+    # If the binary landed inside a subdirectory, hoist it up
+    if not BINARY_PATH.exists():
+        for candidate in dest_dir.rglob(BINARY_NAME):
+            shutil.move(str(candidate), str(BINARY_PATH))
+            log.debug("Moved binary from %s to %s", candidate, BINARY_PATH)
+            break

requirements.txt

@@ -0,0 +1,8 @@
+typer[all]>=0.12.0
+pyrage>=1.0.0
+toml>=0.10.2
+pyotp>=2.9.0
+pexpect>=4.9.0
+APScheduler>=3.10.4
+fastapi>=0.111.0
+uvicorn>=0.29.0

setup.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# MailRelay — virtual environment setup script
+# Creates .venv, upgrades pip, and installs all required dependencies.
+# Run once before first use:  bash setup.sh
+
+set -euo pipefail
+
+VENV_DIR=".venv"
+PYTHON="${PYTHON:-python3}"
+MIN_PYTHON_MINOR=11   # 3.11+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+green()  { printf '\033[0;32m%s\033[0m\n' "$*"; }
+yellow() { printf '\033[0;33m%s\033[0m\n' "$*"; }
+red()    { printf '\033[0;31m%s\033[0m\n' "$*"; }
+die()    { red "ERROR: $*"; exit 1; }
+
+# ---------------------------------------------------------------------------
+# Python version check
+# ---------------------------------------------------------------------------
+
+if ! command -v "$PYTHON" &>/dev/null; then
+    die "'$PYTHON' not found. Install Python 3.11+ and retry, or set PYTHON=/path/to/python3."
+fi
+
+PY_VERSION=$("$PYTHON" -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')
+PY_MINOR=$("$PYTHON"  -c 'import sys; print(sys.version_info.minor)')
+PY_MAJOR=$("$PYTHON"  -c 'import sys; print(sys.version_info.major)')
+
+if [[ "$PY_MAJOR" -lt 3 || ( "$PY_MAJOR" -eq 3 && "$PY_MINOR" -lt "$MIN_PYTHON_MINOR" ) ]]; then
+    die "Python 3.${MIN_PYTHON_MINOR}+ required (found $PY_VERSION)."
+fi
+
+green "Python $PY_VERSION — OK"
+
+# ---------------------------------------------------------------------------
+# Create virtual environment
+# ---------------------------------------------------------------------------
+
+if [[ -d "$VENV_DIR" ]]; then
+    yellow "Virtual environment already exists at $VENV_DIR — skipping creation."
+else
+    echo "Creating virtual environment at $VENV_DIR …"
+    "$PYTHON" -m venv "$VENV_DIR"
+    green "Virtual environment created."
+fi
+
+# Activate
+# shellcheck source=/dev/null
+source "$VENV_DIR/bin/activate"
+
+# ---------------------------------------------------------------------------
+# Install dependencies
+# ---------------------------------------------------------------------------
+
+echo "Upgrading pip …"
+pip install --quiet --upgrade pip
+
+echo "Installing dependencies from requirements.txt …"
+pip install --quiet -r requirements.txt
+
+green "All dependencies installed."
+
+# ---------------------------------------------------------------------------
+# Done
+# ---------------------------------------------------------------------------
+
+echo ""
+green "Setup complete!"
+echo ""
+echo "  Activate the environment:   source .venv/bin/activate"
+echo "  Run first-time setup:       python main.py --setup"
+echo "  Start MailRelay:            python main.py"
+echo ""