Felix Förtsch
7aa3106be3
address IMAP client complexity, remove bidirectional flag sync, add attachment table, error handling, HTML rendering, FTS5 sync notes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
423 lines
18 KiB
Markdown
423 lines
18 KiB
Markdown
# Magnum Opus v0.2 — Native Swift Email Client
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Build a read-only email client in pure Swift that runs on macOS, iOS, and iPadOS. Prove the on-device architecture: IMAP sync → SQLite cache → SwiftUI three-column UI with threaded message view and full-text search.
|
|
|
|
**Architectural pivot from v0.1:** No remote server. All sync, indexing, and storage run on-device. Standard protocols (IMAP) connect directly to mail servers. The v0.1 TypeScript prototype (branch `feature/v0.1-backend-and-macos`) remains as reference.
|
|
|
|
**Design document:** `docs/plans/2026-03-10-magnum-opus-design.md` (architecture section superseded by this document — on-device Swift replaces the Hono/Uberspace backend)
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
Everything runs on-device. The app connects directly to IMAP servers via swift-nio-imap, stores messages in SQLite via GRDB, and renders with SwiftUI. No intermediate server, no CLI tool dependencies, no file-based storage layer.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────┐
|
|
│ SwiftUI Apps │
|
|
│ (macOS / iOS / iPadOS targets) │
|
|
│ │
|
|
│ ┌──────────┐ ┌────────────┐ ┌─────────────┐ │
|
|
│ │ Sidebar │ │ ThreadList │ │ ThreadDetail│ │
|
|
│ └────┬─────┘ └─────┬──────┘ └──────┬──────┘ │
|
|
│ └───────────────┴────────────────┘ │
|
|
│ │ @Observable │
|
|
│ ┌────────┴────────┐ │
|
|
│ │ ViewModels │ │
|
|
│ └────────┬────────┘ │
|
|
├───────────────────────┼──────────────────────────┤
|
|
│ MagnumOpusCore (Swift Package) │
|
|
│ ┌────────┴────────┐ │
|
|
│ │ SyncEngine │ │
|
|
│ │ (coordinator) │ │
|
|
│ └───┬─────────┬───┘ │
|
|
│ ┌────────┘ └────────┐ │
|
|
│ ┌──────┴──────┐ ┌───────┴───────┐ │
|
|
│ │ IMAPClient │ │ MailStore │ │
|
|
│ │ (nio-imap) │ │ (GRDB+FTS5) │ │
|
|
│ └──────┬──────┘ └───────────────┘ │
|
|
│ │ │
|
|
└─────────┼────────────────────────────────────────┘
|
|
│ TLS
|
|
▼
|
|
┌───────────┐
|
|
│ IMAP │
|
|
│ Server │
|
|
└───────────┘
|
|
```
|
|
|
|
**Data flow:** SyncCoordinator triggers IMAPClient → fetches message deltas → writes to MailStore (SQLite) → GRDB ValueObservation pushes updates to @Observable ViewModels → SwiftUI re-renders.
|
|
|
|
---
|
|
|
|
## Package & Project Structure
|
|
|
|
```
|
|
MagnumOpus/
|
|
├── Packages/
|
|
│ └── MagnumOpusCore/
|
|
│ ├── Package.swift
|
|
│ ├── Sources/
|
|
│ │ ├── Models/ ← shared types
|
|
│ │ ├── IMAPClient/ ← IMAP protocol layer
|
|
│ │ ├── MailStore/ ← SQLite + GRDB + FTS5
|
|
│ │ └── SyncEngine/ ← sync coordinator
|
|
│ └── Tests/
|
|
│ ├── IMAPClientTests/
|
|
│ ├── MailStoreTests/
|
|
│ └── SyncEngineTests/
|
|
├── Apps/
|
|
│ ├── macOS/ ← macOS SwiftUI app
|
|
│ └── iOS/ ← iOS/iPadOS SwiftUI app
|
|
├── docs/
|
|
├── Ideas/
|
|
└── scripts/
|
|
```
|
|
|
|
**Dependency direction:** `SyncEngine` → `IMAPClient` + `MailStore` → `Models`. App targets import `SyncEngine` and `MailStore`.
|
|
|
|
---
|
|
|
|
## Data Model
|
|
|
|
### Core Types (`Models` module)
|
|
|
|
- **`Account`** — IMAP server config (SMTP deferred to v0.3). Credentials stored in Keychain, not SQLite.
|
|
- **`Mailbox`** — IMAP folder (Inbox, Sent, Drafts, Archive, Trash, custom). Tracks `uidValidity` and `uidNext` for delta sync.
|
|
- **`Message`** — email with headers, body, attachment metadata. Identified by RFC 5322 `Message-ID`.
|
|
- **`Thread`** — group of messages linked by `In-Reply-To` / `References` headers.
|
|
|
|
### SQLite Schema
|
|
|
|
```sql
|
|
CREATE TABLE account (
|
|
id TEXT PRIMARY KEY,
|
|
name TEXT NOT NULL,
|
|
email TEXT NOT NULL,
|
|
imapHost TEXT NOT NULL,
|
|
imapPort INTEGER NOT NULL
|
|
-- SMTP fields added in v0.3
|
|
);
|
|
|
|
CREATE TABLE mailbox (
|
|
id TEXT PRIMARY KEY,
|
|
accountId TEXT NOT NULL REFERENCES account(id),
|
|
name TEXT NOT NULL,
|
|
uidValidity INTEGER NOT NULL,
|
|
uidNext INTEGER NOT NULL
|
|
);
|
|
|
|
CREATE TABLE message (
|
|
id TEXT PRIMARY KEY,
|
|
accountId TEXT NOT NULL REFERENCES account(id),
|
|
mailboxId TEXT NOT NULL REFERENCES mailbox(id),
|
|
uid INTEGER NOT NULL,
|
|
messageId TEXT,
|
|
inReplyTo TEXT,
|
|
refs TEXT,
|
|
subject TEXT,
|
|
fromAddress TEXT,
|
|
fromName TEXT,
|
|
toAddresses TEXT, -- JSON array of {"name", "address"} objects
|
|
ccAddresses TEXT, -- JSON array of {"name", "address"} objects
|
|
date TEXT NOT NULL,
|
|
snippet TEXT,
|
|
bodyText TEXT,
|
|
bodyHtml TEXT,
|
|
isRead INTEGER NOT NULL DEFAULT 0,
|
|
isFlagged INTEGER NOT NULL DEFAULT 0,
|
|
size INTEGER NOT NULL DEFAULT 0,
|
|
UNIQUE(mailboxId, uid)
|
|
);
|
|
|
|
CREATE TABLE thread (
|
|
id TEXT PRIMARY KEY,
|
|
accountId TEXT NOT NULL REFERENCES account(id),
|
|
subject TEXT,
|
|
lastDate TEXT NOT NULL,
|
|
messageCount INTEGER NOT NULL DEFAULT 0
|
|
);
|
|
|
|
CREATE TABLE threadMessage (
|
|
threadId TEXT NOT NULL REFERENCES thread(id),
|
|
messageId TEXT NOT NULL REFERENCES message(id),
|
|
PRIMARY KEY (threadId, messageId)
|
|
);
|
|
|
|
CREATE TABLE attachment (
|
|
id TEXT PRIMARY KEY,
|
|
messageId TEXT NOT NULL REFERENCES message(id),
|
|
filename TEXT,
|
|
mimeType TEXT NOT NULL,
|
|
size INTEGER NOT NULL DEFAULT 0,
|
|
contentId TEXT, -- for inline images (CID references)
|
|
cachePath TEXT -- path in app sandbox, NULL until downloaded
|
|
);
|
|
|
|
CREATE VIRTUAL TABLE messageFts USING fts5(
|
|
subject, fromName, fromAddress, bodyText,
|
|
content='message', content_rowid='rowid'
|
|
);
|
|
-- FTS sync: the SQL above is illustrative. Implementation must use GRDB's
|
|
-- DatabaseMigrator with db.create(virtualTable:using:) for automatic
|
|
-- trigger generation to keep messageFts in sync on insert/update/delete.
|
|
```
|
|
|
|
### Cross-Mailbox Deduplication
|
|
|
|
The same message can appear in multiple IMAP mailboxes (e.g., CC to self → Inbox + Sent). Each IMAP copy gets its own row in the `message` table (different `mailboxId` + `uid`). Thread reconstruction deduplicates by `messageId` (RFC 5322 Message-ID): when threading, all rows sharing the same `messageId` join the same thread. The `threadMessage` join table references the `message.id` (UUID), so a thread may contain multiple rows for the same logical message — the UI deduplicates by `messageId` when rendering. Messages without a `messageId` (broken mailers) cannot be deduplicated and each gets its own single-message thread.
|
|
|
|
### Thread Reconstruction
|
|
|
|
Simplified JWZ algorithm — no subject-based fallback grouping (produces false matches):
|
|
|
|
1. New message arrives: extract `Message-ID`, `In-Reply-To`, `References`
|
|
2. Look up existing threads where any member shares a reference
|
|
3. If found → add to that thread (merge if multiple threads match)
|
|
4. If not found → create new thread
|
|
5. Update thread metadata (last date, message count, subject from earliest known message in thread — stripped of `Re:`/`Fwd:` prefixes)
|
|
|
|
---
|
|
|
|
## IMAP Client Layer
|
|
|
|
### Complexity Note
|
|
|
|
swift-nio-imap is a low-level protocol encoder/decoder, not a ready-made IMAP client. Building the `IMAPClient` actor requires: TLS negotiation, IMAP state machine management, command-response pairing, untagged response handling, literal continuations, and error recovery. This is the single largest implementation effort in v0.2 and must be broken into multiple implementation tasks with explicit milestones.
|
|
|
|
### Public API
|
|
|
|
```swift
|
|
actor IMAPClient {
|
|
init(host: String, port: Int, credentials: Credentials)
|
|
|
|
func connect() async throws
|
|
func disconnect() async throws
|
|
|
|
func listMailboxes() async throws -> [MailboxInfo]
|
|
func selectMailbox(_ name: String) async throws -> MailboxStatus
|
|
|
|
func fetchMessages(uids: UIDRange, fields: FetchFields) async throws -> [FetchedMessage]
|
|
func fetchNew(since uid: UID) async throws -> [FetchedMessage]
|
|
func fetchFlags(uids: UIDRange) async throws -> [UIDFlagsPair]
|
|
}
|
|
```
|
|
|
|
### Internal Layers
|
|
|
|
The actor is built in layers:
|
|
|
|
1. **Connection** — SwiftNIO `Channel` with TLS (via swift-nio-ssl), handles connect/disconnect/reconnect
|
|
2. **Command pipeline** — sends IMAP commands, pairs tagged responses, handles continuation requests
|
|
3. **Response parser** — processes untagged responses (EXISTS, EXPUNGE, FLAGS updates)
|
|
4. **High-level operations** — `listMailboxes`, `selectMailbox`, `fetchMessages` built on the pipeline
|
|
|
|
### Sync Strategy (Poll-Based for v0.2)
|
|
|
|
1. Connect to IMAP server
|
|
2. `SELECT` each mailbox, compare `UIDVALIDITY` with stored value
|
|
- Changed → full re-sync of that mailbox (rare — means server rebuilt UIDs)
|
|
- Same → fetch messages with UID > last known `uidNext`
|
|
3. Fetch envelopes + snippets eagerly; full bodies lazily
|
|
4. Fetch flag changes from server (read-only — v0.2 does not write flags back to server). Local `isRead`/`isFlagged` state mirrors the server. Opening a message locally does not change flags in v0.2 — read state is purely server-driven.
|
|
5. Disconnect, schedule next poll
|
|
|
|
### Eager vs. Lazy Fetching
|
|
|
|
- **Eager (on sync):** envelope (from, to, subject, date, message-id, references), flags, snippet (~200 chars of body)
|
|
- **Lazy (on open):** full body text, HTML body, attachments — fetched when user opens message, then cached in SQLite
|
|
- **Background fill:** after initial sync, progressively fetch bodies for recent messages (last 30 days) so they're available offline
|
|
|
|
### Attachment Caching
|
|
|
|
Attachment metadata stored eagerly in SQLite. Binary content downloaded on demand, cached to disk in the app sandbox (not SQLite — attachments can be large).
|
|
|
|
---
|
|
|
|
## Sync Engine
|
|
|
|
Orchestrates the full sync lifecycle:
|
|
|
|
```swift
|
|
@Observable
|
|
final class SyncCoordinator {
|
|
private let account: Account
|
|
private let imapClient: IMAPClient
|
|
private let store: MailStore
|
|
|
|
var syncState: SyncState // .idle, .syncing(progress), .error(Error)
|
|
var events: AsyncStream<SyncEvent>
|
|
|
|
// 5-minute interval applies in foreground only.
|
|
// On iOS background, sync uses BGAppRefreshTask at system-determined intervals.
|
|
func startPeriodicSync(interval: Duration = .seconds(300))
|
|
func syncNow() async throws
|
|
func stopSync()
|
|
}
|
|
|
|
enum SyncEvent {
|
|
case newMessages(count: Int, mailbox: String)
|
|
case flagsChanged(messageIds: [String])
|
|
case syncStarted
|
|
case syncCompleted
|
|
case syncFailed(Error)
|
|
}
|
|
```
|
|
|
|
### MailStore — Read/Write Interface
|
|
|
|
```swift
|
|
final class MailStore {
|
|
func threads(in mailbox: String, limit: Int, offset: Int) -> [Thread]
|
|
func messages(in thread: Thread) -> [Message]
|
|
func message(id: String) -> Message?
|
|
func search(query: String) -> [Message]
|
|
|
|
func insertMessages(_ messages: [FetchedMessage]) throws
|
|
func updateFlags(uid: UID, mailbox: String, flags: MessageFlags) throws
|
|
func storeBody(messageId: String, text: String?, html: String?) throws
|
|
func reconstructThreads(for messages: [Message]) throws
|
|
|
|
func observeThreads(in mailbox: String) -> AsyncValueObservation<[Thread]>
|
|
}
|
|
```
|
|
|
|
GRDB `ValueObservation` is the sole mechanism for reactive UI updates. `MailStore` itself is not `@Observable` — it provides `ValueObservation`-based streams that ViewModels consume. This replaces the SSE pattern from v0.1.
|
|
|
|
### Error Handling & Offline Behavior
|
|
|
|
The app must remain fully usable with cached data when offline. Error strategy:
|
|
|
|
- **Connection failure / offline:** `SyncState` moves to `.error`. UI shows a non-intrusive banner ("Offline — showing cached mail"). All cached data remains browsable and searchable.
|
|
- **Authentication failure:** Surface to user immediately with option to re-enter credentials. Do not retry automatically (avoids account lockout).
|
|
- **Mid-sync connection drop:** Abort current sync cycle gracefully. Already-stored messages are kept. Next poll retries from the last successful `uidNext`.
|
|
- **Retry policy:** Exponential backoff starting at 30 seconds, capped at 15 minutes. Reset on successful sync or manual trigger.
|
|
- **TLS failure:** Do not fall back to plaintext. Surface error to user.
|
|
|
|
---
|
|
|
|
## SwiftUI App Layer
|
|
|
|
### View Hierarchy
|
|
|
|
```
|
|
AppRoot
|
|
├── AccountSetupView ← first launch: configure IMAP
|
|
└── MainView (NavigationSplitView)
|
|
├── Sidebar
|
|
│ └── MailboxListView ← Inbox, Sent, Drafts, Archive, Trash, custom
|
|
├── Content
|
|
│ └── ThreadListView ← threads in selected mailbox
|
|
└── Detail
|
|
└── ThreadDetailView ← messages in selected thread
|
|
└── MessageView ← single message (header, body, attachments)
|
|
```
|
|
|
|
### HTML Email Rendering
|
|
|
|
- Use `WKWebView` for HTML email display (works on macOS, iOS, iPadOS)
|
|
- **Block remote content by default** (tracking pixels, external images). Show a "Load remote content" button.
|
|
- For `multipart/alternative` messages, prefer plain text; offer a toggle to view HTML
|
|
- Strip `<script>` tags and event handler attributes before rendering
|
|
- Inline CSS only — no external stylesheet loading
|
|
- Attachments with `Content-ID` (inline images) are resolved from the local cache
|
|
|
|
### Platform Adaptation
|
|
|
|
`NavigationSplitView` handles platform differences automatically:
|
|
|
|
- **macOS:** three-column layout always visible, keyboard shortcuts
|
|
- **iPad landscape:** three columns, collapsible sidebar
|
|
- **iPad portrait / iPhone:** push navigation (sidebar → thread list → detail)
|
|
|
|
### Account Setup (First Launch)
|
|
|
|
1. User enters email address
|
|
2. Auto-discovery: query Mozilla ISPDB (Thunderbird autoconfig database) as primary, DNS SRV records (RFC 6186) as fallback
|
|
3. Fall back to manual IMAP host + port entry
|
|
4. Credentials stored in Keychain
|
|
5. Initial sync begins immediately after setup
|
|
|
|
SMTP configuration is deferred to v0.3 (compose/reply). v0.2 setup only collects IMAP details.
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
SyncCoordinator (background sync)
|
|
↓ writes
|
|
MailStore (SQLite via GRDB)
|
|
↓ ValueObservation
|
|
@Observable ViewModels
|
|
↓
|
|
SwiftUI Views
|
|
```
|
|
|
|
No manual refresh needed. ViewModels observe two sources: **MailStore** (via GRDB `ValueObservation`) for data, and **SyncCoordinator** (via `@Observable`) for sync status (progress, errors). Background sync + GRDB observation = automatic UI updates.
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
| Package | Purpose |
|
|
|---------|---------|
|
|
| [swift-nio-imap](https://github.com/apple/swift-nio-imap) | IMAP protocol layer |
|
|
| [swift-nio](https://github.com/apple/swift-nio) | Networking (transitive) |
|
|
| [swift-nio-ssl](https://github.com/apple/swift-nio-ssl) | TLS for IMAP/SMTP |
|
|
| [GRDB.swift](https://github.com/groue/GRDB.swift) | SQLite, FTS5, ValueObservation |
|
|
|
|
Four dependencies total (two transitive). No bloat.
|
|
|
|
---
|
|
|
|
## Testing Strategy
|
|
|
|
- **`IMAPClientTests`** — mock IMAP server via swift-nio-imap test utilities. Verify connection, fetch, flag sync, UID tracking, error handling.
|
|
- **`MailStoreTests`** — in-memory SQLite. Test insert, query, thread reconstruction, FTS5 search, observation. Most testable layer.
|
|
- **`SyncEngineTests`** — mock IMAPClient + real MailStore (in-memory). Test full sync flow: delta detection, message storage, flag reconciliation, error recovery.
|
|
- **UI tests** — minimal for v0.2. Verify navigation flow on both platforms.
|
|
|
|
Not tested in v0.2: performance at scale, real IMAP server integration.
|
|
|
|
---
|
|
|
|
## v0.2 Scope
|
|
|
|
### In
|
|
|
|
- Single account (IMAP only, read-only — no flag write-back to server)
|
|
- IMAP sync via swift-nio-imap (poll-based, configurable interval)
|
|
- SQLite cache with GRDB (envelope eager, body lazy + cached)
|
|
- FTS5 full-text search
|
|
- Thread reconstruction (Message-ID / In-Reply-To / References)
|
|
- Three-column SwiftUI layout (macOS + iOS + iPadOS)
|
|
- Mailbox navigation (Inbox, Sent, Drafts, Archive, Trash, custom folders)
|
|
- Message reading with HTML/plain text display
|
|
- Attachment metadata display (download on demand)
|
|
- Account setup with auto-discovery (Mozilla ISPDB + DNS SRV)
|
|
- Keychain credential storage
|
|
- Background body prefetch (last 30 days)
|
|
|
|
### Out (Deferred)
|
|
|
|
- Compose / reply / forward (needs SMTP client — v0.3)
|
|
- Triage actions: do / defer / delegate / file / discard (v0.3)
|
|
- Task management / VTODO (v0.4+)
|
|
- CalDAV / CardDAV sync (v0.4+)
|
|
- IMAP IDLE for real-time push (optimization, post-v0.2)
|
|
- Multiple accounts
|
|
- Keyboard shortcuts beyond system defaults
|
|
- Notifications
|
|
|
|
---
|
|
|
|
## Future Evolution
|
|
|
|
v0.2 is the foundation. The path forward:
|
|
|
|
- **v0.3:** SMTP client (SwiftNIO-based), compose/reply/forward, basic triage (archive, delete, flag)
|
|
- **v0.4:** VTODO tasks via CalDAV, unified inbox mixing email + tasks, GTD triage workflow
|
|
- **v0.5:** Contacts via CardDAV, calendar via CalDAV, delegation workflow
|
|
- **Later:** IMAP IDLE, multiple accounts, keyboard-first triage, append-only history
|