Files

Felix Förtsch 8c3f51adb4 fix v0.3 spec: address 11 review issues (concurrency, codable, bcc, folder roles, etc.)

2026-03-13 22:17:49 +01:00

23 KiB

Raw Permalink Blame History

Magnum Opus v0.3 — Compose, Triage, and SMTP

Goal: Turn the read-only v0.2 email client into a fully functional email client. Add SMTP sending (compose, reply, forward), IMAP write-back (flags, move, delete, append), and an offline-safe action queue. Plain text compose only.

Builds on: v0.2 native Swift email client (IMAP sync, GRDB/SQLite, SwiftUI three-column UI, threaded messages, FTS5 search).

Architecture Overview

v0.3 adds three capabilities to the existing architecture:

SMTPClient — new MagnumOpusCore module for sending email
IMAP write operations — extend existing IMAPClient with flag/move/append commands
ActionQueue — persistent queue in SyncEngine for offline-safe action dispatch

┌─────────────────────────────────────────────────────────────┐
│                      SwiftUI Apps                            │
│                                                              │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐             │
│  │ ComposeView│  │TriageBar   │  │ MoveToSheet│             │
│  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘             │
│        └────────────────┼───────────────┘                    │
│                   ┌─────┴──────┐                             │
│                   │ ViewModels │                             │
│                   └─────┬──────┘                             │
├─────────────────────────┼────────────────────────────────────┤
│              MagnumOpusCore (Swift Package)                   │
│                   ┌─────┴──────┐                             │
│                   │ SyncEngine │                             │
│                   │            │                             │
│         ┌─────────┼─────────┐  │                             │
│         │   ActionQueue     │  │                             │
│         │  (offline queue)  │  │                             │
│         └──┬──────────┬─────┘  │                             │
│   ┌────────┘          └──────┐ │                             │
│   │ SMTPClient │  │IMAPClient│ │  ┌──────────┐              │
│   │  (NEW)     │  │(extended)│ │  │ MailStore │              │
│   └─────┬──────┘  └────┬────┘ │  │(GRDB+FTS5)│              │
│         │               │      │  └───────────┘              │
└─────────┼───────────────┼──────┼─────────────────────────────┘
          │ TLS           │ TLS
          ▼               ▼
    ┌──────────┐    ┌──────────┐
    │   SMTP   │    │   IMAP   │
    │  Server  │    │  Server  │
    └──────────┘    └──────────┘

Data flow for write operations:

User action (triage or send) → ViewModel
ViewModel → ActionQueue.enqueue() → local SQLite update (immediate UI feedback)
ActionQueue dispatcher → SMTPClient or IMAPClient (immediate attempt, queue if offline)
Next sync cycle: flush pending actions before fetching new messages

Schema Changes

Migration v2_smtp: Account SMTP fields

ALTER TABLE account ADD COLUMN smtpHost TEXT;
ALTER TABLE account ADD COLUMN smtpPort INTEGER;
ALTER TABLE account ADD COLUMN smtpSecurity TEXT;  -- "ssl" or "starttls"

Migration v2_draft: Local drafts table

CREATE TABLE draft (
    id          TEXT PRIMARY KEY,
    accountId   TEXT NOT NULL REFERENCES account(id),
    inReplyTo   TEXT,
    forwardOf   TEXT,
    toAddresses TEXT,           -- JSON array of {"name", "address"}
    ccAddresses TEXT,           -- JSON array of {"name", "address"}
    bccAddresses TEXT,          -- JSON array of {"name", "address"}
    subject     TEXT,
    bodyText    TEXT,
    createdAt   TEXT NOT NULL,
    updatedAt   TEXT NOT NULL
);

Migration v2_mailboxRole: Special folder detection

ALTER TABLE mailbox ADD COLUMN role TEXT;  -- "trash", "archive", "sent", "drafts", "junk", or NULL

Populated during sync from LIST response attributes (RFC 6154). Falls back to name matching if no attributes are present.

Migration v2_pendingAction: Offline action queue

CREATE TABLE pendingAction (
    id          TEXT PRIMARY KEY,
    accountId   TEXT NOT NULL REFERENCES account(id),
    actionType  TEXT NOT NULL,  -- "setFlags", "move", "delete", "send", "append"
    payload     TEXT NOT NULL,  -- JSON with action-specific data
    createdAt   TEXT NOT NULL,
    retryCount  INTEGER NOT NULL DEFAULT 0,
    lastError   TEXT
);

SMTPClient Module

New module in MagnumOpusCore. Built on SwiftNIO + swift-nio-ssl, mirroring the IMAPClient layering pattern.

Internal Layers

SMTPConnection (actor) — NIO ClientBootstrap with TLS. Handles implicit SSL (port 465) and STARTTLS (port 587). Sends raw SMTP lines, reads status code + text responses.
SMTPResponseHandler (ChannelInboundHandler) — SMTP responses are line-based: <3-digit code> <text>. Multiline responses use <code>-<text> continuation. Buffer lines, deliver complete response via continuation.
SMTPCommandRunner — sends commands sequentially, waits for response. No tags (unlike IMAP) — SMTP is strictly request/response.

Public API

public actor SMTPClient: Sendable {
    public init(host: String, port: Int, security: SMTPSecurity, credentials: Credentials)

    public func send(message: OutgoingMessage) async throws
    public func testConnection() async throws
}

public enum SMTPSecurity: String, Sendable, Codable {
    case ssl       // implicit TLS, port 465
    case starttls  // upgrade after connect, port 587
}

public struct OutgoingMessage: Sendable, Codable {
    public var from: EmailAddress
    public var to: [EmailAddress]
    public var cc: [EmailAddress]
    public var bcc: [EmailAddress]
    public var subject: String
    public var bodyText: String
    public var inReplyTo: String?     // Message-ID for threading
    public var references: String?    // accumulated References header
    public var messageId: String      // generated Message-ID for this message
}

Message-ID generation: Generated at enqueue time by ComposeViewModel, before the action is persisted. Format: <UUID@domain> where domain is extracted from the sender's email address (e.g., <550e8400-e29b-41d4-a716-446655440000@example.com>). This ensures both the send and append actions reference the same Message-ID.

SMTP Command Sequence

connect (TLS or plain)
← 220 server greeting
→ EHLO <hostname>
← 250 capabilities (check for STARTTLS, AUTH mechanisms)
[if STARTTLS: → STARTTLS, ← 220, upgrade TLS, → EHLO again]
→ AUTH PLAIN <base64(username + password)>
← 235 authenticated
→ MAIL FROM:<sender>
← 250 OK
→ RCPT TO:<recipient>  (repeat for each to + cc + bcc)
← 250 OK
→ DATA
← 354 go ahead
→ <RFC 5322 formatted message>
→ .
← 250 OK (message accepted)
→ QUIT
← 221 bye

Message Formatting

The SMTPClient builds the RFC 5322 message internally:

From: "Name" <address>
To: "Name" <address>, ...
Cc: "Name" <address>, ...
Subject: <subject>
Date: <RFC 2822 date>
Message-ID: <generated-id@domain>
In-Reply-To: <original-message-id>    (if reply)
References: <accumulated-refs>         (if reply)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<body text, quoted-printable encoded>

Authentication

Support AUTH PLAIN (most common) and AUTH LOGIN (fallback). Check server's EHLO capabilities for supported mechanisms. If neither is available, throw an error — no CRAM-MD5 or OAuth in v0.3.

Testing

Mock SMTP server using NIO: accept connections, verify command sequence, capture the formatted message. Tests verify:

Connection with SSL and STARTTLS
Authentication (PLAIN and LOGIN)
Message formatting (headers, encoding, threading headers)
Error handling (auth failure, rejected recipient, connection drop)

IMAP Write Operations

Extend the existing IMAPClient actor and IMAPClientProtocol.

New Protocol Methods

public protocol IMAPClientProtocol: Sendable {
    // existing v0.2 methods...

    // v0.3 write operations
    func storeFlags(uid: Int, mailbox: String, add: [String], remove: [String]) async throws
    func moveMessage(uid: Int, from: String, to: String) async throws
    func copyMessage(uid: Int, from: String, to: String) async throws
    func expunge(mailbox: String) async throws
    func appendMessage(to mailbox: String, message: Data, flags: [String]) async throws
    func capabilities() async throws -> Set<String>
}

IMAP MOVE Support

RFC 6851 MOVE command. Check server capabilities at connect time:

If MOVE is advertised: UID MOVE <uid> <destination>
Otherwise: UID COPY <uid> <destination> + UID STORE <uid> +FLAGS (\Deleted) + EXPUNGE

Special Folder Detection

Detect well-known folders from mailbox LIST attributes (RFC 6154):

Attribute	Purpose	Name fallbacks
`\Trash`	Delete target	"Trash", "Deleted Messages", "Bin"
`\Archive`	Archive target	"Archive", "All Mail"
`\Sent`	Sent mail	"Sent", "Sent Messages", "Sent Mail"
`\Drafts`	Drafts	"Drafts"
`\Junk`	Spam	"Junk", "Spam"

Store the detected role in the MailboxRecord.role field (added in migration v2_mailboxRole). The ActionQueue uses role to resolve target folders for archive, delete, and sent mail operations without hardcoding folder names.

Testing

Extend MockIMAPClient with the new protocol methods. Test:

Flag store commands (add/remove flags)
MOVE command generation
COPY+DELETE+EXPUNGE fallback when MOVE not available
APPEND command with message data and flags
Capability detection

ActionQueue

Lives in the SyncEngine module. The central dispatcher for all write operations.

Design

public actor ActionQueue {
    private let store: MailStore
    private let imapClient: any IMAPClientProtocol
    private let smtpClient: SMTPClient?

    /// Enqueue an action — applies locally first, then attempts remote dispatch.
    /// The local SQLite update (phase 1) is synchronous. The remote dispatch (phase 2)
    /// runs as a detached Task on the actor's executor, not the main actor.
    public func enqueue(_ action: PendingAction) throws

    /// Enqueue multiple actions in a single SQLite transaction (e.g., send + append).
    /// Actions are dispatched in array order.
    public func enqueue(_ actions: [PendingAction]) throws

    /// Flush all pending actions to server (called by SyncCoordinator before fetch).
    /// Dispatches sequentially in creation order. Network I/O runs on the actor's
    /// executor, not the main thread.
    public func flush() async

    /// Number of pending actions (for UI badge/indicator)
    public var pendingCount: Int
}

Concurrency note: ActionQueue is a plain actor, not @MainActor. The enqueue() method writes to SQLite (fast, no network). The flush() method performs network I/O via SMTPClient/IMAPClient actors. ViewModels call await actionQueue.enqueue(...) from @MainActor — the hop to the ActionQueue actor is fine for the brief SQLite write. The GRDB ValueObservation on the main actor picks up the local changes immediately after enqueue returns.

PendingAction Types

public struct PendingAction: Sendable, Codable {
    var id: String
    var accountId: String
    var actionType: ActionType
    var payload: ActionPayload
    var createdAt: Date
}

public enum ActionType: String, Sendable, Codable {
    case setFlags
    case move
    case delete
    case send
    case append
}

public enum ActionPayload: Sendable, Codable {
    case setFlags(uid: Int, mailbox: String, add: [String], remove: [String])
    case move(uid: Int, from: String, to: String)
    case delete(uid: Int, mailbox: String, trashMailbox: String)
    case send(message: OutgoingMessage)
    case append(mailbox: String, messageData: String, flags: [String])
}

Storage encoding: actionType is derived from payload on insert (the enum discriminator). The payload column stores JSON using Swift's auto-synthesized Codable with explicit CodingKeys for each associated-value case. JSON shape per type:

{"setFlags": {"uid": 42, "mailbox": "INBOX", "add": ["\\Seen"], "remove": []}}
{"move": {"uid": 42, "from": "INBOX", "to": "Archive"}}
{"delete": {"uid": 42, "mailbox": "INBOX", "trashMailbox": "Trash"}}
{"send": {"message": { ... OutgoingMessage fields ... }}}
{"append": {"mailbox": "Sent", "messageData": "...", "flags": ["\\Seen"]}}

The actionType column is redundant with the JSON discriminator but kept for efficient SQL queries (e.g., WHERE actionType = 'send') without JSON parsing.

Two-Phase Execution

When the user performs an action:

Local phase (synchronous, immediate):
- Update SQLite directly (change flags, update mailboxId for moves, etc.)
- GRDB ValueObservation fires → UI updates instantly
- Create pendingAction record in SQLite
Remote phase (async):
- Attempt to dispatch the action via IMAP/SMTP immediately
- On success: delete the pendingAction record
- On failure: leave in queue, increment retryCount, store error

Ordering and Concurrency

Actions are dispatched in creation order (FIFO), sequential per account
A send followed by append (save to Sent) must execute in order
If an action fails, subsequent actions for the same message are held (no flag change after a failed move)

Failure Handling

Error Type	Behavior
Network timeout / connection refused	Retry on next flush. Exponential backoff: 30s, 1m, 2m, 4m, 8m, 15m cap.
Auth failure (SMTP 535, IMAP NO on LOGIN)	Stop queue, surface re-auth prompt to user
Permanent failure (message doesn't exist)	Mark failed, remove from queue, surface to user
Retry cap exceeded (5 attempts)	Mark failed, surface to user

SyncCoordinator Integration

The existing performSync() flow becomes:

Flush action queue (push local changes to server)
Connect to IMAP
Fetch deltas (existing v0.2 sync)
Disconnect

This ensures the server reflects local changes before we pull new state.

Compose Flow

ComposeViewModel

@Observable @MainActor
final class ComposeViewModel {
    var to: String = ""
    var cc: String = ""
    var bcc: String = ""
    var subject: String = ""
    var bodyText: String = ""
    var mode: ComposeMode = .new

    // Draft state
    var draftId: String?
    var isDirty: Bool { /* compare current to saved */ }

    func saveDraft() throws
    func deleteDraft() throws
    func send() async throws
}

enum ComposeMode: Sendable {
    case new
    case reply(to: MessageSummary)
    case replyAll(to: MessageSummary)
    case forward(of: MessageSummary)
    case draft(DraftRecord)
}

Body Fetch Requirement

Reply and forward modes require the original message body for quoting. MessageSummary.bodyText may be nil if the body hasn't been fetched yet (v0.2 lazily fetches bodies). Before opening compose in reply/forward mode, the ViewModel must:

Check if bodyText is populated
If not, fetch the body via IMAPClient.fetchBody(uid:) and store it via MailStore.storeBody()
If the fetch fails (offline), open compose without quoted text and show a note: "Original message body unavailable offline"

Reply Behavior

To: Original sender (reply) or all recipients minus self (reply-all)
Subject: Re: <original> (strip duplicate Re:/Fwd: prefixes)
Body: \n\nOn <date>, <sender> wrote:\n> <line1>\n> <line2>\n...
Headers: Set In-Reply-To: <original-message-id>, append to References

Forward Behavior

To: Empty (user fills in)
Subject: Fwd: <original>

Body:

\n\n---------- Forwarded message ----------
From: <original sender>
Date: <original date>
Subject: <original subject>
To: <original recipients>

<original body>

Headers: No In-Reply-To. Set References to original Message-ID.

Send Sequence

Two actions enqueued atomically via ActionQueue.enqueue([send, append]) (single SQLite transaction):

send — SMTPClient delivers the message
append — IMAPClient appends the formatted message to the Sent folder with \Seen flag

If send succeeds but append fails, the append stays in the queue for retry. The message is delivered — the Sent folder copy is a convenience, not a requirement.

Draft Auto-Save

Auto-save to SQLite every 10 seconds while isDirty
On explicit discard: delete draft record, dismiss compose
On successful send enqueue: delete draft record, dismiss compose
On close without send: save draft, dismiss compose (draft persists)

ComposeView

macOS: New window via openWindow
iOS: Sheet presentation
Fields: To, CC, BCC (expandable), Subject, body TextEditor
Toolbar: Send button, Discard button
Compose is modal — one compose at a time in v0.3

Triage Actions and UI

Actions

Action	IMAP Operation	Local Effect	macOS Shortcut
Archive	MOVE to Archive folder	Update mailboxId	`e`
Delete	MOVE to Trash (or EXPUNGE if already in Trash)	Update mailboxId (or remove record)	`⌫`
Flag/unflag	STORE +/-\Flagged	Toggle isFlagged	`s`
Read/unread	STORE +/-\Seen	Toggle isRead	`⇧⌘U`
Move to folder	MOVE to chosen folder	Update mailboxId	`⌘⇧M`

All actions go through the ActionQueue for offline safety.

UI Integration

ThreadListView toolbar:

Archive, Delete, Flag buttons (visible when thread selected)
Reply, Reply All, Forward buttons

Swipe actions on thread rows:

Leading swipe: Archive (green)
Trailing swipe: Delete (red)

Keyboard shortcuts (macOS):

Applied via .keyboardShortcut() on toolbar buttons
Only active when the mail view has focus

Move-to-folder picker:

Sheet with searchable mailbox list
Triggered by toolbar button or ⌘⇧M

Auto-advance:

After archiving/deleting, automatically select the next thread in the list
If no next thread, select previous. If list empty, show empty state.

Thread-Level vs Message-Level

Triage actions operate on all messages in the thread that belong to the currently selected mailbox. A thread may span multiple mailboxes (e.g., a message in Inbox and its reply in Sent). Archiving from Inbox moves only the Inbox copies to Archive — the Sent copies stay in Sent. This matches standard email client behavior (Apple Mail, Gmail).

Account Setup Updates

AccountConfig Model

public struct AccountConfig: Sendable, Codable, Equatable {
    public var id: String
    public var name: String
    public var email: String
    public var imapHost: String
    public var imapPort: Int
    public var smtpHost: String?
    public var smtpPort: Int?
    public var smtpSecurity: SMTPSecurity?
}

SMTP fields are optional for backwards compatibility with v0.2 accounts.

Auto-Discovery Extension

AutoDiscovery lives in the App target (Apps/MagnumOpus/Services/AutoDiscovery.swift) and stays there — it's UI-layer configuration logic, not core protocol code.

The existing AutoDiscovery.discoverIMAP(for:) is renamed to AutoDiscovery.discover(for:) and returns a DiscoveredConfig containing both IMAP and SMTP settings:

struct DiscoveredConfig: Sendable {
    var imap: DiscoveredServer?
    var smtp: DiscoveredServer?
}

enum AutoDiscovery {
    /// Discover both IMAP and SMTP settings for an email address.
    /// Replaces the v0.2 discoverIMAP(for:) method.
    static func discover(for email: String) async -> DiscoveredConfig
}

The existing parseISPDBXML already receives the full XML. Extend it to also extract the <outgoingServer type="smtp"> block — the XML structure is identical to the incoming server block. The DNS SRV fallback adds _submission._tcp lookup for SMTP alongside _imaps._tcp for IMAP.

AccountSetupView Updates

Show SMTP fields alongside IMAP in manual mode
Auto-discovery fills both IMAP and SMTP
Test both connections before saving
Existing v0.2 accounts: prompt to add SMTP settings when user first opens compose

Testing Strategy

New Test Targets

SMTPClientTests — mock SMTP server, test connection/auth/send/formatting/errors
Extended IMAPClientTests — test flag store, move, copy, append, capability detection
Extended SyncEngineTests — test ActionQueue enqueue/flush/retry/failure handling
Extended App tests — ComposeViewModel: reply/forward prefill, draft save/load, send validation

Key Test Scenarios

Send plain text email → verify RFC 5322 formatting, SMTP command sequence
Reply to message → verify threading headers (In-Reply-To, References)
Archive while offline → local update immediate, action queued, flushed on reconnect
Send fails (auth error) → queue stops, user prompted, retry after re-auth
Move message → MOVE command if supported, COPY+DELETE fallback if not
Draft auto-save → save on timer, load on resume, delete on send

Dependencies

No new external dependencies. v0.3 uses the same four packages as v0.2:

Package	Purpose
swift-nio-imap	IMAP protocol (existing)
swift-nio	Networking (transitive, also used for SMTP)
swift-nio-ssl	TLS for IMAP and SMTP (existing)
GRDB.swift	SQLite, FTS5, ValueObservation (existing)

The SMTPClient is built on swift-nio and swift-nio-ssl directly — no SMTP-specific dependency needed.

v0.3 Scope

In

SMTPClient module (NIO-based, SSL/STARTTLS, AUTH PLAIN/LOGIN)
Compose: new message, reply, reply-all, forward (plain text)
Local drafts with auto-save (SQLite, not synced to IMAP)
IMAP APPEND to Sent folder after send
IMAP write-back: flag changes, move, delete
IMAP MOVE with COPY+DELETE fallback
ActionQueue: local-first, immediate remote dispatch, offline queue with retry
Special folder detection (Trash, Archive, Sent via LIST attributes)
Triage: archive, delete, flag, mark read/unread, move to folder
Keyboard shortcuts for triage (macOS)
Swipe actions on thread rows
Auto-advance after triage
Account setup extended for SMTP (auto-discovery + manual)
Schema migration for SMTP fields, drafts, pending actions

Out (Deferred)

Rich text / Markdown compose (v0.3+)
IMAP draft sync (local only in v0.3)
Attachment sending (v0.3+)
OAuth / XOAUTH2 authentication
IMAP IDLE push
Multiple accounts
GTD triage (defer/delegate — v0.4)
VTODO / CalDAV (v0.4)

23 KiB Raw Permalink Blame History

Magnum Opus v0.3 — Compose, Triage, and SMTP

Architecture Overview

Schema Changes

Migration v2_smtp: Account SMTP fields

Migration v2_draft: Local drafts table

Migration v2_mailboxRole: Special folder detection

Migration v2_pendingAction: Offline action queue

SMTPClient Module

Internal Layers

Public API

SMTP Command Sequence

Message Formatting

Authentication

Testing

IMAP Write Operations

New Protocol Methods

IMAP MOVE Support

Special Folder Detection

Testing

ActionQueue

Design

PendingAction Types

Two-Phase Execution

Ordering and Concurrency

Failure Handling

SyncCoordinator Integration

Compose Flow

ComposeViewModel

Body Fetch Requirement

Reply Behavior

Forward Behavior

Send Sequence

Draft Auto-Save

ComposeView

Triage Actions and UI

Actions

UI Integration

Thread-Level vs Message-Level

Account Setup Updates

AccountConfig Model

Auto-Discovery Extension

AccountSetupView Updates

Testing Strategy

New Test Targets

Key Test Scenarios

Dependencies

v0.3 Scope

In

Out (Deferred)

23 KiB

Raw Permalink Blame History