Processing Saved Chat Logs Without Losing History
On this page
A chat export gets interesting once I need to actually do something with it.
Find an old decision. Pull out a few fields. Compare edits. Turn a messy conversation into a table. Then wonder why row 184 says something slightly different from what I remember reading.
The bit I care about is tracing that row all the way back to the message that produced it.
saved pages in, a receipt out
The local input is saved JSON, not a Discord login. A page carries cursor, messages and next_cursor; each message keeps its ID, author, channel, timestamp and content.
The pagination fixture supplies 250 identities across three saved pages, then repeats one message and adds a later edit to another. The result keeps 250 identities, with two revisions for the edited message. Supplying only the first page returns partial, even though all of that page's rows are readable. A conflicting cursor also returns partial.
This is an offline completeness claim about the supplied page chain. It cannot establish that the original export included every server message. No network or account access is part of this module.
two jobs, one clean hand off
I split the problem into two parts.
The exporter preserves message history and identity. The parser reads those records and proposes useful fields. Either side should remain inspectable without the other running.
A message record needs its ID, author, channel ID, timestamps, content availability and attachment/embed descriptions. IDs stay strings because turning 0017 into 17 for convenience is exactly the sort of helpful cleanup I don't want.
The export run keeps its scope, cursors and completion state too. A history collected over several minutes isn't one magical atomic snapshot; things can change while I'm paging through it.
pagination gets weird quickly
Say page one ends at m100 and a retry makes page two start with m100 again.
That shouldn't become two messages.
If m100 changed between observations, I also don't want the newer text silently replacing the older one. Same identity, different observed version.
If I cancel after two pages, keep the data and mark the run partial. A file existing on disk does not magically mean the export finished :)
every field should open its source
If I later extract:
company = Cedar Systems
I want the exact message or document span supporting it.
Each field keeps the source ID, value, span or structural path, extractor revision and review status. A generated value with no support belongs in a suggestion queue, not the final table.
I'd parse predictable Markdown/JSON deterministically and only use a model for irregular prose where the rules genuinely struggle.
Faithful JSON should stay beside any flattened spreadsheet export. Formula like text may need presentation escaping, but the original value should remain recoverable.
missing data should stay missing
Discord permissions and Gateway intents affect what an app can observe; the Gateway documentation is the reference I'd use for a live collector.
If message content isn't available, don't turn that into an empty message. not available and "" are different observations.
Tokens also belong in configuration, not URLs or exported diagnostics. Fake pages are enough to debug pagination safely.
Implementation checks and recorded findings
The saved JSON exporter retained all 10,000 unique messages across 100 overlapping pages. This checks reconciliation and export completeness for the supplied inputs, including an edit; it does not measure live Discord acquisition.
Generated saved JSON pages with duplicates and an edit; offline reconciliation only, no Discord connection.
| Recorded metric | Value |
|---|---|
| exported messages | 10000 |
| input unique messages | 10000 |
| pages | 100 |
The comparison record includes the 1 recorded run, measured values, source hashes and dependency versions. This is a single fixed evaluation; no across seed uncertainty is estimated.
what I built from this
The offline exporter now follows explicit cursors, reconciles duplicate message IDs, preserves edit history and records conflicting revisions rather than overwriting them.
A 250 message fixture covers three pages, an edit and repeated delivery. Interrupted or malformed input produces a partial receipt.
See export_logs.py; the module README documents the JSON page format. Output preserves raw message objects and the source file hash.
The next feature I'd add is the review screen: proposed value on the left, exact source text on the right, plus a diff when parser versions disagree.
That's the useful bit. I shouldn't have to trust the parser saying it improved; I should be able to see what changed and why.
A live Discord adapter, PDF adapter and interactive field review UI are still separate work. The current tool only consumes saved JSON pages and does not connect to Discord.