Document Processing and Dataset Quality
Follow a document from collection through extraction, preparation and validation.
Collect the source
Start with paper and book collection if versions or acquisition history are getting lost. Keep the original file and why you saved it before converting anything.
Extract and prepare
Read PDF extraction when layout or reading order matters. Continue to dataset preparation for identifiers, conflicting rows and split rules. Saved chat and contact extraction cover records that must keep a precise source reference.
Validate the output
Use the synthetic data article when you can check answers independently. Image conversion covers dimensions, byte limits and metadata. Note organisation explains how to make a reading copy while keeping the originals.
Articles in this reading path
- Organising Research Papers, Books and Reading NotesKeep selection, acquisition and reading separate in a research collection, with source versions and the question behind each saved paper or book.
- PDF to Markdown: Checking Extraction FidelityCheck whether PDF extraction preserves page evidence, reading order and meaning, and identify when native text extraction needs an OCR alternative.
- Dataset Cleaning Without Losing IDs or MeaningClean datasets with explicit row accounting, preserved identifiers, conflict quarantine and reproducible splits rather than silent type conversion.
- Synthetic Datasets with Independent ValidationGenerate synthetic examples from checkable source facts, validate answers independently and split parent records before producing related variants.
- Image Resizing: File Size, Formats and MetadataChoose image dimensions, fit rules and file size limits, then check the actual output and metadata instead of relying on format assumptions.
- Processing Saved Chat Logs Without Losing HistoryProcess saved chat JSON into traceable fields while preserving message IDs, edits, timestamps and source pages. This is an offline export workflow.
- Extracting Contact Fields with Source EvidenceTurn saved text into contact fields while keeping each value attached to its source passage and separating extraction from verification.
- Organising Research Notes Without Losing HistoryCreate a useful reading copy of research notes while preserving original files, duplicate relationships, conflicting versions and attribution.