About JSONL Merger

Merge multiple JSONL files together. Part of the DevTools Surf developer suite. Browse more tools in the Data / SQL collection.

Use Cases

• Combine JSONL log files from multiple services into a single chronological stream for analysis.
• Merge daily export files into a monthly JSONL dataset for batch processing.
• Combine API response pages (each saved as JSONL) into a single dataset.
• Aggregate JSONL files from distributed processing jobs into a single result file.

Tips

• Validate that all input files use the same schema before merging — schema mismatches produce merged files with inconsistent fields that break downstream parsers.
• Use the deduplication option when merging files that may overlap — dedup on a unique ID field to prevent duplicate records in the output.
• Sort merged output by timestamp or ID field immediately after merging to maintain chronological or indexed order for analytical queries.

Fun Facts

• JSONL's line-delimited format makes it trivially parallelizable — split a large JSONL file into N parts, process each in parallel, and cat the results together. This property made it the preferred format for distributed data pipelines in the 2010s.
• Elasticsearch's Bulk API requires JSONL format for batch indexing — each two-line pair (action metadata + document) is standard NDJSON. The API can ingest millions of documents per second in this format.
• Apache Spark can read JSONL files directly using spark.read.json() and distributes lines across worker nodes automatically — the line-separated format maps directly to Spark's RDD/Dataset partition model.

FAQ

Does it handle files with different schemas?

It merges any valid JSONL files regardless of schema consistency. If schemas differ, the output is still valid JSONL but downstream tools may encounter missing fields. Enable schema validation to flag mismatches before merging.

Can it deduplicate on merge?

Yes — specify a key field (e.g., 'id') and duplicate records (same key value) are deduplicated, keeping the first or last occurrence based on your setting.

Is there a size limit?

Files up to 50MB can be merged in-browser. For larger files, use the streaming CLI alternative: cat file1.jsonl file2.jsonl > merged.jsonl (no tool needed for simple concatenation).