Disclaimer: All material in this post has been used with permission. Certain details have been modified for client confidentiality.
Every engineer has a story about a task that seemed trivial at first glance. Ours started with four words from a client: "Can you just unzip these files?"
Sure. We unzip files all the time. We've processed petabytes of data, wrangled 100TB datasets, spun up GPU clusters. Unzipping some archives? That's a Tuesday afternoon. We almost quoted it as a half-day job.
We're very glad we didn't.
The Initial Request
The client was a legal team preparing a large body of archived engineering data for litigation. The data — roughly 500GB of compressed archives — arrived on a set of external hard drives whose partitions were encrypted with VeraCrypt. Their internal team had tried to extract the files on a few workstations but kept running into problems: corrupted archives, nested layers of compression, and machines running out of disk space mid-extraction. After a few frustrated days, they reached out to us.
"We just need someone to unzip these for us and put them somewhere we can access them," they said. It felt like calling a plumber to turn on a faucet. But we took the call seriously because, in our experience, when someone says "just unzip," what they really mean is "we tried and something went very wrong."
We asked a few clarifying questions: How many archives? What format? How deeply are they nested? The answers were vague — "a few hundred ZIPs," "some are inside other ZIPs," "we're not totally sure what's in there." We decided to do a quick scoping session before committing to a timeline.
That decision saved us.
The Discovery: Turtles All the Way Down
We pulled the drives into a staging environment and started surveying the contents. At first glance, it looked manageable: around 1,200 top-level archive files, mostly .zip with a smattering of .7z, .rar, and .tar.gz files. But when we started extracting, a pattern emerged that made our stomachs drop.
The archives contained more archives. Those inner archives contained even more archives. Some chains went five or six levels deep. It was compression inception — zips within zips within zips.
Here's what a typical nesting chain looked like:
vendor_delivery_2024_batch_003.zip
└── region_northeast/
├── NE_records_001.7z
│ └── NE_records_001/
│ ├── quarterly_reports.tar.gz
│ │ └── quarterly_reports/
│ │ ├── Q1/
│ │ │ ├── report_001.pdf
│ │ │ ├── report_002.pdf
│ │ │ └── ... (4,800 files)
│ │ └── ...
│ ├── raw_exports.zip
│ │ └── ... (18,000 files)
│ └── metadata.rar
│ └── ... (3,200 files)
├── NE_records_002.7z
└── ... (340 more archives)We wrote a quick script to crawl the top-level archives and estimate the total uncompressed size. The number came back: approximately 12 terabytes. Across what we projected to be 132 million individual files.
This was not a "just unzip" job. This was a full-scale data engineering project.
The Technical Challenges
Recursive Archive Extraction
The first and most fundamental challenge was building an extraction engine that could handle arbitrary nesting depth. You can't just run unzip *.zip and call it a day. Each archive type requires a different tool, and the output of one extraction becomes the input for the next.
We built a recursive extraction pipeline in Python. The core logic works like this: extract an archive, scan the output for any files that are themselves archives, and queue those for extraction. Repeat until there's nothing left to decompress.
import os
import subprocess
import zipfile
import tarfile
import logging
from pathlib import Path
logger = logging.getLogger(__name__)
ARCHIVE_EXTENSIONS = {
'.zip', '.7z', '.rar', '.tar', '.tar.gz', '.tgz',
'.tar.bz2', '.tar.xz', '.gz', '.bz2', '.xz'
}
def is_archive(filepath: str) -> bool:
"""Check if a file is a recognized archive format."""
p = filepath.lower()
return any(p.endswith(ext) for ext in ARCHIVE_EXTENSIONS)
def extract_archive(filepath: str, dest_dir: str) -> list[str]:
"""
Extract a single archive to dest_dir.
Returns list of extracted file paths.
"""
extracted = []
os.makedirs(dest_dir, exist_ok=True)
if filepath.endswith('.zip'):
with zipfile.ZipFile(filepath, 'r') as zf:
zf.extractall(path=dest_dir)
extracted = [os.path.join(dest_dir, n) for n in zf.namelist()]
elif filepath.endswith(('.7z', '.rar')):
cmd = ['7z', 'x', f'-o{dest_dir}', '-y', filepath]
subprocess.run(cmd, check=True, capture_output=True)
extracted = list_files_recursive(dest_dir)
elif filepath.endswith(('.tar.gz', '.tgz', '.tar.bz2', '.tar.xz', '.tar')):
with tarfile.open(filepath, 'r:*') as tf:
tf.extractall(path=dest_dir)
extracted = [os.path.join(dest_dir, m.name) for m in tf.getmembers()
if m.isfile()]
return extracted
def process_archive_tree(root_archive: str, output_root: str,
max_depth: int = 10) -> dict:
"""
Recursively extract all nested archives using DFS (a LIFO stack).
Returns extraction statistics.
"""
stats = {"archives_processed": 0, "files_extracted": 0, "errors": []}
stack = [(root_archive, output_root, 0)] # (path, dest, depth)
while stack:
archive_path, dest_dir, depth = stack.pop()
if depth > max_depth:
logger.warning(f"Max depth exceeded: {archive_path}")
stats["errors"].append(f"Max depth exceeded: {archive_path}")
continue
try:
logger.info(f"[depth={depth}] Extracting: {archive_path}")
extracted_files = extract_archive(archive_path, dest_dir)
stats["archives_processed"] += 1
for fpath in extracted_files:
if os.path.isfile(fpath) and is_archive(fpath):
nested_dest = fpath + "_extracted"
stack.append((fpath, nested_dest, depth + 1))
else:
stats["files_extracted"] += 1
except Exception as e:
logger.error(f"Failed to extract {archive_path}: {e}")
stats["errors"].append(f"{archive_path}: {str(e)}")
return statsWe started with a BFS (breadth-first) traversal, but switched to DFS (depth-first, backed by a LIFO stack). Going depth-first let us finish one nested chain completely before moving to the next, which gave us a much more accurate, monotonic progress signal — instead of every branch creeping forward at once, we could see whole subtrees close out.
Dealing with Corrupted Archives
A small fraction of the archives — about 0.2% — were partially or fully corrupted. Some had been copied from failing drives, others had been truncated during transfer. We deliberately didn't build a pre-flight integrity check: test-extracting 12TB of nested archives would have cost roughly as much time as actually extracting them. Instead, we let extraction itself be the test and failed fast:
- Skip the obviously dead: any archive that arrived as a 0-byte file was logged and skipped before we tried to open it.
- Let the extractor be the validator: a bad archive makes
7zexit non-zero. We caught that error, logged it against the work item, and deleted the half-written output so a partial extraction never polluted the results. - Flag the unrecoverable: archives that couldn't be extracted at all were collected into an error report and sent back to the client, who in some cases got replacement copies from the original vendor.
Infrastructure: Building the Pipeline
It became clear early on that this wasn't a job for a single workstation. 12TB of output data doesn't fit on most local drives, and the extraction process is both CPU-intensive (decompression) and I/O-intensive (writing millions of small files). We needed cloud infrastructure.
Storage Architecture
We set up a layered storage strategy on AWS:
- S3 (cold storage): The original 500GB of archives were uploaded to S3 as an immutable backup. Total cost: roughly $12/month.
- EFS (hot processing): We attached an Amazon EFS file system to the workers for active extraction work. The containers shipped with only ~200GB of ephemeral storage, which wasn't enough headroom — a single archive could balloon to hundreds of gigabytes mid-extraction — so EFS gave us the elastic working space the local disk couldn't.
- S3 (output): Extracted files were uploaded back to S3 in a structured directory layout using the AWS CLI's
s3 syncwith aggressive parallelism. S3 was the system of record — the state of the job was simply the state of the files in S3, so we never needed a separate database to track what had been processed.
Worker Architecture
We ran the extraction pipeline across a fleet of containerized workers, coordinated by an SQS queue. Each top-level archive was a discrete work item. Workers would pull a message from the queue, download the archive from S3, recursively extract everything, upload the results back to S3, and then clean up local disk.
import boto3
import json
import shutil
import tempfile
from pathlib import Path
sqs = boto3.client('sqs')
s3 = boto3.client('s3')
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/archive-extraction"
SOURCE_BUCKET = "raw-archives"
OUTPUT_BUCKET = "extracted-output"
def worker_loop():
"""Main worker loop: pull jobs from SQS, extract, upload."""
while True:
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=1,
WaitTimeSeconds=20,
VisibilityTimeout=3600 # 1 hour per job
)
messages = response.get('Messages', [])
if not messages:
continue
msg = messages[0]
job = json.loads(msg['Body'])
receipt_handle = msg['ReceiptHandle']
try:
process_job(job)
sqs.delete_message(QueueUrl=QUEUE_URL,
ReceiptHandle=receipt_handle)
except Exception as e:
logger.error(f"Job failed: {job['archive_key']}: {e}")
# Message returns to queue after visibility timeout
def process_job(job: dict):
"""Download archive, extract recursively, upload results."""
archive_key = job['archive_key']
with tempfile.TemporaryDirectory(dir='/mnt/data') as work_dir:
# Download from S3
local_archive = os.path.join(work_dir, 'source',
os.path.basename(archive_key))
os.makedirs(os.path.dirname(local_archive), exist_ok=True)
s3.download_file(SOURCE_BUCKET, archive_key, local_archive)
# Extract recursively
output_dir = os.path.join(work_dir, 'output')
stats = process_archive_tree(local_archive, output_dir)
logger.info(
f"Extracted {stats['files_extracted']} files "
f"from {stats['archives_processed']} archives"
)
# Upload to S3
upload_directory_to_s3(output_dir, OUTPUT_BUCKET,
prefix=Path(archive_key).stem)
We ran the extraction fleet on AWS — a mix of Fargate tasks and an EKS node group — so we could scale the number of workers up and down against the queue without babysitting individual machines. Each worker handled multiple extraction jobs concurrently, using Python's concurrent.futures to parallelize the decompression step within a single archive. The full extraction pipeline ran for about 72 hours.
Performance Optimization
The Small File Problem
The biggest performance bottleneck wasn't decompression — it was writing millions of small files. Most of the 132 million files were under 100KB. Operating systems are not optimized for creating millions of tiny files: filesystem metadata operations (creating inodes, updating directories) become the bottleneck, not actual data transfer.
We experimented with several mitigations:
- Batch around storage latency: EFS gave us the capacity, but any networked file system carries real per-operation latency when you're creating millions of tiny files. Rather than hammer it one file at a time, we leaned on the batched uploads below.
- Batched S3 uploads: Instead of uploading files one at a time, we bundled small files into tar archives for upload and wrote a manifest file alongside them. This reduced S3 API calls from millions to thousands.
Monitoring and Observability
With the fleet running for 72 hours, we needed real-time visibility into the pipeline. We leaned on CloudWatch for that — SQS queue depth (messages remaining) was the headline progress signal, alongside per-worker CPU, memory, and disk usage.
Watching the queue drain told us exactly how many archives were left, the current extraction rate, and a rough projected completion time. When a worker died at 3 AM because an archive expanded to fill its entire local scratch disk (a 400GB uncompressed surprise inside a 12GB 7z file), a CloudWatch alarm caught it and we had it back online within 20 minutes.
The Final Numbers
When the dust settled, here's what the "just unzip this" project looked like:
- Input: ~500GB compressed
- Nested archives discovered: 14,832
- Output: ~132M files, ~12TB uncompressed
- Corrupted/unrecoverable archives: ~0.2%
- Processing time: ~72 hours
The compression ratio was staggering — roughly 24:1 overall. Every gigabyte of compressed data hid around 24 gigabytes of actual content.
Delivering the Data
Extraction was only half the job. The client needed the fully expanded data in a form they could hand to opposing counsel — and 12TB of loose files isn't something you email. Once everything was extracted and verified, we recompressed it into .tar.zst archives — zstd gave us a good balance of ratio and speed — copied the result onto a physical hard drive, and shipped the drive so the final data set could be extracted one more time on a workstation at the destination. For a litigation deliverable, a drive you can hold is sometimes still the most reliable transfer mechanism there is.
Lessons Learned
1. Always Scope Data Projects Before Quoting
If we had quoted this as a half-day job based on the client's description — "just unzip some files" — we would have been in serious trouble. The gap between the perceived complexity and the actual complexity was enormous. We now have a standing rule: for any project involving data from external sources, we do a paid scoping session first. We'll sample 5-10% of the data, measure compression ratios, check for nesting, and build a realistic estimate before committing to a timeline or budget.
2. Build for 10x the Expected Scale
Our initial extraction script worked fine on a handful of test archives. It fell over immediately at scale. The worker architecture, SQS coordination, and disk-space guardrails were all things we had to build after the naive approach failed. Now, even for seemingly simple data tasks, we design for at least 10x the expected volume from the start. The marginal cost of building a scalable pipeline upfront is tiny compared to the cost of rebuilding mid-project.
3. Small Files Are the Silent Killer
We've worked with 100TB datasets before, but those were large files — images, videos, parquet files. 132 million small files is a completely different beast. Every layer of the stack — filesystem, S3 API, metadata operations — has per-operation overhead that dominates when files are tiny. If your data is lots of small files, plan your infrastructure around IOPS and metadata operations, not raw throughput.
4. Invest in Observability Early
Our CloudWatch monitoring was an afterthought — we bolted it on after the first few hours of running blind and not knowing whether things were working or silently failing. On the next project, observability was the first thing we set up. Being able to see exactly what's happening in real time is worth its weight in gold, especially for multi-day processing jobs.
5. Keep the Raw Data Sacred
We uploaded the original 500GB of compressed archives to S3 before touching anything. Every operation was non-destructive. When we discovered halfway through that our extraction script was incorrectly handling a specific RAR variant (RAR5 with solid compression), we could re-extract those archives from the originals without going back to the client. Immutable source data saved us at least two days of back-and-forth.
The Takeaway
"Just unzip this" is the data engineering equivalent of "it's just a small change" in software development. The simplicity of the description hides an ocean of complexity. Nested archives, encryption, format diversity, corruption, scale — any one of these would complicate a simple extraction. All of them together turned a faucet-turning job into a 72-hour distributed systems project.
But that's exactly the kind of problem we love solving at Fan Pier Labs. We take the gnarly, underestimated, "how hard could it be?" problems and build real engineering solutions for them. This project started as a favor and ended as one of our favorite war stories.
Next time someone tells you to "just unzip this," ask how deep the rabbit hole goes. You might be surprised.
Have a data challenge that's bigger than it looks? We specialize in making the impossible feel routine. Get in touch — we'd love to hear about it.