Python – Writing large ZIP archives without memory inflation

Buzon - ZipFly

ZipFly is a zip archive generator based on zipfile.py. It was created by Buzon.io to generate very large ZIP archives for immediate sending out to clients, or for writing large ZIP archives without memory inflation.

Requirements

Python 3.5+

Install

pip3 install zipfly

Basic usage, compress on-the-fly during writes

Basic use case is compressing on the fly. Some data will be buffered by the zipfile deflater, but memory inflation is going to be very constrained. Data will be written to destination at fairly regular intervals.

ZipFly default attributes:

• paths: [ ]
• mode: w
• chunksize: (bytes) 16384
• compression: Stored
• allowZip64: True
• compresslevel: None
• storesize: (bytes) 0

paths list of dictionaries:

• fs (filesystem): path from your disk
  
• n (name): final path in zip file

import zipfly
    
    paths = [ 
        {
            'fs': 'home/user/Videos/jupiter.mp4', 
            'n': 'movies/jupiter.mp4', 
        },       
        {
            'fs': 'home/user/Documents/mercury.mp4', 
            'n': 'movies/mercury.mp4', 
        },          
    ]

    zfly = zipfly.ZipFly( paths = paths )

    generator = zfly.generator()
    print ( generator )
    # <generator object generator at 0x7f85aad60b13>

    with open("test.zip", "wb") as f:
        for i in generator:
            f.write(i)

Examples

Create a ZIP file with size estimation Use the BufferPredictionSize to compute the correct size of the resulting archive.

Streaming a large file Efficient way to read a single very large binary file in python

Streaming multiple files in a zip The easiest is to use the Django or Flask built-in streaming feature

Maintainer

Santiago Debus (@santiagodebus.com)

Santiago's open-source projects are supported by his Patreon. If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.

License

This library was created by Buzon.io and is released under the MIT. Copyright 2019 Grow HQ, Inc.