Finally, I got some time to update the
async-unzip Python module to add support for
aiofiles. One friend asked about this, as his project is
aiofiles, and he doesn’t understand my love for
aiofile. It is elementary; I prefer a thread-free file I/O operation in Linux with caio/libaio native asynchronous async I/O operations by
async-unzip was built for some operations with
arq to not get AsyncioTimeout issues on extended sync operations with unzipping big files. Also, I wouldn’t say I like massive memory usages, like a project on enormous NPI and other healthcare data imports in a pharmacy near me. So,
async-unzip helped with that too.
I will have another post to compare
aiofiles later, but, just to confirm from the start, they will always be slower than the standard way to work with files. We had another aim here 😉
So, it took some time for debugging to find that
seek call is synchronous in
aiofile and requires
aiofiles, and I finally got out of crazy Zlib errors.
So, version 0.3.6 of
async-unzip was released, feel free to take it from PyPi by :
pip install --upgrade async-unzip
P.s. Was quite surprised by results of libaio usage, on my Intel-based server on Debian Linux the time of data import with the usage of async-unzip and asynchronous file-reading operations without threads decreased from 56 to 44 minutes. However, I need to cut the part of actual files I/O operations to generate some synthetic, but close to be real test with possibly parallel work of the tasks inside one process.