Finally, I got some time to update the async-unzip Python module to add support for aiofiles. One friend asked about this, as his project is aiofiles, and he doesn’t understand my love for aiofile. It is elementary; I prefer a thread-free file I/O operation in Linux with caio/libaio native asynchronous async I/O operations by aiofile.
So, async-unzip was built for some operations with arq to not get AsyncioTimeout issues on extended sync operations with unzipping big files. Also, I wouldn’t say I like massive memory usages, like a project on enormous NPI and other healthcare data imports in a pharmacy near me. So, async-unzip helped with that too.
I will have another post to compare aiofile and aiofiles later, but, just to confirm from the start, they will always be slower than the standard way to work with files. We had another aim here 😉
So, it took some time for debugging to find that seek call is synchronous in aiofile and requires await in aiofiles, and I finally got out of crazy Zlib errors.
So, version 0.3.6 of async-unzip was released, feel free to take it from PyPi by :
pip install --upgrade async-unzip
P.s. Was quite surprised by results of libaio usage, on my Intel-based server on Debian Linux the time of data import with the usage of async-unzip and asynchronous file-reading operations without threads decreased from 56 to 44 minutes. However, I need to cut the part of actual files I/O operations to generate some synthetic, but close to be real test with possibly parallel work of the tasks inside one process.
No Comments, Be The First!