This was a good read. I’ve definitely done my share of hasty pickling when training models and it was a surprise when the old ones didn’t load anymore: the files store only class names and I had changed constructor arguments or something.
For quick caching it’s invaluable. For example I recently wrote a script that processed images and constructed a large matrix whose contents were only dependent on the image’s dimensions. It was easy to memoize the function with pickle for quicker experimentation.
By the way, if you need to analyze data written from native code in Python, you can use numpy’s flexible dtypes for that very cleanly.