ParquetDatasetWriter.flush() named each shard from the chunk index plus the number
of files in the worker directory. When two flushes split the chunks in different
ways, a new shard could get the name of an existing shard and replace it. The
data in the old shard was lost without an error.
New shards now get numbers after the highest existing shard number in all worker
directories. One helper writes each shard to a temporary file and renames it. It
raises FileExistsError instead of an overwrite. flush(write_remainder=True) now
also counts the remainder rows in its return value.
Chanu Ollala
3c713d808a