Cassandra 1.2 migration : Broken pipe
We migrated our cluster to Cassandra 1.2 (from 1.1.7) and we tried to run a repair after the migration (we run them periodically, as recommended).
The repair failed due to the SSTable version being different, so we used nodetool upgradesstables and nodetool scrub and retried.
But then the repair failed with this exception :
java.io.IOException: Broken pipe
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
We also noticed that before dying, nodetool netstats shows weird statistics (SSTable transfers being at 756845%…).
It seems to be a problem with the transfer of compressend SSTables, I will try to disable compression on this ColumnFamily and retry.
Disabling compression fixed the problem so we tried to re-enable it and got a
Last written key DecoratedKey(97115[…]3136) >= current key DecoratedKey(8…
The issue was caused by a bug in Cassandra 1.2, affecting transfers of compressed column families between nodes. It was fixed in 1.2.2 => Jira entry for this bug.