On the other hand, when I rewrote some code to manually read the input instead of using binary it gave an 8x speed up:
I think it should be possible to write a more efficient package (maybe it already exists). flatparse is a gold standard for parsing, but it is not really meant for binary input.