Get rid of the repeated bytesRead requests

We use `isolate` so we know exactly how far we've advanced, we can trivially
compute the correct position.