Fix a race condition with SegmentFetcher