Resolved -
This incident has been resolved.
Jul 15, 17:37 UTC
Update -
The backfill just finished and all systems metrics are nominal since recovery.
We expect no further issues and thus updates to this incident. We will continue monitoring the systems actively for a while.
Jul 15, 15:32 UTC
Monitoring -
The API is working again and we are backfilling data. Current estimates are that backfilling will take 1-2h.
Jul 15, 14:12 UTC
Update -
We are having an avenue to recovery now, if it works out we expect starting to backfill data within the hour.
Jul 15, 14:00 UTC
Update -
We are still working on recovery, during which the historical positions API is fully unavailable. There is no certain timeline to recovery, we are doing our best to make it as soon as possible and will update as soon as we know more on the timeline.
Jul 15, 12:06 UTC
Update -
The original assessment of the scale of the issue was incorret: All historical position data is missing since roughly Saturday midnight.
Jul 15, 08:33 UTC
Identified -
There is an ongoing issue where parts of the data in the historical positions API is not available. We are working on recovering it. In the process full interruptions to API access and delayed data updates are to be expected.
Jul 15, 08:27 UTC