Why an Apple Health export adds up to too many steps

An iPhone and an Apple Watch both count your steps, and the export keeps every record from both. The Health app shows one source for each stretch of time; adding up the file counts every walk twice. To get close to Health's figure, total each source separately for each day and take the largest, not the sum. VITALS does that in your browser tab, without uploading the file.

What is in the file

The export is a folder, usually zipped as export.zip, with a large file called export.xml inside. Every measurement is a Record line with a type, a value, a start and end time, and a sourceName: the device or app that wrote it. A walk with a phone in your pocket and a watch on your wrist produces two sets of step records for the same minutes, one named after the phone and one after the watch. A running app, a pedometer app or a smart scale that also writes steps adds a third.

The Health app does not add those together. It keeps an order of sources, which you can see and change under Health, Browse, Activity, Steps, Data Sources & Access, and for each stretch of time it takes the steps from the highest source that has any. The merged figure it shows is worked out when it is shown. It is not written into the export.

The three things in the file that count twice

Sleep has a second catch. A night that starts at 11 pm is split across two calendar dates, so totalling sleep by the date each record starts gives short nights on some days and long ones on others. It reads better to give a night to the morning it ends.

Doing it in a spreadsheet

An export.xml of a few years is usually hundreds of megabytes and often several gigabytes. Excel stops at 1,048,576 rows, and a heart rate record every few minutes passes that within a year or two. Even when it fits, the XML has to be turned into rows first. If you already have the step records as rows with a date, a source and a value, the fix is a pivot table with date as rows and source as columns; the largest column in each row is the day's figure.

Doing it in Python

Most scripts online do df.groupby("day")["steps"].sum(), which is the double count. Grouping by day and source first, then taking the largest, is the fix:

import xml.etree.ElementTree as ET
import pandas as pd

rows = []
for _, el in ET.iterparse("export.xml"):
    if el.tag == "Record":
        if el.get("type") == "HKQuantityTypeIdentifierStepCount":
            rows.append((el.get("startDate")[:10], el.get("sourceName"), float(el.get("value"))))
        el.clear()

df = pd.DataFrame(rows, columns=["day", "source", "steps"])
by_source = df.groupby(["day", "source"])["steps"].sum()
daily = by_source.groupby(level="day").max()

iterparse reads the file as a stream, and el.clear() drops each record once it is read, so a large export does not have to fit in memory. startDate[:10] is the date in the time zone the record was made in, which is how Health dates it after a trip. On the test export VITALS is checked against, where the phone goes on most walks with the watch, the plain sum is 73% too high.

Why the largest source is a little under Health's figure

Taking the largest single source never counts a step twice, but it is not quite Health's merge. On a day the watch was charging for an hour, Health takes that hour from the phone and the watch for the rest. The largest source for the day is the watch alone, without that hour. Usually the difference is small; on a day with a long gap it can be a few thousand steps. Reproducing Health exactly would mean merging minute by minute in your own source order, and the export does not record the order.

Without writing any code

Drop the export.zip on VITALS. It reads the file in your browser, lists every kind of record with its sources, and shows each day's figure with the largest source by default, the sum of every source, or one source alone. Sleep is counted as time asleep with the night given to the morning, and the grouped blood pressure copies are left out. Any of it saves as CSV. It reads the zip as it goes, so a file of a gigabyte or more is fine, and nothing is uploaded: the export holds years of heart rate and sleep, plus the date of birth and sex set in Health.