Test iOS latency percentiles on a cloud Mac

For the same batch of requests, your local report shows a P95 of 190 milliseconds, but the pipeline produces a different value. Before blaming the cloud Mac's performance, check whether one side uses nearest rank and the other uses linear interpolation. When running statistical acceptance checks on VMFly physical nodes, first pin down the algorithm, sample scope and empty-set semantics. Only then should you investigate whether the API has slowed down. This article provides a Swift script that requires no network services and fits into the regression workflow for an iOS project's statistics module.

Define the percentile calculation contract

This implementation accepts only integer percentiles from 1 to 100. Sort the nonnegative, finite durations in ascending order. For percentile q and sample count n, the nearest rank is ceil(q × n / 100), using one-based indexing. Return the actual sample at that position, without interpolating between adjacent values.

For example, with 20 increasing samples ranging from 10 to 200 milliseconds, P50, P95 and P99 are 100, 190 and 200 milliseconds, respectively. With just one sample, all three results equal that sample. This follows from the algorithm's definition; it does not mean that tail latency has been adequately observed.

An empty set has no percentile. Reporting zero makes a window with no successful requests look faster than a normal window.

A negative value, infinity or NaN causes the entire calculation to fail. Do not silently filter out invalid values: doing so disguises collection errors as a normal report. If a production system quarantines invalid samples instead, it must track the rejected sample count separately.

Calculate ranks without floating-point multiplication

Save the following as percentile-check.swift. The script uses Swift from the full Xcode toolchain and requires no changes to the app project. All durations are in milliseconds. The function cannot infer units from bare numbers, so callers must perform any necessary conversion.

import Foundation

enum MetricError: Error, Equatable {
    case invalidPercentile
    case invalidDuration
}

func percentile(_ values: [Double], q: Int) throws -> Double? {
    guard (1...100).contains(q) else {
        throw MetricError.invalidPercentile
    }
    guard values.allSatisfy({ $0.isFinite && $0 >= 0 }) else {
        throw MetricError.invalidDuration
    }
    guard !values.isEmpty else {
        return nil
    }

    let sorted = values.sorted()
    let n = sorted.count
    let rank = (n / 100) * q + ((n % 100) * q + 99) / 100
    return sorted[rank - 1]
}

func expectError(_ expected: MetricError,
                 _ operation: () throws -> Void) {
    do {
        try operation()
        preconditionFailure("Expected metric error")
    } catch let error as MetricError {
        precondition(error == expected)
    } catch {
        preconditionFailure("Unexpected error type")
    }
}

let samples = (1...20).map { Double($0 * 10) }
let p50 = try percentile(samples, q: 50)
let p95 = try percentile(samples, q: 95)
let p99 = try percentile(samples, q: 99)
precondition(p50 == 100 && p95 == 190 && p99 == 200)

let empty = try percentile([], q: 95)
let single = try percentile([7], q: 99)
let repeated = try percentile([4, 4, 4], q: 95)
let zero = try percentile([0], q: 50)
let reversed = try percentile(Array(samples.reversed()), q: 95)
precondition(empty == nil && single == 7)
precondition(repeated == 4 && zero == 0 && reversed == p95)

for q in [0, 101] {
    expectError(.invalidPercentile) {
        _ = try percentile(samples, q: q)
    }
}
for value in [-1.0, Double.nan, Double.infinity] {
    expectError(.invalidDuration) {
        _ = try percentile([value], q: 95)
    }
}

The rank calculation splits n into whole hundreds and a remainder, avoiding the rounding errors that Double multiplication can introduce near integer boundaries. It also avoids calculating n × q directly, which could overflow. Because q is no greater than 100, the intermediate result for the remainder is bounded, and the final rank never exceeds the sample count.

Run deterministic acceptance checks on the node

Run the following commands from the repository root. The script produces no output on success and exits with a nonzero status on failure. The pipeline can use that exit status directly, without parsing localized logs.

set -eu
xcrun swiftc percentile-check.swift -o /tmp/percentile-check
/tmp/percentile-check

The example uses a fixed temporary path, which is suitable for a single-job demonstration. Concurrent pipeline jobs should place the executable in their own working directories to avoid overwriting one another's files. Do not add -Ounchecked: that mode removes the precondition checks used here as the acceptance gate.

The standalone script validates the example implementation. Once it is integrated into the project, move the same test vectors into the test target and call the production statistics function directly, rather than maintaining a separate copy of the algorithm solely for testing. If you add permutation tests with a fixed random seed, retain the original inputs as well so failures can be replayed.

Cover window and sample boundaries

Before deploying, check at least these four cases:

• Shuffling the input does not change the result, and the statistics function does not modify the input array itself.
• Repeated durations still count as separate samples when determining rank; do not deduplicate first.
• Ranks are correct at 99, 100 and 101 samples, with no off-by-one errors at the hundred-sample boundary.
• Increasing the percentile never decreases the result, and every result belongs to the original sample set.

All percentiles should come from the same immutable snapshot. If the collector appends to the array while P50 and P99 are calculated separately, the two results may represent different windows. Copy the samples within the collector's synchronization boundary, then sort outside that boundary so sorting does not hold up collection.

Do not add timed-out requests to the array as zero-duration samples. If you measure only successful requests, label the metric explicitly as “successful-request latency” and report failure and timeout counts alongside it. Otherwise, latency may appear to improve as the success rate drops.

Make regression reports verifiable

For each report, save the algorithm identifier, unit, window start and end, successful sample count, failure count, timeout count and all three percentiles. Fixed fixtures can retain the complete vector. For real requests, keep only the data needed for troubleshooting, without request credentials or bodies.

Do not average P95 values from multiple windows and present the result as the P95 for the combined window. To combine windows, recalculate from the merged samples or use a statistical data structure that explicitly supports merging and documents its error bounds. The sorting implementation in this article is suitable for exact acceptance checks, not an indefinitely growing data stream.

Finally, keep “the statistics implementation is correct” and “performance meets the target” as two separate gates. The first uses exact comparisons against fixed vectors; the second compares results against application-specific thresholds using sufficient samples and consistent windows. That separation makes it possible to identify whether a failure comes from an algorithm regression, a collection anomaly or a genuine change in latency.

Frequently asked questions