Every number in the note, and where it came from

The source is Algoseek US Equity Trades and Quotes at nanosecond resolution, drawn from the consolidated SIP feed (CTS, CQS, UTDF, UQDF) collected at Equinix NY2 and NY4. Each symbol-day is one gzipped CSV with the schema Date, Timestamp, EventType, Ticker, Price, Quantity, Exchange, Conditions, where Timestamp is HH:MM:SS.nnnnnnnnn and Conditions is a 32-bit sale-condition bitmask in hexadecimal.

Trading days are enumerated from the per-day manifest at .index/YYYYMMDD.csv.gz rather than by listing the bucket, which is denied on this product. One small GET returns bucket, prefix, file, compressed_bytes, uncompressed_bytes for the whole day, where column 3 is the compressed size and column 4 the uncompressed one. Reading that ordering backwards is what produced an early 300 GB/day size estimate for a product that is 27 GB/day.

Getting either of these backwards silently corrupts every volume and order-flow feature, and neither is obvious from the vendor specification.

The tape is the union of TRADE and TRADE NB, and the two are disjoint. The specification describes TRADE NB as a trade at or within the NBBO and TRADE as everything else, which leaves open whether the second re-emits the first, so the question was settled by measurement.

Exact-tuple overlap on timestamp, price, quantity and venue comes out at 0. A near-match test asking whether each TRADE NB print has a TRADE print of the same price, quantity and venue within 1 ms returns a hit rate of 0.0000, against 0.0000 for a control whose TRADE timestamps are shifted forward by 5 seconds.

Structurally the two are disjoint too: 95% to 98% of TRADE prints carry the odd-lot flag at bit 31 against 0.00% of TRADE NB prints, and Algoseek's own trades-only product contains exactly this union, 83,244 rows against the union's 83,243 for JPM on 2019-03-15. Using one half halves the tape, and treating them as duplicates halves it too.

Official open and close restatements must be dropped. Bit 24 (tOfficialClose) and bit 26 (tOfficialOpen) flag a market centre restating the auction price, not a new execution. For F on 2024-03-15 the NYSE closing print of 36,626,227 shares appears five times, once as the auction execution and then again at 16:01, 16:10, 18:30 and 19:00, so summing the union naively gives F 238 million shares that day where dropping bits 24 and 26 gives 88 million, which is what F actually traded.

A third correction is smaller but worth stating: the original brief asked to exclude FINRA, CRF and other off-exchange venues, yet CRF does not exist as a label in this feed. All 19 SIP market-centre identifiers were enumerated across the three years, and Algoseek consolidates every off-exchange report under FINRA (SIP code D), so excluding FINRA is the operative form of that instruction.

Below is the full filter, applied identically to both datasets so the OHLC bars underlying arms A and B are bit-identical.

The national best bid and offer come from the QUOTE BID NB and QUOTE ASK NB rows, forward-filled per side, with rows sharing a timestamp collapsed to their last state first. That collapse matters because one side can print before the other, so forward-filling the raw sequence manufactures transient locked or crossed books that never existed at a rate near 5% of updates.

Time-weighted statistics are computed exactly by injecting a synthetic state row at every minute boundary, so each weight interval falls wholly inside one bar.

Arm B's 53 features cover bar geometry (range, body, shadows, close location value and gap), the Parkinson, Garman-Klass and Rogers-Satchell volatility estimators, moving-average deviations and crossovers at 5, 15, 60 and 390 bars, momentum and rate of change out to 1,950 bars, rolling volatility and skewness, RSI at 14 and 60, MACD with signal and histogram, ATR, Bollinger %B and bandwidth, Stochastic %K and %D, Williams %R, CCI, Donchian position, TRIX, Ulcer index, new-extreme frequency, a short-to-long volatility ratio and lagged returns.

Volume is deliberately absent because it is not an OHLC quantity, which excludes on-balance volume, money flow and VWAP-based indicators from arm B by construction.

Arm A adds 151 tape features in three families. Trade-side columns carry trade count, volume, dollar volume, average, median and maximum trade size, VWAP deviation from close, buy and sell volume under a Lee-Ready classification against the prevailing NBBO, signed volume and signed dollar flow, count imbalance, the fractions executing at or through each side of the quote and strictly inside it, effective spread both simple and volume-weighted, realised variance and bipower variation from trade prices, Amihud illiquidity, Roll's implied spread, Kyle's lambda estimated within the minute, inter-trade duration mean and dispersion, odd-lot and block share, venue count, venue Herfindahl index, top-venue share and a one-second markout.

Quote-side columns carry the time-weighted quoted spread in bps with its minimum, maximum and dispersion, time-weighted depth on each side, size imbalance, microprice and its deviation from the midpoint, midpoint open, high, low and close, realised variance of the midpoint, the fraction of time locked or crossed, NBBO update counts by side, quote-to-trade ratio and Cont order-flow imbalance accumulated over consecutive NBBO states. Twelve of those signals additionally enter at lags 1, 2, 3 and 5 and as trailing means over 5, 15 and 60 bars, giving arm A the same depth of history arm B's rolling indicators already have.

Pooling an $11 stock with a $520 ETF whose message rates differ by two orders of magnitude means level-valued columns cannot enter raw. Price levels become basis-point deviations from the bar close and count and size columns become trailing z-scores over a five-day window, with flows normalised by trailing turnover, where every rolling window ending at t uses only information available at t.

Three gradient-boosting implementations, LightGBM, XGBoost and CatBoost, all at depth 4, 15 leaves, a minimum of 500 samples per leaf, learning rate 0.02, 300 trees, column subsampling 0.5 and an L2 penalty of 10, with SEED = 20260731 everywhere.

One-minute equity returns carry a signal-to-noise ratio of order 1e-3, so ordinary tabular capacity memorises the training window: a smoke test with 63-leaf trees returned an out-of-sample R-squared of -0.60 where the regularised configuration returned -0.046 with non-degenerate predictions. Those values were fixed before the full run from the known properties of the target, never from any arm's performance, then applied identically to every arm, horizon, block and fold, so there is no hidden search to deflate.

Each symbol-block is divided into five contiguous segments giving four expanding folds, with training always preceding testing in time. Overlapping targets are the standard way to leak in a horizon study, since at h = 390 a training row near the boundary has a target extending deep into the test window, so every fold discards the final h bars of its training range and a fold is skipped when purging leaves fewer than 2,000 training rows, which removes most folds at one month.

Arm B's longest indicator needs 1,950 bars of history where arm A's needs 60, so without a shared warm-up the two panels would differ in how many rows carry missing values, which is a difference in the data and not in the information. The first 1,950 bars of every symbol-block are therefore dropped from all arms.

That loop produced 459 fits across 3 blocks x 5 horizons x up to 4 folds x 3 arms x 3 models.

Out-of-sample R-squared is measured against a zero forecast rather than the test-period mean, since the latter is a look-ahead constant. Alongside it sit the Spearman rank IC, directional accuracy and per-ticker breakdowns, with the headline being the paired difference A minus B inside each matched block, horizon, fold and model cell, which removes the period and fold effects that dominate the levels.

Overlapping h-bar targets supply roughly n/h independent observations, and five correlated tickers do not supply five independent series. Measured mean pairwise one-hour return correlation is 0.32, with AAPL against SPY at 0.85, which by the standard variance argument gives 2.19 effective series.

Confidence intervals come from a bootstrap that resamples contiguous time slices carrying all five tickers together. An interval built by resampling rows instead came out 1.7 times too narrow, at a 95% width of 0.184 where the joint time-slice bootstrap gives 0.310.

Fixed thresholds were replaced entirely by an empirical null, which needs no assumption about effective sample size. The same fold is re-run against a target rotated circularly inside each symbol-block, destroying the feature-target correspondence while preserving every marginal distribution, the autocorrelation of the target and the cross-ticker correlation, after which a real result is judged by its percentile against that spread.

That ran 480 refits, 20 rotations for each block, horizon and arm. Rotation is the only structure-preserving relabelling available when observations cannot be exchanged freely, which is what makes the resulting distribution the right null for this design.

Statistical detectability is not tradeability, so at one minute the sign of the forecast is traded and reported gross, then net of the effective spread the tape actually printed in that minute plus a fee allowance, with the cost measured from the data and not assumed.

Minutes with no lit prints, about 0.01% of bars, inherit the previous close and are marked with a zero trade count, so their effective spread is null and gets filled with the sample median in this analysis only.

Three gates had to pass before any comparison was reported, and the pipeline exits non-zero on failure.

Two bugs in the gates themselves are recorded here because the difference between a gate that passed and a gate that could not fail is the whole value of a gate. The first was written as if python3 leak_tests.py 2>&1 | tee -a "$LOG"; then, which tests the exit status of tee and not of the script, and was fixed with set -o pipefail and ${PIPESTATUS[0]}.

The second was the shift-null, which used shift(-k) and therefore nulled the last k rows of each group, and since the test evaluates the final fold its test set was empty and the function silently returned nothing. Fixing it took a circular rotation plus fill_nan(None), because polars treats NaN and null as distinct and drop_nulls sails past rolled-in NaNs.

That working null then exposed an inference error, not a pipeline error. Its first run scored R-squared = +0.042 on a target that cannot contain signal, because the effective sample size was being computed as rows/h, treating five tickers as five independent replicates, which overstated every n_eff by about 2.4x and produced bootstrap intervals 1.7x too narrow, both corrected as described above.

Every model is seeded and the bootstrap and rotation draws use numpy.random.default_rng(SEED) with SEED = 20260731, so a rerun on the same raw data reproduces the tables exactly. Total run cost was $0, since the S3 reads are Algoseek-billed, and total wall clock was about 90 minutes.

The charts in the note are drawn from the published result tables, which are the same tables the numbers in the text come from.

This page is enough to check every number in the note by hand, or to rebuild an equivalent study from the same licensed source. The production scripts (the sizer, the puller, the tick-to-bar feature build, the panel builder, the walk-forward runner, the null, the analysis and the figures) are not attached, so email daniel@daru.finance and I'll send them over.