How Accurate Are Our EOI Forecasts? We Tested — Here Are the Numbers

Before showing you a single prediction, we made our forecasting engine compete against nine alternatives — including machine-learning and deep-learning models — across thousands of blind tests on real SkillSelect data. This page shows exactly how it performed, where it’s reliable, and where it isn’t.

The result: pool-size forecasts

The Immitrend Forecast Engine beat every alternative at every horizon beyond one month — and the gap widens the further out you look. For the large occupations most users care about, production accuracy is better still: 3.2% at one month, 9.9% at six.

How we test a forecast honestly

Any model looks good if you grade it on data it has already seen. We use rolling-origin backtesting — the standard from forecasting science:

• Stand at a past month — say, December 2024 — and hide everything after it.
• Fit the model only on what was known up to that month.
• Predict 1 to 12 months ahead, then compare against what actually happened.
• Step forward and repeat — across every occupation, every visa subclass, every month of history.

That produces thousands of genuinely blind predictions at every horizon. We score them with MAPE — mean absolute percentage error. A MAPE of 5% means our forecasts were, on average, within 5% of the real number.

What we tested the engine against

The Immitrend Forecast Engine competed against nine alternatives on identical data, under an identical protocol, with strict train/test separation:

• Two naive baselines (the pool stays flat; the recent trend continues as a straight line)
• Linear regression at every window length
• Gradient-boosted decision trees trained across all occupations (the method that wins most tabular ML competitions)
• A pretrained time-series foundation model — a 200-million-parameter transformer
• Four neural network architectures trained on the pooled data, including residual networks and a transformer
• Plus ensembles, hybrids, and per-occupation model-selection strategies

Three blind-test case studies

Numbers in aggregate can feel abstract, so here are three of the most accurate results among the ten largest 189 occupations, shown in full: the engine was trained only on data up to December 2025, predicted the next six months, and then we let reality arrive. The solid line is what actually happened — including the six months the engine never saw. The dashed line is what it predicted.

These are strong examples, chosen to show the engine at its best — the averages in the table above include every occupation, including the hard cases (one accelerated 18% past every model we tested; another was whipsawed by an invitation round). That spread is exactly why every forecast ships with a range.

How far ahead can we see? The 12-month test

We extended the blind protocol to a full year. Total-pool forecasts hold up: at 12 months the engine averages 17.9% error on 189 pools — against 39.7% for “nothing changes” — and its error curve flattens after month six rather than compounding.

Trained only on data through June 2025, then twelve hidden months. Pool-size outlooks run to 12 months; queue forecasts stop at 6 — beyond that their errors grow past the point where we consider them decision-grade, and we would rather cap the product than publish noise.

The harder problem: the queue above your score

For 189, what matters isn’t the total pool — it’s how many EOIs sit at or above your points, because invitations go top-down. These queues are smaller, faster-moving, and driven by human behaviour (people re-sitting English tests to jump bands), so errors are honestly larger:

We label the ≥90 queue low confidence everywhere it appears — no model we tested (including the neural networks) can reliably predict a small queue that doubles when a cohort re-sits an English test. We’d rather tell you that than pretend otherwise.

Worth noting why the ≥90 queue is predictable here: Systems Analyst’s pool is unusually top-heavy (nearly half its ≥80 queue already sits at 90+), so its ≥90 queue is large and stable — unlike the small, behaviour-driven ≥90 queues that earn the low-confidence label elsewhere.

Why every forecast ships with a range

A single number pretends to a precision no forecast has. Every Immitrend forecast comes with a typical range — calibrated so that roughly 8 times in 10, the real outcome lands inside it. We validated this the hard way: our first two range algorithms failed the calibration test and were rejected before launch. The one that ships is the one that passed. Ranges widen the further ahead you look, because that’s the truth.

The part most forecast products skip

Backtests grade the past. From launch, every forecast we publish is written down first and graded later: when the next month’s official data arrives, we score every stored prediction against reality — including whether it beat the “nothing changes” baseline and whether the actual landed inside our range. That verification runs automatically every month, on data that did not exist when the forecast was made. If the engine ever loses its edge, this page will show it.

Where you’ll see it

Forecasts appear on the EOI Trends page (a 6-month outlook per visa subclass), and on every occupation page: the pool trend forecast (up to 12 months) and, for 189, the Projected Pool Distribution — including the projected queue at or above each points band, with ranges, round assumptions, and confidence labels.