FMAX Tactic Lab
Test method
Every score on the leaderboard comes from the same official test. This page sets out the conditions it runs under, and what it can’t tell you.
- 1Same conditions for everyone
Whoever runs it, in whatever save, the test uses the same clubs, the same players and the same opponents.
- 2Identical players
Every player on both sides is the same, so the tactic is what’s left in the result.
- 3It has limits
Tactics built around a particular kind of player don’t show their edge here.
The same test for everyone
FMAX has your own game play the matches. So that every save gives the same conditions, it sets the values only while the test runs and puts them back when it ends. The players and clubs in your save are left as they were.
- Clubs
- Two clubs fixed by the test sheet: the tested side and the opponent. You can’t pick them. In a save where you manage one of them, a stand-in club takes its place, and its players are set to the same squad described below.
- Match engine
- Only results from the same match engine version go on the same leaderboard. When a patch changes the match engine, a new leaderboard starts.
- Tactical familiarity
- Both sides start with full familiarity in their tactic.
- Set pieces
- Both sides use the same baseline set-piece routines.
- Live tweaks
- FMAX’s Live tweaks that can affect a match are switched off for the test.
- Line-ups and subs
- FMAX doesn’t pick them. The game’s AI manager decides who starts and when to make substitutions.
The squad: identical players
Both squads are made of the same player. A matchday squad is 20 per side: 2 goalkeepers and 18 outfield players. Every other player at the club becomes the same outfield player too, so whoever the game picks, it’s the same man.
Outfield player (same on both sides)
- Positions
- Natural in every outfield position. Whatever the formation, nobody plays out of position.
- Attributes
- Every visible outfield attribute is 13, except Determination at 12. Goalkeeping attributes are 1. Hidden attributes are the same for every player too.
- Feet
- Both feet Very Strong.
- Build and age
- 182 cm and 26 years old, with the same body type for every player.
- CA and PA
- Both 150.
- Personality
- Every player has the same personality.
- Player traits
- None.
Goalkeepers (two per side, all the same)
A right-footed keeper with strong goalkeeping attributes and weak outfield ones. Build, age, ability and personality are the same as the outfield player’s.
Before kick-off (every player)
Every player starts with full condition and match sharpness, and no fatigue. Morale is the same for everyone too.
Opponents
The opponents use the game’s own stock tactics, one from each of the ten styles the AI managers play. Each is met home and away, the same number of times. Their players are the same as yours, so every match is between two sides of equal strength.
The ten opponent tactics on the current test sheet
- Catenaccio
- Control Possession
- Direct Counter-Attack
- Fluid Counter-Attack
- Gegenpress
- Park the Bus
- Route One
- Tiki-Taka
- Vertical Tiki-Taka
- Wing Play
From one match to the next
- Every match starts fresh
- Each fixture is set up once, and every match is played on a fresh copy of it. So every match starts from the same state, and fatigue, injuries and cards never carry over to the next one.
- Cards and injuries
- On. Red cards and injuries happen as in any match and count in that match’s result.
- Random seed
- The server hands out a new random seed for every test, after the tactic is locked in. Each test plays different matches, and the tester can’t choose the seed.
Number of matches
Pick one of three when you start. The more matches, the less luck is left in the score, and the narrower the margin of error (±) shown next to it.
Repeat runs of the same tactic are added together. There is a limit, though, to how far one person’s runs can narrow the margin, and when different people’s results disagree it can get wider instead. The score is the points the tactic would be expected to take over a 38-game season.
How results are pooled
- One tactic
- Results measured with the same tactic file are pooled into one tactic, whoever measured them. The more people measure it, the less any one game’s small quirks matter.
- The server checks
- Every time a result comes in, the server checks the test conditions. A result that differs from the official conditions is recorded with a mark saying how.
- All measurements
- The view the leaderboard opens on. As long as the match engine is the same, it includes results measured with a stand-in club, on game files that aren’t the original, or with mods on.
- Official conditions only
- The view without any marked result: only the sheet’s own two clubs, unmodified game files, Compatibility mode off (the FMAX setting that lets it run on game files another mod has patched), and no plugins or match-related file mods switched on. Game files that aren’t the original means files with FMAX DoF on, files another mod has patched, and any other changed file.
How many people and matches it takes to be ranked, and how results from different people are combined, is in the test guide.
How the score is worked out
Where the points and the margin of error on the leaderboard come from, step by step. The statistical methods are named as we go, and the original source of each is listed at the end of this page.
- Wins, draws and losses only
- The score uses only whether each match was won, drawn or lost: 3 points for a win, 1 for a draw, 0 for a defeat. Goals don’t go into the score. They’re used only to order tactics that are level on points.
- All 20 fixtures count the same
- There are 20 fixtures: each of the 10 opponents, home and away. Points per match are averaged within each fixture, the 20 averages are averaged again, each counting the same, and the result is scaled to a 38-game season. A fixture with a few more or fewer matches doesn’t tilt the score towards it. Statisticians call this a stratified mean.
- The margin of error is a 95% confidence interval
- The ± next to the points is a 95% confidence interval. It is worked out so that, if the same kind of test were run 100 times and the range worked out afresh each time, about 95 of those ranges would hold the true points. It is an approximation: when few people have measured a tactic, or saves differ a lot, the real share can be lower (see What this method can’t see below). It isn’t a forecast that next season’s points will land inside it. Its width comes from Student’s t distribution, and the Welch–Satterthwaite method allows for results spreading more in some fixtures than in others. The averages and spreads are built up one result at a time with Welford’s method.
- No one person can swing it
- When several people measure a tactic, each person’s value is worked out on its own and then combined. Once three or more people have measured it, each person’s share is capped, so however much one person measures, they can’t decide the score alone. Values far from the middle also count only up to a set distance, so one outlier can’t move the score much. Statisticians call this kind of approach influence-limited, or robust, estimation.
- Differences between people go into the range
- With three or more people, the value is worked out again leaving each person out in turn, and how much that moves it goes into the range (the jackknife). That’s how differences between people’s saves get into the range. With two people, the range is always wide enough to cover both their values.
- Estimated ranks come from simulation
- Rank ranges such as ‘est. 2nd–5th’ come from a Monte Carlo simulation. Each tactic’s score is drawn at random many times in line with its points and margin of error, the tactics are ranked each time, and the places holding the middle 95% of those ranks are shown. If the tactic’s rank falls outside them, the range is widened to include it.
- Our own calculator
- No outside statistics package is used. The methods above are implemented in FMAX’s own code, and the app and the server run the same code. The server doesn’t use the score the app sends: it works the score out again from scratch from the wins, draws and losses it receives for each of the 20 fixtures.
How we checked it
What we did to check that the calculation and the test work as they should. Every number is taken straight from the records of those checks. ‘Simulation’ means match results generated by computer; ‘real game’ means matches played in the game itself.
- How often the range holds the true value (simulation)
- We generated 200- and 1,000-match tests under five different ways of producing match results, 2,000 times for each setting. The 95% range held the true points between 94.55% and 95.65% of the time in every setting.
- Pooling several people’s values (simulation)
- We generated a number of set-ups measured by two or more people, 3,000 times each. With no difference between saves, the range held the true value more than 95% of the time in every set-up. Even with three or more people, a spread of about 10 season points between saves brought that down to between 74% and 92%, depending on the set-up. With three people it allows generously for differences between them, so the range was about twice as wide as the matches alone would make it.
- How often an honest value is wrongly marked (simulation)
- When everyone measured honestly, the uploader’s value was wrongly marked ‘Not reproduced’ 0.00% of the time, and at most 0.07% of the time with a small difference between saves added in.
- Same conditions, same matches (real game)
- With the same save, the same pair of clubs and the same random seed, 50 out of 50 matches were identical down to every event. They were still identical after FMAX was restarted and the matches set up again. The game stayed open throughout.
- Same matches after a patch (real game)
- After a game patch, an official test played before the patch was replayed with the same save and the same seed, and all 200 matches came out the same. That was checked with one pair of clubs, one tactic and one save.
- The app’s value and the server’s (real game)
- For 200 matches played in the real game, the app and the server each worked out the score on their own, and the points and the margin of error matched to two decimal places.
- Differences between pairs of clubs (real game)
- Measuring the same three tactics over 400 matches with each of three pairs of clubs, two of the three tactics’ points differed by 14 to 16 between pairs and the third’s by about 8, and the order of tactics sometimes flipped. That’s why everyone tests with the one pair of clubs fixed by the test sheet.
- Differences between saves (real game)
- Measuring the same pair of clubs in two saves from different careers, 400 matches per tactic, the three tactics’ points differed by at most 4.7, all within the margin of error. That many matches can’t pick out a difference smaller than about 7 points, though, and only two saves were compared.
What this method can’t see
Making every player the same has a cost, and there are things the calculation can’t tell you. Keep them in mind when you read the leaderboard.
- Tactics that rely on player profiles
- With every player the same, the gap between tactics that get the most out of a particular kind of player, a quick winger or a tall striker, doesn’t show. The flip side: a tactic that falls apart without that kind of player doesn’t show its weakness here either.
- Your save may differ
- In a real save, players differ in ability and positions, opponents differ in strength, and fatigue and injuries build up over a season. A tactic that ranks high here isn’t guaranteed to do as well with your squad.
- Only equal-strength opponents
- The opponents use the same players, so results against much stronger or weaker sides aren’t measured.
- Set-piece routines
- Both sides use the same baseline routines, so differences between set-piece routines don’t enter the score.
- One pair of clubs
- The test is played with one fixed pair of clubs. Even with the same players, a different pair can give different results.
- Scores from one or two people
- The margin of error of a tactic measured by only one or two people makes no separate allowance for differences between saves. In simulation, with two people measuring 400 matches each, a spread of about 5 season points between saves brought the share of ranges holding the true value down to about 70%. The ‘One tester’ mark and the filter on how many people measured a tactic let you pick these out.
- Estimated ranks are per tactic
- A range such as ‘est. 2nd–5th’ is worked out for each tactic on its own. It doesn’t mean every tactic on the leaderboard sits inside its range at the same time.
- Made-up values cannot be ruled out entirely
- The server checks the test conditions and works the score out again, but it can’t entirely stop a modified app from sending false results. That’s why measurements from several people are pooled, and the number of people who measured each tactic is shown.
What may change
We’re testing ways to close these gaps: a standard squad of specialist players with different strengths for each position, and opponents of different strength. Depending on what the tests show, the current method may stay as it is. If it does change, it will change in one go at the next match engine patch, when the leaderboard starts fresh anyway, and the current leaderboard will be kept as a record.
Methods and their original sources
The statistical methods the score uses, and the original source each one first appeared in. The calculator is FMAX’s own implementation of these methods.
- Student’s t distribution
- Sets the width of the margin of error.
- Student (1908). The probable error of a mean. Biometrika 6(1). doi:10.1093/biomet/6.1.1
- Welch–Satterthwaite method
- Adjusts that width when results spread more in some fixtures than in others.
- Welch, B. L. (1947). The generalization of “Student’s” problem when several different population variances are involved. Biometrika 34(1–2), 28–35. doi:10.1093/biomet/34.1-2.28
- Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin 2(6). doi:10.2307/3002019
- Welford’s method
- Builds up averages and spreads accurately, one result at a time.
- Welford, B. P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics 4(3), 419–420. doi:10.1080/00401706.1962.10490022
- Stratified mean
- Averages the 20 fixtures, each counting the same.
- Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley. ISBN 978-0-471-16240-7
- Influence-limited (robust) estimation
- Caps how much any one outlying value can count. We don’t use this paper’s estimator as it stands; we follow its idea of limiting the influence of a single value.
- Huber, P. J. (1964). Robust estimation of a location parameter. Annals of Mathematical Statistics 35(1), 73–101. doi:10.1214/aoms/1177703732
- Jackknife
- Measures the spread between people by working the value out again with each person left out in turn.
- Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika 43(3–4), 353–360. doi:10.1093/biomet/43.3-4.353
- Efron, B. & Stein, C. (1981). The jackknife estimate of variance. Annals of Statistics 9(3), 586–596. doi:10.1214/aos/1176345462
- Monte Carlo simulation
- Produces the estimated rank ranges.
- Metropolis, N. & Ulam, S. (1949). The Monte Carlo method. Journal of the American Statistical Association 44(247), 335–341. doi:10.1080/01621459.1949.10483310
Further reading
A paper on the uncertainty in league tables. The calculation above isn’t taken from it.
- Goldstein, H. & Spiegelhalter, D. J. (1996). League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society, Series A 159(3). doi:10.2307/2983325