Evaluation · NASA CMAPSS FD001
Model performance
How well the LSTM + Attention model does on engines it never saw during training.
RMSE
On average the guess is off by about this many cycles. Big misses count extra. Smaller is better.
MAE
The plain average of how far off each guess is, in cycles. One or two odd cases bother it less than RMSE.
NASA Score
A score made for engines. Guessing too high (saying an engine is fine when it is not) is punished harder. Lower is better.
Within ±10
Out of every 100 engines, how many we got within 10 cycles of the real answer.
The data
Where the numbers come from
We use NASA's C-MAPSS FD001 set. These are simulated jet engines that were run until they broke, so the real life left is known for every one. We learn from one group of engines and test on a completely separate group.
How we cleaned it up
- ›We show the model the last 30 flights at a time, then slide the window forward.
- ›7 of the 24 readings never change, so we drop them and keep the 17 that do.
- ›Every reading is squeezed into a 0 to 1 range so no single sensor takes over.
- ›We cap "life left" at 125 flights. Above that, an engine just counts as healthy.
What the model sees
It looks at the last 30 flights
For each guess the model reads the last 30 flights of the engine, with 17 sensor readings per flight (from the fan, compressor, combustor and turbine). It uses LSTM layers, the type of network built for data that comes in order over time, because wear builds up flight after flight. An attention step then helps it focus on the flights and sensors that matter most.
Attention
Which flights it pays attention to
The attention step gives each of the 30 flights a weight, a score for how much it matters to the final guess. Taller, brighter bars mean the model leaned on that flight more. It usually cares most about the recent flights, where wear shows up. (This is an example of the shape; wiring the live weights from the model is a quick next step.)
Architectures
Four models, ranked
RMSE by model · lower is better
MAE by model · lower is better
GRU has the lowest error (13.57). We still run LSTM + Attention because it can show which flights it looked at, which matters more to us than a tiny bit of accuracy.
What each model is
The simplest model that reads data in order. It has a short memory, so over 30 flights it forgets the early ones. That is why it scores worst here.
A smarter version with small "gates" that choose what to keep and what to drop. It is light and quick to train, and it got the best score.
Like a GRU but with a separate memory line that holds information for longer. Good at remembering wear that started many flights ago.
An LSTM with an extra "attention" step that points to the flights that matter most. This makes it easier to explain, so it is the one we run live.
Convergence
Training vs validation loss
"Loss" is how wrong the model is. Lower is better. The training line is the error on the data it studies. The validation line is the error on data it has not seen. We want both lines to go down and stay close. If the validation line starts going up while training keeps dropping, the model is just memorising the answers instead of learning. That is called overfitting, and we hold it back with the tricks taught in class (dropout, early stopping, and saving the best version).
Calibration
Predicted vs actual RUL
Every dot is one test engine. Across the bottom is the real answer, up the side is the model's guess. A dot on the dashed line is a perfect guess. Above the line means the model guessed too high (it thinks the engine has more life left than it does, which is the risky kind of mistake). Below means it guessed too low (it plays it safe). The closer the dots hug the line, the better.
100 test engines, scored by the live model. 60 out of 100 land within 10 cycles of the real answer, 92 within 25. Dot colour shows the size of the miss (green is close, red is far).
Right bucket?
Did it sort each engine correctly?
In real life you mostly care about the action, not the exact number. So we sort every engine into three buckets by life left: OK (80+), Warning (30 to 80) and Critical (under 30). Each row is where an engine really belongs; each column is where the model put it. The boxes down the diagonal are correct. The box in the bottom-left is the dangerous mistake: a truly critical engine the model called OK.
Cost planning · example
When is the best time to fix the engine?
Cost at each choice
best ≈ 30 cyc · $77,025Fix it too early and you throw away good flights. Leave it too late and you risk a $500,000 surprise breakdown. The sweet spot saves about $422,975 each time.
The failure chance comes from how much the model usually misses by (about 15 cycles).