Data splits, leakage, and imbalance: why scores look better than they are

This post covers the problems that come up most often with data when building AI models. I also wrote down what I missed on the practice questions. It covers how to split data, leakage, class imbalance, bias, distribution shift, and data that leaves through prompts. It does not cover evaluation metrics in detail, feature engineering, or library code.

1. Data sets the ceiling

A model can only learn the patterns that are in its data. If the data is wrong or skewed, the model is wrong and skewed in the same way.

Adding more data does not fix that, because it only adds more of the same errors. So quality comes before quantity: whether the labels are correct, and whether the sample represents what actually happens on the floor.

2. Split the data three ways

Once you start adjusting the model while watching the test score, the test set has effectively become training data. The score no longer tells you how the model does on data it has never seen.

Overfitting is when the score on training data is very high but drops sharply on new data. The model has memorized the answers in the training set. A large gap between training and validation scores is the sign to look for.

3. Data leakage

In the leakage practice set, I got the question wrong about feeding a "failure cause code" into an equipment-failure model. I picked "not leakage." The answer was leakage: the cause code for this failure is written down during maintenance, after the failure, so it does not exist at prediction time.

On the question about using the average chamber pressure over the 24 hours before a failure, I answered "leakage, because the failure comes 24 hours later." The verdict was right. More precisely, the problem was that the 24-hour window was cut to line up with the failure time, which nobody knows in advance on the floor.

Leakage is information that should not be available at evaluation time getting mixed into training. Scores come out better than they really are, and performance drops sharply once the model is deployed. One question is enough to check:

At the moment the prediction is actually made, is this information in hand? If not, it is leakage.

Four cases come up often with semiconductor fab data.

The same batch lands in both train and test. A lot, the batch of wafers that goes through processing together, shares the traces of the same tools, recipe, and time window. If chips are shuffled and split without regard to lot, the model can score well on test just by memorizing what makes each lot different. On the floor, new lots arrive all the time, so that score does not hold. Split by lot or by wafer instead.

Time order is ignored. A model trained on March data and tested on January defects has used the future. With time series, train on the past and test on the future.

A feature is derived from the answer. Values recorded after a defect is confirmed, such as "rework flag" or "defect disposition code," let the model read the answer to predict the answer. The failure cause code above is this case.

The same feature name can go either way, depending on when the value is recorded.

So instead of judging a feature by its name, check when the value is recorded. If it is unclear, it is a good idea to ask whoever provided the data: "When does this value get recorded?"

Preprocessing is done on the full dataset. If you compute the mean and standard deviation for normalization, or fill missing values with a mean, using all the data including test, information from test flows into training. Split first, compute the statistics on train only, and apply them unchanged to test.

Fix the prediction time first

The 24-hour pressure question belongs here. When building the training data, the failure time is already known, so the window was cut backward 24 hours from it. On the floor, nobody knows when the failure will happen, so the only window you can cut is 24 hours back from now.

How the training data was built
  |------- 24 hours -------|
                           failure time   <- you need the answer to cut here

How it is used on the floor
  |------- 24 hours -------|
                           now            <- you don't know when it will fail

Built this way, failure samples are always the window right before a failure, while normal samples are windows cut at arbitrary times. The model learns patterns that only appear right before a failure. And because the window ends at the moment of failure, even a correct prediction leaves no time for maintenance.

The fix is to choose a prediction time t first and split features and label around it.

  |-- features: 24 hours before t --|-- label: does it fail within 48 hours after t? --|
                                    t

• Compute features only from data accumulated up to t.
• Set the label by whether a failure happens within a fixed period after t.
• Pick t by the same rule for failure and normal samples alike.

Training and real use then run under the same conditions, and you get a 48-hour window for maintenance.

4. Class imbalance

Semiconductor defects are rare. Take 100,000 chips with a 0.1% defect rate (100 chips) and compare two models.

Recall is the share of real defects that were caught. Precision is the share of alarms that were real defects. By accuracy alone, model A, which catches nothing, beats model B. That is why accuracy alone cannot be used to judge a model on imbalanced data.

Model B's precision of 3.85% is worth a look too. Of 2,078 alarms, 80 are real defects. When defects are rare, even a small false-alarm rate produces far more false alarms than real defects. Choosing the right metric is the topic of the next post.

On the data side, the usual responses are:

• Sample the rare class more, or the common class less (oversampling, undersampling).
• Penalize mistakes on the rare class more heavily (class weights).
• Generate synthetic defect samples. Synthetic data can differ from the real distribution, so evaluate on real data.

5. Bias

Bias comes from how the data is collected.

Collecting more data the same way makes the same bias bigger. Reducing it means changing how the data is collected.

6. Distribution shift

When a process recipe changes, a tool is replaced, or the raw material changes, the distribution of the input data moves away from what the model was trained on. This is called distribution shift, or drift. A model that was accurate yesterday starts giving wrong answers without raising any error.

So after deployment, keep monitoring the input distribution and the model's performance, and retrain when they change significantly.

LLMs have the same problem. Facts after the training cutoff, and internal company documents, are not in the model. RAG, covered in the previous post, is how that gap gets filled.

7. What goes into a prompt is data too

Training data is not the only concern. Whatever you paste into a prompt is sent to and stored on an external server, and once it is in, it cannot be taken back.

In March 2023, after Samsung Electronics' semiconductor (DS) division allowed employees to use ChatGPT, Korean media reported two leaks of equipment information and one leak of meeting content. The material reportedly included program content related to equipment measurement and yield and defects (Economist Korea). The company limited input size, and in May restricted the use of generative AI on company PCs (Financial News, SBS Biz). Reports differ on the details: some describe the leaked material as "source code," others as an "equipment measurement program."

Process conditions, yield, and customer information are confidential. Which AI the data goes into, and under what contract terms, is part of handling data.

Leakage practice

Decide whether each case is leakage. The test is the one question from section 3.

1. A wafer yield model uses "final test bin result" as an input feature.

2. The model is trained on equipment sensor logs from January to June and tested on July logs.

3. Chip-level data is shuffled without regard to lot and split 80:20.

4. Measurements taken right after etch are used to predict whether the chip will ultimately be defective.

5. The data is normalized with the mean and standard deviation of the full dataset, then split into train and test.

6. Settings were changed twenty times while watching the test score, and the best score was reported as final performance.

7. An equipment-failure model uses the "failure cause code" from the maintenance record of the very failure it is trying to predict.

8. A failure model uses "average chamber pressure over the 24 hours before failure," with the window cut backward from the failure time.

To sum up

• Data quality sets the model's ceiling, and quantity cannot make up for quality.
• Look at the test set once, at the end.
• Check for leakage by asking "is this information in hand at the moment of prediction?" Check when a feature is recorded rather than going by its name, split by lot and by time, and compute preprocessing statistics on train only.
• On imbalanced data, accuracy alone cannot tell you whether a model is good.
• Bias goes down only when the way data is collected changes.
• The distribution keeps moving after deployment, so keep watching it.
• What you put into a prompt is also data sent to an outside server.