A probability-of-default model, and an honest answer about whether the clever one wins.

The data, and what is wrong with it

A thousand loan applications from a German bank, twenty attributes each, labelled good or bad. It is the most-used credit dataset in teaching, which is the reason to use it and also the reason to be careful with it: everything below is a demonstration of method on a set that no longer describes any real lending market.

Information value is the first thing a credit analyst computes and the last thing a machine-learning course mentions. It measures how far an attribute separates the two outcomes, on its own, with no model in the way, and the industry has read it off the same thresholds for thirty years: under 0.02 useless, 0.1 weak, 0.3 medium, above 0.5 strong enough that the honest first reaction is to check for leakage rather than to celebrate.

The pipeline, in the order that matters

Model one: the scorecard

Logistic regression, fitted by iteratively reweighted least squares, which is Newton's method on the log-likelihood:

ln( p / (1 − p) ) = β₀ + β₁x₁ + … + βₖxₖ

Every coefficient is a log odds ratio against its reference level, every one has a standard error from the inverse of the Hessian, and the whole thing converts into a points table where the odds double every twenty points. That last part is not decoration: a scorecard is what lets a lender tell a declined applicant which two things cost them the most, which in most places that lend money is a legal requirement rather than a courtesy.

A ridge is on by default. Without it, a category where every applicant happened to default sends its coefficient to infinity, which is called complete separation and looks from the outside exactly like a very confident model.

Model two: gradient boosting

The algorithm XGBoost and LightGBM implement, written out rather than imported, because a page served as static files cannot run a C++ library. Each tree fits the errors of the ones before it, using the second-order approximation to the loss. A split is worth taking when

gain = ½ [ G²ₗ/(Hₗ+λ) + G²ᵣ/(Hᵣ+λ) − G²/(H+λ) ] − γ

is positive, and the value that lands in a leaf is −G/(H+λ), where G and H are the sums of the gradient and the curvature of the loss over the rows that reach it. Features are pre-binned into a histogram before any tree is grown, which is the trick that makes LightGBM fast and costs nothing to do here.

The validation curve is drawn on rows carved out of the training set, never on the test set. Choosing the number of trees by watching a test curve is how a test set stops being one, and it is the most common way a model's reported performance ends up better than its real performance.

So does the clever model win?

On one split, usually, by a little. The question is whether that little survives being measured properly, and there are two ways to find out. DeLong's test compares two AUCs measured on the same applicants, which is the right test because the two models agree about the easy cases and differ only on the hard ones. Cross-validation asks the same question by refitting everything, five times, on different folds.

Where the cut-off goes, and what it costs

Everything above is ranking. None of it is a decision. The decision is a number: above this probability of default, decline. The dataset ships its own cost matrix, and with it the answer follows from arithmetic rather than taste. Declining a good applicant costs 1. Lending to a bad one costs 5. So decline when

p × 5 > (1 − p) × 1, which is p > 1/6 = 0.167

and the whole apparatus of AUC has no opinion about it. A model that ranks better lets you sit at a lower cut-off for the same loss rate, which is where its value actually shows up.

A ranking is not a probability

AUC only knows about order. A model can rank every applicant perfectly and still say 4% when it means 20%, and for a PD model that is not a detail: the number goes into a provision, a capital calculation and a price. Boosted trees are usually the worst offenders, because the loss they minimise rewards confidence on the training rows.

The fix is a one-variable logistic regression of the outcome on the model's own log odds, fitted on held-out rows. It cannot reorder anybody, so AUC, Gini and KS are untouched by construction. What changes is whether the number means what it says.

Explaining one decision

A model that cannot say why it declined someone is a model that cannot be deployed in consumer lending. The scorecard answers by construction: the points are additive, so the reasons are the rows with the most negative points. The boosted model has to be taken apart, and for an additive ensemble of trees that can be done exactly rather than approximately: walk each tree, and give every split the change in value it caused. The parts add to the prediction, which the test file checks.

–

The attributes a lender is not allowed to use

This dataset ships sex, age and foreign-worker status, and a model handed all three will use all three. Every jurisdiction that regulates consumer credit restricts at least two of them, so the first thing a real project does is take them out. The toggle above does that, and the interesting part is what happens next: performance barely moves, and the disparity does not disappear, because fifty other columns rebuild what they can from what is left.

What this is not