Credit Card Fraud Detection
https://www.kaggle.com/mlg-ulb/creditcardfraud
This Python script builds several classifiers for the highly imbalanced “creditcard” dataset and compares them on held-out data. It covers preprocessing, logistic regression (with different imbalance strategies), random forests, and two PyTorch models (a logistic regression and a small MLP) trained with a class-imbalance-aware loss.
-
Setup and data loading
– Resets variables (IPython “%reset”)
– Imports NumPy, pandas, scikit-learn utilities, and later PyTorch
– Loads the CSV file creditcard.csv into a pandas DataFrame
– Splits columns into features X (all columns except “Class”) and target y = “Class”
– Prints class counts to show the severe imbalance (very few frauds)
-
Feature scaling
– Fits a StandardScaler on X (mean 0, variance 1) and transforms X to float32
Note: in a fully rigorous pipeline, the scaler should be fit on the training set only, then applied to the test set
-
Random under-sampling to create a balanced subset
– Counts the number of fraud cases and gets their indices
– Randomly selects the same number of non-fraud indices
– Concatenates both to form a balanced subset (under_sample_data)
– Extracts X_undersample and y_undersample from that subset
-
Train/test splits
– Splits the full data (X, y) into train and test (70/30)
– Splits the balanced subset similarly
– Prints class counts in each split for both the original and the undersampled data
-
Logistic regression (scikit-learn)
(a) Baseline on the original imbalanced data:
– Trains LogisticRegression without regularization
– Predicts on X_test and prints a confusion table (y_test vs predictions)
(b) Class-weighted logistic regression:
– Trains LogisticRegression with class_weight=“balanced” (the minority class is up-weighted automatically)
– Predicts on X_test and prints the confusion table
(c) Logistic regression on the undersampled (balanced) subset:
– Trains LogisticRegression on X_train_undersample, y_train_undersample, with more iterations and a Newton solver.
– Predicts on X_test_undersample and prints the confusion table
-
Random forest (scikit-learn)
(a) Trains a RandomForestClassifier on the original imbalanced training set (100 trees, max_features ≈ sqrt(30), OOB enabled, parallel jobs)
– Predicts on X_test and prints the confusion table
(b) Trains the same RandomForestClassifier on the undersampled training set
– Predicts on X_test_undersample and prints the confusion table
-
PyTorch setup
– Selects the Apple Silicon Metal backend device (“mps”) if available
– Converts train/test arrays to PyTorch tensors; moves the test tensors to the device
– Builds a TensorDataset and DataLoader for the training data (batch size 1024, shuffled)
– Computes pos_weight = (#negatives / #positives) from y_train. This ratio is used by the loss to counter the imbalance
-
Generic PyTorch training loop (train_model)
– Moves the model to device and defines Adam optimizer
– Uses BCEWithLogitsLoss(pos_weight=pos_weight): this expects raw logits and internally applies a stable sigmoid + binary cross-entropy; pos_weight increases the penalty for misclassifying the positive class
– For a given number of epochs:
• Iterates over mini-batches: forward pass → compute loss → backpropagate → optimizer step
• Tracks mean training loss per epoch
• Periodically evaluates test loss on the full test set (still with BCEWithLogitsLoss)
– Returns the trained model
-
PyTorch logistic regression
– Defines a single Linear layer with output size 1 (equivalent to logistic regression without explicit sigmoid on the last layer, because the loss uses logits)
– Trains it with train_model (50 epochs, lr=1e-3)
– Switches to eval mode, computes logits on X_test_t, applies sigmoid, thresholds at 0.5 to obtain class predictions, and prints the confusion table against y_test
-
PyTorch MLP
– Defines a small feed-forward network: Linear → GELU → Linear → GELU → Linear → GELU → Linear(→1)
– Trains it with weight decay for mild regularization (80 epochs, lr=1e-3, weight_decay=1e-5)
– Evaluates as above: logits → sigmoid → threshold 0.5 → confusion table
What the script demonstrates
– Several standard ways to handle class imbalance: raw training on imbalanced data, class weighting, and random under-sampling
– Comparison of linear (logistic regression) and non-linear (random forest, MLP) models
– How to train PyTorch classifiers for imbalanced binary classification using BCEWithLogitsLoss with a positive-class weight, and how to evaluate them via a confusion table on a held-out test set
%reset
import numpy as np
import pandas as pd
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
data = pd.read_csv("/Users/marin/Documents/TEACHING/2526/M2-GLM-HAX912X/TP/creditcard.csv")
data.head()
X = data.loc[:, data.columns != 'Class']
y = data['Class']
pd.Series(y).value_counts()
scaler = preprocessing.StandardScaler().fit(X)
X = scaler.transform(X).astype(np.float32)
number_records_fraud = len(data[y == 1])
fraud_indices = data[y == 1].index
normal_indices = data[y == 0].index
random_normal_indices = np.random.choice(normal_indices, number_records_fraud, replace = False)
under_sample_indices = np.concatenate((fraud_indices,random_normal_indices))
under_sample_data = data.iloc[under_sample_indices,:]
X_undersample = under_sample_data.loc[:, under_sample_data.columns != 'Class']
y_undersample = under_sample_data['Class']
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.3, random_state = 2003)
X_train_undersample, X_test_undersample, y_train_undersample, y_test_undersample = train_test_split(X_undersample,y_undersample,test_size = 0.3, random_state = 2005)
print("Train:", pd.Series(y_train).value_counts().to_dict())
print("Test:", pd.Series(y_test).value_counts().to_dict())
print("Train undersample:", pd.Series(y_train_undersample).value_counts().to_dict())
print("Test undersample:", pd.Series(y_test_undersample).value_counts().to_dict())
logit = LogisticRegression(penalty=None)
logit.fit(X_train,y_train)
y_chap = logit.predict(X_test)
table = pd.crosstab(y_test,y_chap)
print(table)
logit = LogisticRegression(penalty=None,class_weight="balanced")
logit.fit(X_train,y_train)
y_chap = logit.predict(X_test)
table = pd.crosstab(y_test,y_chap)
print(table)
logit = LogisticRegression(penalty=None,max_iter=5000,solver="newton-cg")
logit.fit(X_train_undersample,y_train_undersample)
y_chap_undersample = logit.predict(X_test_undersample)
table = pd.crosstab(y_test_undersample,y_chap_undersample)
print(table)
from sklearn.ensemble import RandomForestClassifier
from math import sqrt
forest = RandomForestClassifier(n_estimators=100, max_features=int(sqrt(30)),oob_score=True,n_jobs=-1)
forest.fit(X_train,y_train)
y_chap = forest.predict(X_test)
table = pd.crosstab(y_test,y_chap)
print(table)
forest = RandomForestClassifier(n_estimators=100, max_features=int(sqrt(30)),oob_score=True,n_jobs=-1)
forest.fit(X_train_undersample,y_train_undersample)
y_chap = forest.predict(X_test_undersample)
table = pd.crosstab(y_test_undersample,y_chap)
print(table)
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
device = torch.device("mps")
X_train_t = torch.from_numpy(X_train.astype(np.float32))
y_train_t = torch.from_numpy(y_train.values.astype(np.float32)).unsqueeze(1)
X_test_t = torch.from_numpy(X_test.astype(np.float32))
y_test_t = torch.from_numpy(y_test.values.astype(np.float32)).unsqueeze(1)
X_test_t = X_test_t.to(device)
y_test_t = y_test_t.to(device)
train_ds = TensorDataset(X_train_t, y_train_t)
train_loader = DataLoader(train_ds, batch_size=1024, shuffle=True, drop_last=False)
n_pos = (y_train == 1).sum()
n_neg = (y_train == 0).sum()
pos_weight_value = torch.tensor([n_neg / n_pos], dtype=torch.float32, device=device)
pos_weight_value.item()
def train_model(model, epochs=50, lr=1e-3, weight_decay=0.0):
model.to(device)
opt = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight_value.to(device))
for epoch in range(1, epochs+1):
model.train()
running = 0.0
for xb, yb in train_loader:
xb = xb.to(device, dtype=torch.float32)
yb = yb.to(device, dtype=torch.float32)
opt.zero_grad()
logits = model(xb)
loss = criterion(logits, yb)
loss.backward()
opt.step()
running += loss.item() * xb.size(0)
train_loss = running / len(train_loader.dataset)
if epoch == 1 or epoch % 5 == 0 or epoch == epochs:
model.eval()
with torch.no_grad():
logits = model(X_test_t)
test_loss = criterion(logits, y_test_t).item()
print(f"Epoch {epoch:02d} | Train loss = {train_loss:.5f} | Test loss={test_loss:.5f}")
return model
logreg_torch = nn.Linear(X_train.shape[1], 1)
sum(p.numel() for p in logreg_torch.parameters())
train_model(logreg_torch, epochs=50, lr=1e-3, weight_decay=0.0)
logreg_torch.eval()
with torch.no_grad():
logits = logreg_torch(X_test_t)
y_pred = (torch.sigmoid(logits).squeeze(1) >= 0.5).int()
y_pred = y_pred.detach().cpu().numpy().ravel()
table = pd.crosstab(y_test,pd.Series(y_pred, index=y_test.index))
print(table)
mlp = nn.Sequential(
nn.Linear(X_train.shape[1], 128),
nn.GELU(),
nn.Linear(128, 64),
nn.GELU(),
nn.Linear(64, 32),
nn.GELU(),
nn.Linear(32, 1)
)
sum(p.numel() for p in mlp.parameters())
train_model(mlp, epochs=80, lr=1e-3, weight_decay=1e-5)
mlp.eval()
with torch.no_grad():
logits = mlp(X_test_t)
y_pred = (torch.sigmoid(logits).squeeze(1) >= 0.5).int()
y_pred = y_pred.detach().cpu().numpy().ravel()
table = pd.crosstab(y_test,pd.Series(y_pred, index=y_test.index))
print(table)