My From-Scratch Phishing Classifier Matched scikit-learn. The Best Part Was a Bug.

I finished Andrew Ng's Supervised Machine Learning: Regression and Classification course from DeepLearning.AI and Stanford Online on Coursera this month (certificate). Courses are good at teaching you the math. They are less good at showing you what goes wrong when you point that math at real data.

So I built something small to find out. It's called PhishGuard, and the code is here: github.com/WasimSheikh/phishguard.

The idea is simple. Take a URL. Look only at the string itself, things like its length, its characters, its structure. Then decide: phishing or not. No page content, no network lookups. Just lexical features.

I wanted two versions of the same model. One written from scratch in NumPy, with my own sigmoid, cost function and gradient descent. One using scikit-learn's LogisticRegression. If my version was right, the two should land close to each other. If they didn't, I'd have a bug somewhere and I'd have to go find it.

Building it twice

The data is the PhiUSIIL Phishing URL Dataset from the UCI Machine Learning Repository (link), released under CC BY 4.0. It's a labeled set of legitimate and phishing URLs, which is exactly what this kind of exercise needs.

Writing logistic regression by hand is not hard. It's maybe the first real model the course has you build. But writing it is different from trusting it. Gradient descent will happily run and give you numbers even if a sign is flipped or a learning rate is off. The loss goes down. Everything looks fine.

That's why I kept scikit-learn next to it. Not to replace my code, but to check it. A reference implementation is a cheap way to verify your output. If your from-scratch model and a well-tested library agree on the same features and the same split, you have much more reason to believe both.

The feature that was too good

Before the comparison meant anything, I hit the more interesting problem.

One of the features was simple: does the URL use https? It sounds reasonable. Phishing sites have historically been more likely to skip TLS, so it should carry some signal.

In this dataset it carried all of it. Every plain-HTTP URL in the data was labeled phishing. Not most. Every one.

That means the https feature wasn't a hint about the label. For a big slice of the data, it was the label. A model trained with it would look great on the test set and learn almost nothing useful about URLs. And in the real world plenty of phishing pages use https now, since certificates are free and easy to get. The model would be confident for the wrong reason.

So I dropped it.

This is the part I keep thinking about. When a feature looks too good, my first instinct is to be happy. The better instinct is to be suspicious. A suspiciously strong feature is usually a bug in the data, not a breakthrough in the model. Nothing crashed. No warning showed up. The only way to catch it was to look at what the data was actually saying, not just whether the pipeline ran.

That ties back to something I've been saying lately: verify outputs, not just processes. A clean training loop is a process. A metric that is too good to be true is an output, and it deserves a second look.

The results

With the https feature removed, I trained both models and tested them on a 12,000-URL holdout set.

The two are close. ROC-AUC is 0.842 for mine and 0.844 for scikit-learn. F1 is 0.777 against 0.787. My version is a bit more precise, and scikit-learn catches a bit more phishing. Small differences in optimization and defaults can push things around at that level, so I'm not reading much into which one "won."

What matters to me is that they agree. That's the check I wanted. My gradient descent is doing what it's supposed to do.

These numbers are also honest in a way they wouldn't have been with the leaky feature. They're not spectacular. A recall of around 0.71 to 0.74 means a fair share of phishing URLs still get through. That's what you'd expect from a linear model that only sees the URL string.

What this is and what it isn't

PhishGuard is a learning project. It's lexical-only. It doesn't look at page content, domain age, hosting, or anything else a real filter would use. It is not a production phishing filter and I wouldn't put it in front of users as one.

What it is: a clear, small example of building a model from scratch, checking it against a trusted library, and catching a leak before it made the results look better than they were.

If you're working through the same course, I'd recommend doing something like this. Build the model yourself. Then build it again with a library and compare. And when one feature looks like it solves the whole problem, go look at why.

The code, the notes and the comparison are all in the repo: github.com/WasimSheikh/phishguard. Questions and pull requests are welcome.

Data: PhiUSIIL Phishing URL Dataset, UCI Machine Learning Repository, CC BY 4.0.

Try it: paste a URL in the live demo.