The 3am Attack That Taught Me Not to Trust a Server to Look After Itself

The alert came in just after three in the morning.

The server was unresponsive. SSH was timing out. The hosting panel showed CPU usage I'd never seen before — not a spike, but sustained, brutal load that had been building for hours while I slept.

What I found was my Laravel application being hammered by a coordinated bot campaign. Not the low-level scanning I'd got used to. This one had found something: the contact form. No rate limiting, no CAPTCHA, no validation beyond the basics. To an automated script, an open door.

Nearly a million emails went out before I managed to shut it down.

The damage

The server got blacklisted — not by one service, but by several: email providers, spam databases, reputation services.

I spent the next week in a kind of focused misery. Days on the construction site, evenings contacting blacklist operators, waiting for reviews, and putting in everything that should have been there from the start:

• Rate limiting on every form
• CAPTCHA where it mattered
• SPF, DKIM and DMARC configured properly
• Monitoring on outbound email volume

Each fix felt like closing a door that should never have been open.

The real lesson: nobody was watching

The attack itself wasn't the worst part. The worst part was that nothing had told me it was happening until the server was already on its knees.

While I was repairing things, I started a list: everything that could go wrong on a server like mine, and what would need to be true to catch it early. The contact form, the login endpoint, the registration page, the email queue, database connections, CPU, disk, SSL certificate expiry, the running services.

Nothing I could find watched all of it. So I started building something that did.

From a cron script to Sentinel

The first version was not impressive: a Python script on a cron job, checking a list of things every few minutes. But it ran, and it found things. A service that had quietly crashed seven times in an hour. A database query taking twelve seconds. Disk usage heading towards full.

Small things. The kind that turn into crises if nobody notices them.

That script grew into Sentinel. Today it runs as a long-running service rather than a cron job, and on the security side it integrates with Fail2Ban, CrowdSec and UFW. It detects the services on your server and automatically installs Fail2Ban jails for them (sshd, nginx, caddy, apache, mysqld). Across a fleet, banned IPs are shared through Queen, Sentinel's central server, so one server's attacker gets blocked on the others too.

On one of my production servers, the lifetime counter of blocked attacks has passed 300,000. That's not unusual — it's what being on the internet looks like. The difference is that now I can see it.

What I'd tell anyone running their own server

• Assume you're being probed right now. You are.
• Protect every form, not just the login page. My open door was a contact form.
• Set up email authentication (SPF, DKIM, DMARC) before you need it, not after a blacklist.
• Make sure something is watching, so you hear about problems at the first sign, not at 3am when the server is already down.

Try Sentinel

• Basic: free for 1 production server
• Pro: $49/month for up to 5 servers
• Enterprise: $149/month for up to 10 servers, with AI Chat, AI Healer and Cross-Fleet IP Reputation

Sign up at sentinel-ai.info/pricing to get a personalised one-line install command. Installation takes about 2–4 minutes.

Built by FixFlex Ltd, West London.