As promised, a little Blackhat anecdote.
I thought a bit about posting it, was unsure. It’s kind of recent activity, it’s kind of close on what I’m doing. But if I ever finish the about page, you’ll understand. It’s about letting that all out.
Like with most of the other stuff I’m telling you, it’s not to teach you an actual method, but a way to think. So you can come up with your own methods. Everything else would be pointless.
Lets begin…
Google announced shutting down the ping API this fall. The reason? Spam (https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping).
Most ‘real’ users are submitting their sitemaps through the Search Console, if they are using Sitemaps at all.
But spammers, man… we LOVE(D) the ping API. 🥰
No account, no login, just straight pinging like there’s no tomorrow.
I always wondered why it was so ‘open’. You could ping it all day long, Google accepting your pings. There were literally no restrictions.
The Problem
I think, after all those years, it was end of 2021 when it wasn’t working as before. I don’t want to bore you with technical details, because your attention span is the same as Google’s ping API: limited.
They limited the API requests on a domain level. It was maybe 5 pings per domain before you got blocked. Once blocked it took like 24h before you could ping your domain again. This dramatically reduced the amount of URLs you were able to index.
Simple math: 25k urls per Sitemap x 5 pings = 125k urls with some flimsy crawl rate of maybe 50% max and some indexing rate in the same range over the first 24 hours. Could work, but I’m not here to play.
The Solution
The solution was both, simple and expensive. To send tens of thousands of pings per day again I would need tens of thousands of domains. Cheapest I could find was half a dollar. Not what I was looking for on the long run.
So I came up with this stupid idea to dust off the good old domains from Freenom again, i.e. .ml, .cf, .ga and all that crap.
If you don’t have the background: Those domains are free for the first year. They are not only used for tons of webspam campaigns, but also for malicious activities. Compared to .ml/.cf/.ga, .xyz is like virgin Mary, when it comes to trustworthiness.
Who ever dealt with Freenom knows how reliable they are. Sometimes it works, sometimes not. Same here, not what I was looking for on the long run.
I found two suppliers far in the east who supplied me with Freenom domains, so I wouldn’t have to deal with accounts and their API.
And Google ate that crap.
Booom, back in business, baby.
The lesson
So, here’s the lesson. The lesson is not in how to index or a certain method. It’s in the numbers and the scale.
We (including myself) or often limited in our beliefs. We assume things can only be done in a certain way, and there’s some ceiling. It’s everywhere. Even if we might not be conscious of those self-implied limitations, they are still there.
You think it’s hard to do more than a certain amount of money. You think this will only work when done in a certain way. You think this or that is not possible. For others it is.
I experienced this plenty of times during my ‘career’. I sit there and think, “nah… that’s not how it should be done”. This was one of the times I just said, “fuck it – whatever it takes”.
I have no idea where this shit is coming from. Some were born or raised with more of this shit in them, some with less. No matter how little of that crap in you, if you aren’t Elon Musk, you believe there’s some limit somewhere.
A friend of mine quit due to this kind of thinking. “It’s over”, “You can’t beat this!”, “You would need such a large scale”, “There must be easier ways”, “Maybe I’ll do crypto trading…”….
I overcame that kind of thinking (for that moment) and just did it.
The numbers I give you are not to brag. I don’t care what you think of me. They are to inspire you, to teach your mind you could go higher. Just because you saw someone else did it.
What I did was:
A total of 121.000 of Freenom domains running for indexing.
4 servers just for indexing
12 servers with 1gbps ports, mostly maxed out, for splogs
300-500 new domains for new splogs per day
Here’s a screenshot I once took for a friend in the beginning. As you can see, it’s ‘only’ 8 servers there, but they are busy 😂
So, those were my numbers. That could be your new ceiling. Hope you surpass those numbers and set a new ceiling for everybody to see.
Hope you enjoyed.
Until the next one.
Update: Just saw it was actually 12 servers for splogs. I updated that in the numbers list, but when you find 10 servers somewhere else in the text, it’s actually 12 😂