This start to be ridiculous

12 posts / 0 new
Last post
#1 Fri, 12/23/2016 - 01:46
Diabolico
Diabolico's picture

This start to be ridiculous

The amount of spam and topics all over the forum regardless if they are in appropriate section or not, start to be ridiculous. Right now there are tons of spam post still sitting and some of them are there for days if not weeks. Yes i reported majority of them but until now no one deleted them.

If i want to subscribe to Virtualmin section i can expect that biggest part of email notifications would notifying me about spam or subject what doesnt belongs there. Seriously guys, its time to woke up. On one side you act as professionals on other side things are falling apart. Before some of you jump in with "more urgent things", "lack of time", "drupal is bad"... well how about not using this shitty forum and instead go for something better. Whatever you choose just clean this mess once for all.

Last but not least, how about to force classic forum list view so no more new post in the middle of several page long topic.

Sun, 12/25/2016 - 00:01
andreychek

Howdy,

We do get rid of quite a bit of spam, but some does get through. I just went through the full list of reported posts again and got rid of all the remaining ones I saw there.

-Eric

Sun, 12/25/2016 - 03:10
Joe
Joe's picture

It's not entirely lack of time or Drupal being bad, at this point. The tools for fighting human spammers (non-bot, actual people who setup email addresses, verify them, and then create content) are really just not very effective. We have almost every Drupal anti-spam measure enabled, plus third party anti-spam tools. I've spent hours on anti-spam efforts just this month. It's just a really hard problem.

And, for every spam you see, we've probably killed three or four more within an hour or two of it arriving. It's just a deluge. For whatever reason, some high volume spammers have decided that Virtualmin.com is a high value target; relatively high PR ranking, I guess. The abuse reporting mechanism, unfortunately, is very poorly implemented...it doesn't provide tools to acknowledge reported posts, so now we've got a huge list of posts to look at (having to ignore the ones that weren't supposed to be flagged because they aren't actually spam, just annoying to someone for whatever reason), and kinda parse out which ones are new and which are not needing attention. I kinda pieced it together out of some half-assed existing modules, and it doesn't really hold up to use at our scale (we're finding lots of things in Drupal don't do well at our scale, even though our scale isn't actually all that big).

I keep working on it, and there is another anti-spam module that doesn't actually work right now that I've been working on fixing (I mean the module has been unmaintained for a couple of years, and does not work with current Drupal versions, but I'm working on updating it). It would provide us some additional tools to blacklist specific words and phrases, as well as implement our own local bayesian database. The paid third party service we use produces more false positives than accurate blocks, so it's kind of proving to be less than worthless, which is a real pain in the ass.

We honestly have put a lot of effort into spam prevention. It isn't good enough, but I don't know what else to do. I'm sitting here typing this at 1AM on Christmas Eve, after working all day, and with plans to work half of Christmas, too. We just don't have any more hours to put into play. We're four guys, one of whom has a full-time job in addition to Virtualmin (and the rest of us also have some other stuff going on to keep the bills paid), supporting about a million, or two, users, spread across project made of hundreds of thousands of lines of code. Sometimes the choice is between making the website better or making Virtualmin better or directly supporting a paying customer. I try to find a good balance.

--

Check out the forum guidelines!

Sun, 12/25/2016 - 09:01
Diabolico
Diabolico's picture

Find 2-3 people to clean up the forum, split old topics when necroing and move new one to appropriate section (e.g. general is a total mess)?

Ninja edit: To be sure who you get doing this job just ask for copy of government ID and no need to fear if that person will do something bad. Really, you should not avoid this part if you want to sleep in peace.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Wed, 12/28/2016 - 22:55
Diabolico
Diabolico's picture

Oh wow, 22 new spam topics. Can i give you few suggestions:

  1. Email verification (in case you dont have it already), so new member must verify his email before second one is sent with his login details.

  2. Block sending emails to disposable emails (domains), no registration emails no user registration. The list can be found somewhere on the internet.

  3. Post time limit, e.g. at least 5-10 min (10 min is better) between each post. This can be lifted for "well known users" but its optional.

  4. Block non-latin characters, e.g. chinese, arab, etc...

  • Optional:
  1. Based on your sale block spam regions at server level, e.g, if in 10+ years no one bought VmPro from Vietnam free to block that country.

  2. Better forum?

This should limit or at least make it harder to spam on the forum. Just think in this way, so much work around new drupal, coding, modifications, problem solved... and then new user/potential client comes here and see a forum full of spam - all your work for nothing as no one will take you seriously.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Wed, 12/28/2016 - 23:37 (Reply to #5)
Joe
Joe's picture
  1. We've never not had email verification. We also have CAPTCHA on register, honeypot, spamicide (another kind of honeypot module), and probably a couple of other bits of protection for preventing spam registrations.
  2. That's probably smart, but I'm hesitant to do it...I don't really oppose someone using a disposable email if they want to post but don't like giving out an email address. But, then again, we never spam and we never sell email addresses, so it's probably not unreasonable to expect a little trust from users who want to contribute here.
  3. That's probably a good idea; I don't see any way to implement it in Drupal without writing a custom module. Interestingly, many of the spammers here created their account 1+ months ago, and only begin posting when it is an older account. So, we can't just "age" a user into being unrestricted. I'll see if I can find an existing solution for this.
  4. I hate to do that since we have users all over the world, but I can't remember the last time we had a post that needed non-latin characters. Again, I dunno how to implement this in Drupal, and don't see off-the-shelf solutions for it.
  5. We don't have any regions that would fall under that ban; except maybe North Korea, but we don't get spam from there either. Virtualmin is surprisingly popular in some places that are considered "spam havens", like Russia, Ukraine, southeast Asia (I think that might be a carryover from our popularity in Australia because Jamie is from there).
  6. I'm bettering it as fast as I can. I'm not willing to use something completely separate from Drupal, as I really want unified user accounts, look-and-feel, search, and notifications handling (though we already have two separate sets of notifications because the issue tracker implemented its own rather than using the same methods the forums use). I can say that the next time we upgrade (not anytime soon), it's almost certainly not going to be another Drupal version (I have not looked deeply into Drupal 8, yet, but the D6->D7 transition was the saddest, most frustrating, several months of my professional life).

Edit: I should point out that with regard to #2, ~85% of our spam comes from users with a Hotmail or Yahoo account. Another 10% from GMail accounts. I don't remember the last time it was someone with a disposable email address, though I know it has happened in the past.

--

Check out the forum guidelines!

Thu, 12/29/2016 - 00:21
jimdunn

JOE:

-- I'd just like to say that we appreciate all your efforts -- and I'm sorry for the extra work of cleanup that it causes you -- when I see spam, I just ignore it -- if I don't click it, I don't have to read it -- thank you for providing both the paid and free versions of your products, you guys ROCK!

EVERYONE:

-- remember that there's no need to post a comment complaining about spam, that's just more spam -- sometimes I do click the spam, and then click the REPORT link so the admins have some help in cleanup -- it's a great feeling to be a part of the solution

Thu, 12/29/2016 - 00:50
Diabolico
Diabolico's picture
remember that there's no need to post a comment complaining about spam, that's just more spam

Spam - is the use of electronic messaging systems to send an unsolicited message (spam), especially advertising, as well as sending messages repeatedly on the same site.

Forum - public meeting place for open discussion / medium (as a newspaper or online service) of open discussion or expression of ideas.

Have a discussion about something is not same as spamming, learn the difference. When people are discussing about some problem is more likely they will find a solution than just pretending everything is good.

@Joe: I can only imagine how complicated must be to handle Drupal but anything you can take from my suggestions it should make at least some difference.

  1. There is no valid reason to use disposable email, especially with so many free email services like Gmail. If anyone is afraid to net get spam why not open another email account just for random website registration all over the web? Like i previously said, no valid reason to have disposable email.

  2. Custom module is the only thing what comes into my mind.

  3. Maybe it could be done with some MySQL modifications to simply reject any input containing non latin characters. Honestly not sure on this one but i saw it in the past on some forums and even registration forms.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Thu, 12/29/2016 - 11:55
jimdunn

Thanks again, Joe and team, for all you do for us!!

Fri, 12/30/2016 - 06:09
Kintaro

https://www.drupal.org/project/spambot

stopforumspam.com work very well in a forum I admin and catch all the spammers that try to register. I don't know if the spam sources are the same but maybe you can give it a try. If it's already installed... ouch!

Fri, 12/30/2016 - 16:55 (Reply to #10)
Joe
Joe's picture

Hey, that looks great!

It's not one we've enabled yet. I'll put it on my todo list for deploying sometime soon. It'd be really nice to have spam solved once-and-for-all. As it is, I keep revisiting it every few months when our existing tools stop working (either because the modules stop working with new versions of Drupal, or the spam service starts giving too many false positives, or whatever), and we still end up with more spam than I'd like. I don't know why we're such an attractive target for spammers; our forum isn't that popular, our page rank isn't that high, and our audience isn't gullible enough to fall for any of the spammed products. But, for whatever reason, the spammers send real humans (or very good bots) to our website hundreds of times a day to try to create accounts (occasionally succeeding), and to try to send spam (often succeeding, once they make it past the registration).

--

Check out the forum guidelines!

Fri, 12/30/2016 - 06:12
Kintaro

Just for information: There's a bridge that sync users with xenforo: https://www.drupal.org/project/xenforo

Xenforo is a great forum script

Topic locked