A crowdsourced response to whois.spam

The calls are starting to die down now, and I only get 5 emails a day now, rather than 20. That's some comfort, I guess.

I've had to ensure the same process that most proud new domain owners endure. Calls from allegedly local numbers from offshore web developers, SEO services, social media marketers and app builders offering their services towards making my new domain the successful new venture that they assume i want it to be. Emails too. Many more emails than phone calls. In the week following the registration of three domains- one '.com' , one '.co.uk' and one '.xyz' - I was receiving around 20 emails a day, but a mere 5 or 6 daily calls.

Much has already been written on the issue of whois spam, and whether the balance between the integrity of the web and privacy. I would suggest that having a name and postal address would be sufficient in order to bring accountability to domain ownership and ensure that those with valid reasons for contacting you are able to do so. This adds a small cost to the contact process and avoid email spam and nuisance calls.

Domain registrars acknowledge this issue with the whois registration, and use it to promote their various domain privacy options, which often cost more than the domain itself. No doubt, on a day with 7 nuisance calls and having my inbox filled with dozens of spam emails, I would have told you that the price of these domain privacy packages was absolutely worth it. But then again, why should we have to pay a significant levy on the price of registration to avoid a form of harassment that is uninvited and entirely caused by an ill-considered public register policy.

So I wondered what I could do about it. How could I devise my own system that detected and filtered out spam and nuisance calls, while still providing full disclosure of ownership and a means to contact the owner with legitimate queries. So I wrote down patterns and wondered how i might address them. If I was to create a phone number and email address that served an as intermediary to my usual number and email address, what are the means by which I might be able to ensure that I might be able to filter out dubious connections and allow through correspondence I need to know about. Because the email address and phone number would only be used for whois publication, I might be able to apply customised filters that would be impractical to everyday email or phone call screening processes

 Pattern Problem Solution
 Calls often received without caller ID  This may suggest that the user is calling from a VOIP service or is actively obscuring their number. Although there may be legitimate reasons for doing this, it usually suggests that the caller has a reason for not wanting to reveal their identity  The phone number listed on the whois register would take the caller to an Interactive Voice Response system that would advise them that calls without caller ID are not permitted and that they should call back from a disclosed number
 Emails often come from public email provider addresses

 It's not without irony that people purporting to be web professionals send you their sales pitch from a gmail or hotmail account.

Along with the fact that these emails rarely contain the name of the company or links to their portfolio, it indicates that the sender is aware that they are engaging in spam.

 The email system would add a greater level of scrutiny to email messages coming from these addresses. Senders would be sent a captcha image which they would need to respond to and confirm by email in order for their message to be received. This added burden makes blanket emails impractical.
 Emails contain similar keywords Emails offering web services contain the same words and phrases.   A database can be easily created with keywords that indicate a sales email and this can be run against all incoming email to filter it appropriately

 So, I've created domainbeard.com, where I'm staging my fightback against whois spam. I have a IVR call system set up that leads all legitimate callers to a voicemail which alerts me to new messages by email, and i can pick up at any stage. I also have a dedicated email address which I've used in the registration of a few new domains. I'm monitoring and configuring an email system that necessitates validation and emails e senders a response requiring their action under certain circumstances. I'm just learning now, but I see a place for a new free service for all domain registrars to use that will alleviate the pain of whois spam.

Allowing legitimate correspondence

I figured I could not only create a system that filters out junk, but also identifies high priority calls and emails

 Pattern Solution
 Calls from bonafide sources come from recognised numbers A database of numbers can be built up of both known nuisance callers, and conversely of known domain regulators and legal bodies. This could rely on the reporting of users and be built up over time 
 Emails from bonafide sources come from recognised email addresses The same applies to email. A database of known spammers and known legitimate sources would be built up which would enable better filtering as time goes on.

 At the moment, I'm gathering data and experimenting with different approaches, but I'll be opening up Domainbeard to any domain registrant early in 2018 with a few main goals:

  • Alleviate the pain of spam email and nuisance calls by providing an intermediary address and number specially configured to identify the legitimacy of sources
  • Allow users to report on the legitimacy of callers and email senders to improve the filtering algorithm
    • This can be used to save future email registrants from being bothered by known spammers
    • It can also be used to build a case of evidence to provide to ISP and telephony providers in order to have the account of spammers stopped.
    • Provide statistics and advice related to whois spam that is far more authoritative and quantitative than the anecdotes of a frustrated domain owner.
  • To be free- given the enormous wealth of data that could be collected, its ability to improve the integrity of the web, and a common acceptance that people ought not be subjected to nuisance calls and emails, why should people have to pay for this? 


Visit Domainbeard


RESTful API added to Geodata Solutions

If you've been wondering why a world of web site developers, CMS teams, shopping cart providers, and blue chips are all maintaining their own independent geographical databases, then you're not alone. 

Imagine if every website wanting to display the weather forecasts, forex information, or sporting results all had to download CSV files from a github repository and then commit themselves to maintaining their own lists forever more. 

Countries, states and cities may not pop into existence as often as the weather changes, but the idea that every website should maintain their own database seems crazy. It's the kind of thing that is ideally suited to a web service.

Updating country lists may be a mere irritation, an occasional chore, for a website administrator. After all, the last time a new country needed to be added was back in 2011, when South Sudan gained independence from Sudan. However, states and their names change much more frequently, and towns and cities even more so. And then there's additional information about locations, such as their population, that change even more frequently.

The point of Geodata Solutions is to make this tedious task a job for one organisation, rather than for millions around the world. I plan to build on the existing database I've created, keep it current, and build on its assets. My hope is that this project will change the way geographical information is sourced in web applications. Syndicated usage of the system will alleviate the duplicated burden of maintaining local data stores, while allowing error reporting and updates to be submitted by millions of users. 

The RESTful interface is in 'Beta' at the moment. It offers country lists, state lists and city lists, along with some basic additional location information. Developers wanting to use the system should get in touch for an API key, as well as requests for new search types or filtering, or new pieces of data. There's also quite a big job in completing gaps in the data set, of which population data is a significant one.


Outgrown Joomla?

Most of my clients are companies that started off with a simple Joomla website years ago, either because they arrived at it by evaluating the CMSs strengths against other systems, or because the initial designer they worked with worked with Joomla, and so they ended up with the system by default. Fortunately, these businesses grew, and the requirements they demanded from their Joomla site became more specialised. 

While new businesses often buy commercial extensions that are by nature a one-size-fits-all setup, in a growing business it's less acceptable to have your business systems dictated by your Joomla extensions. So what can businesses who feel like they've outgrown Joomla do?


Read more: Outgrown Joomla?

I'm pleased to announce that version 1.0.1 of the Geodata Solutions Location List Custom Fields Plugin is now completed.

Joomla site owners can now add location data to their content, categories, user and contacts components.

There were some difficulties along the way. Given the newness of the custom fields functionality, there is little documentation for the new plugin type, and the routine for adding new custom field types differs from the custom fields types offered in the default Joomla package, which have their templates hard-baked into the core. I have to thank people on the Joomla forum, who were able to assist with questions and offer support.

This is just the first iteration of the extension, and there are many improvements to come, within the functionality of the plugin's code, and also in the web service offered on the Geodata Solutions side.

A few 'hacks' were required in this version:

  • The jQuery chosen library had to be subverted in order to get the ajax functionality of the lists (as coded in general-use code snippet version) to work within Joomla's back end. The ability to 'destroy' the -chzn elements was available in Joomla 2.5, but it seems this was removed in J3.x. This version of the extension uses a hack that hides the chosen elements and unhides the original form fields offered by cavarief here.
  • The system uses the simple javascript request method that the general use code snippet uses. The disadvantage of this is that it doesn't allow for secure serverside authentication. At current usage, there is no need to throttle access to the web service, but I anticipate that when usage increases, I'll need to switch to a superior solution. Some things to note:
    • Authentication currently checks the HOST header in the js request- it's the crudest way of recording the usage of each separate implementation of the system, but hardly the most reliable
    • Proposed solution in the next release would be to have a key generated with each download of the plugin (I think I can do this without requiring registration with Geodata Solutions- something that I think is a barrier to quick implementation) which the user could simply enter into the plugin configuration
    • Very high volume sites might have to subscribe to support the service at a later stage
  • There are issues related to the output/override of custom fields in the user component. This is reported issue, and one that I hope will be corrected in future Joomla version. Until then users will have to put up with the pipe-separated format used for stage within their user profile views in the front end (eg. Australia|Victoria|Ballarat ) or create their own template override. If you want some help implementing this, or want to share your solution, then please get in touch.

The plugin was submitted to the JED on 4th July, and at time of writing is pending approval, but you can download Location Lists Custom Fields Plugin  and read the (minimal) Location Lists Documentation


Why would any website administrator or developer want to 'outsource' something as simple as a country list to a web service?

It sounds like a valid question. Implementing these isn't rocket science, and there are a load of sites with detailed instructions on how this can be achieved with a bit of jQuery, HTML, serverside code (whatever flavour) and some mysql data dumps. I began this process recently when needing a set of dynamic country- state (county for some places) - city dropdowns for a site I was working on. Although there's some great code snippets out there being offered graciously (and freely!) by fellow developers, I wondered why we're all re-inventing the wheel.


Read more: Select lists as a web service?! Really?

Here is an example of the select lists possible with only a few lines of code...

<select name="country" class="countries presel-byip" id="countryId">
    <option value="">Select Country</option>
<select name="state" class="states" id="stateId">
    <option value="">Select State</option>
<select name="city" class="cities" id="cityId">
    <option value="">Select City</option>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script> 
<script src="https://geodata.solutions/includes/countrystatecity.js"></script>
Back to Top