SearX for Anonymous Search

If you have read my past posts on search engines you would have realised that I am always looking to use a search engine other than Google search – well anything other than Google in everything I do!

In the past I have recommended Scroogle which unfortuately died in 2012, ixquick.com a European based search engine, duckduckgo.com which is still going and growing in strength and most recently privatesearch.io which also seems to have disappeared although the excellent sister site privacytools.io is still going – they have great advice on which software and services to use to be as private as possible (and move away from Google).

All these services have one thing in common, they anonymize your searching to mitigate against your personal life being tracked. Results are not produced by profiling the user (tracking), every user will get the same search results when entering the same search terms.

Scroogle used a hack into Google’s search engine but suffered from being reliant on Google not changing their API to thwart this kind of third party service.

Duckduckgo relies on Bing, Yahoo and crowdsourced sites like Wikipedia for it’s search results.

Privatesearch.io did something similar but also used Google for results.

Not sure what happened to Privatesearch.io but it was based on the opensource asciimoo/Searx project: https://github.com/asciimoo/searx. This works in a similar way to the other engines – anonymizes your search and uses Yahoo, Bing and Google (it shows which search engine the results have come from and also uses many other sources). It is completely open source and there is a thriving community – https://twitter.com/Searx_engine – where many people have created their own search engine sites for general search or for specific search criteria: http://stats.searx.oe5tpo.com/ This link will list all the current Searx sites available (and whether they are up – note the down signal for privatesearch.io). The one I use the most is the general Searx.me site.

searx.png

 

ixquick.com is still going, and it has somehow acquired the domain startpage.com. It is owned by a commercial outfit – Surfboard Holdings B.V. in the Netherlands. It has had much more written about it and many awards and plaudits associated with it: https://en.wikipedia.org/wiki/Ixquick. It’s based in Europe and adheres to European privacy rules plus a lot more. The only drawback is that it is a little slow.

startpage

What I found with all these engines was that results were very much steered towards the USA – I’m in the UK so I wanted results more relevant to my local area. Searx.me does offer preferences where you can change the country to United Kingdom and I found that the results are very good using this engine. Ixquick also now has the ability to change preferences to UK English and even has a ‘Pages from the UK’ button automatically appearing (I assume that changes according to which country you are in) and I have found the results very relevant.

So what would I recommend now?

At this time I would definitely recommend startpage.com from ixquick as your default search engine but if speed is important and you like the idea of using an opensource solution rather than commercial then go with SearX – that’s what I use.

Note: when saving preferences they are usually saved in a cookie. If you delete cookies on exiting your browser have a look at the selectivecookiedelete add-on for Firefox detailed in a previous post: My Firefox settings – retaining some cookies whilst deleting everything else on exit

 

Advertisements

New anonymous search tool

I’m always on the look out for better search engines – not just with the results they provide but that also don’t track or store your searches like Google, Bing et al do.

We used to have Scroogle but they died some years ago. Since then I have been using Duckduckgo.com as my default engine. It’s ok but suffers from being US based – i’m in the UK so I do find myself using google.co.uk when i’m searching for something specific to the UK.

But now we have something much better – privatesearch.io

Not only is it an anonymous search engine but it also collates it’s results from all the other search engines. If you search for something then the results are listed with a text icon indicating where the result has come from – Google, Duckduckgo, Wikipedia (directly!) etc. All this can be customised in the preferences – so if you don’t want Google results you can specify that. It’s also very fast. Fantastic!

It’s too early to say if it solves the location problem of getting too many US based results but i’ll report back on that once I have used it for some time. I highly recommend you set this as your default search engine.

It’s also part of a privacy advisory site – privacytools.io -which is a site dedicated to giving you the latest advise on privacy software tools – encryption, VPNs, browsers etc. An excellent resource and well worth checking out.

 

Search Engines – what are they good for?

Absolutely nothing in some cases!

One increasingly well known fact is that paywalls – those barriers that some websites put in the way to try and get you to hand over money or register to gain access – are these days, very porous.

They’re designed that way so that they can charge for access, either by charging or selling your details, but do not close themselves off to the rest of the internet i.e. search engines.

They still want search engines to scan their site and therefore have to open themselves up to them whilst blocking anyone else. Google’s T&C’s even state that websites must not block access when referred from Google search results if they want to appear in those results.

That’s the loophole that apps like BreakthePaywall! (http://www.breakthepaywall.com) use to circumvent the paywalls – they impersonate a search engine.

For example: if you go to a site and notice an article that, when you click on it, displays a paywall blocking page or popup then if you copy that same article heading and paste it into a search engine then, more often than not, the link in the search result list to the same article, when clicked, will result in the article being displayed without any problems.

That’s because the website is seeing that you have come from a search engines results page – using the ‘Referer’ header that is sent by the linking website.

This is how paywalled websites keep themselves at the top of search results but also block people that have gone to the website directly and/or click on subsequent articles. This is usaully referred to as opening themselves up a little bit in order to gain search engine traction but still make money out of the ignorant. It must be rather galling for loyal customers who stump up annual subscription fees to discover that others can get it for free – hey, that’s the mad world of the internet!

Another technical aspect is that websites need to allow the search engine robots unfettered access. These robots go out onto the internet and scan websites and report back to the databases that collate the results. They have their own special names that websites can pick up and therefore allow access.

So, search engines are useful after all, but interestingly, not all of them…

Here’s a list of the top search engines in the world according to Wikipedia:

Search engine Market share in June 2014
Google 68.69%

 

Baidu 17.17%

 

Yahoo! 6.74%

 

Bing 6.22%

 

Excite 0.22%

 

Ask 0.13%

 

AOL 0.13%

 

https://en.wikipedia.org/wiki/Web_search_engine

As you would expect Google is way out in front. Experimenting with the method of copying a header, paste into search engine and see if the resulting link to the paywalled website works it’s not surprising that using Google as the search engine always works. Most websites want to be on Google’s search results so they allow full access from Google. But what about the other search engines:

Baidu – is of course the Chinese based search engine – try a paywalled western based news organisation  website on there and you don’t get anywhere. Consequently they don’t appear anywhere near the top of the search results (you have to put in the website domain to get anywhere near). Yes, it seems like some websites are not interested in a few billion chinese customers.

Yahoo next – again, no joy at all. It turns out that Yahoo doesn’t actually do its own search results. They currently use Microsoft Bing’s search results – they used to use Google up until 2004. So, it seems some sites have excluded Yahoo from their allow lists, which seems strange as it is the 3rd largest search provider.

Microsoft Bing – works!

Excite, Ask, AOL and any other smaller search engine – DuckDuckGo uses Bing as well as other sources for its search results – do not work, or rather, are ignored by paywalled websites.

So paywalled websites want to expose themselves and effectively give free access but don’t want to do it for everybody – maybe they are just as ignorant of what they are doing as their customers.

But it also highlights the fact that for non-Chinese internet users at least there is really only two search engines – Google and Bing. What a stitch up!

Scroogle is no more!

My favourite search engine Scroogle.org has died:

http://www.theregister.co.uk/2012/02/21/scroogle_dead/

http://www.betabeat.com/2012/02/21/scroogle-privacy-first-search-engine-shuts-down-for-good/

Quite why some hackers had it in for Scroogle I have no idea and who exactly were they??!

To be honest it had been going down so often, due to Google throttling, that I was thinking of looking for an alternative anyway. Here is an interesting take on alternatives:

http://www.betabeat.com/2012/02/21/scroogle-privacy-first-search-engine-shuts-down-for-good/

So use duckduckgo.com for your day to day searches as it has full privacy like Scroogle but if you must have Google search results, like Scroogle gave you, then use their encrypted version – Google will know what you are doing but at least no one else will.

The article above also has a link to instructions on integrating them into your browser but not for IE (especially IE9 which seems to make it difficult to add custom search engines). Use this page, despite it referring to IE7 this works for all IE versions including IE9: http://www.enhanceie.com/ie/searchbuilder.asp

By the way I have seen posts that warn against using duckduckgo.com as it is owned by Amazon – that is not the case, they simply use the Amazon EC3 cloud service to host their servers. Again use the https version and you should be fine – at least as private as you can be these days!

Anonymous searching

Came across some recent tech news items regarding Scroogle this week – http://www.scroogle.org/cgi-bin/scraper.htm – don’t use .com, that’s a porn site! Apparently Google has switched off a search scraping service which allows you to use Google search results through your own interface. This was apparently due to Google dropping support for IE6 – how these two relate I don’t know.

This caused Scroogle to stop working but it piqued my interest enough to investigate further, especially when a a few days later Google re-instated the service and Scroogle was back up and running.

Scroogle is a means of obtaining Google search results anonymously – they use various techniques like IP obsfuscation and cookie manipulation to make sure Google cannot track your searches i.e. cannot link it with your Google account etc. Scroogle delete all the info they collate within 48hrs of your search and you don’t get Google advertisements on your search results page – it feels just like the early days of Google, no clutter, just basic results. Marvellous! So I started using it and have not looked back.

They provide a helpful set of instructions for adding Scroogle to the IE search provider list which enables you to change the default search engine and the search engine box, at top right of IE, to Scroogle:

1. Go to Microsoft’s add search provider page: http://www.microsoft.com/windows/ie/searchguide/en-en/default.mspx

2. Paste in the test search link to the URL box: http://www.scroogle.org/cgi-bin/nbbw.cgi?Gw=TEST

Microsoft Search Provider Web Page

3. Type in ‘Scroogle’ for the name.

4. Hit the Install button.

5. Tick to make default search provider and hit Add button.

 

Adding Scroogle Dialog

That’s it – your default search provider is now Scroogle – you should now see a red, green and black crossed through ‘G’ icon in your search box.

The only problem I have found is that you do not get the ‘Cache’ link appearing on the results for the Google cached content – this is sometimes useful when trying to access cloaked content i.e. Experts-Exchange.com, but otherwise I have found the service faultless.

At the time Scroogle went down they were advising people to use another alternative anonymous search engine which you may want to check out as well:

http://www.ixquick.com/

And they have a page for adding it to IE here: http://us2.ixquick.com/eng/download-ixquick-plugin.html