New anonymous search tool

I’m always on the look out for better search engines – not just with the results they provide but that also don’t track or store your searches like Google, Bing et al do.

We used to have Scroogle but they died some years ago. Since then I have been using Duckduckgo.com as my default engine. It’s ok but suffers from being US based – i’m in the UK so I do find myself using google.co.uk when i’m searching for something specific to the UK.

But now we have something much better – privatesearch.io

Not only is it an anonymous search engine but it also collates it’s results from all the other search engines. If you search for something then the results are listed with a text icon indicating where the result has come from – Google, Duckduckgo, Wikipedia (directly!) etc. All this can be customised in the preferences – so if you don’t want Google results you can specify that. It’s also very fast. Fantastic!

It’s too early to say if it solves the location problem of getting too many US based results but i’ll report back on that once I have used it for some time. I highly recommend you set this as your default search engine.

It’s also part of a privacy advisory site – privacytools.io -which is a site dedicated to giving you the latest advise on privacy software tools – encryption, VPNs, browsers etc. An excellent resource and well worth checking out.

 

Google at Work

I was at the Google offices in London, UK yesterday for a Google at Work roadshow hosted by one of their premier partners ‘Cloud Solutions’.

Google at Work is the rebranded Google Apps for Business containing cloud based apps like Gmail for Business (Email), Calendar (Shared Diaries), Drive (Cloud storage), Docs (Spreadsheet, Word Processor, Presentations) and Hangouts (Remote Desktop and Conferencing).

After being greeted by various young, attractive Google employees, or ‘Googlers’ as they are called, a group of us were ushered into a small part of the chocolate factory – tin foil wallpaper, captain nemo style doors, strange sculptures hanging from the ceiling, rather like being in some 1960s Beatles acid trip. Pleasing but slightly disconcerting!

Having been suitably lubricated we were then given a brief run-down of all the Google apps with Q&As after each session.

A number of the participants seemed to be able to answer questions which the panel of speakers could not – it seemed a number of them were already using Google apps within their companies so quite why they were there I wasn’t sure. I did start to get the feeling that they were perhaps ‘plants’, brought in the beef up the message and show how you can become a Google evangelist too.

It quickly became apparent that Google do have something to offer. But they are rather delusional about how much impact they can make in the SMB market which is where we were all coming from. You see, with Google it’s an all or nothing approach – if you decide to use Google apps then their impression is that you will want to move completely to the Google platform, there is no hybrid state.

None of the Googlers could bring themselves to utter the word ‘Microsoft’ – the corporate training is obviously very effective. What they don’t seem to realise is that most companies, and especially SMBs, do use Microsoft products, extensively, and they will carry on using Microsoft products for various reasons – familiarity, adversity to change, cost of migrating but more importantly functionality.

There were many questions, after the Gmail run through, regarding how you could use Microsoft Outlook with Gmail. Apparently you can through the IMAP interface but you don’t get the full functionality. Another question asked was regarding whether you can connect Google Sheets or any of the apps to back end databases or web based data services – the short answer was no, not out of the box. If you want to do something like that you have to delve into the programming (Javascript) API and possibly import your databases into Google’s cloud database service – Google Engine.

The viewpoint was that anything beyond the basic functionality was considered to be a separate self-contained application that you would need to create with programmers and application designers. This is in stark contrast to Microsoft’s approach where, for instance, a typical user may only use 5% of the functionality of Excel but, the difference is, that functionality is available straight away when they do come to need it. And invariably they don’t need to involve programmers or the IT department. Microsoft have also created self-service business intelligence portals into their office products – again providing further functionality that the users can tap into out of the box and enable the IT department to control access to backend systems. The reality is that most applications are not self-contained and have to fit into a wider company eco-system.

Imagine the scenario a short time after your company has embraced the Google cloud mantra and started solely using Google for Work: a user approaches the IT department requesting the kinds of functionality described above only to be told ‘well, you can’t really do that in Google for Work, you’ll have to use those old programmes that we were told were legacy and redundant and the old way of working or you could hire a bunch of programmers to create it from scratch for you’!

However, there were many good aspects as well – document collaboration and live editing by multiple users is impressive. Google Hangouts looks like an excellent tool for conferencing and remote support. The Vault with archiving and versioning offered excellent backup. These are the kinds of apps that fit naturally into the cloud but what Google fails to realise is that they also need to fit in with existing systems.

The fact is that most companies want systems that are as flexible as possible. Enlightened ones will realise that they should not embrace one technology completely replacing another. They will use the best tool for the job – that might be local apps on desktops and/or cloud apps on tablets. They will use Google apps as an extra tool not as a replacement.

Companies like Google need to cater for this reality. Gmail with full functionality via Outlook – yes please. Document collaboration, archiving and versioning with native Excel files stored on a companies’ local storage system – yes please. Google mapping for SQL Reporting Server – yes please. They need to offer extra functionality that fits seamlessly with existing software – Google need to embrace Microsoft technology and enhance it rather than trying to replace it.

BreakthePaywall now available in Google Chrome

BreakthePaywall is now available as a Google Chrome Extension.

Visit the Breakthepaywall home page: http://www.breakthepaywall.com

Or search in the Chrome store for ‘BreakthePaywall’

Or click this direct link: 

https://chrome.google.com/webstore/detail/breakthepaywall/ddlcmnkkbojfnfclfhbdfkcmhfaebhgc

The Chrome version utilises referer and agent hacks but does not perform any cookie deletion or other storage manipulation. However, this should be enough for most paywall websites.

The Extension adds a simple BTPW button to the Chrome toolbar which you can use to toggle BreakthePaywall on and off. When it’s coloured blue it’s on and grey is off.

Search Engines – what are they good for?

Absolutely nothing in some cases!

One increasingly well known fact is that paywalls – those barriers that some websites put in the way to try and get you to hand over money or register to gain access – are these days, very porous.

They’re designed that way so that they can charge for access, either by charging or selling your details, but do not close themselves off to the rest of the internet i.e. search engines.

They still want search engines to scan their site and therefore have to open themselves up to them whilst blocking anyone else. Google’s T&C’s even state that websites must not block access when referred from Google search results if they want to appear in those results.

That’s the loophole that apps like BreakthePaywall! (http://www.breakthepaywall.com) use to circumvent the paywalls – they impersonate a search engine.

For example: if you go to a site and notice an article that, when you click on it, displays a paywall blocking page or popup then if you copy that same article heading and paste it into a search engine then, more often than not, the link in the search result list to the same article, when clicked, will result in the article being displayed without any problems.

That’s because the website is seeing that you have come from a search engines results page – using the ‘Referer’ header that is sent by the linking website.

This is how paywalled websites keep themselves at the top of search results but also block people that have gone to the website directly and/or click on subsequent articles. This is usaully referred to as opening themselves up a little bit in order to gain search engine traction but still make money out of the ignorant. It must be rather galling for loyal customers who stump up annual subscription fees to discover that others can get it for free – hey, that’s the mad world of the internet!

Another technical aspect is that websites need to allow the search engine robots unfettered access. These robots go out onto the internet and scan websites and report back to the databases that collate the results. They have their own special names that websites can pick up and therefore allow access.

So, search engines are useful after all, but interestingly, not all of them…

Here’s a list of the top search engines in the world according to Wikipedia:

Search engine Market share in June 2014
Google 68.69%

 

Baidu 17.17%

 

Yahoo! 6.74%

 

Bing 6.22%

 

Excite 0.22%

 

Ask 0.13%

 

AOL 0.13%

 

https://en.wikipedia.org/wiki/Web_search_engine

As you would expect Google is way out in front. Experimenting with the method of copying a header, paste into search engine and see if the resulting link to the paywalled website works it’s not surprising that using Google as the search engine always works. Most websites want to be on Google’s search results so they allow full access from Google. But what about the other search engines:

Baidu – is of course the Chinese based search engine – try a paywalled western based news organisation  website on there and you don’t get anywhere. Consequently they don’t appear anywhere near the top of the search results (you have to put in the website domain to get anywhere near). Yes, it seems like some websites are not interested in a few billion chinese customers.

Yahoo next – again, no joy at all. It turns out that Yahoo doesn’t actually do its own search results. They currently use Microsoft Bing’s search results – they used to use Google up until 2004. So, it seems some sites have excluded Yahoo from their allow lists, which seems strange as it is the 3rd largest search provider.

Microsoft Bing – works!

Excite, Ask, AOL and any other smaller search engine – DuckDuckGo uses Bing as well as other sources for its search results – do not work, or rather, are ignored by paywalled websites.

So paywalled websites want to expose themselves and effectively give free access but don’t want to do it for everybody – maybe they are just as ignorant of what they are doing as their customers.

But it also highlights the fact that for non-Chinese internet users at least there is really only two search engines – Google and Bing. What a stitch up!

Scroogle is no more!

My favourite search engine Scroogle.org has died:

http://www.theregister.co.uk/2012/02/21/scroogle_dead/

http://www.betabeat.com/2012/02/21/scroogle-privacy-first-search-engine-shuts-down-for-good/

Quite why some hackers had it in for Scroogle I have no idea and who exactly were they??!

To be honest it had been going down so often, due to Google throttling, that I was thinking of looking for an alternative anyway. Here is an interesting take on alternatives:

http://www.betabeat.com/2012/02/21/scroogle-privacy-first-search-engine-shuts-down-for-good/

So use duckduckgo.com for your day to day searches as it has full privacy like Scroogle but if you must have Google search results, like Scroogle gave you, then use their encrypted version – Google will know what you are doing but at least no one else will.

The article above also has a link to instructions on integrating them into your browser but not for IE (especially IE9 which seems to make it difficult to add custom search engines). Use this page, despite it referring to IE7 this works for all IE versions including IE9: http://www.enhanceie.com/ie/searchbuilder.asp

By the way I have seen posts that warn against using duckduckgo.com as it is owned by Amazon – that is not the case, they simply use the Amazon EC3 cloud service to host their servers. Again use the https version and you should be fine – at least as private as you can be these days!

Anonymous searching

Came across some recent tech news items regarding Scroogle this week – http://www.scroogle.org/cgi-bin/scraper.htm – don’t use .com, that’s a porn site! Apparently Google has switched off a search scraping service which allows you to use Google search results through your own interface. This was apparently due to Google dropping support for IE6 – how these two relate I don’t know.

This caused Scroogle to stop working but it piqued my interest enough to investigate further, especially when a a few days later Google re-instated the service and Scroogle was back up and running.

Scroogle is a means of obtaining Google search results anonymously – they use various techniques like IP obsfuscation and cookie manipulation to make sure Google cannot track your searches i.e. cannot link it with your Google account etc. Scroogle delete all the info they collate within 48hrs of your search and you don’t get Google advertisements on your search results page – it feels just like the early days of Google, no clutter, just basic results. Marvellous! So I started using it and have not looked back.

They provide a helpful set of instructions for adding Scroogle to the IE search provider list which enables you to change the default search engine and the search engine box, at top right of IE, to Scroogle:

1. Go to Microsoft’s add search provider page: http://www.microsoft.com/windows/ie/searchguide/en-en/default.mspx

2. Paste in the test search link to the URL box: http://www.scroogle.org/cgi-bin/nbbw.cgi?Gw=TEST

Microsoft Search Provider Web Page

3. Type in ‘Scroogle’ for the name.

4. Hit the Install button.

5. Tick to make default search provider and hit Add button.

 

Adding Scroogle Dialog

That’s it – your default search provider is now Scroogle – you should now see a red, green and black crossed through ‘G’ icon in your search box.

The only problem I have found is that you do not get the ‘Cache’ link appearing on the results for the Google cached content – this is sometimes useful when trying to access cloaked content i.e. Experts-Exchange.com, but otherwise I have found the service faultless.

At the time Scroogle went down they were advising people to use another alternative anonymous search engine which you may want to check out as well:

http://www.ixquick.com/

And they have a page for adding it to IE here: http://us2.ixquick.com/eng/download-ixquick-plugin.html

Setting up google search on wordpress

When setting up this blog I hit a brick wall when trying to implement site search.

It quickly became apparent that the out of the box solution was not good enough – didn’t work for a start! – and that implementing search using Google’s custom search facility would be the best and easiest to implement solution.

You can create a Google custom search facility for any website. It gives you the code for the search form and results page which you can just plug into your website code. But for WordPress content management systems I found you have to go through a few hoops – particularly if you are a novice, like I was.

Firstly create a Google Custom Search account – as with all Google sites this is different to a normal Google account. Goto: http://www.google.co.uk/cse/ and hit the ‘Create a Custom Search Engine’ button and then hit ‘create an account now’ if you do not have one already.

Once logged in select create new search engine and fill out the form (just standard, you don’t want to pay for the Business edition unless you don’t want ads – you can monetize ads to your adsense account later on):

Google Custom Search Engine Creation

Once setup goto Look and Feel on the left and select Iframe for the hosting option:

Google Custom Search Engine Setup Look and Feel

Then hit the Get Code button and fill in the URL path and where you want the ads to appear. The URL path is the Permalink for the search results page that you will define later in WordPress, as you type it will automatically be inserted into the code. In our case we use Numeric Permalinks (changed from default Page ID method under Settings, Permalinks in WordPress admin). Putting ads on the Top and Bottom gives you a minimum width of 500px – you can change the default width of 600 after you have copied the code, having ads on the right limits you to 735px which is too wide for our site:

Google Custom Search Engine Setup Get Code

You can now copy the code for the search form to the default WordPress search form – in WordPress admin goto Appearance, Editor and choose Searchform.php. (If no Searchform.php exists in your theme you can copy it from the default – just copy it to your wp-contentthemesyour-theme folder). Overwrite the default code:

Wordpress Search Setup Search.php

To get the form to appear on your pages you need to edit the Appearance, Widgets – drag and drop the search option to the right hand side. This will then appear on your sidebar:

Wordpress Search Setup Search Widget

Now at this point you would think all I have to do is copy the Google Search Results code to the Search.php file in the same way but apparently this doesn’t work – search.php seems to be reserved for the default search process. You need to create a new page – in your wp-contentthemesyour-theme folder copy page.php (again if you don’t have one copy from default) to a new file – we chose Googlesearch.php

Once created this file will be listed in Appearance, Editor.

 

  • Insert the following code at the top (this will identify it as a template file):

 

 

<?php
/*
Template Name: Google Search Page
*/

    • Copy the Google Results Page code after the PHP Content command.

 

You should now have something like this:

Wordpress Search Setup GoogleSearch.php

Next create a new page under Pages, we named it Search. This should match the Permalink URL path you entered in the Google code setup. Select the Google Search template you created above from the template option on the right:

Wordpress Search Setup Search Page and Permalink

Note the Permalink at the top – same URL as you entered in the Google Search code setup page.

That’s it! You should now have a Search Form on your sidebar which loads a search results page (/index.php/search in our case).

To monetize the search result ads select Make Money from the Google Custom Search Engine control panel and follow the instructions to associate your search engine with your Adsense account.