The Beast With 1,000 Heads

Share Button

Hi Team,

After my last blog post on the new clickable results arrows inside Market Samurai, I wanted to give you some insights into search engines that will save you a lot of unnecessary confusion and frustration – and give you a better understanding of the way search engines work.

Now that you’re getting closer to the raw data sources, and doing deeper analysis, it’s important that you know about the way that Google, Yahoo and other search engines keep their data.

Otherwise, some of their “quirks” can cause confusion.

The Beast with 1,000 Heads…

Most people believe that Google is a single mega-computer.

This is untrue.

In fact, it’s a series of perhaps thousands of computers (servers) – all working together to provide Google users with results for their search queries.

Keeping thousands of computers up-to-date with all of the changes that are happening online is a mammoth task!

These thousands of servers are constantly being updated with new information – but it doesn’t all happen at once.

This means, depending on which server Google gets your results from, you can see slightly different results server-to-server.

Which Data Source is Correct?

It can be confusing for some people, when they see an SEO Competitor count of [say] 104,000 for a keyword inside Market Samurai, and 102,000 when doing a search manually on the same keyword – in “quotes”, just like Market Samurai does.

They ask “Which data source is correct?” or start to wonder if they can trust Market Samurai results since they’re different to what they see manually.

It can be confusing for people if they don’t understand the way servers are structured – but the truth is simple if you know…

Both numbers are accurate.

They both come from the same data source (Google) – and an end user is just as likely to see either.

When you’re doing analysis, it’s rare that the numbers will be substantially different (normally they’ll be fairly close to one-another if there is ANY difference at all.) So it shouldn’t make too much difference to any analysis.

However, if you have concerns, it’s wise to use the more conservative number – the number that works least in your favor.

And now that you know this little tit-bit about how search engines operate, you now know more about the technology behind search engines than many professional SEO’s. :)

Brent

Brent Hodgson a co-founder of Noble Samurai, and an internet marketing specialist.

Brent has written 68 post(s) for Noble Samurai

25 Responses to “The Beast With 1,000 Heads”

  1. 1
    On June 10th, 2009 at 1:56 pm
    Raza Imam said:

    I’ve been wondering why results in Google’s Keyword Tool and their Search Based Keyword Tool are so off.

    For example, if you type “strength training” in the Keyword Tool, you’ll see 74,000 monthly searches. But if you type the same phrase in the SBKT, you’ll see a few thousand at most. What’s the deal?

    Why the huge variance and which one can you trust?

    Raza

  2. 2
    On June 10th, 2009 at 1:58 pm
    Raza Imam said:

    I’m sorry, my numbers were wrong… Searching “strength training” in the Keyword Tool will say there were 450,000 monthly searches. But it will show a few thousand in the SBKT. Why?

    Thanks,
    Raza

  3. I believe this is to do with the way data is analyzed.

    It’s not uncommon to see Google Trends or Google Insights to have slightly different results either.

    If you consider the huge load of computer power it would take to analyze billions of keywords, and the huge expense that would involve, it’s likely that Google is analyzing the results to [say] a 99% certainty.

    That means most results will be very close to accurate, but [in this case] 1% of results will be out by a margin of error.

    We use the Google Synonym Tool (the Keyword Tool, not the Search Based Keyword Tool) for traffic data. Our analysis has found it to be the most accurate source of data available today (which is why we use it above any other tool).

    Brent

  4. Thanks for clarifying. I even emailed Google to ask why there was such a HUGE variance (3,000 vs. 74,000) for the exact same keyword phrase and their answer was not very clear. They didn’t really explain the difference so I naturally relied on the Synonym/Keyword tool rather than the SBKT, until I read your posts about LSI, uncovering niche ideas, etc. using the SBKT.

    I’m probably not the only one that feels this way, but I always look forward to your posts. I have you in my RSS reader on my Blackberry and usually read your posts more than once.

    You should keep your personal blog up to date as well, there’s some real gold on it too.

    Best,
    Raza

  5. I also thank you very much for the clarification.
    I am another who has wondered.
    Victoria

  6. Hi brent, would it be possible to make the estimates more accurates for local searches ?

    for example i work a lot with the ITALIAN language inside the Italian country.

    Your estimates are based on this kind of query (even when my project is set to italy/italian ):

    http://www.google.com/search?q=allintitle%3A%2Bb%26b%20%2Broma%20%2Bcentro&btnG=Search&meta=15&filter=0&num=10&start=0&safe=off&gl=IT&hl=en&cr=countryIT

    but i get more accurate results this way

    http://www.google.com/search?q=allintitle%3A%2Bb%26b%20%2Broma%20%2Bcentro&btnG=Search&meta=15&filter=0&num=10&start=0&safe=off&gl=IT&hl=IT&cr=countryIT

    Thanks for this great software!

  7. The other option of course would be to run an Adwords campaign for a few days then extrapolate the data for the whole month to see if it compares to the keyword tools. In my experience the synonym tool is definitely closer to the real numbers.

    Thanks for the post
    Michael

  8. [...] too much about that. Here is a great Blog post by Brent from Noble Samurai explaining the above: The Beast With 1,000 Heads | Noble Samurai Conclusion > Use the MS stats and numbers as a guide, but always do further research as Caro [...]

  9. [...] that people have begun noticing this though) Here’s a blog post explaining how Google works: The Beast With 1,000 Heads | Noble Samurai Tip: If you click on the arrow beside the result, you’ll almost always get an identical result [...]

  10. [...] keyword on 4 different servers. Here’s a blog post I wrote a few weeks ago about this EXACT issue: The Beast With 1,000 Heads | Noble Samurai Here’s an extract: [...]

  11. [...] a look at this great Market Samurai blog post on data source confusion by Brent Hodgson The Beast With 1,000 Heads | Noble Samurai May help explain the data inconsistencies that crop up all the time! Bill __________________ [...]

  12. [...] once in the SEO Competition module, once with Rank Tracker all hit 3 different data centers – The Beast With 1,000 Heads | Noble Samurai ) __________________ Brent Hodgson Market Samurai team Last edited by BrentH; Today at [...]

  13. [...] at Google to the one you connect to when you search manually. (Here’s a post that explains it all: The Beast With 1,000 Heads | Noble Samurai ) So you can get slightly different results in Market Samurai – but both the results you see [...]

  14. [...] The other is changes in Google’s own data itself – and these changes are impossible to avoid: The Beast With 1,000 Heads | Noble Samurai Brent __________________ Brent Hodgson Market Samurai [...]

  15. [...] expire, and it was re-registered by Palo during the 30 Day Challenge. daiquiri – is this helpful?: The Beast With 1,000 Heads | Noble Samurai Brent __________________ Brent Hodgson Market Samurai [...]

  16. Why does Market Samurai say that my website is not in DMOZ and YAHOO Directory when it is – I have checked several times.

    Today for the first time ti told me I was in DMOZ but not YAHOO for one keyword but not another?

    Could this be a case of MS talking to one google server that has not been updated with my DMOZ and YAHOO listings? I have been in both directories for a number of years, I wouldn’t have thought it would take google to update the info across the servers.

    On
    Brent Hodgson replied:

    Hi Raymond,

    I’m looking at the site you left in your comment now – and I see Market Samurai reporting that it IS in DMOZ, but not in the Yahoo Directory.

    When I check manually, I see that this is indeed the case.

    So it looks like everything’s working fine.

    There IS a .com version of your .co.uk site (same name, just ends with .com, not .co.uk) that is NOT listed in the Yahoo Directory OR DMOZ. Perhaps this is what has caused confusion?

    Whatever the cause – if you’d like us to have a look into this in detail, make sure you contact us via support:

    http://www.noblesamurai.com/support/helpdesk/

    If you’re still having trouble – give us the details of the URL you’re looking at, and a screenshot (if you can), we can look into this for you and find the answers.

    I hope this helps you out.

    Brent

  17. Hi, thanks for that, my website is http://www.polycrete.co.uk NOT .com. Although there is a .com site this is nothing to do with me or my company.

    I have a screenshot of a previous MS SEOC analysis which shows the polycrete.co.uk site as NOT being in DMOZ and I have been frustrated to see this every time that I have used MS, except today for the first time it DID confirm the site was in DMOZ. i also thought it was in Yahoo but today when I checked, it is not. Could it have been de-listed? or perhaps I was mistaken. I have tried o submit the site to Yahoo today but not sure if my submission went through as I just got a blank screen after I clicked submit. It also said it would charge me $299 for the submission which I know I have never paid before, have they always charged? I thought that charging was just for an expedited submission for premium members. It s a bit of a mystery as to why MS reported my site not being in DMOZ when it was – I also have a screenshot of the DMOZ site confirming the entry which was taken at the same time – many weeks ago – when MS said the site was not listed.

    There are many other inconsistencies, between different keywords for example, MS will tell me that I have 20 BLEG links for one keyword but none for another. I know exactly what the BLEG links are because I created them on dot edu sites and no time did I use a keyword in the link they were just links to the domain.

    Anyway thanks for the reply and if you have any further useful information , I would be grateful to hear it

    On
    Brent Hodgson replied:

    The BLEG’s – Are they two different pages ranking?

    Market Samurai will report the backlinks to a page for BLEG, rather than to a domain.

    It’s also important to remember that Market Samurai will also only ever report what it finds. It’s a machine, or a robot – and its function is to gather information faster than anyone can do manually (from the same sources that you’d use manually) – and report it back to you.

    (You can click on the arrows beside most results inside Market Samurai to be taken to the raw source)

    If search engine numbers change, or are reported differently, then Market Samurai will see different numbers – but Market Samurai has no capacity to create the numbers itself.

    I mention this because it’s one of the most common reasons why people receive unexpected results. (The other reason is mis-targeting results – e.g. looking at results targeted to “United States” or Australia” for example, instead of “All Countries and Territories” – but again, this is just Market Samurai reporting what it sees)

    We’re always happy to look into any discrepancies that you spot – and see if we can trace the issue to a cause.

    But please – make sure you use support:

    http://www.noblesamurai.com/support/helpdesk/

    I don’t always catch every question left in blog comments – and people tend to get annoyed when they don’t get a response.

    However, the support area is always staffed – and we respond to all support tickets that come in.

    Brent

  18. [...] to find more threads. Here is a post from Brent of Noble Samurai on their own blog addressing it: The Beast With 1,000 Heads | Noble Samurai In either case, 30DC recommends your SEOC be <30,000, so the example KW in your question [...]

  19. That is why it is so important to know you methodology.
    If in doubt check the data in multiple sources.

    In my experience market samurai is very close to the manual results other software i have used has had much wider discrepancies

  20. Hello. I have a question in regards to the SEOC Discrepancies.Why when you search for SEOC by putting the keyword in quotations does it pull it’s data from (lets use the keyword car brake parts)

    http://www.google.com/#hl=en&source=hp&q=%27%27car+brake+parts%27%27&aq=f&aqi=&oq=&fp=fceeb2705a2dd16

    Which says there are 11 million results

    However, if you click on the SEOC google link in Marlet Samurai, it goes to this URL Which says 4220 results.

    http://www.google.com/search?q=%22car%20brake%20parts%22&btnG=Search&meta=15&filter=0&num=10&start=0&safe=off&gl=US&hl=en

    Now, you can do any search with this URL and replace the car brake parts with your phrase and the results will match market samurai.

    Why is that?

    What kind of search is Market Samurai doing?
    What is the meta=15 mean?????

    On
    Brent Hodgson replied:

    I just checked these two sets of results and found 4,170 for both.

    The reason why?… I don’t want to repeat it – but it’s the exact issue addressed in this blog post (above). Make sure you re-read the post because it has the answers you need.

    I hope this helps you out.

  21. 21
    On October 14th, 2009 at 5:58 am
    Ken Morgan said:

    I am quite often getting SEOTC greater than SEOC which makes no sense. If I go to Google I may get a similar result, yet if I page through to the end of the listings (including similar pages excluded at first), the number of results is always less than SEOC and so consistent. If the problem is different data centers, how come this method always seems to give a reasonable result whereas the number on the first page of the results can be 1000s of times greater?

    On
    Brent Hodgson replied:

    You’re right – SEOTC will sometimes get a higher figure than SEOC.

    That’s because the way SEOTC is gathered, it is looking for all of the words in the title (not necessarily in the phrase order) – but SEOC is looking for phrases appearing on websites – two slightly different things.

    For the response to your 2nd question (why does the figure change when you get to the last page of results), check out this:
    http://www.thirtydaychallenge.com/forums/general-chat/14681-interesting-search-result-phenomenon-google-guru.html

    Brent