If you’ve tried to launch and use Market Samurai in the last few days you’ll have noticed that there are large areas of functionality that are broken.
As you know, Market Samurai pulls data from dozens of different search data providers so that you have all the data you need in a single place to make the most informed decisions about your online marketing opportunities.
In order to obtain this data we often need to perform large numbers of queries and searches through free services such as Google, Yahoo! or Bing on behalf of our users.
For example, tracking the rankings for even a few keywords can require thousands of searches. By performing these queries on your behalf, using our servers, we saved you hours, days or sometimes even weeks of time compared with doing these queries by hand.
Several days ago, Google made some significant technical changes to its services that make it impossible to reliably perform large volumes of free queries.
As large portions of Market Samurai currently use Google’s service, this has temporarily broken a range of functionality in Market Samurai (and related products, including modules from Domain Samurai and Article Samurai – although the core syndication functionality in Article Samurai is unaffected).
It’s also worth noting that most other SEO software has been affected in some way by these changes. Edit: we had received reports that other software was affected.We have since discovered that this is incorrect and other SEO packages are not being impacted by this change.
What We’re Doing About It…
We’re working tirelessly around the clock to restore the vast majority of this functionality, and we estimate that most of the application will be functional within the next few days.
As there is such a large amount of functionality that is currently built upon Google’s services, we’re prioritizing the high usage modules such as Keyword Research and SEO Competition, and then moving on to the other modules.
What Are We Going to Use In Place of Google?
It’s important to note that much of the data that we supply in Market Samurai could have easily been provided from other search engines. (eg. locating articles about “dog training” in our Find Content module).
In these simple cases we’re going to just replace Google with Bing.
As you know, Bing has made some big improvements in recent years and there are many who are saying that Bing’s algorithm produces more relevant results than Google for many keywords.
As an SEO marketer, it is important to understand that Bing’s index is rapidly approaching the size of Google’s, and as they also power Yahoo!, they have rapidly grown to capture over 30% of the search queries in the US.
But why is this important?
The increased use of Bing as a search engine is important because competition metrics, such as SEOC (“SEO Competition”) and SEOTC (“SEO Title Competition”), measure the scarcity of keywords on the Internet. The idea behind keyword scarcity is that keywords that have a high scarcity are likely to be more useful keywords for your marketing campaign. A good candidate for a more detailed competition analysis, by using our SEO Competition module matrix, would be a keyword where there aren’t many pages on the Internet containing that keyword in the title or body of a page – i.e., keyword with a low SEOC/SEOTC value.
These high-level competition “scarcity” numbers can also come from another search engine, as long as the index is of a comparative size to Google. We believe this is the case with Bing.
It’s important to note that making a keyword decision based solely on a scarcity metric such as SEOC or SEOTC will provide poor results. It’s not a substitute for doing a detailed analysis by using the SEO Competition matrix and a backlink analysis.
The scarcity metrics help us quickly filter out keywords that are highly likely to be TOO hard to rank for (because they have a lot of relevant content out there competing for rankings). They DON’T tell you that a keyword will be easy to rank for. (i.e. use these numbers only to disqualify tough keywords)
Until this change is implemented, we’ve unfortunately had to introduce a manual check for SEOC values, similar to that for the SEOTC/SEOUC values. We understand that this is frustrating and time-consuming, but it was the only way we could keep this data available.
Changes to Rank Tracker
The module most affected by these Google changes will be the Rank Tracker module.
The Bad News…
With over 400,000 users now, tracking the ranks of your keywords on your behalf results in literally millions of Google searches every day. Google’s changes mean that it will be simply impossible for us to continue doing this on your behalf.
Unlike Bing and Yahoo!, Google has no API (commercial or otherwise) which is suitable for tracking rankings. The APIs that do exist for Google often produce results that are significantly different from actual rankings, making them useless for SEO purposes.
There are third-party companies that do have accurate APIs to Google, and we are currently in negotiations to get access to their data.
The particular company that we are talking to provides the ranking data to major SEO tools such as the excellent Raven Tools product, for example.
However, this data is expensive (Raven Tool’s cheapest plan costs $99 a month to track 1,000 keywords), and the underlying cost of ranking data is a large part of the reason for this price.
We’ve evaluated many options and it seems, at this stage, that the only way for us to offer Google Rank Tracker functionality to our users is to buy this data.
However, with a fixed-price product and so many users, it’s simply an economic impossibility to provide the same rank tracking functionality on an ongoing basis. In fact, it would cost us over $40,000 a day to purchase this data!
So, we’re left with the difficult decision of having to pass this costs on to our users in some fashion (most likely a monthly paid service).
Please know, if you’re an existing Market Samurai buyer we’ll try to keep this cost as low as possible. Potentially we will even try to offer a free level of service for a small number of keywords.
Unfortunately, this change will take a while to implement, at least several weeks. So until that time you may need to check your rankings manually, or perhaps try the 30 day trial at Raven Tools until we can get the new version of Rank Tracker out the door.
We’ll be releasing an updated version of the existing Rank Tracker module shortly, that will enable you to check a very small number of keyword rankings in Google (up to 10), by using your own connection rather than our servers to perform the searches. This is a temporary workaround until we can put a more permanent solution in place.
The Good News…
However, the news is not all bad. Rank Tracker has always been a very slow module to use due to the higher numbers of searches it requires.
The new Rank Tranker module built on the paid API will enable us to check your rankings in the background from our servers each week, so that when you open Market Samurai, you’ll instantly be able to see your rankings report—instead of waiting upwards of 10–30 minutes to check all your keyword rankings.
What To Do Next
We know the implications of these recent Google changes are very disappointing and will cause some anger amongst our users.
We understand this, and while we these changes are beyond our control, I want to apologize on behalf of the whole Noble Samurai team. We’re very grateful for the support of our customers over the years – you’re the reason we can do what we do! We deeply appreciate your patience through this difficult time.
Our current estimate is that we’ll have the vast majority of Market Samurai’s functionality working again within the next few days. The revised Rank Tracker module may take us several weeks to fully implement, though you can still check small numbers of keywords manually in Rank Tracker, as well as checking Bing and Yahoo! as normal.
Please be assured that as a small software development team, we understand that delivering great value is what keeps us alive and what built this business in the first place. When the dust has settled, we’re determined to do whatever it takes to ensure the investment you made to own Market Samurai is the best Internet Marketing investment you’ve made.
However, if at any time you don’t think Market Samurai is worth your initial investment please contact our help desk for a partial or full refund—just be aware that our help desk has a high volume of tickets at the moment, so it may take a little longer than usual to get to your requests.
Once again, I’m very sorry for the inconvenience that this has caused our customers.
We’re working around the clock to get this fixed.
Please leave any questions, concerns or encouragement (we could use a little bit right about now!) in the comments below.