Add handling for visits referred by Google with no search query provided

Hello!!

I thought this might be helpful for some who are struggling with "not provided" and NetInsight. Please know that there are multiple steps that can be taken to mitigate against the issue, and I will be posting more over time... just didn't see a reason to hold this helpful tidbit back.

While Google is masking search terms from landing sites, They are making an effort to provide insight into the fact that the visitor did come from Google search by setting a referrer that is equal to https://www.google.com (or https://www.google.co.uk, etc.). This has also been handled by removing the q= parameter value from the URL while leaving everything else intact, which will be a follow up either here or at tbdac.com.


What this JS modification does is pushes a "?q=not provided" (a la Google Analytics) query string on the referring URL if the document.referrer is equal to http://SOMEGOOGLEDOMAIN/ or  https://SOMEGOOGLEDOMAIN/, with nothing after the first slash following the domain... where SOMEGOOGLEDOMAIN is www.google.something (e.g. www.google.com, www.google.com.mx, www.google.co.uk, etc.)

var rx = new RegExp("https?://www.google.[^/]+/$");

if (rx.test(document.referrer)) {

NTPT_GLBLEXTRA += "&rf=" + encodeURIComponent(document.referrer) + "%3Fq%3Dnot%20provided";

}

Depending on your implementation, this type of JS snippet should be added immediately to the end of the ntpagetag.js... or to some other custom JS include that is called shortly after ntpagetag.js (it is dependent on NTPT_GLBLEXTRA having been set by ntpagetag.js). For vanilla implementations (where the page tag request occurs as the JS loads) you may need to move to something that allows for more customization by splitting the JS load from the page tag image request... see NTPT_NOINITIALTAG.

I realize this could be rewritten to be more fault tolerant. For example, we could test existence of NTPT_GLBLEXTRA... but if this is being called in tandem with ntpagetag.js that should not be necessary.

I'm open to any other suggestions/tips if anyone has them! Please feel free to write back with any questions or comments.

Views: 367

Reply to This

Replies to This Discussion

Hi, 

I seem to have a simple alternative for the same. 

Add the below to your Search & Replace with other required changes.

<urlsearchrule type="referrer" search="(\&amp;q\=\&amp;)" replace="&amp;q=GUser-Keyword-N/A&amp;" name="Google Loggedin">

                        <member method="contains">google.</member>

                </urlsearchrule>

Regards, 

Naveen.KK

Oh .. perfect!  I was looking for this exactly for a client who hasn't caught up with this change yet!  

Naveen I was leaning towards the URL search & replace idea as well.  I will give this a try!  IF anyone has updates or modifications to this let me know.

Elizabeth

That's great, Naveen! Not sure why I had it in my head that it couldn't be done with URL S&R, unless we simply elected to skip it in favor of a retroactive fix. 

Naveen,

If you have any more insight into exactly how your formatted the search & replace strings that would be be helpful.  I tested it as you propose but so far have not been able to get it to work.  The NI administrator's guide also says not to use encoded URL formats for the URL search & replace filters.. but I still can't seem to get it right yet.

Elizabeth

Here you go (had to test it to see!) -- this fixes one, the original post here talks about another that could be addressed... I'm not sure of the numbers coming in today on that one, where the referring URL doesn't have the empty q= parameter, in fact there is no query string at all (this was mostly an IOS issue, I believe). It would be interesting to hear how many Google referrals you are seeing that aren't tracked with a keyword, even after fixing this q=& issue.


Thanks Todd!  I had the regex right , but did not have the encoding set up for the replacement value.  Perfect I will set this up.

Here you go (had to test it to see!) -- this fixes one, the original post here talks about another that could be addressed... I'm not sure of the numbers coming in today on that one, where the referring URL doesn't have the empty q= parameter, in fact there is no query string at all (this was mostly an IOS issue, I believe). It would be interesting to hear how many Google referrals you are seeing that aren't tracked with a keyword, even after fixing this q=& issue.

Todd you are right -- we are only able to strip the q= parameter when the traffic passes through the redirect page google.com/url.  I notice Chrome traffic is not passed through the redirect.  The referrer in that case is simply http://www.google.com.  There is some discussion about this Google 'feature' here:

http://www.webmasterworld.com/google/4431397.htm

I was trying to figure out a way to distinguish https Google traffic as its own 'referrer group' in the reporting interface.  Then we could at least make the assumption it was all hidden organic search traffic.

I'm shocked this has to be so complicated -- so far reaching out to IBM has not lead to any tips on how to modify the tool to deal with this.  Out goal is rather straight-forward.. trend organic traffic over time, but the setting we have in place to do this so far cannot handle Google's change in not passing entry keywords.  The 'not provided' gets us closer.. but not all the way there yet.

I need a beer in front of me before I am willing to talk about IBM's handling of NetInsight. Sorry to hear, but take solace in the fact that you aren't being treated any differently than anyone else. 

In a separate post yesterday, I had provided a few alternatives... one of them being a UDP. This is not a bad way to approach things, but there is a challenge in that you must first have a "not provided" keyword in the database for it to work. Then, it works retroactively. So, if you've implemented the rule we were talking about above, and then you added this UDP (as a post-import UDP) to the profile... all of the history would be fixed, too:

update %profilename%_visits set keywordsid = (select keywordsid from %profilename%_keywordsid where keywords = 'not provided') where referrerbreakdownid in (select referrerbreakdownid from test_refbrkdwnid where referrerbreakdown like '%&q=&%' or referrerbreakdown like '%?q=&' or regexp_like(referrerbreakdown, 'https?://www.google.[^/]+/(search)?$')) and keywordsid = 0 and referrerid = (select referrerid from %profilename%_refid where referrer = 'Google');

That UDP would correct:

http://www.google.com

https://www.google.com

http://www.google.com/search

https://www.google.com/search

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&sou...

I will post full implementation details on tbdac.com before the end of the week... in case UDPs are a mystery as there is no UI for them and I'm not sure they are documented. Pretty NetInsight-y.

I will try and wrap these comments and threads into a single post still, but wanted to get you this UDP information since it does seem like it works well.

In case you have not worked with a UDP... you need access to the NetInsight installation to edit files.

Create a "udp" directory under your NetInsight program home directory, and place a "notprovided.udp" text file there that contains this: 

update %profilename%_visits set keywordsid = (select keywordsid from %profilename%_keywordsid where keywords = 'not provided') where referrerbreakdownid in (select referrerbreakdownid from test_refbrkdwnid where referrerbreakdown like '%&q=&%' or referrerbreakdown like '%?q=&' or regexp_like(referrerbreakdown, 'https?://www.google.[^/]+/(search)?$')) and keywordsid = 0 and referrerid = (select referrerid from %profilename%_refid where referrer = 'Google');

(replace %profilename% for the short name of the profile in question)

Add a <userdefinedprocesses> section, or a <process> to an existing userdefinedprocesses section in your profile config:

<userdefinedprocesses>
<process phase="postimport" type="sql">./udp/notprovided.udp</process>
</userdefinedprocesses>

If you've already put the rule from above in place, this UDP will now retroactively correct history. If you do not want to use S&R rules, you can use the UDP after you first insert a "not provided" record into the keywords table and update the misc table so that NetInsight is aware that you've done that... let me know if you have questions on that.

Thanks -- we'll have to buy you that beer after your help with this!  I am passing these instruction onto the data architect who will be reprocessing the last year of data.  I am not familiar with UDP but I am sure her his. 

Meanwhile, I'll document what I doing on the reporting side to clean up metrics for Organic Search in NetInsight, in case anyone else is looking for tips.  The existing metric was defined as "contains entry keywords, does not include paid keywords", which of course no longer suffices.  I renamed that metric Organic Search (basic).  It now includes the (not provided) keywords we could grab from google.com/url thanks to the S&R filter.  I created another calculated metrics ('Google https') for https google referrals (looking for referrer breakdown starts with https://google. (AND where entry keywords are NOT available and paid keywords are NOT available) to capture the Google referrers where there is no q= parameter at all, but to avoid double counting any of the search traffic in our other buckets. I created a calculated metric to combine those two,which is our new total for Organic Search.  For Google this total Organic Search metric, plus our paid search traffic, gets us to 98.5% of all visits from Google.  The remainder are the random other referrals (images etc, which are excluded from our Organic Search metric). 

I would also caution that the exclusion list should be checked.  I noticed that a while back the google.com/url URL had to be deleted from this URL (it was included by NI by default).  The client I am working this still had this in place.

Thanks!

Hi Todd

I’m responding on your post re the UDP solution for the Google Keywords not provided.
We are using SQL Server and after trying this suggested solution the Netinsight application returned an error
[Microsoft][ODBC SQL Server Driver][SQL Server]’regexp_like’ is not a recognised built-in function name. (42000)

When searching we are advised that SQL Server 2008 does not support regular expressions directly, but you can use CLR.

Do you have a solution for SQL Server.
I would be very grateful for your help
regards
Ali

Reply to Discussion

RSS

© 2017   Created by Wendy Ertter.   Powered by

Badges  |  Report an Issue  |  Terms of Service