Live Webcast: Setting Standards and Making Sense of Web Analytics

Hi guys! I know its Friday and Fridays are usually reserved for talk about puppies and boys and baseball (Sox! W00t!), but when I heard that the Web Analytics Association was giving a Webcast to discuss industry standards and the recently released WAA Standards Analytics Definitions Volume I, I thought it was worth liveblogging. So you can read all about that below. And don’t worry, the puppies and boys and baseball-related stuff will come later (Sox! W00t!).

Introductions all around. Introductions. Today’s panel of experts includes Jason Burby, Avinash Kaushik, and Angie Brown. Brian Induni will be acting as moderator.

Understanding Web Analytics

Jason Burby is up first to talk about the importance of understanding Web analytics. Analytics experts need to understand what is going on in order to make improvements. Marketers also need to understand what is going on in their business. What the WAA committee is finding is that these people are speaking two different languages to the determinant of their sites and their business. How can we better educate people and get them to speak the same language?

You have to break down the barriers. There are more people in Web analytics, with greater sophistication in tools and Web analytics. There are also many more people wanting to look at Web analytics data that aren’t currently, and won’t ever be, Web analysts. There are too many silos of data and insight (Behavioral, attitudinal, competitive, social networking, transaction, etc). Combing this information together allows you get better results, but in order to do this everyone needs to be speaking the same language

Jason identifies five core problems or "threats of confusion". They are the misinterpretation of data (people looking at the same data points and getting different results), wrong decisions being made as a result, not leveraging all available insight, a lack of trust in people and data, and no action.

By removing these threats, businesses will be able to make smarter decisions, better understand their site visitors, improve the overall site experience, impact offline decision, increase ROI and improve resource allocation.

Those are just some of the things the committee has been talking about in terms of clarity. It’s all about getting people to speak the same language regardless of who they are – an analyst, a marketer, etc. They want to meet all audiences.

What Makes a Great Metric?

Next up is Avinash to talk about what makes great KPIs. Avinash says he wants to share some of the learning and perspectives of things he’s had and to "demystify" the myths related to metrics.

What makes a great metric? One of the beautiful things about the Internet is that you can put a piece of JavaScript on a page and 16 seconds later you’ll have a million different kinds of data. But what are the important metrics to help you understand your business?

He applies four best practices to each of the metrics that people have or would like to have to see if it passes the "sniff test".

According to Avinash, great metrics are:

1. Uncomplex: He’s trying to rank for the term "uncomplexify". It’s common for people in our industry to take five different metrics and add/subtract/multiply them, turn it into mush, and then try to make it an index. As you consider what metrics are important for your business, the first thing you have to do is "uncomplexify" it. If your metrics cannot pass the Occam’s Razor test then you don’t have the makings of a great metrics.
2. Relevant to you: Even though Best Buy and Circuit City are the same type of business, the key metrics for them are radically different. This is the one thing he wants people to understand and apply. People have a tendency of researching other people’s metrics and then applying them to their business. Resist this tendency. Don’t take metrics that work for other people and simply slap it on your business. You have to find the metrics that work for you.
3. Timely: Get information back in a timely manner. For the most part, if you want to execute data driven process, it’s important to know that most of your metrics should be available in a timely fashion so that you can understand what is going on. You don’t want to wait 16 days or 16 weeks to get them.
4. Instantly Useful: You are the analyst. You understand everything. And yet, when you send a dashboard, the person at the other end knows none of that. If you display great dashboards, try and apply the instantly useful test. This means that as soon as people look at the numbers, they should instantly know what it means and what action they need to take. You should be able to look at a metric, know what the number is and if it’s good or bad. Instantly useful means it drives action.

An example of a great metric is bounce rate.

It’s uncomplex, easy to understand (bounce rate is people who only have one page view in their session. The geniuses understand that and so does your CEO.), it’s relevant in any business because it helps you locate the crap (hee!), it’s timely and standard in all tools, and it’s instantly useful.

Avinash sums this up by sharing three lessons learned from a tough life.

1. What works for Jack…might not work for Jane: You are unique and your business is unique.
2. Perfection is…the enemy of good enough: If you have 90 percent confidence in your data, move on. It’s not worth it for you to get the last few percentage points. Embrace incompleteness.
3. Critical few, baby, critical few: When the crap hits the fan, what are the metrics that are most important to your business? At the end of the day, there are probably only 1-3 metrics that will indicate the key success of your business. You should know them.
4. You have one best friend – The Metrics lifestyle process. Define a metric, measure it, analyze it, take action against it, then improve or eliminate it and start the process again. Good companies have at least a 30 percent turn in metrics every year. If your metrics are the same every year, you’re probably not thinking hard enough about your visit.

The Web Analytics Association’s Mission

Jason is back up to discuss the intent behind the WAA creating standard definitions for Web analytics. They wanted to:

• Define common Web analytics terms – Easy to understand both by marketers and Web analytics professional.
• Balance the need for specific definitions while offering flexibility based on changing technologies and individual company needs.
• Help people "speak the same language".
• Definitions created by a diverse group of people, not just a few experts.

The new definitions that have been released are a work in progress. It’s a starting point to help people speak the same language. They’re building the framework.

An Update on the WAA Committee

Angie Brown is up to give us an update on the WAA standards committee.

The status there is that they released 26 definitions back in August. What they’re doing right now is reviewing the feedback they’ve received from the Web analytics community to see if they need to rework or clarify things. They’re trying to add new definitions as time permits. The new version of the document will be focused on improving things, not necessary adding new terms.

Angie explains that in the definitions document, each definition is broken down into different parts:

Type – count, ratio, KPI, dimension
Universe – aggregate, segmented, individual
Definition – meant to be meaningful even if read by itself.
Comments – clarifies the definition and/or describes where customization is often available.

As has been mentioned earlier, the definitions are needed to help people speak the same language. Definitions lay the groundwork for methodologies. They recognize that some conflicting methodologies can be useful to different stakeholders. The key is to be clear about which concept is being reported. Your metrics need to be relevant to your particular business. In some cases your tool will allow you to customize things for your business. You need to be clear what concept is being reported on. If you’re reporting visits, it should be called visits, not visitors or elephants.

Angie offers up some examples of conflicting methodologies:

Question: Should a count of page views include PDF requests?

Angie: That’s up the analyst and to how the business thinks of these types of content. Could also be events if you prefer.

Q: How long can a visit remain active before being cut off and considered "done"? Until the end of day? 8 hours? 12 hours? 24 hours? Forever?

A: That’s an open issue. The tools available to use right now are all over the place. Realistically, one cut-off is probably no less useful than another. In fact, if you have sessions that last for too many hours, you should probable revisit your spider/bot filtering scheme.

Q: If I land on the home page of your site, refresh the page in my browser, and then leave, will the activity count as a bounce? Will my visit be included in the single page visit count?

A: Your activity is a single page visit (ie only one page was viewed, even though it was viewed twice). However, it is not a bounce (aka "single page view visit).

She mentions that the committee will probably issue a new release of these terms by the end of the year. Some of the 26 terms will be updated based on feedback. They’re looking to publish quarterly. There’s a chance for a possible Web 2.0 subcommittee. Opportunity to establish standards in parallel with technology invention instead of after the fact.

Question/Answer Session

Q: Are there any efforts going on by the WAA to translate its analytics terminology into other languages?

Jason: That is not something we’ve explored, but it is something he thinks they should do. They’ll bring that up at their next meeting. If you’re bilingual and want to help with that, let them know.

Q: Can you answer why analysis from a log file state tool differs from analysis from page tagging?

A: There could be a whole host of reasons why the data will be differently. Think of it as the way you fundamentally collect the data is dramatically different. It could be due to cashing, session-ization, cookies, the way they deal with people turning JavaScript/cookies on and off, etc. Those are just some quick reasons. If you want more info, search for Web analytics in Wikipedia. It’s a very complex topic.

Q: What kind of measurement standards are being developed for AJAX, Web gadgets, etc?

Angie: That’s one of the things we’re looking into right now. Members have expressed interest in forming a subcommittee.

Avinash: Web 2.0 as a term is very fluid in regards to what it means and what it’s trying to accomplish. We’re trying to figure out how to get ahead of the train instead of getting to the party after the fact.

Q: How do you handle AJAX in terms of page views?

Jason: We’ve talked about that in the page views definition itself. We tried to structure it to allow flexibility for the person/company to define what they consider a page view. It’s something we’ve begun to discuss but we need to spend more time on it.

Q: There are many different kinds of Web sites out there. What are some of the top KPIs to measure for content/information Web sites?

Avinash: The definitions document that was released actually had metrics broken down based on different types of metrics.

Jason: There are three types of content sites — ad based, subscription based, engagement based. Even within that it differs significantly.

Q: How would one measure online viral marketing in something like SecondLife?

Avinash: I think what people are doing is still trying to figure out how to hack at some of the ways to track Web pages. There aren’t any standards. There aren’t a lot of people who have figured out what SecondLife is or how to take advantage of it. It’s very much virgin territory for now. The way that viral marketing is being tracked is that it depends. If you’re sending out an HTML rich email and it’s telling you to forward to the Nigerians to send you money, you can tag it. Other viral campaigns are things like YouTube where they’re encoding events into the action or video to allow the event logs to capture it that way. You still have to decide what your campaign is after and then use the standard ways.

Jason: We’re looking at what people are doing once they come back to your site. How do their behaviors differ when they’re coming from a site like Facebook? Look at their search queries, where they were, what they did and then tune the site experience based on that. Those people perform an act very different than your "average visitor".

Angie: The WAA does have a social media committee.

Q: Now that there are standards being created and generated, what do you think the chances of adoption are?

Avinash: I’m very optimistic. Many of the vendors have already reached out to the committee and are referencing the work of the committee. At eMetrics, several people from Google and Yahoo mentioned the standard definitions that were released. Now that we have created a baseline, we’re confident that we will get participating from the vendors. For better or for worse, it’s a great stick to have.

Thanks, guys! You can get a recording of the Webcast later today on the Web Analytics Association Web site. You should go and listen when it comes out, if only because Avinash is hilarious.

Lisa Barone is a writer, content marketer & VP of strategy at Overit Media. She's also a very active Twitterer, much to the dismay of the rest of the world.

See Lisa's author page for links to connect on social media.