Sampling and pop-up surveys

Introduction

Polls and surveys created with ViewsFlash are usually published on the web site in places where they can be seen and voted on as often as possible. There are times, however, where you may want to be as discreet as possible, such as when collecting marketing data. For example, you may want to ask visitors what they think about this section of the site and use that information to improve it. Or, you may want to know why they pressed the Cancel button instead of the Order Now button. You could even ask them, perhaps offering them an incentive, to provide you with personal information for demographic analysis or profiling.

In all of these uses, it may not be desirable, at least at the outset, to force every visitor to answer these questions. You can always invite them, by offering them a "take a survey" link or banner. Another effective method involves having a survey, or a survey invitation, pop up for some, but not all, visitors. ViewsFlash provides the capabilities needed to both pop up a survey, and most importantly, to present the survey to a statistically valid sample of visitors.

Sampling on the Internet has often been shortchanged, without sufficient care being paid to its fundamentals. First, ViewsFlash helps you decide how many samples you need to have a satisfactory confidence level, paying due attention to correcting sample size for small populations, which can affect the results of sites or sections of sites that are not heavily visited yet -- which is precisely one of the better times to ask people for feedback, for example. Second, ViewsFlash is careful to present the survey only once to each visitor, and most importantly, gives every visitor the same chance of being chosen as any other visitor, thus fulfilling the fundamental premise required for a representative sample.

Other software and services often miscalculate the sample size required when the number of visitors is small, typically claiming that size doesn't matter. More dangerously, they also present a survey every Nth page view, which will result in a sample heavily biased towards the views of frequent visitors, who will be much more likely to be chosen for the survey. While the views and habits of frequent visitors are important, any survey that tries to understand those views from all visitors to the site must be able to give equal consideration to infrequent visitors, whose views may be just as important to the growth of the site. Worse, not being aware of the significance of this sampling methodology can cause you to make decisions based on information that can be completely wrong, as the views and profiles of frequent visitors are likely to be significantly different from those visitors who are just trying out the site or use it not very often.

Methodology

Recapping, ViewsFlash's two objectives are to provide a sufficiently large sample, and to provide a representative random sample. The Calculate button on the Define Sample page calculates the correct sample size, given what you know about visitors. For situations where the number of visitors is small, this calculator can give very different answers from other sample size calculations which ignore the correction required for small populations. For example, here are the recommended sample sizes to be 95% confident that the answers to a question will be within 5% of the answers if everybody had been surveyed instead of asking only the people in the sample.

As expected, the correction is negligible for large numbers of visitors, but when the number of visitors is small the correction is very significant. If you only expect 100 visitors, the standard answer of 384 can only mean that you must interview everybody, whereas you only need to ask 80 people. For 500 visitors, only 217 interviews are needed, as opposite to 384, and so on.

To make the sample random, it is not sufficient to write JavaScript that pops up a window when a random number produces a certain result. This will result in two problems: a visitor may be asked for an interview more than once, and frequent visitors are more likely to be asked. ViewsFlash solves this problem by providing an HTTP API that, when implemented on a web page, causes a window to pop up for every Nth visitor, and tagging visitors with cookies as they go by so that once a visitor has been a candidate for a survey, he is only given one chance, no matter how often he comes by.

Configuration Summary

Fill out the settings on the Define Sampling page and submit it. Then, add JavaScript to the page or pages where the survey may pop-up. Last, Publish the poll. A detailed explanation follows:

How to do it, step by step

1 The Define Sample screen first asks to specify the Sampling Method. There are three choices:

Don't sample. This is the default setting, which disables these options.

Fast Sampling presents every visitor with the survey in a pop-up window once, until the desired number of completed surveys have been collected or the Publication ending date is reached. This is a terrific way to get a representative sample of a desired size, provided that the site does not exhibit a pattern of different use during the time it will take to collect the sample. Since this is not usually the case, this method should be used only when gathering the information as fast as possible is more important than having results that are representative of visitors' views over a given period, such as a week. For example, after installing a new video capability on the front page which is technically challenging, it would be useful to know what the next 1,000 people think about it so that it can be removed if it is providing a less than satisfactory user experience.

Besides specifying the number of surveys to collect, you will need to set up JavaScript, as explained below. You can also use the calculator in Random Sampling and copy the calculated sample size to this section.

Random Sampling.

The first two fields, collect __ responses and pop up the survey every __ th visitor, can be filled in by hand, or the can be calculated by filling in the other fields in this section and pressing the Calculate button. Here's how that works. You chose the confidence level (90, 95, or 99%) for your results from the pull down menu, and the degree of precision for your answers as plus or minus a percentage. Then you estimate how many unique visitors will be viewing the page where the pop-up survey can appear during the period when the survey will be Published. You also estimate how many people will be actually completing the survey. When you press the Calculate button, the first two fields will be filled with the corresponding values. ViewsFlash will use the every Nth visitor number as its criteria for deciding who should be presented with a survey and who should not.

2 Displaying the survey in a pop-up window defines the actual JavaScript code that will run on the browser for those visitors who are chosen to be presented with the survey. There is no need to modify it. If you do, examine it carefully first. Note that very creative applications of this technology are possible by modifying this script.

3 Press Submit to set the options you select.

4 Very Important! Add a snippet of JavaScript to the page or pages where the survey may pop up. The JavaScript to copy is presented on a color background at the Define Sample page, from where it can be cut and pasted into the <head> section of the page. The snippet typically looks like this:

<script language="javascript" src="http://yourdomain.com/ViewsFlash/servlet/viewsflash?vwfsample=yoursurvey">
</script>

This snippet calls the ViewsFlash server and asks for a further bit of JavaScript to execute when the page is loaded. ViewsFlash determines whether this visitor should be presented with a survey or not, and then return either a single space, which means there's nothing to do and no survey will pop up, or the JavaScript specified in the Define Sample page, which typically looks like this after ViewsFlash has removed all the [/...] tags and replaced them with the appropriate values:

window.open ('http://yourdomain.com/ViewsFlash/servlet/viewsflash?cmd=getsurvey&pollid=pollingplace!poll&userid=1',
'vwfsurvey',
'width=400,height=300,x=100,y=100,resizable=1,scrollbars=1,menubar=0,toolbar=0')

This JavaScript opens a window named "vwfsurvey", with dimensions and properties as given, and in that window the survey will be displayed by the http reference to viewsflash with the "getsurvey" command. Simultaneously, ViewsFlash will toss a cookie to the visitor's browser to prevent her from being eligible to participate in the survey twice.

5 Finally, go to the Publish page and set up the starting and ending dates and times for the survey. The start time should be about the same time when you insert the JavaScript into the web pages where the survey will pop up. It does not matter if it is slightly before or after. No surveys will pop up until the JavaScript is installed in the page where the survey can pop up AND the Publication opening date arrives. Similarly, once the JavaScript is removed or the Publication end date arrives, no more surveys will pop up, nor will any more surveys pop up once the requested number of completed surveys (responses) has been gathered.

Schedule the survey for a period of time that is representative of the life cycle of your site's content. For example, a stock market site could run a survey for one 24-hour week day, as visitors each day of the week should be the same as any other weekday. A news site could be treated as a daily site or a weekly site, if its content changes significantly for weekend programming. Other sites may be steady-state sites, where there is no reason to believe that visitors change from one day to another. The period of time should also be long enough to make sure that there will be enough visits to the hosting page so as to provide the desired number of surveys; analyzing web logs for that page is one way to ascertain this information.

Monitoring the survey

Once the survey is up and running, visitors will be presented with the survey and some of them will be responding. You can get a good idea of how the survey is progressing by using the View Log command on the toolbar at the left of the screen. This log is updated periodically as voting progresses, and is updated with a final report when the Publication ending date is reached, or the requested number of surveys are completed, whichever happens first.

Changing settings after going live

You can hasten the end of a survey by changing its Publication ending date. Sampling begins anew whenever you change Publication dates.

You can also change the Sampling parameters while the survey is in progress. For example, if you are not gathering surveys fast enough, you can change the sampling rate (every Nth visitor) to accelerate the pace at which they are gathered. Some of the statistics in the log will be reset, but sampling will not be affected.

Technical Considerations

When used for sampling, the ViewsFlash server is required to respond to every page where a survey may pop up. This response is very fast, but the amount of work required can be huge. For example, installing this on the front page of a portal site could result in millions of hits per day and incapacitate even a powerful ViewsFlash server. On a 300 MHZ Intel machine, ViewsFlash completes these requests in under 10 milliseconds CPU time. Use this as a guideline for where to put these surveys. In a very large site, ViewsFlash should be run in a distributed configuration; see Architecture.

Next: Printing