Methodology

The Employer Database

Sampling

ERISS conducts large-scale, representative sample surveys using a technique known as "stratified proportional quota sampling." This technique enables researchers to accurately represent the major characteristics of a population by sampling a proportional amount of each. Pre-determined population characteristics (such as company size, industry, and region) serve as stratification criteria.

The result of this sampling design is an optimized allocation of the sample among the various strata. Additionally, this technique allows researchers a great deal of flexibility in terms of representation of pre-chosen strata within the population. For example, a particular region may have relatively few large companies, but these companies account for a large proportion of regional employment.

Purposive over sampling of larger companies ensures the employees and occupations at these companies are represented in the survey to the degree they impact the region, not just their proportionate representation in the overall population.

Finally, ERISS combines this sampling method with a "census" survey approach to ensure as many employers are contacted as possible. To this end, ERISS attempts to contact every employer in the employer database with five-or-more employees, and achieves response rates of 20% to 40%. This means that for an area with 20,000 employers, we would obtain responses from between 4,000 to 8,000 employers. This is conservatively four to ten times as many responses as would be obtained using a traditional stratified random sampling technique.

The higher number of responses ensures sufficient data is gathered to provide detailed local information, even when stratified by sub-regions, industry classifications employer size and even customized industry sub-clusters. The result of such a large sample and the use of stratified proportional quota sampling results in representative data on a number of dimensions. Random samples can also be stratified on a multiple dimensions, but because fewer employers are surveyed in a random sample, these stratifications are based on fewer responses and may have a higher level of error.

Methodological Tradeoffs

There are inherent advantages and disadvantages to any research methodology. There is no single method best suited to all types of research. Labor market research is typified by a series of tradeoffs that are best characterized in terms of cost/benefit. Research goals, intended use of the data and available resources largely dictate the optimum method of choice. The method that returns the most value is often not the most theoretically desirable, but rather the most valid for the purpose and budget of the survey.

Obtaining a true random sample along all desired stratification levels usually requires the expenditure of resources greatly out of proportion to the expected returns. For the applied purposes of most workforce professionals, surveying the greatest number of employers possible in a representative, valid, and cost effective manner is of greater importance than achieving a true random sample that allows for the use of statistical techniques that are often of little applied value to the purposes of the survey on a local workforce development level. Therefore, for research involving job markets, sample representativeness is often a more important consideration than true randomization (which is better suited to research more experimental in nature and design, and where timeliness of the data is not an issue).

Additionally, due to selection bias, non-response, inaccurate or non-representative database source, attrition and other factors, studies originally designed as random sample surveys often do not qualify as random after the survey is complete. Due to a dependence on randomization to ensure the generalizability of survey results (rather than targeting individual strata or other techniques to ensure representativeness), the final result of some "random sample" surveys are samples that are neither random nor representative.

Assuming a true random sample is achieved, what are the advantages and disadvantages of true random sampling over other techniques such as proportional quota sampling? The primary advantage is that statistical theory can be used to make generalizations from the sample of employers to the population. However, sampling theory dictates that the more units sampled from a population, the closer the sample comes to approximating the characteristics of the population (assuming sampling is equivalent across various demographic dimensions, such as company size). When a large proportion of the employer population is surveyed, the ability to make accurate statements about those employers is largely equivalent to making statements about the population as a whole. Additionally, measures to ensure and gauge external validity can be utilized. For example, a comparison of ERISS data for a key factor such as salary, with pre-existing salary information about the target population can add confidence to the survey data.

In sum, as compared to a random sample survey, the primary advantage of our approach is a representative and valid sample containing a higher number of responses, representing more occupations, with a lower cost, and most importantly, more timely results. Our surveys are completed within 16-weeks, hence workforce professionals have the most recent job market information available.

Admittedly, some sacrifices are made in terms of generalizability of the survey results as compared with surveys using probabilistic sampling techniques. The information provided by the Bureau of Labor Statistics utilizing random sample survey techniques cannot be replaced by any other method in terms of value for strategic decisions and planning. However, the information required for more tactical decisions (such as those common to local "One-Stop centers") must be more current and represent a large number of local employers. Ideally, the timely, representative data provided by ERISS in conjunction with random sample survey data (such as provided by the BLS) supplies the necessary information workforce professions require for both tactical and strategic labor market solutions.

Tracking of Stratification Parameters

As previously described, the sample is stratified in terms of industry, company size, and region. This is accomplished by means of real-time data tracking. The emerging sample is inspected daily, and resources are adjusted and reallocated to ensure representativeness in terms of each stratification parameter. For example, if an employer database indicates that the Business Services industry accounts for roughly 8% of all employers in a region, steps are taken to ensure this industry represents approximately 8% of all industries in the final survey data. Similarly, company size (regardless of industry assignment), and regional representation are tracked and adjusted. The survey is not considered complete until all stratification parameters fall within the predetermined ranges, and the overall required response rate has been achieved.

The Staffing Pattern

ERISS employs sophisticated staffing patterns to determine which occupations to survey for each individual employer. These staffing patterns are based upon interviews with more than 2,000,000 employers, crossing four-digit SIC codes to enhanced, 8-digit O*NET codes (ERISS uses customized eight-digit O*NET codes to account for new and emerging occupations not yet represented in the standard O*NET coding system).

Considering the volatility of occupational titles and the evolution and development of new industries and categories of businesses, staffing patterns tend to lose their ability to reflect occupational trends over time. Therefore, the ERISS staffing patterns are modified and updated to accurately reflect occupations in the current labor market. This is accomplished by continuous adjustments and improvements as we incorporate information from each completed survey to reflect new occupations or existing occupations appearing in new industrial classifications.

Occupational Targeting

ERISS deploys both preset and dynamic targeting criteria for occupational selection. Preset targeting criteria include: · Sponsor requests - occupations that are selected as "special interest" occupations by our customers. Occupational "rarity" - for example, when surveying fire stations, the occupation "fire chief" would be given top priority since they are found nowhere else. Number of potential employers - the more potential employers, the higher the priority.

The ERISS dynamic targeting process is designed to adjust the parameters of the survey in real-time, as the survey is being conducted. The targeting program works by following the pre-set parameters to ensure the proper amounts and types of industries, occupations and company sizes are surveyed. These parameters operate at multiple levels simultaneously and the number of surveys required for each occupation varies according to the fulfillment of certain preset requirements as well as the characteristics of the developing survey data

To determine the proper number of surveys required for each occupation, dynamically adjusting thresholds, referred to as "floors" and "ceilings," are used. The "ceiling" is defined as the point at which the collection of more data for a particular occupation will no longer significantly impact the results. The "floor" refers to the minimum number of occupational surveys required for occupational data to be considered valid and publishable.

In response to live survey data, the program adjusts these thresholds for each occupation. For example, if the existing data for a particular occupation is displaying a high amount of salary variability, the program adjusts the floor so that a higher number of these occupations are targeted for surveying, thereby increasing the confidence in the data for that occupation. Occupations with very little salary variability require fewer data points to achieve the same level of confidence. As more occupational surveys are collected, and the dynamic minimum threshold is reached, the probability that the occupation will be selected for surveying declines until, after the maximum threshold is reached, the probability becomes zero, and the occupation is no longer surveyed.

Additionally, before the survey, targeting parameters can be manually adjusted to ensure that certain "VIP" businesses or other high-priority targets, whether they be industries, regions, occupations or even specific firms or companies, are sampled in sufficient numbers. One special feature of the program is the ability to know when enough occupations have been surveyed using more than simple "counts." For instance, some occupations typically have a great deal of salary variability (i.e. large range between the highest and lowest salaries). These types of occupations require more data in order to make confident statements concerning the average and median salaries. Using existing data from the Bureau of Labor Statistics and other sources, these occupations are identified prior to the start of the survey and assigned higher completion thresholds ensuring adequate numbers are surveyed according to the salary characteristics.

Data Validation

Data is validated at three points during the survey process; during the live survey (real-time validation), spot checks of completed surveys (usually within 24 hours of the survey) and after the survey calls are complete (post-survey validation).

Real-time validation is conducted along several dimensions, the first being during keyboard entry. The ERISS CATI (Computer Aided Telephone Interview) system performs extensive logic and range checking both in terms of salary points, hiring and turnover numbers, etc. During the survey, ERISS surveyors are also subject to both visual (observations of screens) and audio ("silent" listening) monitoring. Experienced ERISS team-leaders, trained in management and leadership listen-in on randomly selected survey calls. This monitoring not only ensures that data is collected in a standardized and reliable manner, but also that surveyors are polite and professional when interviewing employers. Additionally, ERISS survey team-leaders visually monitor the surveyors computer screens to track survey progress and method.

Also during the live-survey, survey statistics are monitored in real-time to track trends in the developing database. For each individual surveyor as well as at the aggregate level, these statistics document such factors as number of calls, average length of call, successful surveys, phone appointments, and various other factors crucial to tracking the live survey. This information is also used to target surveyors for observation.

In addition to the live-survey monitoring, daily tabulations of the survey data are conducted. At the end of each day of survey operations, a review and analysis of survey statistics and other statistical information collected to date is conducted. These "daily" statistics are used to analyze and interpret trends in key areas such as "representativeness" by size and industry. This information is used to devise strategy and targeting adjustments for the following days survey.

Also, daily validation calls are made to participants to spot-check data. These calls are made by experienced ERISS surveyors to randomly selected employers. This not only provides a valuable validity check on the collected data, but it also provides a check on the accuracy of individual surveyors.

Post-Survey Validation

Lastly, the resulting data sets are evaluated against initial targeting parameters as agreed upon with the survey sponsor. In consultation with the sponsor, a determination is made as to whether the survey is complete, or whether to continue the survey to gather additional information.

Guidelines for Publication of Data

After the survey is completed, not all data will meet minimum requirements for publication. Although publication can take the form of an Internet application, or a written report, these guidelines remain the same.

One requirement is that there be enough observations to validly represent a particular occupation as determined by the occupational "floors" discussed above. As such, the required number of observations will vary from occupation to occupation. For example, if we obtained four observations for "Police Officer" from a total of five possible precincts, we publish. On the other hand, a higher publishing criteria would exist for occupations such as "General Managers and Top Executives," where there are many more employers and higher data variability.

In order to ensure confidentiality, ERISS will, under no circumstances, publish occupational data representing less than four employers, nor any occupation where just one employer represents an overall weight of 80% or more.