Skip to: [Printable Version (PDF)]

2003 SSBF: Methodology Appendix EE Screen Reader version ^♣

Appendix EE

Final Data Collection Debriefing Memo

Memorandum

February 15, 2004

To: John D. Wolken

From: Nancy Potok

Re: 2003 Survey of Small Business Finances

Summary Notes of Final Telephone Interviewer Debriefing

Participants:

Federal Reserve Board: Traci Mach, John Wolken

NORC: Bob Bailey, Mireya Dominguez, Terri Kowalczyk, Nancy PotokBill Sherman, Ben Skalland, NORC telephone interviewers and supervisors

1. Introduction

On February 9, 2005, NORC conducted a six-hour SSBF interviewer debriefing session. Six interviewers and four supervisors participated; all participants had extensive SSBF experience. The session was moderated by five members of the project team and observed by Traci Mach and John Wolken of the FRB. The session was held in NORC's call center in Downers Grove, Illinois.

Findings from the debriefing are qualitative. Interviewers were invited to the debriefing partly because of the breadth or depth of their SSBF experience - they brought a richness of commentary or insight to the discussion - and do not necessarily represent the population of SSBF interviewers.

1. General Observations
   • Call it a study. Interviewers say it is more effective for gaining cooperation to refer to SSBF as a study rather than a survey. The word survey was sometimes a roadblock, making some respondents think the call was for market research or telemarketing. The word study, on the other hand, tended to convey a more serious, academic purpose.
   • Respondents vary greatly in how much they know. Respondents' knowledge of business and financial terminology - and their interest in the financial end of their business - varied widely. Some respondents appeared to be insulted or bored when having terminology read to them. Other respondents needed long definitions. An interviewer suggested two versions of the questionnaire, one shorter for more knowledgeable respondents, one longer for respondents who need all of the definitions.
   • Many respondents dislike the balance sheet and income statement sections. Some respondents said upfront that they would only give ranges, and were then put off by having to go through a long series of amount and estimation questions. Interviewers noted, however, that sometimes in these situations respondents would cave in and start giving exact amounts just to move it along.
   • Teamwork leads to better individual work. The interviewers enjoyed and appreciated the support from supervisors and other interviewers.
   • Put greater emphasis on the study's purpose in the questionnaire. It is important for interviewers to be able to explain the purpose of the survey to the respondents, to make respondents feel their participation is worthwhile. Interviewers would like to see the purpose stressed more in the questionnaire introduction, since many respondents didn't read or skimmed the advance mailing. A supervisor suggested a national public-service announcement about SSBF, so that the survey would be in respondents' minds when then were called.

3. Training

• Having representatives of the FRB at training sessions is valuable and appreciated by interviewers.
• Give more attention to question A1. Emphasize that the preloaded owner's name is not always correct, and that there can be other owners who can do the survey if the preloaded owner is unavailable. Interviewers should be encouraged to read the preamble before dialing. When multiple pieces of preloaded information are incorrect, it is can be hard to locate the correct business and owner.
• Consider presenting an entire scripted main interview from start to finish. It is daunting to try to absorb all of the details given during training. Several interviewers thought it would be helpful to watch an entire interview from beginning to end, so that trainees can get a general sense of what they will be doing before being given details and theory. One interviewer suggested taping a real interview for this purpose.
• Consider shorter training sessions over more days. Shorter training sessions, spread over more days, might be more helpful than the long training sessions. Many interviewers agreed that a more general training should be given first, followed by shorter, specific training after the interviewers have a chance to get a little experience.
• Provide more challenging mock interviews. Interviewers at the debriefing thought that the mock interviews were too easy. A supervisor, however, noted that not every SSBF interviewer thought the mocks were so easy. It was suggested that there be mocks at different levels of difficulty, so that trainees can progress through easy to more difficult mocks as they gained experience and became comfortable with the questionnaire.
• Consider a longer break between screening and main training. A supervisor thought there should be more time between screener training and main interview training, so that as trainees are doing screener interviews, they can listen to experienced interviewers do main interviews before being trained on the main themselves.
• Shadowing is popular. Interviewers suggested having a new trainee shadow an experienced interviewer before trying to do interviews by him- or herself. Bob Bailey thought this would help limit damage done by inexperienced interviewers. Shadowing could also help teach new interviewers in gaining cooperation - actually hearing the words and techniques that are effective on real calls.
• Give more attention to refusal aversion - learning to leave the door open for the next interviewer. Call notes show a fair amount of respondent alienation. Knowing what not to say is important. Interviewers, even if they cannot gain cooperation, should know how to avoid final refusals and leave the door open for the next interviewer's attempt. (Work on refusal aversion first, then refusal conversion.) Working pass-two cases helped interviewers hone their gaining-cooperation skills.
• Improve round robins. Interviewers said the round robins were probably the least effective training technique. Trainees tend to zone out when it is not their turn to read a question. Interviewers agreed that round robins would be more effective if trainees were called in random order.
• Be selective about detailed question-by-question analysis. One interviewer felt that the analysis of each question during training was too much. Another said he appreciated question-by-question analysis. A third suggested that this detailed analysis should be delayed until the interviewers are more comfortable with the questionnaire in general.

4. Gaining Cooperation During Screening

Participants identified numerous techniques for gaining cooperation during screening with gatekeepers, proxies and owners. Techniques that worked for some interviewers did not necessarily work for others. Generally, these experienced interviewers tended to find their own voice early on in the study, and spent the rest of the project perfecting their voice and adapting it to new situations.

• Explain the purpose of the study as early as possible.
• Sound professional and knowledgeable. When speaking to a gatekeeper, this tone helps get through to the owner.
• Be matter of fact, calm, serious and confident.
• Say that a firm was chosen, not randomly selected. De-emphasize the word randomly in the introduction and instead emphasize that a firm was chosen to participate. Explain that the selection was based on firm type, size, and geographic location.
• Say that the call is to qualify the firm, or to confirm where to send information to allow the respondent to make an informed decision. Say that we were just trying to verify basic information. The downside to this tactic is that when eligibility was mentioned, some respondents would try to disqualify themselves by guessing which answers would disqualify their firm.
• Try mentioning the Federal Reserve Board. If that doesn't work, ask if the respondent has heard of Alan Greenspan. Some interviewers found that even if respondents had not heard of the FRB, or were unsure of its function, most had heard of Alan Greenspan, and the reference to Mr. Greenspan could sometimes lend authority and star power to the study.
• Put more focus on closing eligible firms. The screener close sets the tone for the next phase of the relationship between the firm and NORC, and can have significant effect on the extent to which respondents are willing to cooperate in completing the main interview. Interviewers can be better trained to read the close slowly, carefully and with commitment; they need to be trained to avoid rushing the close. General comments and suggestions about the eligible-respondent screener close are below:
  • At the close, use the opportunity to ask if the respondent has questions or concerns. If so, the interviewer would give the respondent information about the website and SSBF hotline. One interviewer thought it would be good to have a question written into the close, asking if the respondent had questions or concerns.
  • It was good to get a time for the main interview during the screener and to put this in the call notes.
  • Many objections came at the end, during the close.
  • Many respondents, after hearing the incentive amount (near the end of the study when the respondent fees increased to $200 or $500), wanted to set appointments for the main interview at the close of the screening interview.
  • If talking to a proxy in the screener interview, it was helpful at the close to ask the proxy to make sure the owner knew about the study and the materials he or should would be receiving from NORC
  • Some screener-only interviewers were not conversant about the next steps and the content of the main interview, and consequently were not able to explain what happens next at the screener close, compared to interviewers who did both screener and main interviews.

5. Gaining Cooperation During Main Interview

• Reassure respondents about the worksheet. It helped to emphasize that completing the worksheet was not absolutely necessary to do the interview; tax returns could also be used. Half of the worksheet covered balance sheet and income statement, which led some respondents to think that half of the interview would be about those financials.
• Be ready for privacy concerns. Privacy issues are more of a problem in the main interview than in the screener. It helps to make it clear that estimates rather than specific amounts may be given, and to refer respondents to privacy-assurance information on the SSBF website. One interviewer says that privacy laws should be on FAQs (Frequently Asked Questions) as a quick reference.
• Embarrassment can be a barrier. Some interviewers thought that owners of firms that were not doing well were more reluctant to participate than owners of more successful firms.
• Respondent fees work. However, interviewers said that some respondents showed little interest in the incentive until it reached $200.
• Increasing respondent fees has some negative consequences. Changing the incentive amounts made some respondents question if NORC actually intended to send them any money. Other respondents were cynical. ("Call me when it hits $1,000.")
• Explain the purpose of the incentive. It helps to give a reason for offering a financial incentive, e.g. "We have great respect for your time, sir." If the respondent feels that we are just looking for his price, this could be insulting.
• Offering a choice of incentives is good, but the D&B brand comes with baggage. Mentioning Dun & Bradstreet might be more hurtful than helpful. For some respondents, it may have brought a marketing or commercial flavor to the study. Some respondents were concerned that their financial information would be shared with D&B. Interviewers liked the idea of two options, and they liked the idea offering something that might help a respondent's business, like a Wall Street Journal subscription.
• Improving cooperation in answering sensitive financial questions is possible.
  • Toward the middle of the data collection period, a memo was distributed to interviewers to focus on getting respondents to give fewer "don't know" and "refuse" responses to questions in sections P, R, and S. A meeting was held to walk interviewers through the memo. After the meeting, the completeness percentage noticeably improved.
  • It helps to emphasize to respondents that accuracy is important.
  • One interviewer did not give some respondents the option to refuse. Instead she told them that they could give an amount, an estimate, or a range. The tactic worked.
• Interviewers mentioned other tactics that helped gain cooperation.
  • Reading call notes beforehand to see why a respondent has refused in the past, so that a different approach could be attempted.
  • When getting the same gatekeeper repeatedly, setting callbacks and developing rapport, so that the gatekeeper would be less likely to feel harassed.
  • Faxing a signed confidentiality statement to the respondent.
  • Emphasizing that it the banking industry being evaluated, not the firm.
  • When encountering a gatekeeper, the interviewer should not identify him- or herself until asked. Instead, just ask to speak with the owner.

6. Screener Questionnaire

• Interviewers had numerous suggestions and comments about the screener questionnaire. The list should not be construed as NORC's recommendations to the FRB for changing the instrument. Rather, this list is of observations from a group of seasoned interviewers who each administered the instrument hundreds of times.
  • Verify the owner's name during the screener introduction instead of waiting until the main questionnaire. (On the other hand, the verification might make gatekeepers suspicious about the nature of the call or the caller's lack of familiarity with the owner.) One suggestion was to ask in the introduction: "May I please speak with the owner, (NAME)?"
  • If an interviewer learns during the introduction that the firm is ineligible, there is no way to jump to the right eligibility question and enter the appropriate information.
  • A1 should be simplified and possibly divided into three screens.
  • At A1, sometimes "Owner not available" was selected inappropriately, so an interviewer talked to a proxy before the fourth attempt to reach the owner.
  • Sometimes on the first call attempt, a gatekeeper would direct the interviewer to a proxy. It is awkward to say on each of three calls that we cannot speak to the proxy, only to come back on the fourth call and say that speaking with the proxy is okay.
  • When the preloaded information is missing or erroneous, the questions as read can be awkward. For example, if the preloaded business name is the name of the owner, the confirmation of the business name question (A2) is awkward.
  • One interviewer would like a prompt or an option for doing business as a different firm name, e.g., DBA.
  • The office or branch question (A9.1) is not always clear. For example, for a restaurant with a separate business office, is the restaurant considered a branch? If yes, the firm becomes ineligible.
  • A3 is really two questions combined into one question. Some interviewers believe that not all respondents heard both parts of the question. They recommend that for future SSBF studies, A3 be broken into two questions. Several interviewers also recommend having a clearer or more detailed definition of what it means to be "in operation."
  • Put LLP and LLC answers on a different screen.
  • There were no problems with the "number of employees" question or in distinguishing for-profit firms from not-for-profit firms.

7. Main Questionnaire

• Interviewers had numerous suggestions and comments about the main questionnaire. The list should not be construed as NORC's recommendations to the FRB for changing the instrument. Rather, this list is of observations from a group of seasoned interviewers who each administered the instrument hundreds of times.
  • At A1, consider verifying the name of the owner during the introduction. There needs to be a way to update the name of the owner at A1, to make corrections if the case was screened improperly. Also, some interviewers did not like reading the sentence that the owner may be expecting the interviewer's call. They felt that for some gatekeepers the sentence rang hollow.
  • The wording of F3.2 and F6.2 might be improved. If the respondent says that credit cards are typically paid in full (F3 and F6), then currently F3.2 and F6.2 ask for the interest rate paid. Paid should be changed to used or charged. A few interviewers suggested that the worksheet provide space to record the interest rate of a firm's credit card. Many respondents took a long time providing the information.
  • The exception key was used a lot at F3.1 and F6.1. The range check should be removed.
  • It was problematic to read a question after which followed an explanatory sentence, before expecting an answer. Some respondents would try to answer immediately after the question was read and not wait to hear the additional explanation.
  • There was confusion between credit cards and lines of credit. READ11 is confusing. Interviewers read two sentences about lines of credit and then say "do not include credit card accounts." One recommendation is to exclude credit cards at the beginning of READ11 and then reiterate it at the end.
  • Asking MRL questions is sometimes seen by respondents as repeating earlier questions that are similar to those asked in section MRL.
  • There should be an automated CATI process for capturing a new name and address to which to send the Federal Express package, if the respondent has not received it.
  • A definition of paid day laborer should be embedded in question A10.7.
  • For the number of owners question (A10.1) "worked" should be highlighted on the CATI screen, and interviewers should get better training in reading the word with emphasis.
  • One interviewer liked the simplicity and structure of the language in A10.5 and thinks similar questions should use this language.
  • An interviewer reported a shift in verb tense following C20 if the firm has more than one stockholder.
  • At A7, most respondents from trust-owned firms didn't consider a trust to be a company. If a trust is the only stockholder, the CATI tries to get demographic information about the trust. There were similar problems with employee stock ownership plans.
  • If the largest shareholder is a company owning more that 50%, the CATI should kick the case out as ineligible. This would be a double-check on the screener and section A.
  • No exception is allowed at D1, and an interview can get stuck if the respondent insists on answering zero at D1.
  • Many interviewers felt that question N1 was premature and that some respondents used the question as an excuse to postpone the rest of the interview, when, according to some interviewers, the worksheet is most necessary for sections P, R, and S. It was suggested that N1 be moved to before section P.
  • The XXX fake institution name convention worked well.
  • Some respondents confused auto loans with capital leases. A better explanation or an additional question would help.
  • The institution look-ups were easy to use and worked well. Interviewers thought that too much time was spent in training on look-ups. Interviewers who did have problems were trying to input all of the information into a single search, instead of using simpler search techniques.
  • There was a lot of estimation in section L. Possibly some of this information could be entered on the worksheet in advance of the main interview.
  • Many respondents didn't know what was meant by cost of doing business, even with prompts. There should be a more simple definition and/or some examples for very small businesses.
  • At R6 many respondents misunderstood book value of land and gave the current land value, not the value at the time the land was acquired.
  • In the loans and line of credit section, the loan amount the owner said he received was almost always the same amount he said he asked for. Asking what amount the owner believed he should have received, rather than what he asked for, might be more useful information for the FRB.
  • Some interviewers thought the close did not work well. Many respondents never completed a worksheet, and some of them were annoyed when they were asked about the worksheet at the end of the interview. Sometimes a respondent was not the person who possessed the worksheet. "Can I count on you...," was seen by a few interviewers as patronizing to respondents.
  • Instead of telling a respondent that we have provided him with a postage-paid envelope, we should ask if he has the postage-paid envelope, and if not, then the interviewer should give him the address.

8. Job Aids

• The job aids and CATI did not always match in their references to the worksheet.
• One interviewer thought that the answering machine script did not flow well.
• One interviewer thought that SSBF FAQs should return to their original format, which was divided into sections like the REACH FAQs.

9. Contact Materials

• Interviewers think the advance mailing is effective, when respondents read it. Among respondents who read the advance letter and were expecting a call, the response rate was high, according to interviewers. The nearer the call followed the advance letter, the better the likelihood of a completed interview. Several interviewers thought that the advance letters were very well written. Sometimes interviewers had to fax the advance mailing to a respondent, to help orient him or her.
• Some respondents were intimidated by the worksheet. However, it was good for interviewers to be able to refer to the worksheet package materials. Interviewers were unsure of how many respondents actually read the materials included with the worksheet. Interviewers did not strongly suggest that the package contain fewer materials or that the worksheet be shortened. One interviewer reported that a respondent had called the design of the worksheet "genius."
• Some owners mailed in the worksheet before doing the main interview. It should be made clearer that the worksheet should not be returned until after the main interview.
• Have a form letter for accountants and lawyers. A supervisor suggested sending a letter to lawyers and accountants in addition sending one to owners, because some lawyers and accountants were advising their clients - with insufficient information -to not do the survey.

10. Interviewer Incentives

• Posting daily goals is an effective motivator. Interviewers liked the use of white boards in the production center on which goals by batch, day and hour were posted. Many said it was a terrific motivational tool.
• Paid lunches and relatively small monetary rewards are appreciated. Interviewers had more comments about the daily-goal boards than about the extra money they could earn by working more hours, but they still appreciated the lunches and the monetary incentives.

11. Locating

• The 5% follow-up locating went quickly and smoothly once all of the systems were in place. For the locating part of fieldwork, one interviewer would have liked an all-encompassing check list to help ensure that she/he had performed all possible locating steps.

^♣ This version is optimized for use by screen readers. Descriptions for all mathematical expressions are provided in LaTex format. A printable pdf version is available. Return to Text

Home | Surveys | OSS
Accessibility | Contact us
Last update: November 1, 2006