Since the beginning of the COVID-19 pandemic, park and recreation agencies have faced numerous challenges with adapting activities and programming to ensure, first and foremost, the safety of staff and community members. Not unlike the programmatic nature of park and recreation agencies, evaluation and community needs assessments are even more important than ever before. In recent weeks, NRPA’s evaluation team has been approached by agencies struggling to assess and evaluate their work on the ground. Agencies can no longer rely on the tried-and-trusted methods, such as in-person community meetings, face-to-face surveying and the like, which makes evaluation even more uncertain in a climate where agencies also face increasing demands and simultaneous budget cuts.
NRPA has developed this guide to help your agency adapt evaluation and monitoring of project and programmatic progress in the age of COVID-19. To reduce unwanted contact with community members and ensure safety comes first, NRPA’s evaluation team has compiled some ideas for low-risk evaluation and assessment activities that can help agencies in their efforts to continue to serve constituents across the country. Not only can these tools help you track social progress, but they can be helpful in evaluating services and needs that are directly related to the pandemic.
Objective: Identifying use and satisfaction, as well as unmet needs
Surveys
Surveys can often be conducted safely at a distance and may require little, if any, modification to reach your audience. You want to ensure that your survey is concise and clear. See NRPA’s customer feedback surveys for tips on survey development. You may want to limit the number of questions to just a handful of check box-type or Likert scale-type questions and one or two open-ended questions based on the information you seek. By articulating a clear purpose statement for the survey, including how the information will be used to guide programmatic development, you can help improve response rates.
- Online surveys: You can create online surveys using a free service, such as Google Forms. There are other free and low-cost service providers (see this summary blog here on different surveying tools). One difficulty may be in reaching constituents. By providing incentives, such as a raffle for gift certificates to local businesses, you can boost your survey response rate. Agencies may use Facebook, NextDoor and other social media platforms to reach residents, however not all constituents may be active online or have internet access, and it may be worth considering complementing with other surveying methods to reach a wider audience.
- Telephone surveys: Telephone surveys have several advantages over online surveys. If you have a list of constituents and their contact information, it is a great way to solicit feedback from returning customers. Although this approach is more time consuming for staff, you may have a higher response rate than relying solely on passive surveying techniques, like online surveys. This is because you already have a relationship with people you are reaching out to one-on-one. When you target people who have used your services in the past, you should focus questions on regular programming and services, as well as unmet needs. One challenge with telephone surveys is that recent increases in robot calls may mean fewer people pick up their phones when they are not familiar with the number.
- Mail-in or drop-off surveys: While mail-in surveys often suffer from high costs and low response rates, you can combine the delivery of certain services, including food pick-ups and drop-offs with a short survey. When providing food in bags, consider stapling a thank you letter with the short survey and a pre-stamped envelope to the bag. This will reduce barriers for survey returns and by directly associating the survey with a provided service, community members may be more likely to respond.
- Dot surveys: When you have a few questions you’d like to ask people attending a space such as a park or farmers market, you can use the dot survey technique. Find a good location at an entrance or exit to prominently display easels with white paper where you have written out survey questions with pre-defined responses. People respond by placing a dot sticker under their appropriate response category. To keep this surveying technique as safe as possible, make sure that you devise a contactless way of providing stickers to participants and can offer hand sanitizer or a hygiene station following participation.
Systematic observation in public places
Using and modifying systematic observation protocols, like SOPARC, can help you understand what people do in key sites, including farmers markets, parks and other facilities. Physical distancing and entrance restrictions present an opportunity for observations of places that may be typically crowded and can allow you to conduct more than a headcount. For instance, in farmers markets you may be able to combine a headcount with other observation techniques to assess interactions and use of space. You may be interested to learn where crowding is likely to occur, or whether people are washing their hands at the entrance and exit. This information can help inform messaging and other logistics to reduce unwanted behaviors and bolster desired behaviors. For example, observation can help you identify prime locations for placement of a hygiene station or a controlled entrance/exit based on where people are likely to access a public space.
Objective: Understanding unmet needs, gaps in services, and building from strengths
Virtual interviews
Moving interviews online is easy with help from new communication tools, like Zoom and Google Meet. You can also easily record your sessions through several different platforms, which will give you the opportunity to ensure you have accurately understood participants. Be sure to ask for permission first before recording and be prepared in case a participant refuses permission. Like with surveys, it is important to be clear and direct with participants about the purpose of the interview and what you are hoping to understand from their contributions. While it is best practice to transcribe the entire interview before analyzing, we recognize that park and recreation agencies are often short-staffed and under-resourced. As a shortcut to transcription and analysis, we recommend creating a table of the key categories of information you are looking for and filling it in with the comments and suggestions from your participants as you listen to the recording following the interview. You may want to consider incentives for participating community members, but do not forget that representatives of community groups, non-profits and government agencies are likely to provide invaluable information from a perspective that is complementary to that of community members.
- Focus groups: Many of the logistics are the same as in-person focus groups, though virtual focus-groups should be condensed to no more than an hour and a half due to screen fatigue. By carefully selecting 4-6 participants who are knowledgeable about a certain topic you can maximize the quality of information you receive. This guide provides helpful guidance on some considerations for running a focus group.
- One-on-one interviews: Virtual interviews easily replace in-person interviews. This guide provides helpful guidance on some different types of interviews and other important considerations. For interviews, it is critical to think about who you want to talk to before you begin so you can decide how to identify people. There three main approaches for deciding who to talk to, all detailed here.
Participant observation
When you want to understand what an experience is like from an inside perspective, participant observation is very useful. Want to understand what the customer service is like at a farmers market or food share point? Go through the line as a customer on several occasions and at different times! This can help you identify where long waits may deter people from accessing resources. When you combine participant observation with other low-risk evaluation methods, for instance, a survey or virtual focus group, you can identify not only where things work well and need improvement, but how to make those improvements. Participant observation is often used in action research, including in the classroom setting, and can easily be adapted to any parks and recreation setting.
Participatory asset mapping and calendars
There are many different kinds of mapping, including geospatial and asset. Mapping is a useful technique in action research, and when you involve different target stakeholders in the process, it can help your agency better understand 1) key players and services that support the work that you do, and/or 2) key gaps in services. For instance, you may want to know where people access food when they are food insecure but do not qualify for federal meal programs. This information can be gathered alongside interview data, but the product is a map of assets instead of a transcript. Alternatively, if you want to know where would be the best location to establish an orchard based on locations where residents harvest other wild-growing foods, you likely want a geospatial map accompanied by a calendar to tell you when and where people harvest food. You can make this activity as interactive as you like, sending to people general maps with colored pencils and an instruction sheet, or you can facilitate this effort through an online collaborative session with interested participants. For a useful toolkit that walks you through participatory mapping, download this pdf. Remember that maps do not have to be detailed or accurate. They just have to give you the information you are looking for.
Objective: Monitoring primary and secondary data of key metrics
Monitoring of primary data
Some of the services you provide are already collecting information that can be useful to understand when and where people use services. For instance, some of your recreation facilities may track user data through personal IDs, like swipe cards. For other programs, you may be able to build in a monitoring system, for example, to track how much SNAP/WIC money is spent at farmers markets. Capitalize on that information by making it as easy to collect and analyze as possible and building it into your regular evaluation plan.
Monitoring of secondary data
A lot of organizations are reporting data about your county right now. These data pertain to projected food insecurity, vulnerability to COVID-19, needs for childcare, unemployment, among others. In addition to national-level data sets, there are many state-wide or county-specific data sets that give a snapshot of current socio-economic trends impacting your constituents. When you track multiple data sets side-by-side across time, you can get a picture of what is happening in certain neighborhoods. For instance, socio-demographic data can be tracked alongside changes in rent prices in areas around park developments to identify if gentrification may be occurring as a result of capital investment. Additionally, some other organizations may have already collected and compiled much of the information you are looking for. For instance, local hospitals are often required to conduct regular community needs assessments to identify gaps in health-related service. Reach out to other relevant partners to find out what has been done already in your area!
Adapting your evaluation and assessment approach will serve both your constituents and your agency!
NRPA recognizes that the COVID-19 pandemic changes the reality of your work on a day-to-day basis, however, assessment and evaluation is more important than ever. This is especially true as your agency makes the case for funding and support despite the economic downturn and uncertain future. By bringing a little bit of creative thinking and problem solving to your assessment and evaluation approach, you can ensure that you continue to serve your constituents in a way that serves their best interests even during the hardest of times.
Lauren Redmore is NRPA’s Evaluation Manager.