How might we find a way to support better referrals to health and human service providers in NYC?

Situation

The lack of accurate NYC health and human services information undermines the well-being of service providers and residents. People need help to get referrals; organizations providing referrals spend their limited and valuable time tracking down poorly maintained information, and innovation in this sector is nearly impossible.

Analysis

We conducted a variety of discovery activities, including finding users, conducting interviews a community summits, data discovery and transformation and capturing feedback on the product

Solution

We built out the ReferNYC three-part platform solution:

• The HHS Referrer Federation (the participants in the community who would co-design the product),
• the Open Data Exchange (the code to create clean referral data)
• The public data platform (to interact with the product). I created this diagram to help stakeholders understand the parts

Results

This is the first comprehensive view of health and human services in New York City, and as a result the solution was integrated into the City’s procurement systems which manage data about non-profits and those providing services. Eventually, the ReferNYC Open Data Exchange can be scaled out to the larger community for public use, including as part of the City’s Open Data work. Part of the work ahead for the City is to understand how to grow the customer base externally, or use it internally within the City to first have the best data set.  The data allows the City to understand geographic gaps where services aren’t available, to have accurate information about service providers, and to help lay a foundation where service providers can seamlessly share what they know about themselves and benefit others.  Non-profit organizations, the City of New York and New Yorkers all benefit from better referrals to lead happy, healthy lives.

In detail

Situation

• Over 3 million New Yorkers access some form of social support services. New York City has one of the broadest safety nets in the country.  To access a social service, often social service agencies will provide people with a referral – information about another service in the community, but maintaining up-to-date data about organizations is a challenge. Which organizations provide what services to whom? Where are they, and when are they open? What languages do they speak? These details are constantly in flux, and there is a baffling array of call centers, directories, websites, and apps, each with duplicative and often contradictory information. Referrers in the health, human, and social services space need a better way to provide, fund, or refer people to services, but they lack a means to share or maintain this information. 
• People who require health and human services in New York depend on referral directories – documents that list vital community services like homeless shelters, food banks, community centers, and other services. However, referral directories that list the information must be more consistent and accurate, and often require digitization. Nonprofits are struggling with the directories, and the end users – New Yorkers – often lack the vital information they need.
• The lack of accurate NYC health and human services information undermines service providers’ and residents’ well-being. People need help to get referrals; organizations providing referrals spend their limited and valuable time tracking down poorly maintained information, and innovation in this sector is nearly impossible.

How might we use technology to create a digital social safety net about referrals?

Analysis

The who

• My role was as the product manager (and project manager) to conduct research and help develop the roadmap for the main application for the Data Exchange – to allow any non-profit in New York City to share info about their organization and benefit from the shared directory. We would be using the Human Services Data Specification (HSDS) to create a better, normalized, and standardized set of data. I managed the product, set up product design best practices, and managed an offshore vendor that built out the product. I also managed the research, including interviewing non-profits to ensure they were suitable for participating in our pilot.  Eventually, any organization in New York could benefit from the Web application we were building. Still, our goal was to find three organizations to partner with and pilot the intervention with.  As Director of Product Design, I was also involved in managing the Statement of Work for the Vendor and liaising with the executive sales team to ensure team performance
• Stakeholders were a partnership between Civic Hall Labs and the District Attorney’s Office of Manhattan through the Criminal Justice Investment Initiative at CUNY’s Institute for State and Local Governance and the City of New York. Our steering committee also involved the City of New York’s Human Resources Administration members and multiple offices within the Mayor’s Office.

The why

• Vision: Our goal was to understand how access to resources like referrals affects New Yorkers, especially those vulnerable to becoming homeless, involved in the Criminal Justice system, or any other health and human services scenarios. The work of Civic Hall Labs was tied to these Civic Determinants of Health to understand how to scale out participatory design within an R&D project. The vision was an interoperable, comprehensive, and accurate directory for non-profits by developing an Open Data Exchange. 
• Goals : We identified the goals – the things we wanted to help non-profits in New York City with:
  • Cost: Saving time and money so organizations can re-allocate precious resources away from duplicative data management toward helping people.
  • Improve service delivery: making it easier to find, access, and provide a range of services to New Yorkers in need.
  • Open the doors to innovation: By providing the foundational and previously inaccessible data, you will encourage an ecosystem of innovators to develop products that deliver this information to users.
• Metrics:
  • Increased knowledge of gaps in data that organizations weren’t aware of
  • Increased new referrals for services, increased referrals to other boroughs etc.
  • Increased accuracy and timeliness of data including being updated more than once a year
  • Decreased time for staff doing data entry, on phone verifying referral information, other metrics etc. 

The how

Our product plan included the fillowing milestones

• Q1: Stakeholder Recruitment – finding community partners to work with
• Q1 & Q2: Data Discovery – understanding stakeholder challenges
• Q2: Data Transformation – transforming stakeholder data into ‘clean data’
• Q3 & Q4: Data Exchange Prototyping – building the final prototype of the service
• Q4: Stakeholder Summit & Prototype Feedback
• Year two: Data Exchange Release & Iteration, User Acquisition & Growth, Sustainability Working Groups and a Final Stakeholder Summit and to scale this outside of NYC.

Research

Co-designing

One of our success metrics included using Participatory Design as a research method, so we ensured community-based design was part of our research and product evangelism.  A community outreach coordinator and I conducted these through the project to build buy-in for more participation. It remains a professional highlight that we could work with the community in building something With Not For, especially since Civic Hall was such a beautiful meeting place for entrepreneurs, government officials, non-profits, and anyone who believed in Civic Tech for good.

Thematic Network Map

Approach

The product development process

I managed the product development by setting up best practices to build out all parts of Refer NYC, including:

Define

• Review previous work to identify goals, hypotheses, success metrics, and personas
• Identify the design process – methods, timelines, resources required, acting as project manager/Scrum manager in partnership with our vendor
• Get aligned with the vendor, Civic Hall Labs team, and clients on the process.  We had frequent meetings with both the vendor and our client stakeholders to inform them of our work
• Identify non-profit organizations to participate in research and use the product

Discover

• Create Discovery questions for non-profits and organize interviews.
• Conduct interviews at the referral organization’s offices using interviews and contextual inquiry to observe the referral process.
• Manage community town halls: Capture ideas from a lunch series and learn where community members outside of our end user non-profits provided their perspective, including Open Data advocates in Civic Hall.
• Create a light Product Requirements Document (PRD) as requested by stakeholders to understand the scope.
• Synthesize research into hypotheses and product roadmap, features, user stories, and prioritization

Design and develop

• Synthesize research into hypotheses and product roadmap, features, user stories and prioritization (timelines and scope of MVP, first release etc.)

• Create user stories and sprint backlog into product roadmap and build out the data exchange. A sample user story was “As a data manager, I want to view a list of which fields don’t map to HSDS so I can make changes and improve the quality of data”
• Capture feedback:
  • Identify prototype feedback process and guidelines for feedback
  • Install prototype onto machines, use the prototype and provide feedback

Reflect

• Analyze feedback:
  • Determine  how  to  prioritize  feedback  into  product  backlog  via The Systemico Model to help prioritize and balance new and existing customers
  • Attend Federation convenings  to  determine  new  product  feature  priorities  and  various  governance and  sustainability  decisions and share findings at  Community Summits
• Measure  and  evaluate  success  metrics: Ensure our work was valuable to the community and stakeholders
• Transition: Transfer technology over to the City of New York for implementation (see results)

Solution

The research pointed us to three components for our design intervention:

One of the challenges was that stakeholders were not familiar with what the technology part of our work was, so these diagrams helped explain what exactly our technical vendor was building

Results

This is the first comprehensive view of health and human services in New York City, and as a result the solution was integrated into the City’s procurement systems which manage data about non-profits and those providing services. Eventually, the ReferNYC Open Data Exchange can be scaled out to the larger community for public use, including as part of the City’s Open Data work. Part of the work ahead for the City is to understand how to grow the customer base externally, or use it internally within the City to first have the best data set.  The data allows the City to understand geographic gaps where services aren’t available, to have accurate information about service providers, and to help lay a foundation where service providers can seamlessly share what they know about themselves and benefit others.  Non-profit organizations, the City of New York and New Yorkers all benefit from better referrals to lead happy, healthy lives.

Data quality

• Insight: End users – non profit community organizations—didn’t have the time to update information about other nonprofits to refer people to.  They didn’t want to just ‘Google’ it, and wanted to be able to have custom CRM that had organizational history while also having referral info.
• Activities that solved it: We reduced the time it took to update information by creating a data exchange that would automate much of the basic information about non-profits, including what services were provided.

Innovation

• Insight: Independent developers and organizations in the Civic Tech community at Civic Hall were interested in clean data about referrals
• Activities to solve it: We could increase innovation by creating an API that others could use to both create new applications and improve existing ones 

Management

• Insight: The City of New York was finding the idea of a data exchange to understand ALL non-profit community service providers incredibly valuable for them to manage contracts with those vendors
• Activities to solve it: Implementation: We ended up having the City of New York ‘own’ the project so they could not only benefit from clean data but also use it to reduce the challenges around contract management

Reflection

This was a complicated but deeply meaningful product in that it was a true ‘marketplace’ – we were able to help New Yorkers, non-profit organizations and the City of New York itself.  Some reflections include:

I like

• I was happy to work as a product manager: I was able to not only research, but create the product roadmap and identify the priorities for the product.  A key highlight was reading about the Systemico Model, which was something far more powerful than RICE (‘reach, impact, confidence, and effort’) or MoSCoW (‘must have, should have, could have and will not have”). (More on prioritization frameworks on Folding Burrito).  I advocated for its use to ensure we were helping multiple users – data managers, as well as the City of New York. Systemico advocates to understand across core functionality, use, engagement and exploration across user goals so we could ensure engagement across both new and existing customers.
• Co-designing builds trust and a great product: We chose to have a Federation of participating organizations help shape the product roadmap and leaned on participatory design (‘co-designing) rather than traditional user research like individual interviews. All of us were uncertain if our three participating organizations and stakeholders would be open to trying co-design, but there was a sense of shared ownership among the organizations, and people loved how their ideas were incorporated into the work in subsequent updates. It was incredible to see the excitement as they shaped the roadmap and helped ensure transparency and sense of community about the product.

I wish

• Building requires much partnership: part of our challenge when building a product is understanding when to focus on building and pivoting into marketing and outreach. There were partners like the Open Data teams in NY and at the federal and state levels who would be great to work with. We only scratched the surface of what we could do in time, which allowed us to build more momentum with our steering committee within the city. Rather than identifying users, we should actively look for partners and have dedicated staff to help scale out that work. Elizabeth Adams talks about the key ingredients for civic tech projects and identifies partnerships as key. 
• Data ecosystem products are valuable but misunderstood: this was my first data product as a product manager, and I was excited to see the efforts grow. At the same time, many folks see Web and mobile products as a priority in civic tech, and behind-the-scenes services like ReferNYC could be seen as less urgent. There is always a sense that infrastructure products may seem easier to build, but I now wonder if we could get more excitement for the work if it had a consumer-facing angle. While we were helping New Yorkers, the end users were also non-profits, and more marketing to them of the benefits could help with evangelism that data products are valuable. 

I wonder

• Customer growth is always a concern: I wondered how evangelism of the product would work – while we had three enthusiastic non-profits as our first customers, according to the Human Services Council, there are over 200,000+ employees in New York City working for health and human services organizations – I’m not even sure anyone has an accurate number of how many organizations there are. The challenge would be about Go To Market – how people would hear about the product and how we could scale out the team size to meet the research and implementation time needed.

In the same spirit, we ought to consider taking on roles with an eye on long-term changes and impact we can help get off the ground or simply push forward. We may overestimate how much progress we can bring about within the first year, but I believe a career focused on advancing projects bigger than anyone one person could do alone can become deeply gratifying. – Daniel SchnelbachT, 2020 (Medium)