On the surface, the job of a statistical agency like the U.S. Census Bureau is deceptively simple: gather some data, compute interesting statistics from that data, and release them to the public. Of course, it’s not that simple.

The data you download from our website or API and the statistics you see cited in the media result from a complex set of interactions between people, software and systems. At the Census Bureau, we refer to the software and systems we use to gather, process and disseminate data as our Business Ecosystem (BE). In this blog, I want to update you on how we’re evolving the BE to better measure our dynamic population and economy. 

For decades, the Census Bureau utilized siloed systems and software tools to collect and process data across different census and survey programs. This, in part, reflected different funding streams and methodological approaches. This resulted in costly duplication and meant some programs adopted innovations like internet data collection before others. Moreover, it also meant that our statisticians, demographers, economists and other subject matter experts didn’t always have access to all the source data relevant to their work. You could say we had many business ecosystems.

There are several factors leading us toward a more integrated and modern BE. These include greater use of internet data collection by our surveys, increased reliance on direct access and data feeds from businesses and government agencies, modernized methods like AI, and the need to make all the relevant data and information available to our teams so they can better improve the statistics we deliver to the public with blended estimates.

The BE is evolving to efficiently address these needs through four major components:

• First is Data Ingest and Collection for the Enterprise (DICE), our suite of tools to ingest and perform basic curation regardless of mode (online/paper/in-person surveys, data feeds from businesses, imagery, etc.).
• Next is the Enterprise Data Lake (EDL), our cloud-based data repository and scalable compute environment. 
• Frames are authoritative and up-to-date lists of addresses, businesses, jobs and demographics that provide the skeleton from which we construct blended data products as well as the universes we draw survey samples from.
• Finally, Census Enterprise Dissemination Services and Consumer Innovation (CEDSCI) is how we deliver statistical products to our users whether on data.census.gov or via our API or other tools.

While there’s a lot happening under the hood, it’s helpful to think of the BE as a holistic framework that connects and integrates the many sources of data we collect, process and publish. It’s not just a database or a new software platform – it’s a different way of thinking about our data, enabling us to produce richer, timelier and more relevant statistical products. The BE allows us to efficiently scale common tools and open-source software applications across all our programs, while still permitting program-specific methodological approaches.

On this last point, most improvements in statistics are made possible by improved source data like new surveys or newly available administrative data. An excellent example from earlier in my career is the Longitudinal Business Database (LBD), which a colleague and I created from linked annual snapshots of the Census Bureau’s business frame. The LBD facilitated research and statistical products that fundamentally altered our understanding of the role of entrepreneurship and business dynamics in economic growth – new data leads to new knowledge. 

I’ll write more about the great work being done to bring new source data to bear in my next blog. Just note that the BE is critical to these efforts and will permit much more efficient ingest, curation and utilization of new source data – certainly easier than when we were building the LBD back in the day.

We’re already putting our modernized BE to work, and the benefits are clear. Frames and CEDSCI are already widely used across all our programs, and they benefit as we continuously upgrade and modernize those BE components.

Moving programs to DICE and EDL requires considerable engineering to optimize performance in the cloud. Our new Annual Integrated Economic Survey (AIES) is the first survey to fully utilize DICE and EDL. But several DICE components, especially those involving Internet data collection, have been in use for some time by our business surveys and notably the 2020 Census. We’re currently using DICE for our mid-decade special censuses and will be using it in 2030 Census testing.

We are working hard to migrate our remaining surveys to DICE and EDL. We are currently testing, for the first time, internet data collection on the Current Population Survey (CPS). We jointly conduct the CPS with the Bureau of Labor Statistics (BLS) and it’s the source data for BLS’s monthly Unemployment Rate and the Census Bureau’s annual Income and Poverty estimates. Like all surveys, CPS response rates have been declining over time. BLS and the Census Bureau are exploring data sources and methods to ensure sustainability of these crucial data. Internet response is a critical first step that will help mitigate increasing costs and give household more options to respond.

We’re still in the early phases of the BE journey. Over the next few years, we’ll be migrating more of our work to the BE. This includes, of course, the 2030 Census, which will leverage the BE to help us reduce costs and enhance data quality by building on the successful innovations of the 2020 Census – such as online self-response, field automation and administrative records.  

We’re excited to share more about the BE as it develops. We’re especially excited about how the BE will better enable us to utilize new data sources to produce better statistical products – the topic of my next blog. So, stay tuned for updates.