Smart Cities - The Rise of Aggregated Open Data Portals, No More Silos

The rise of open data began with widespread computerization of city administrations in the 90s, followed by rapid digitization of hardcopy records in the early 2000s. With digitized data, individual city agencies began to open up pockets of data to interested parties. One common example is the geo-spatial land-use data, usually provided by land planning agencies for a fee.

In the late 2000s and early 2010s, cities began investing in sensors and IoT (Internet-of-Things) infrastructure, enabling the collection of real-time data to improve city awareness and operations. However, over time, the volume, velocity and variety of data began to outpace and overwhelm the city government’s capability to manage.

In the same time, the world witnessed a growth spurt in the number of software and application development professionals, partly triggered by the feverish rush into innovative mobile app development in the late 2000s.

With massive amounts of data on one hand and a growing developer ecosystem on the other, the path to overcome the city government’s limitation of producing innovation from data is clear – open up access to city data publicly and let the developer ecosystem produce the innovation.

By releasing city data publicly, cities can realize highly strategic benefits.

“Whole-of-government” Initiatives.
Most government employees are not aware of the range and depth of data that other departments possess. By encouraging each department to release open data, the city government will itself benefit in the long term.

With shared open data, government employees gain knowledge and awareness of cross-departmental data. Instead of working on traditional departmental initiatives, the government is now positioned to plan and roll out “whole-of-government” initiatives.

“Let a Thousand Flowers Bloom” Ecosystem.
Smart cities and innovation are symbiotic. Innovation helps make cities smarter and smarter cities feed resources and infrastructure that further enable innovation. One such critical ingredient is the growing mountains of data that a city has – from traffic conditions, buildings, tourism, air quality, transportation and many aspects of a city's operations and environment.

Instead of looking inward for ideas (narrow, slow approach), governments can draw from the city’s broader entrepreneurial and developer ecosystem for innovative ideas. Service providers and application developers in the innovation ecosystem can then harness the open data to provide useful services to city residents and workers, making the city "smarter" in a quicker way.

Trust and Transparency.
With open data, the city government will inevitably open up its departments to public scrutiny. This adds another layer of self-review and integrity-checking for the city departments. In the long run, the transparency, due diligence and scrutiny reduces corruption and builds trust with the public, enabling the government to roll out new data-driven policies with higher acceptance.

The release of open data can be done using a silo-ed, departmental approach which is a traditionally straight-forward approach when some departments have digitized their data earlier than the others.

However, in the situation where most or all city departments have completed their data digitization and have commenced regular data collection, cities should aggregate and release all their open data in a single website portal. Aggregated open data portals bring clear and highly productive benefits to both city government and data users.

Obvious Simplicity.
The most obvious benefit to anyone is simplicity i.e. all the city’s open data, no matter how massive and varied, is always and simply aggregated in one place. The simple knowledge of a single website will enable anyone, government workers and data users alike, to productively find just about any data that is released openly by the city.

Consistency and Standardization.
Having all open data in a single portal enables the city government to easily ensure that the data is available in common and machine-readable structured formats and provide standardized single and public point of API access to access the data. Government users can be easily trained on standardized workflows and methods to maintain and update the data.

Single Point of Review and Audit.
Likewise, it becomes more effective and easier to review, filter and screen the released open data to avoid inadvertent disclosure of sensitive or un-anonymized data, and measure open data performance metrics for each department, such as relevance of data, timeliness of data updates, integrity of data and others.

However, an aggregated open data portal does carry performance and security risks, such as being a usage bottleneck, single point of failure and single target of cyber-attacks. This can result in higher cost investments in fault-tolerant, load-balanced, scalable and highly secure portal infrastructure.

Today, there are more than 200 major cities, states and provinces around the world that have rolled out on aggregated open data portals. At RList Insights, we surveyed the major cities and have compiled an updated list of most major cities and their open data portal URLs (yes, they are all valid links, as of Oct 2019).

There are 81 major smart cities in North America with aggregated open data portals (68 from United States and 13 from Canada). In Europe, there are 65 major smart cities with aggregated open data portals, mainly from Italy (13 cities), Spain (12 cities) and United Kingdom, France and Germany (7 cities each). In Asia, there are 40 major smart cities with aggregated open portals, primarily from Japan (14 cities), South Korea (9 cities) and Taiwan (7 cities).

It is important to note that not all countries have rolled out city-level aggregated open data portals and have only deployed country-level aggregated open data portals. We focus only on city data portals in this article and will cover country data portals in a separate article. In addition, our team has also excluded data portal URLs that belong only to a specific department of a city (for example, the data portal of the land use agency providing open geo-spatial data.).