As Polkawatch turns 3, seems like a good time to look back and check progress made.

Why measuring decentralization in the first place? Many data analysts, when asked the question, will reply that you can only improve what you measure.

Since the Polkadot community has been measuring decentralization for 3 years now, it is fair to ask the question: Have decentralization improved?

A quick look at now and back then charts leaves no shade of doubt:

Back when we started, a handful of operators dominated most of the validation process. Fast forward 3 years and the situation has improved significantly, while there are still different sizes of operators, including some leaders, taking control of half of the network would imply taking control of more than 2 dozens of node operators.

And what about data-centers and IP networks used during the validation process?

Significant progress can also be appreciated on the IP Networking front, with significant more players controlling half of the validation process by stake.

With regards to regional decentralization:

Significant progress was made, with big increase of Asian and South American validation. However, we continue to validate too much from Europe.

When replying the question above, its fair to say that Yes! The Polkadot community is continuously improving the decentralization of its network.

A Community Effort

Unlike first generation blockchains, that use POW (proof of work) to make it difficult for opponents to attack the Network, Polkadot uses NPOS (nominated proof of stake) instead. Nominators elect validators with their stake, and the aggregated backing is used during the validation process to "vote" for validity of each network block.

So, nominators collectively decentralize the network by adjusting their nominations, using their stake to "vote" where validation will take place.

Polkawatch allows nominators to visualize the decentralization of the network, and the decentralization of their own nominations, allowing them to take the staking decisions that benefit the network the most.

With the Polkawatch API, community projects can embed decentralization analytics in their DAPPs, allowing our nominators to visualize the Decentralization Analytics right where staking decisions are made. A great example is Polkadot Staking Dashboard:

Other community initiatives are geared towards decentralization, a good example is Web3 Foundations decentralized nodes initiative, "A Delegation Program that Supports Validators in the Polkadot Ecosystem" ensuring that nodes "Are sufficiently decentralized in all aspects".

So it is fair to say that de progress made in decentralization is a community effort from Nominators to DApps, to the Web3 Foundation.

JAM and the Future of decentralization

We are used to general purpose computers that can do lots of different stuff, it took decades before general purpose computers, both personal computers and servers were introduced. Purpose specific computers were the norm for decades.

Pioneering blockchains such as Bitcoin, Ethereum or Polkadot, can in fact be thought of as purpose specific decentralized computers. A decentralized computer resource, builds on top of "nodes" which are built on top of independent general purpose servers.

The Polkadot ecosystem introduction of JAM will provide the first general purpose decentralized computing resource. A resource on which a "service" can be deployed, but this time the service could be similar to Ethereum or Bitcoin. The capability to host Polkadot blockchains on JAM is under development, while the community is looking for more examples of decentralized services that could be run on JAM, with an open mind.

As for decentralization of the novel computing resource, JAM must inherit the capability that Polkadot has to continuously improve its own decentralization.

Furthermore, there are more dimensions to decentralize than just Network Validation, in fact JAM starts by funding several teams to develop independent implementations of the protocol, a good start.

About Polkawatch

Polkawatch is an analytic tool designed to measure effective decentralization of Polkadot’s Validation process.

Polkawatch is a project originally supported by the Web3 Foundation grants program and currently supported by the Polkadot treasury.

In our last Kusama Treasury proposal we agreed to improve Polkawatch observability. In an attempt to make this work more useful to our community we decided to share all the details in this Operational Insight blog post.

We love tools like Grafana and Prometheus, most of our components are integrated with them including our Polkadot and Kusama Archive nodes.

While we think these tools excel at monitoring all the insights of each individual component, we find it difficult to get an idea of how our full-stack service is performing, in general.

Operating our Stack

Polkawatch Stack is similar to a Substrate indexing application with a DAPP as frontend. However, our stack includes also archive nodes, 2 levels of indexing (chain facts + analytics) and the production of data packs for the DAPP which are published over IPFS.

Many things can go wrong, at each level. And we don't want to have to monitor key metrics of every module to keep our operation agile.

Black box observability for Decentralized Applications

How can we observe the maximum number of components in our stack in the simplest possible way?

We find it useful to publish via Polkawatch own API metrics about the dataset. For example, the timestamp of the last processed Reward Event. If ANY component in our stack misbehaves, that timestamp will start to drift back in time.

In Polkadot, for example, Rewards are issued every day, so, a 24h age of the last Processing event can be considered normal, 48h means something has failed, perhaps not permanently, 72h signals a deeper problem, etc.

Furthermore, because our DDPs are published via IPFS, during monitoring we can request the API information via Public IPFS gateways, that way we also verify our IPFS infrastructure at the same time.

Introducing Data Quality Observability

We are now also introducing a second metric related to data quality. Our API includes now a metric which reports the percentage of rewards that could be successfully traced. The feature is introduced in our Kusama stack first, Polkadot will follow.

A reward event is successfully traced when its Network, Country, Operator, etc can be determined. This process can fail, for example, when a geolocation IP database is missing information. A small percentage of rewards not traced can be considered normal.

Data Quality and Dataset age metrics allow us to deliver black box full stack observability without a single direct request to our stack.

Getting information that matters

Polkawatch is not such a big project as to justify full time operational staff. We manage operations part-time while developing new features for Polkawatch and/or developing other new Projects.

For us, it is important to get the operational information that actually matters via the channels at which we are normally present, for example Matrix/Element messenger.

In the past, for IOT projects we have used Node-Red to quite easily maintain visual workflows that integrate with multiple components. This software is ideal to implement observability in a distributed system that it is easy to maintain.

It was great to find out that Zooper Corp team, details below, had already implemented Polkadot support for Node-Red.

It was very straight forward to adopt Node-Red while implementing observability the way we like it. Here is an example of the flow discussed above:

Every 30 minutes a Public IPFs endpoint is used to get information about the dataset, the Age of the last Reward event is selected. If the event is too old a message (1 max per 12h) is delivered via Matrix to the team. If Age is OK, a different message is logged via Syslog.

We find it useful to generate intuitive messages that include also intuitive severity, as shown. In our case the number of hours in age is easy to understand. Message frequency is also limited based on severity.

Sometimes our stack may run for weeks without a hiccup, and for those periods we find it very useful to leave an equally easy to read trail of syslog messages that serve as testimony that everything is going just fine, just a few messages per day. For example:

polkawatch[0]: DDP Generation on kusama finished OK
polkawatch[0]: Kusama Data Pack seems OK, currently 23 Hours old.
polkawatch[0]: Kusama node is in sync, delta is 3.
polkawatch[0]: Our Kusama Node v0.9.41 is up to date.

Observability in a decentralized ecosystem

With this setup we can very easily observe our stack as a member of our distributed community.

For example, we find it useful to observe if the last block of our nodes drifts when compared to reference nodes in our ecosystem, i.g. those of our top RPC operators.

We can also check if the Node version drifts when compared to reference node operators in our ecosystem, as follows:

In this flow, the RPC interface of our node and a reference node is used to retrieve the nodes versions. Semantic versioning is used to calculate the delta between the two versions and its type (patch, minor, major). Priority is assigned based on this delta, and intuitive messages are delivered based on this information.

The message above will be sent maximum once every 3 days, with the current flow setup.

Our takeaway

Observability in our decentralized ecosystem has its peculiarities many of which help us monitor our infrastructure easier than in other environments.

Node-Red is a fantastic tool to create agile workflows that allow us to observe our infrastructure very easily and intuitively while taking advantage of decentralization. Node-Red workflows are easy to maintain.

When something fails, we have Grafana, Prometheus, and log servers to have a closer look, until the issue is resolved

About Polkawatch

Polkawatch is an analytic tool designed to measure effective decentralization of Polkadot’s Validation process.

Polkawatch is a project originally supported by the Web3 Foundation grants program and currently supported by the Polkadot and Kusama treasuries.

About ZooperCorp

Zooper Corp is the Interplanetary PoS Blockchain Validator. A team of DevOps experts that maintains several DotSama validators including Polkadot, Kusama, Moonbeam and Moonriver.

Zooper Corp is part of the Web 3 Foundation 1KV Validator Program and is the original contributor of the Node-Red plugin for Polkadot, they can be reached on Twitter.

About Node-Red

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

While polkawatch provides us with multiple analytics of the effective decentralization achieved by the polkadot network, we still believe that some relevant information may get lost in the details.

Polkawatch was created with the idea that each member of the community could see how their actions are impacting all of us, providing an opportunity to act accordingly.

By providing detailed analytics by nearly 28.000 entities, including regions, countries, operators, networks, nominators etc, we may fall into a situation where the trees don't allow us to see the forest.

Therefore, we decided to go back to the analytic database and compile a report with the following main goals in mind:

• Long term trends: where is decentralization heading to?
• Events: what remarkable decentralization events took place recently?

We also try to make sense of the data, share opinions received while discussing decentralization with the community and brainstorm possible actions that could help decentralization going forward.

The ultimate goal of the report is to generate debate in our community.

Download the report here:

We are happy to announce Nomination Pools decentralization analytics for Kusama.

Earlier this year the Kusama Network announced the general availability of Nomination Pools.

Polkawatch is an analytic tool designed to measure effective decentralization of Polkadot and Kusama’s Validation process.

It uses the Reward event as main measure of Computing Effort and it complements the information with networking information (IP address) of validator nodes which is then crossed with external geolocation datasources.

With the support of Nomination Pools, Polkawatch allows Pooled Nominators to analyze the effective decentralization of the particular pools they join.

Pool operators may also tune their Pools to improve their decentralization metrics.

Kusama Nomination Pools decentralization analytics are available at:

https://kusama.polkawatch.app/pools

About

Kusama is a scalable network of specialized blockchains built using Substrate and nearly the same codebase as Polkadot. The network is an experimental development environment for teams who want to move fast and innovate on Kusama, or prepare for deployment on Polkadot.

It is our pleasure to announce the release of Kusama decentralization analytics!

Polkawatch is an analytic tool designed to measure effective decentralization of Kusama’s Validation process.

It uses the Reward event as main measure of Computing Effort and it complements the information with networking information (IP address) of validator nodes which is then crossed with external geolocation datasources.

Polkawatch allows us to navigate our rewards, or validation operation, by geography, computing network and node operators.

Polkawatch, introduced last June, was originally developed for Polkadot, but in motion #521 the Kusama community decided to roll it out on Kusama too.

Polkawatch is bringing awareness about the decentralization challenges that can potentially take place in novel Proof-of-Stake (PoS) blockchains, when the majority of nominations are directed to a few Operators, Networks or Geographical regions.

This issue is far from specific to the Substrate ecosystem and is already making headlines around the world. After the release of Ethereum Proof-of-Stake capability it was observed that 64% of staked ETH is controlled by 5 entities, furthermore, the issue of regional node clustering is already been argued in court as grounds for jurisdiction over Ethereum.

We hope the Kusama community will take action and fully realize its decentralization potential.

In this sense Polkawatch allows every single nominator to analyse the decentralization of its own nomination and make changes accordingly.

Kusama decentralization analytics are available at:

https://kusama.polkawatch.app

About

Kusama is a scalable network of specialized blockchains built using Substrate and nearly the same codebase as Polkadot. The network is an experimental development environment for teams who want to move fast and innovate on Kusama, or prepare for deployment on Polkadot.

Validator Alliance Panel+QA: Why is it important to rely on independent validators?

The panel discussed the importance of the Validator role in the dotsama community, comparing it to other cryptos. Validators share the responsibility of validating transactions as a public service to our community.

More technical aspects such as decentralization versus latency were also discussed.

Polakawatch decentralization analytics were also discussed.

If you missed the conference you can watch this panel here:

About parisDOT

3 days of conference and workshops sessions organized by some of the leading projects of the Polkadot and Kusama ecosystems.

Offering an inclusive space to meet, greet, talk and exchange thoughts, and perhaps kick off the next interesting collaboration. Targeting the wider crypto community.

About DOT Validator Alliance

We are a group of community leaders who decided to dedicate their professional lives to the Polkadot and its diverse ecosystem.

Founded by Polkadotters, Promo Team, Repe, Pathrock Network, Stakenode, Bld nodes, Crifferent and Iceberg Nodes.

Polkawatch provides decentralization analytics for Polkadot and potentially for any blockchain built on the Substrate framework.

Polkawatch is a project supported by the Web3 Foundation and was part of the Wave 13 of the grants program.

We covered the motivation behind creating a decentralization analytics platform for Polkadot and showcased some of the available data when Polkawatch was introduced last month.

In this post we would like to cover the technology behind Polkawatch and what it brings to the Polkadot ecosystem.

Polkawatch Delivery

Polkawatch benefits from the rich Substrate ecosystem and builds on top of other community assets: the substrate archive node and substrate indexer.

The substrate indexer is used to extract canonical blockchain events from the archive node and put them in a workable format: SQL.

Polkawatch runs a second-pass indexing that traces relationships between canonical events and complements them with external data sources such as GeoIP databases. The second pass indexing is delivered as a Lucene inverted index which is easy and efficient in terms of implementing complex real time analytics over the dataset.

Polkawatch provides a Live Query Server that allows for analytic queries to be run over the inverted index in real time.

Polkawatch also provides a Distributed Data Packaging solution that allows for the distribution of analytics over IPFS, bundled with a visual DAPP:

Polkawatch.app is currently in production and delivering decentralization analytics about Polkadot. The dataset is distributed as a DDP updated daily and can be browsed with a DAPP.

For additional details about the technology stack visit the polkawatch gitlab page.

Next Steps

We plan to deliver Decentralization analytics for Kusama with support for newly released staking pools.

Some parachain projects have also reached out seeking to introduce decentralization analytics for their collator operation too.

We will continue to use decentralization as the main use case for analytics. However, we foresee parachain project interest in other analytics related to the nature of the parachain itself.

Contributing to the Polkadot Ecosystem

With Polkawatch, the community can now see the decentralization status of the blockchain as well as their own contribution to decentralization.

The Polkadot Rely Chain can now visualize the status of decentralization and motivate operators to cover regions and/or network operators to balance network resources.

Validators can ask for nominator support as they pioneer new regions.

Nominators can also see the impact of their nomination in the overall decentralization of the blockchain.

In the near future parachains will also be able to assess the level of decentralization of their collators.

Accessing Polkawatch

Polkawatch can be accessed at polkawatch.app .

About

Polkadot is a scalable sharded chain and the first protocol that provides a secure environment for cross-chain composability across multiple shards. Polkadot also introduces a highly advanced, open governance system that will allow the network to innovate and grow at a much faster pace than legacy networks. Applications from DeFi to energy to gaming will thrive on Polkadot, challenging the centralized platforms of Web 2.0.

Web3 Foundation funds research and development teams building the technology stack of the decentralized web. It was established in Zug, Switzerland by Ethereum co-founder and former CTO Gavin Wood. Polkadot is the Foundation's flagship project.

Valletech AB, the developer behind Polkawatch, is a Product Development Lab that helps customers innovate Software Products that nicely fit into leading OperSource stacks, including latest Web 3 technologies.

Today we are introducing Polkawatch, decentralization analytics for Polkadot, and we would like to share how this project started:

Like many other crypto enthusiasts I have followed the development of Bitcoin with excitement for years. Most of those years I thought that Blockchain technology was still too early for mainstream adoption.

It all changed with the introduction of PoS and Decentralized Finance. It was clear to me that mainstream adoption was becoming increasingly feasible. In this context, Polkadot seems to me like the rising star.

While reading Crypto news an article really caught my eye: A SEC boss claimed that DeFi platforms are actually “highly centralized”. He further elaborated that while these platforms may provide some decentralized aspects they are highly centralized in some others. And then I started to wonder: Are they?

Decentralization, a pragmatic approach.

When we, as a community, create a decentralized system we think on things like p2p networking, consensus algorithms, cryptography and security. We design and architect for decentralization. In a way our community is building a system with decentralization potential.

A very different thing would be to analyze the resulting “systems”, as in all required subsystems for operation, and measure their degree of “effective” decentralization, meaning decentralization that has been already realized.

For example, Polkadot is a computer program made of lines of code. Who wrote those lines? Which companies do they work for? How many companies account for 75% of the lines of code? If the result were 3, for example, we would have a highly centralized blockchain in terms of its engineering maintenance.

A similar exercise can be done for our computing operation. Our nodes run on computers, on which networks are those computers running? Which companies operate the networks running our computing nodes? How many companies host more than 50% of our computing? If the result is 3 that means that despite our potential our computing operation remains centralized.

Similar exercises could be done for all the systems and subsystems our activity depends on regardless of their nature: algorithmic, infrastructure, regulatory, people, marketing, communication, etc.

I guess decentralization works like a chain. Since a chain’s strength is that of its weakest link, it would be fair to say that a system is as decentralized as the most centralized subsystem it depends on.

Decentralization Matters

While most experts agree that the blockchain/cryptocurrencies are here to stay that doesn’t mean that it will be our technology, that it will be us running it or that it will be used in the way we intend.

As a community, We have tones of reasons to be very skeptic and must work towards effective decentralization. In my opinion, our worst nightmare would be that our blockchain technologies end up been used by dystopian societies to “administer” freedom: Imagine Panem's Capitol administering obligations and rights of the 13Districts in a “decentralized” manner.

We should learn from previous technology waves, such as the Internet and Social Media. Even with their technical shortcomings they could have produced significant Social benefit. It is clear that in many ways they are been used against their initial purpose and creators intent.

Will the blockchain be any different? I hope so.

At least we can learn from the past, and hedge against concrete risks, such as:

Regulatory Uncertainty: Blockchain is in its infancy, and its regulation is unclear in many administrations. Furthermore, sudden changes are often introduced in a way that force a shutdown of entire network segments while our participants implement or challenge compliance.

Deplatforming: There are plenty of examples of deplatforming of key actors in previous technology waves, some are: Wikileaks, Parler, Trump. Specially remarkable is the speed of the deplatforming process and the absence of due-process. Whether one agrees or not with the ideas of the entities being deplatformed is not the point, the absence of due process is.

We should also pay attention to the historic moment we are living. Many experts consider our monetary era coming to an end, some predict hardship or even collapse. Some crypto enthusiasts may see this possibility as an opportunity, however history reminds us that this kind of situation was not easy for alternative stores of value, just consider US Executive Order 6102. The incremental adoptionof sanctions versus using due-process or international arbiters to resolve conflicts point in this direction.

As a community we must realize our decentralization potential making it effective in all required areas and we should hedge for the risks that we have observed in the past.

Introducing Polkawatch

Polkawatch is an analytic tool designed to measure effective decentralization of Polkadot’s Validation process.

Polkawatch is a project currently supported by the Web3 Foundation grants program.

It uses the Reward event as main measure of Computing Effort and it complements the information with networking information (IP address) of validator nodes which is then crossed with external geolocation datasources.

Polkawatch allows us to navigate our rewards, or validation operation, by geography, computing network and node operators.

This not only allows us to measure the realization of our decentralization potential, but it also allows us to measure the rewards produced by our own nomination, which is useful for us to know if we are contributing to decentralize Polkadot or not.

Polkawatch can be used to hedge against the risks exposed above. Deplatforming risk can be minimized by watching the percentage of rewards generated on each computing network provider.

Regulatory related risks by watching the amount of rewards generated from different countries. Absence of due-process risks can be mitigated by not letting any operator become too big, too critical for the operation, etc.

Current Decentralization Status of Polkadot’s validation

Overview

The current overview of Polkadot’s validation network as of Era 746 looks like this:

We can see how rewards are handed to different participants. Rewards or Validator Commissions to Public Validators or to Custodial Validators. Custodial Validators are those that set a 100% commission, and normally stake funds of their customers.

Rewards to Custodial participants make 60% of our network with over 30% of rewards made by nominators on public validators. Of course other rewards go to node operators (commissions) and perhaps there is an extra category of “independent” validators, public validators without an identity.

Polkawatch focuses on the “Public Validation Network”. While the activity custodial participants is significant their exposure to risk is different, and so are their own mitigation tools.

Geographical distribution of public Validation

During the last 60 eras there are currently 6 regions generating public rewards, which is impressive, however most of it is generated from Europe and US, with more than 75% generated from Europe alone.

If we drill down into Europe we can see that participation of countries, the main source of regulatory related risk, is not balanced either.

It is fair to say that Germany, UK, France and US host most of the Public Validation of Polkadot.

However it is very encouraging to see that there were no less than 24 countries participating in reward generation from 4 regions during the last 60 Eras.

Computing Network Providers

There was 40 network providers hosting public validator nodes, and their reward distribution is:

Once again most of the activity is concentrated in a few entities. Note that huge network operators may have more than one network in their name.

Node Operators

As for public reward generation there were 168 operators running 246 validator nodes that generated public rewards during the last 60 eras. And their weight distribution looks like this:

All in all, it is very encouraging to see the numbers of participating entities for such an incipient community:

However it is clear than in order to realize our decentralization potential we need to balance our resources effectively.

Accessing Polkawatch

The whole point of polkawatch is to empower the community to take decentralization in our own hands so:

How decentralized is your nomination?

Find out at polkawatch.app by analyzing your own nomination.

About

Polkadot is a scalable sharded chain and the first protocol that provides a secure environment for cross-chain composability across multiple shards. Polkadot also introduces a highly advanced, open governance system that will allow the network to innovate and grow at a much faster pace than legacy networks. Applications from DeFi to energy to gaming will thrive on Polkadot, challenging the centralized platforms of Web 2.0.

Web3 Foundation funds research and development teams building the technology stack of the decentralized web. It was established in Zug, Switzerland by Ethereum co-founder and former CTO Gavin Wood. Polkadot is the Foundation's flagship project.

Valletech AB, the innovator behind Polkawatch, is a Product Development Lab that helps customers innovate Software Products that nicely fit into leading OperSource stacks, including latest Web 3 technologies.