Filtering

Filtering language

Filtering is applied through the --common-include-filter-expr and --common-exclude-filter-expr parameters. They are both CEL programs.

Filtering in the dfuse platform uses the Common Expression Language, developed by Google, and used in high throughput environments (like their ACL checking routers).

It is a very simple, yet powerful language. It resembles all the languages you already know (Java, JavaScript, C, C++, etc.)

Important: They filter what gets processed by different components across dfuse for EOSIO stack. They are not a language you can use to query dfuse Search indexes.

CEL is Google’s Common Expression Language:

• Introduction
• Language reference
• Google Open Source CEL Website
• Go implementation used here

Components

Filtering is applied on certain components, while other components are happy to accept filtered data and process it.

relayer

The relayer is the first component to filter in a deployment.

It will filter blocks it receives (from either another relayer or a mindreader), and serve filtered blocks through the same BlockStream gRPC interface. Each block within will be marked as FilteringApplied == true (one of the values in the EOSIO-specific protobuf Block).

It uses primitives available to all components, but when the relayer does the filtering, it cuts on much of the network traffic that follows, thereby shrinks much of the RAM needs of all components downstream (like search-live, dgraphql, blockmeta, etc…)

search

The search will only index actions that matched the inclusion filter and did not match the exclusion one.

The forkresolver, the indexer and the live components of search will all potentially need the filtering criterias, as they might pick up merged block logs from Object Storage, and index things on-the-fly. In that case, they will first apply the filtering criterias.

trxdb-loader

The trxdb-loader component will only save transaction traces in the database that contains at least 1 matching action.

This means that, once filtered, the resulting trxdb will not contain all transactions. Will not be able to resolve transaction traces requested by dgraphql or pointers reported by search unless they are also in this database.

statedb

statedb will be affected by filtered data coming from a relayer that relays filtered blocks.

You need to be diligent in what you filter, to not miss table changes that would otherwise make your view of tables inconsistent.

You can use db.table and db.scope in your filtering criteria to make sure transactions that mutate certain tables always make it through filtering.

Identifiers

An similar identifiers available for searching in dfuse Search is available for filtering but with more capabilities because any action’s data field can be search for whereas in dfuse Search, only a subset of all fields is possible.

For example, a dfuse Search of:

receiver:eosio.token data.from:bob

would be filtered as:

receiver == 'eosio.token' && data.from == 'bob'

See the Search Terms page for all EOSIO terms that can be filtered.

Examples

Showcase examples here are given as examples, mainly for syntax purposes, so you can see the full power of the CEL filtering language

There might be new stuff to add to certain examples, like spam coins or new system contracts, this is not a fully accurate document for those stuff, you are invited to make your own research to ensure completeness based on your use case.

EOS Mainnet Spam

Here an example to filter out spam transactions on the EOS Mainnet:

account == 'eidosonecoin' || receiver == 'eidosonecoin' || (account == 'eosio.token' && (data.to == 'eidosonecoin' || data.from == 'eidosonecoin'))