Named Entity Recognition
Info
Entity extraction builds on top off how ingest-file originally extracted mentioned Entities. Read more
Originally ingest-file filtered the entities returned by spaCy with a custom schema prediction model trained on existing FollowTheMoney data. Based on that, Mention-Entities are created. These mentions are resolved into actual Entities (e.g. Company, Person) during cross-referencing datasets.
This creates a problem for "smaller" OpenAleph instances: If there is not enough data to cross-reference with, these Mention entities would never resolved. As well when using the analysis standalone.
ftm-analyze introduces an improvement to this problem: Extracted names can be compared against juditha, and if they are known, the resolved entities are returned instead of mentions.
juditha allows a fast lookup (based on tantivy) against a set of known names (from FollowTheMoney data). The index can be populated by reference datasets such as company registries, sanctions lists, or PEPs.
Set up juditha
Configure the juditha store uri:
export JUDITHA_URI=/path/to/store.db
For example, to load all PEPs by OpenSanctions:
juditha load-dataset -i https://data.opensanctions.org/datasets/latest/peps/index.json
juditha build
When using ftm-analyze now, it will turn known person names into actual Person entities (instead of mentions) if they are within this PEPs list (including fuzzy matching).
To link location mentions as well, load places from GeoNames as Address entities (this needs the juditha[geonames] extra):
juditha load-geonames
juditha build
juditha server
Instead of opening the store in every ftm-analyze worker, juditha can serve it (read-only) over gRPC. Point JUDITHA_URI at the server and all juditha-backed features (resolve_mentions, extract_known_names) run against it, no other configuration needed:
juditha serve # next to the store, binds localhost:50051 by default, see --host / --port
export JUDITHA_URI=grpc://juditha-host:50051
If the server is unreachable, the analysis fails (instead of silently skipping juditha).
Extract known names
NER only finds what its model recognizes, and may cut or mistype a name it does find, even if the name is in the juditha index. With extract_known_names enabled, ftm-analyze additionally scans every text in one pass for all names juditha knows, using the Aho-Corasick automaton that juditha build writes into the store (automaton.txt).
export FTM_ANALYZE_EXTRACT_KNOWN_NAMES=true
or per run:
ftm-analyze analyze -i entities.ftm.json --extract-known-names
How it behaves:
- Matching is case and accent insensitive and aligned to token boundaries. Only multi-token names of at least 8 normalized characters are in the automaton (single-token names are too noisy), see
JUDITHA_MIN_TOKEN_LENGTH.ftm-analyzealso ignores hits of a single word shorter than 8 characters, these are left to NER, which sees the context. - It runs in addition to the configured NER engine, on the same schemata plus
Table(NER skipsTable,Folder,Package,Workbook). Its hits win over NER results at the same or an overlapping position: a knownBodo Ramelowreplaces an NERMinisterpräsident Bodo RameloworRamelow, and an NER locationThüringer Landtagbecomes the knownPublicBody. A single-word hit only wins at the same position: an NERAlternative für Deutschlandis kept over a knownDeutschland. - Hits skip the (opt-in) fastText confidence filter
ner_type_model_confidence, as they are exact matches of known names. - For a
Table, each (non-numeric) cell is matched on its own, so a name split across two cells is not found. With a juditha server this means one request per distinct cell value. - Hits are emitted as entities with the schema juditha knows for the name (e.g.
Person,Company,PublicBody, orLegalEntityif the sources disagree) instead ofMentionentities, the same wayresolve_mentionsdoes it. The entity carries the source document asproof. Ifresolve_mentionsis enabled as well, the juditha lookup still runs for the persons, organizations and locations among them to pick up the canonical name. - Where the automaton lives depends on
JUDITHA_URI. With a local store path, each worker process loads it lazily on the first extraction and keeps it for its lifetime, which for large juditha stores costs memory per worker and some startup time on the first document. With a juditha server (grpc://…), extraction runs on the server, which holds the automaton once for all workers. - If the juditha store has no automaton (
juditha buildhas not run), nothing is extracted and no error is raised.