Our open source stack

An overview about the libraries we develope and maintain to do our data and research work. They are the building blocks of OpenAleph but most of them can be used standalone as well.

1 · Storage

• anystore: store anything anywhere, with one interface to keep and retrieve keys, values and files in local folders, Redis, SQL databases, over HTTP or in cloud storage. (GitHub)
• putfs: fast HTTP blob storage on a plain filesystem, with simple PUT and GET requests, files on disk served by nginx, and no SDK or metadata database.

2 · FollowTheMoney core

• ftmq: queries and filters FollowTheMoney entities from files, streams or statement stores, with one query object that also compiles to SQL and to OpenAleph API parameters. (GitHub)
• ftm-lakehouse: a data standard and archive storage for FollowTheMoney entities, documents and their metadata, from leaks to public collections, that tools like investigraph, memorious and OpenAleph write to and read from. (GitHub)

3 · Processing queue

• openaleph-procrastinate: OpenAleph's task queue for its processing jobs, built on the PostgreSQL-based procrastinate library. (GitHub)

4 · Data creation

• ingest-file: extracts the contents of documents, archives and emails of many file types into FollowTheMoney entities, keeping their folder structure, ready for OpenAleph. (GitHub)
• memorious: a lightweight toolkit to crawl and scrape websites into document collections, continuing the discontinued original memorious. (GitHub)
• investigraph: an ETL framework that turns any data source into FollowTheMoney datasets, configured in YAML and extensible with Python where a source needs it. (GitHub)

5 · Analysis

• ftm-analyze: analyzes FollowTheMoney entities during ingestion: detects languages and countries, extracts names, emails, phone numbers and IBANs, resolves names against known entities with juditha, and annotates them for search. (GitHub)
• juditha: a fast lookup of canonical names that tells whether a name found by entity recognition matches a known company, person or sanctioned entity, even when spelling, accents or word order differ. (GitHub)

6 · Application

• OpenAleph: the platform on top, which securely stores large amounts of documents and data and makes them searchable for investigative teams to work on together. (GitHub, about OpenAleph)
• openaleph-search: the Elasticsearch-based module that indexes, searches and matches FollowTheMoney entities in OpenAleph, and works on its own too. (GitHub)