Skip to content

Settings

Configuration is handled by pydantic-settings in ftm_analyze.settings. Every option below can be set via an environment variable or via a .env file in the working directory, using the prefix FTM_ANALYZE_ (e.g. FTM_ANALYZE_NER_ENGINE=flair).

Print the currently resolved settings with:

ftm-analyze --settings

Pipeline toggles

These flags control which optional resolution stages run. They default to false so a vanilla install behaves like the original ingest-file analyzer; opt-in stages light up as soon as the corresponding flag is on.

resolve_mentions

Env FTM_ANALYZE_RESOLVE_MENTIONS
Type bool
Default false

Resolve known mentions to real entities via juditha.lookup. When the lookup returns a hit above its score threshold the analyzer emits the resolved entity (Person, Organization, …) instead of a Mention entity. Only person, organization and location mentions are looked up, and only against known entities of the same type, so the lookup never changes the NER label. Locations are matched against Address entities in juditha, a hit emits an Address entity. See NER for how to populate the juditha index.

extract_known_names

Env FTM_ANALYZE_EXTRACT_KNOWN_NAMES
Type bool
Default false

Extract known names in one pass via the Aho-Corasick automaton that juditha build writes next to the juditha index. Runs in addition to the configured ner_engine, and both results are merged by position. Hits are emitted as entities with the schema juditha knows for them (Person, Company, …) instead of Mention entities. See NER.

annotate

Env FTM_ANALYZE_ANNOTATE
Type bool
Default false

Rewrite text properties with per-word ZWJ annotation markers consumed by openaleph-search. See annotate and annotations.md for the format spec.

overwrite_lang

Env FTM_ANALYZE_OVERWRITE_LANG
Type bool
Default false

Drop any pre-existing detectedLanguage values before running language detection, so the property always reflects the analyzer's own decision.

NER backend

ner_engine

Env FTM_ANALYZE_NER_ENGINE
Type "spacy" \| "flair" \| "bert" \| "gliner"
Default "spacy"

Selects the named-entity recognition engine. Non-spaCy backends require their respective extras (pip install ftm-analyze[ner-flair], [ner-gliner], [ner-transformers]).

ner_default_lang

Env FTM_ANALYZE_NER_DEFAULT_LANG
Type str (ISO 639-3)
Default "eng"

Fallback language used by the spaCy backend when an entity carries no detected/declared language.

ner_type_model_confidence

Env FTM_ANALYZE_NER_TYPE_MODEL_CONFIDENCE
Type float
Default not set (disabled)

Opt-in: minimum confidence of the fastText classifier model for an NER result to survive filtering (e.g. 0.85), results labelled trash are always dropped. Lower values let more mentions through. Known names (extract_known_names) skip the filter. The model judges the name alone, without its context, and in our benchmarks on German documents it removed more real names than noise, so it is disabled by default.

ner_type_model_path

Env FTM_ANALYZE_NER_TYPE_MODEL_PATH
Type Path
Default ./models/model_type_prediction.ftz

Path to the fastText classifier used by ner_type_model_confidence. The model (about 460 MB) is not part of the Docker images, download it from https://cdn.investigativedata.org/ftm-analyze/model_type_prediction.ftz.

lid_model_path

Env FTM_ANALYZE_LID_MODEL_PATH
Type Path
Default ./models/lid.176.ftz

Path to the fastText language-identification model used by the language detector.

translation_chunk_size

Env FTM_ANALYZE_TRANSLATION_CHUNK_SIZE
Type int
Default 512

Character size of the substrings text is split into for language detection.

Spacy models

The spaCy backend picks a model per detected language. Override individual languages with FTM_ANALYZE_SPACY_MODELS_<ALPHA3> (the lookup is nested through pydantic-settings).

Env Default
FTM_ANALYZE_SPACY_MODELS_ENG en_core_web_sm
FTM_ANALYZE_SPACY_MODELS_DEU de_core_news_sm
FTM_ANALYZE_SPACY_MODELS_FRA fr_core_news_sm
FTM_ANALYZE_SPACY_MODELS_SPA es_core_news_sm
FTM_ANALYZE_SPACY_MODELS_RUS ru_core_news_sm
FTM_ANALYZE_SPACY_MODELS_POR pt_core_news_sm
FTM_ANALYZE_SPACY_MODELS_RON ro_core_news_sm
FTM_ANALYZE_SPACY_MODELS_MKD mk_core_news_sm
FTM_ANALYZE_SPACY_MODELS_ELL el_core_news_sm
FTM_ANALYZE_SPACY_MODELS_POL pl_core_news_sm
FTM_ANALYZE_SPACY_MODELS_ITA it_core_news_sm
FTM_ANALYZE_SPACY_MODELS_LIT lt_core_news_sm
FTM_ANALYZE_SPACY_MODELS_NLD nl_core_news_sm
FTM_ANALYZE_SPACY_MODELS_NOB nb_core_news_sm
FTM_ANALYZE_SPACY_MODELS_NOR nb_core_news_sm
FTM_ANALYZE_SPACY_MODELS_DAN da_core_news_sm

Flair models

Same nesting convention as the spaCy mapping, via FTM_ANALYZE_FLAIR_MODELS_<ALPHA3>. Only languages with a maintained Flair NER model are listed; others fall back to whatever flair's defaults are.

Env Default
FTM_ANALYZE_FLAIR_MODELS_ENG ner
FTM_ANALYZE_FLAIR_MODELS_DEU de-ner
FTM_ANALYZE_FLAIR_MODELS_FRA fr-ner
FTM_ANALYZE_FLAIR_MODELS_SPA es-ner-large
FTM_ANALYZE_FLAIR_MODELS_NLD nl-ner

BERT / GLiNER

bert_model

Env FTM_ANALYZE_BERT_MODEL
Type str
Default "dslim/bert-base-NER"

HuggingFace model id used when ner_engine="bert".

gliner_model

Env FTM_ANALYZE_GLINER_MODEL
Type str
Default "urchade/gliner_small-v2.1"

Model id used when ner_engine="gliner".

gliner_threshold

Env FTM_ANALYZE_GLINER_THRESHOLD
Type float (0–1)
Default 0.5

GLiNER confidence threshold below which predictions are discarded.

device

Env FTM_ANALYZE_DEVICE
Type str
Default cuda if available, otherwise cpu

Torch device for the flair, bert and gliner engines, e.g. cpu, cuda or cuda:1. Has no effect on spacy.

Model cache locations

flair_cache_root

Env FTM_ANALYZE_FLAIR_CACHE_ROOT or FLAIR_CACHE_ROOT
Type Path \| None
Default None

Override the Flair model cache directory. Useful when mounting an external volume in containers.

hf_home

Env FTM_ANALYZE_HF_HOME or HF_HOME
Type Path \| None
Default None

HuggingFace cache directory used by the BERT and GLiNER backends.