Ingestion hooks

Ingestion turns a source's records into the graph in three steps: it extracts the useful entities, collapses duplicates into one canonical node, and links entities by their relationships. The defaults work without configuration. To change a step, override the matching hook method on your VirgoDataSource, or on a single entity.

Where you can intervene

• chunking_hook() returns a ChunkingHook: how unstructured content is split into chunks.
• triple_extraction_hook() returns a TripleExtractionHook: what entities and relationships are pulled from content.
• A connector can also override embedding_hook() and linking_hook(); an entity can override splitting_hook().

Each returns a hook instance, or None to keep the default.

@datasource
class ConfluenceDataSource(VirgoDataSource):
    name = "confluence"
    identifier = "confluence"
    # ...
 
    def chunking_hook(self):
        return ConfluenceChunkingHook()

Reach for a hook only when a source's shape calls for it; the defaults cover the common cases.

Related

• Chunking hooks: customize how content is split.
• Entity extraction hooks: customize what is pulled out.