Defining your data source

A data source teaches Virgo how to read one external system. A connector is a class decorated with @datasource, extending VirgoDataSource, that declares what the source contributes to the graph: its entities, the relations between them, and the exposure policies that let other sources link to it. Once defined, ingestion turns the source into part of the graph.

What a connector declares

@datasource
class GithubDataSource(VirgoDataSource):
    name = "github"
    identifier = "github"
    description = "GitHub datasource"
    webhook = Webhook()
    entities = [GitRepository(), PullRequest(), GitUser(), GitCommit()]
    relations = [Author(), ReviewedBy(), HasCommit()]
    exposure_policies = [ExposePullRequest(), ExposeGitCommit()]

• identifier names the source across the graph.
• webhook keeps the source current after the first backfill; sources without one are polled on a schedule.
• entities and relations are the node and edge types this source contributes. Define them in Data source entities and Entity relationships.
• exposure_policies let other sources reference this one. See Cross-domain linking.

Customizing ingestion

A connector can override ingestion hooks to change how its content is chunked, extracted, and linked. The defaults apply when you do not.

Related

• Data source entities: model the nodes your source produces.
• Entity relationships: connect them.
• Pull on installation: the initial backfill.