Skip to main content

Github

This page contains the setup guide and reference information for the GitHub source connector.

Prerequisites​

  • List of GitHub Repositories (and access for them in case they are private)

For Calabi Connect:

For Calabi Connect:

Setup guide​

Step 1: Set up GitHub​

Create a GitHub Account.

Calabi Connect additional setup steps

Log into GitHub and then generate a personal access token. To load balance your API quota consumption across multiple API tokens, input multiple tokens separated with ,.

Step 2: Set up the GitHub connector in Calabi Connect​

For Calabi Connect:​

  1. Log into your Calabi Connect account.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select GitHub from the Source type dropdown.
  4. Enter a name for the GitHub connector.
  5. To authenticate:

  • For Calabi Connect: Authenticate your GitHub account to authorize your GitHub account. Calabi Connect will authenticate the GitHub account you are already logged in to. Please make sure you are logged into the right account.

  • For Calabi Connect: Authenticate with Personal Access Token. To generate a personal access token, log into GitHub and then generate a personal access token. Enter your GitHub personal access token. To load balance your API quota consumption across multiple API tokens, input multiple tokens separated with ,.

  1. GitHub Repositories - Enter a list of GitHub organizations/repositories, e.g. airbytehq/airbyte for single repository, airbytehq/airbyte airbytehq/another-repo for multiple repositories. If you want to specify the organization to receive data from all its repositories, then you should specify it according to the following example: airbytehq/*.
caution

Repositories with the wrong name or repositories that do not exist or have the wrong name format will be skipped with WARN message in the logs.

  1. Start date (Optional) - The date from which you'd like to replicate data for streams. For streams which support this configuration, only data generated on or after the start date will be replicated.

  • These streams will only sync records generated on or after the Start Date: comments, commit_comment_reactions, commit_comments, commits, deployments, events, issue_comment_reactions, issue_events, issue_milestones, issue_reactions, issues, project_cards, project_columns, projects, pull_request_comment_reactions, pull_requests, pull_request_stats, releases, review_comments, reviews, stargazers, workflow_runs, workflows.

  • The Start Date does not apply to the streams below and all data will be synced for these streams: assignees, branches, collaborators, issue_labels, organizations, pull_request_commits, repositories, tags, teams, users

  1. Branch (Optional) - List of GitHub repository branches to pull commits from, e.g. airbytehq/airbyte/master. If no branches are specified for a repository, the default branch will be pulled. (e.g. airbytehq/airbyte/master airbytehq/airbyte/my-branch).

For Calabi Connect:​

  1. Navigate to the Calabi Connect dashboard.
  2. Click Sources and then click + New source.
  3. On the Set up the source page, select GitHub from the Source type dropdown.
  4. Enter a name for the GitHub connector.

Supported sync modes​

The GitHub source connector supports the following sync modes:

Supported Streams​

This connector outputs the following full refresh streams:

This connector outputs the following incremental streams:

Entity-Relationship Diagram (ERD)​

Notes​

  1. Only 4 streams (comments, commits, issues and review comments) from the listed above streams are pure incremental meaning that they:

    • read only new records;
    • output only new records.

  2. Streams workflow_runs and workflow_jobs are almost pure incremental:

    • read new records and some portion of old records (in past 30 days) docs;
    • the workflow_jobs depends on the workflow_runs to read the data, so they both follow the same logic docs;
    • output only new records.

  3. Other 19 incremental streams are also incremental but with one difference, they:

    • read all records;
    • output only new records. Please, consider this behaviour when using those 19 incremental streams because it may affect you API call limits.

  4. Sometimes for large streams specifying very distant start_date in the past may result in keep on getting error from GitHub instead of records (respective WARN log message will be outputted). In this case Specifying more recent start_date may help. The "Start date" configuration option does not apply to the streams below, because the GitHub API does not include dates which can be used for filtering:

  • assignees
  • branches
  • collaborators
  • issue_labels
  • organizations
  • pull_request_commits
  • repositories
  • tags
  • teams
  • users

Limitations & Troubleshooting​

Expand to see details about GitHub connector limitations and troubleshooting.

Connector limitations​

Rate limiting​

You can use a personal access token to make API requests. Additionally, you can authorize a GitHub App or OAuth app, which can then make API requests on your behalf. All of these requests count towards your personal rate limit of 5,000 requests per hour (15,000 requests per hour if the app is owned by a GitHub Enterprise Cloud organization).

info

REST API and GraphQL API rate limits are counted separately. The REST API uses a request-based limit, while the GraphQL API uses a point-based limit where each query costs a calculated number of points. Streams that use the GraphQL API include pull_request_stats, reviews, pull_request_comment_reactions, issue_reactions, releases, and projects_v2.

tip

In the event that limits are reached before all streams have been read, it is recommended to take the following actions:

  1. Utilize Incremental sync mode.
  2. Set a higher sync interval.
  3. Divide the sync into separate connections with a smaller number of streams.

Refer to GitHub article Rate limits for the REST API.

Releases stream asset limit​

The Releases stream uses the GitHub GraphQL API and fetches up to 100 assets per release. Releases with more than 100 assets will only include the first 100. Sub-pagination for release assets is not currently supported.

Permissions and scopes​

If you use OAuth authentication method, the OAuth2.0 application requests the next list of scopes: repo, read:org, read:repo_hook, read:user, read:discussion, read:project, workflow. For personal access token you need to manually select needed scopes.

Your token should have at least the repo scope. Depending on which streams you want to sync, the user generating the token needs more permissions:

  • For syncing Collaborators, the user which generates the personal access token must be a collaborator. To become a collaborator, they must be invited by an owner. If there are no collaborators, no records will be synced. Read more about access permissions here.
  • Syncing Teams is only available to authenticated members of a team's organization. Personal user accounts and repositories belonging to them don't have access to Teams features. In this case no records will be synced.
  • To sync the Projects stream, the repository must have the Projects feature enabled.

Troubleshooting​

  • Check out common troubleshooting issues for the GitHub source connector on our Calabi Connect Forum