Skip to main content

Quickstart

Not availableCloud AvailableSelf-Managed Community (OSS)Not availableSelf-Managed Enterprise

Airbyte Open Source is a reliable and extensible open source data pipeline.

If you're getting started with Airbyte Cloud, you can skip ahead to moving data by adding your first source.

This quickstart guides you through creating a locally deployed instance of Airbyte in just minutes using abctl (Airbyte Command Line Tool). You'll be able to move data with minimal setup while you're exploring what Airbyte can do!

If you've already set up an Airbyte instance using Docker Compose and want to move to abctl, see the section on migrating from Docker Compose.

tip

When you're ready to put an Airbyte instance into production, you'll want to review our guides on deployment.

For the best experience, we recommend Deploying Airbyte on Kubernetes via Helm.

On a local deployment, Airbyte's default behavior is to store connector secrets in your configured database. These secrets are stored in plain text and are not encrypted. Refer to the Secret Management documentation to set up an external secrets manager.

If setting up an Airbyte server does not fit your use case needs (i.e. you're using Jupyter Notebooks or iterating on an early prototype for your project) you may find the PyAirbyte documentation useful.

Prerequisites​

  • To use abctl, you'll need to have Docker Desktop installed. See Docker's instructions for installation: Mac, Windows, Linux

1: Install abctl​

The easiest method for installing abctl for Mac and Linux users is to use the following command:

curl -LsfS https://get.airbyte.com | bash -

If you would rather install abctl yourself, follow the instructions for your operating system:

We recommend that Mac users use Brew to install the abctl command.

brew tap airbytehq/tap
brew install abctl

With Brew, you can keep abctl up to date easily, by running:

brew upgrade abctl

2: Run Airbyte​

Ensure that Docker Desktop is up and running. Then, with abctl installed, the following command gets Airbyte running:

tip

By default, abctl only configures an ingress rule for the host localhost. If you plan to access Airbyte outside of localhost, you will need to specify the --host flag to the local install command, providing the FQDN of the host which is hosting Airbyte. For example, abctl local install --host airbyte.company.example.

By specifying the --host flag, Airbyte will be accessible to both localhost and the FDQN passed to the --host flag.

abctl local install

Your browser may open automatically to the Airbyte Application. If not, access it by visiting http://localhost:8000.

You will be asked to enter your email address and an organization name. Your email address will be used to authenticate to your instance of Airbyte. You will also need a password, which is randomly generated as part of the install command. To get your password run:

abctl local credentials

Which should output something similar to:

{
  "password": "password",
  "client-id": "client_id",
  "client-secret": "client_secret"
}

Use the value in the password field to authenticate to your new Airbyte instance. If you wish to configure authentication follow the documentation on the Authentication Integration page.

As long as your Docker Desktop daemon is running in the background, you can use Airbyte by returning to http://localhost:8000.

If you quit Docker Desktop and want to return to your local Airbyte workspace, just start Docker Desktop again. Once Docker finishes restarting, you'll be able to access Airbyte's local installation as normal.

3: Move Data​

In the Building Connections section, you'll learn how to start moving data. Generally, there are three steps:

1: Set up a Source

2: Set up a Destination

3: Set up a Connection

Customizing your Installation with a Values file​

Optionally, you can use a values.yaml file to customize your installation of Airbyte. Create the values.yaml on your local storage. Then, apply the values you've defined by running the following command and adjusting the path to the values.yaml file as needed:

abctl local install --values ./values.yaml

Here's a list of common customizations.

Migrating from Docker Compose (Optional)​

If you have data that you would like to migrate from an existing docker compose instance follow the steps below:

  1. Make sure that you have stopped the instance running in docker compose, this may require the following command:
docker compose stop
  1. Make sure that you have the latest version of abctl by running the following command:
curl -LsfS https://get.airbyte.com | bash -
  1. Run abctl with the migrate flag set with the following command:
abctl local install --migrate
note

If you're using a version of Airbyte that you've installed with abctl, you can find instructions on upgrading your Airbyte installation here.

Using an EC2 Instance with abctl​

This guide will assume that you are using the Amazon Linux distribution. However. any distribution that supports a docker engine should work with abctl. The launching and connecting to your EC2 Instance is outside the scope of this guide. You can find more information on how to launch and connect to EC2 Instances in the Get started with Amazon EC2 documentation from Amazon.

  1. Install the docker engine:
sudo yum install -y docker
  1. Add the ec2-user (or whatever your distros default user) to the docker group:
sudo usermod -a -G docker ec2-user
  1. Start and optionally enable (start on boot) the docker engine:
sudo systemctl start docker
sudo systemctl enable docker
  1. Exit the shell and reconnect to the ec2 instance, an example would look like:
exit
ssh -i ec2-user-key.pem ec2-user@1.2.3.4
  1. Download the latest version of abctl and install it in your path:
curl -LsfS https://get.airbyte.com | bash -
  1. Run the abctl command and install Airbyte:
tip

By default, abctl only configures an ingress rule for the host localhost. In order to ensure that Airbyte can be accessed outside of the EC2 instance, you will need to specify the --host flag to the local install command, providing the FQDN of the host which is hosting Airbyte. For example, abctl local install --host airbyte.company.example.

tip

By default, abctl will listen on port 8000. If port 8000 is already in used or you require a different port, you can specify this by passing the --port flag to the local install command. For example, abctl local install --port 6598

Ensure the security group configured for the EC2 Instance allows traffic in on the port (8000 by default, or whatever port was passed to --port) that you deploy Airbyte on. See the Control traffic to your AWS resources using security groups documentation for more information.

abctl local install --host [HOSTNAME]

Editing the Ingress​

note

The latest versions of abctl support a --host flag replacing the need to manually modify the ingress rules.

For example, if you are hosting Airbyte on the FDQN of airbyte.company.example, you would execute the following command: abctl local install --host airbyte.company.example

By default abctl will install and Nginx Ingress and set the host name to localhost. You will need to edit this to match the host name that you have deployed Airbyte to. To do this you will need to have the kubectl command installed on your EC2 Instance and available on your path.

If you do not already have the CLI tool kubectl installed, please follow these instructions to install.

Then you can run kubectl edit ingress -n airbyte-abctl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig and edit the host key under the spec.rules section of the Ingress definition. The host should match the FQDN name that you are trying to host Airbyte at, for example: airbyte.company.example.

Uninstalling​

If you want to remove Airbyte from your system, consider which of the following two options you would like to use.

1: Run the following command to stop all running containers that abctl has created while preserving any data you've created:

abctl local uninstall

2: If you want to clear the persistent data in addition to stopping containers, run:

abctl local uninstall --persisted

As a last step, to clear out any additional information that abctl may have created, you can run:

rm -rf ~/.airbyte/abctl

Troubleshooting​

Unable To Locate User Email​

note

In abctl v0.11.0, support for basic-auth was removed (as basic-auth support was removed from the Airbyte Platform in v0.63.11, and replaced with a more secure randomly generated password. When logging into Airbyte, the email (provided during registration) should be automatically populated and the randomly generated password can be fetched by running abctl local credentials.

Airbyte is aware of situations where the email is not be automatically populated and we are working on addressing this within the abctl tool. In the interim, some manually steps are required to retrieve the authentication email address when it is unknown.

If the email address for authenticating is not automatically populated, and you are unsure what the login email should be, the email can be retrieved from the database used by Airbyte. If using an external database, you will need to connect to the database and execute the query SELECT "email" FROM "user". If using the database automatically included with the abctl local install command, you will need to install kubectl and execute the following command:

kubectl exec --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl -it airbyte-db-0 -- psql -U airbyte -d db-airbyte -t -A -c 'SELECT "email" FROM "user"'

The password for this user can be retrieved by running abctl local credentials.

Additional Resources​

There are several channels for community support of local setup and deployment.

GitHub Airbyte Forum's Getting Started FAQ:
Search the questions others have asked or ask a new question of your own in the GitHub forum.

Airbyte Knowledge Base:
While support services are limited to Cloud and Enterprise customers, anyone may search the support team's Help Center.

Community Slack:
Helpful channels for troubleshooting include:

  • #ask-community-for-troubleshooting: Where members of the Airbyte community can ask and answer questions.
  • #ask-ai: For quick answers sourced from documentation and open support channels, you can have a chat with our virtual Airbyte assistant.

Introductory Course:
On Udemy, The Complete Hands-on Introduction to Airbyte is a convenient and hands-on introduction to Airbyte that includes setting up example source and destination configurations. You'll also go on to use it in conjunction with Apache Airflow, Snowflake, dbt, and more.

Bug Reports:
If you find an issue with the abctl command, please report it as a github issue here with the type of 🐛 [abctl] Report an issue with the abctl tool.

Releases:
If you'd like to select which release of abctl to run, you can find the list of releases here.

Was this page helpful?