Installation
Getting PipelineWise
PipelineWise source code is available on GitHub at https://github.com/transferwise/pipelinewise and can be installed in the following methods:
PipelineWise is a collection of pre-selected and customised
singer
components with a config management and runtime framework on top of it. When installing PipelineWise you will also install a bunch of Singer components into a well defined location. PipelineWise installs Singer.iotaps
andtargets
components into multiple virtual environments to avoid conflicting dependencies between any of the components. It is common that multiple Singer components use the same python library with different versions. This also means PipelineWise is not distributed on PyPI as a Python package and cannot be installed bypip
. PipelineWise can run from Docker or can be installed from source. In both cases the build and install scripts will create all the virtual environments at the right place in the right format and will take care of finding them at runtime.
Installing from source
When building PipelineWise from source make sure that every OS dependencies are installed:
Ubuntu:
apt-get install python3-dev python3-pip python3-venv
MacOS:
brew install python
Clone the PipelineWise git repository and run the install script that installs the PipelineWise CLI and every supported singer connectors into separated virtual environments:
$ git clone https://github.com/transferwise/pipelinewise.git
$ cd ./pipelinewise
$ ./install.sh
Press Y
to accept the license agreement of the required singer components. To automate
the installation and accept every license agreement run ./install --acceptlicenses
.
$ ./install.sh
(...installation usually takes 5-10 minutes...)
--------------------------------------------------------------------------
PipelineWise installed successfully
--------------------------------------------------------------------------
To start CLI:
$ source /Users/jack/pipelinewise/.virtualenvs/cli/bin/activate
$ export PIPELINEWISE_HOME=/Users/jack/pipelinewise/.virtualenvs
$ pipelinewise status
--------------------------------------------------------------------------
Once the install script finished, you will need to activate the virtual environment
with the Command Line Tools and set the PIPELINEWISE_HOME
environment variable
as it is displayed above at the end of the install script:
$ source /Users/jack/pipelinewise/.virtualenvs/cli/bin/activate
$ export PIPELINEWISE_HOME=/Users/jack/pipelinewise/.virtualenvs
$ pipelinewise status
Tap ID Tap Type Target ID Target Type Enabled Status Last Sync Last Sync Result
-------- ---------- ----------- ------------- --------- -------- ----------- ------------------
0 pipeline(s)
If you see that above output saying that you have 0 pipeline in the system then the Installation was successful.
Cool, what's Next?
From this point, you can go to the creating pipelines` section to create pipelines and to start replicating data.