Devlake is an open-source dev data platform that ingests, analyzes, and visualizes the fragmented data from DevOps tools to distill insights for engineering productivity. Devlake is enterprise-ready, with a particular focus on DevOps tools. This blog talks more about Devlake and Devlake Deployment which allow customers to set up pipelines and collect data in just a few clicks.

DevLake is a new product and the only product of its kind. Devlake is building a data platform that can offer the environment a data engineer needs. Devlake aims to provide a platform that addresses the unique challenges of everyday business, together with operational peace of mind.

Devlake is extensible and customizable. A data engineer should be able to deal with various issues and requirements, Be easy to deploy, manage, and monitor Towards the above goals, Devlake has successfully implemented and delivered a product that is really to use and is effective. For keeping the data fresh users can also create recurring pipelines and run them periodically.

The Architecture

A DevLake installation typically consists of the following components:

• Config UI: A handy user interface to create, trigger, and debug data pipelines.

• API Server: The main programmatic interface of DevLake.

• Runner: The runner does all the heavy-lifting for executing tasks. In the default DevLake installation, it runs within the API Server, but DevLake provides a temporal-based runner (beta) for production environments.

• Database: The database stores both DevLake's metadata and user data collected by data pipelines. DevLake supports MySQL and PostgreSQL as of v0.11.

• Plugins: Plugins enable DevLake to collect and analyze dev data from any DevOps tools with an accessible API. DevLake community is actively adding plugins for popular DevOps tools, but if your preferred tool is not covered yet, feel free to open a GitHub issue to let us know or check out our doc on how to build a new plugin by yourself.

• Dashboards: Dashboards deliver data and insights to DevLake users. A dashboard is simply a collection of SQL queries along with corresponding visualization configurations. DevLake's official dashboard tool is Grafana and pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to swap for their own choice of dashboard/BI tool if desired

Deploying Devlake

Using DevLake Locally

Environment Preparation Please ensure that the docker and docker-compose are installed on the machine before the trial. If the docker-compose has not been installed, you can refer to the links below to complete the installation.

Windows Install Docker Desktop on Windows

OS X Install Docker Desktop on Mac

Linux Install Docker Desktop on Linux

After Installing the docker install docker-compose - Install Compose through Docker Desktop

Quick Launch For quickly launching devlake download docker-compose.yml and env-example.env from the release page. Linux users can simply run the commands below:

https://github.com/apache/incubator-devlake/releases/download/v0.10.1
/docker-compose.yml

wget
https://github.com/apache/incubator-devlake/releases/download/v0.10.1
/env.example

Mac users run mv env.example .env in the terminal.

Windows Renameenv.example to .env.

To use DevLake locally, visit config-ui by opening your favorite browser and visiting localhost:4000 by simply typing in the search bar localhost:4000.

Setting up DevLake GitHub Plugin

We will choose Github to get its data in this blog. Click on the GitHub logo and this will redirect to the GitHub integration page

follow the guide for generating personal access token. Copy the newly generated token and paste it in the Auth Token(s) section after clicking on the link that would redirect to your ‘Manage GitHub Settings’ of DevLake config-UI and then test the connection to make sure the connection is working and save it.

Initiate Pipelines Runs by the new “Create Run” Interface. Simply enable the Data Connection Providers you wish to run collection for, and specify the data you want to collect. press Create Run to start/run the pipeline. Pipelines are accessible from the main menu of the config-UI.

click on the view dashboard to access the Grafana for visualization you can use the prebuild dashboard or can create your own for understanding the engineering metrics.

If you wish to learn more about the implementation details, please visit
Github repository or join the slack channel from website channel, where the engineering team is available.