If you are applying to an entry-level Data Engineering role and you are being interviewed exclusively by someone who is from the engineering or data team, the recruiters only have two criteria-

1. They need a few basic DE specific skills
2. They need something that they NEED

1. The few thangs!

The standard DE tools are Airflow, SQL, Python+Pandas, and dbt. And you need to cover a DE cloud stack. For GCP it is: BQ and Cloud Storage. For the ETL process, it could be anything: Cloud Function, Compute Engine or Dataflow.

BI tool... to be honest, you don't need a BI tool that much in DE. The BI tool domain is a slippery slope for DEs because the list of BI tools is endless and BI is a distinct field from DE. A serious recruiter wouldn't demand anything other than that you have heard of the tool they are using. But it is always good if you have built a dashboard or something, but in my opinion, it isn't super necessary.

It's almost the same advice for Spark or Kafka. Get an understanding of what they do, and have some idea of the syntax. If the data team is using Spark or Kafka, they wouldn't expect you to touch those projects in your first 2-3 months.

2. The need

I think you can get operational knowledge of the basic DE skillset in 6 months. But the catch is the NEED factor. The NEED factor can be many things. Every company is different, and they all need that thing that makes them different.

Like Scala. If you know Scala and someone is using Scala in their stack, they will try their absolute best to hire you even if you don't have the hands-on Spark experience. If they are using something like Rust, where there are only a handful of DEs who use Rust, they will try to hire you.

If they are using a method extensively that is often not prioritized in DE, they will try to hire for it. In my case, it is web scraping. Because I am confident in my web scraping skills, companies with large web scraping operations try their best to hire me even though I don't have much DE experience. I also got a few interviews because they were dealing with obscure documentation. I said, I have built a project based on the OLD YouTube API docs (those who knows, knows). They tried their best to hire me.

Now the interesting part is that BI/Dashboard and Spark/Kafka fit in here. In my opinion, they don't fall under the basic needs criteria. These are specialized needs for some DE teams. If you have used them or built projects using them, you would have an edge in a team that uses those tools extensively. But they wouldn't give you an edge everywhere, as not all DE teams use them. That need factor is why some DE recruiters would prefer to have programmers instead of data analysts or vice-versa, because the skillset a recruit brings meets their specific needs.

Essentially, I am saying the need criteria is so diverse, you can't predict your way into it. So specialize in something that relates to DE that you enjoy. It sounds like feel-good advice, but that is just what I figured. Someone who is able to write Java shouldn't focus on building dashboards; they can focus on learning Scala or Rust. If you have data analytics or BI interests, you are wasting your time if you attempt to learn Scala without getting paid for it.

Author: anyfactor