Data Platform Engineering

The Data Platform Engineering (DPE) Team refines and builds the services at the foundation of the AutoScout24 Data Landscape. DPE Engineers continually work to innovate the data tools and serviced used to both ingest and consume data from the data lake, the central repository of data that powers our reporting, analysis and machine learning products. The team is driven to enable the use of vast amounts to data by many AutoScout24 teams.

The data lake currently contains nearly 700 TB of data with an average daily growth rate of 3.9 TB. This growth stands as evidence of the number of teams that are working to ingest vital data to help AutoScout24 be poised to have all critical information available to address a range of applications.

We find that users from a variety of teams and backgrounds are readily making use of data. The usage of our Business Intelligence is frequently used to assess data consumption and we have discovered a growth of nearly 40% of unique active users per month in the first quarter of 2021.

We do often marvel at the number of sources and volume of data available via the data lake. However, we understand that this data needs to be easily discoverable and, subsequently, readily accessible to realize its value. DPE Engineers are currently creating new data lake infrastructure to provide improved data governance. Data will still be easy for teams to ingest, but will only be moved to the data lake once ownership, description details, and defined attributes are provided and validated. We will always strive to make sure important data is available to be enriched with other sources, but will now do so after we have collected additional metadata which also holds value for the teams ultimately working to answer questions and discover ways to support our customers.