Data Science Hackathons with Impact, at AutoScout24
Allison Merritt and Scott Stevens · December 12, 2022 · 7 min read
Hackathons are exciting. In a nutshell, they are fast-paced, high-intensity, 3-day periods during which we on the Data Science team do our best to bring a few projects all the way from a cool idea (a mere twinkle in a data scientist’s eye) to a functioning prototype ready to be presented to management. Hackathons provide an opportunity for us to work on something new and different from our current day-to-day tasks, to think more innovatively and a bit further “outside of the box” than usual, and to try out new skills and technologies. Being very team oriented, they are also an opportunity to work on our collaboration and communication skills, as well as an opportunity to pair with colleagues that we might not usually work with. Finally, hackathons are a way for ideas that are more daring or creative to get some attention. Often, product ideas can come to a Data Science team from other teams, but hackathons provide a route for us to generate ideas ourselves and to pitch them in a form where the groundwork to demonstrate viability is already done (making the pitch much stronger).
In short, they’re a fun activity that, when done correctly, helps the team to develop while jumpstarting projects with a real chance of full development that will provide business value for the rest of the company.
Pitching Ideas
When the announcement for the August hackathon came, we had a week to start brainstorming project ideas. The announcement was quite open ended, stipulating only that the theme be deep learning, since this provided an opportunity to enhance overall capabilities in that area on the team level and also for individual team members to learn from their more-experienced peers. Anyone on the team was free to pitch an idea.
On the morning of the first day, the whole team gathered in a large conference room in person or remotely via Zoom. One by one, each person with a project idea gave their elevator pitch to the group, describing:
- the problem they’re trying to solve
- the solution and how it addresses the larger problem
- the difficultly in implementing it
- the anticipated impact
We spent the next half hour discussing the details of each project in groups before voting for the ones we wanted to work on.
One of the key elements of the way our team runs hackathons is that they are very interest driven. As much as possible—making sure the team is distributed across multiple projects, and nobody works alone—we want everyone to be working on the project that they think is most exciting and motivating. This time it worked out perfectly: four projects, each with 2-4 people working on them.
Making Space for the Hackathon
Just as a marathon in a city means blocking roads and redirecting traffic, space needs to be made to enable the team to focus on the hackathon. Other than incidents in production, all other duties were postponed. If the hackathon is properly planned ahead of time, the impact on other projects can be minimized. And, as mentioned above, the break from the normal routines can have a huge psychological benefit for the team, who are charged up by the new tasks, new skills, new partners, and new ways of working. (Just as it’s not advisable to sit still at our desks for too long, there are benefits to not doing the same set of tasks for too long!)
So, with the hackathon as the top priority, we were able to create a space where we could develop rapidly and creatively. Our time was filled with brainstorming, planning, coding, testing, evaluating, iterating, and more coding.
Throughout the hackathon we maintained the flexible working-space structure that our team has adopted: some groups preferred to come in to meet in person so they could sit near each other for easy access to feedback and new ideas, while others took advantage of the deep concentration possible while working from home and kept open Slack and Zoom connections to stay aligned. In addition to impromptu and asynchronous communication, we had daily full-team check-ins so that each group had enough opportunities to give updates and to request input from the other groups.
The Projects
After three days we had four working prototypes, all of which featured deep-learning models. One team built a tool to improve image quality and consistency by locating a car within a given image so that the image can be cropped and resized automatically to improve attractiveness.
A second team also built a prototype related to image classification, but in their case to blur the background of the image as well as any logos present.
The third group evaluated a method that reads text from images imported by the users in order to build a tool that makes creating a listing on our platform easier.
Finally, the fourth group trained a model to learn complex representations of user behaviour on the site (searches, clicks, engagement, etc.) and ran a clustering algorithm on top of this to define user segments in support of better personalization.
An extra benefit not mentioned above is how exciting and empowering it was to see just how much our team can accomplish in such a short amount of time. This boost in seeing what is possible has continued after the hackathon is done.
Outcomes
And, of course, the impact of the hackathon extended beyond our team. The final stage of the hackathon consisted of a short presentation to potential stakeholders in which the product was pitched, and a brief demonstration made. For the project with the best reception (in this case the easy listing insertion tool), a second, larger-scale and more polished demo was created in collaboration with the UX team for an all-hands, company-wide presentation. In fact, all four projects found their way into the roadmap for the next quarter in one way or another, and the interest they generated paved the way for increased collaboration between teams and even joint initiatives in some cases.
Lessons Learned
The prototypes we built were not perfect, but that wasn’t the goal. A hackathon fast-tracks the product lifecycle, and, due to this acceleration, the result is just a proof-of-concept and not a polished product. Demos help us make our ideas more tangible and are useful for communicating with stakeholders or other interested parties. When it’s over, though, promising projects can (and should!) still benefit from additional discovery, evaluation, and potential upgrades.
The high intensity “hackathon mode” would probably not be sustainable as a standard or default way of working, but it does naturally support a workflow that we encourage within the team: start by building a minimal but functional product, and then iteratively build up features from there. Working this way helps avoid premature optimisation and to keep things moving. We have to look several steps ahead and move to address or circumvent bottlenecks and barriers quickly.
Final Thoughts
Hackathons can be a highly effective way to energize a team, build up technical and collaboration skills, and to jumpstart products that might otherwise never receive the attention needed to grow. It allows us to try new things, to work in new ways, and to be the origin of bottom-up product generation. It’s empowering to be able to pitch ideas and see them built, and inspiring to see just how much our team can accomplish in only three days.
