Our journey to becoming data-driven

Our mission is to help salon owners grow their business. In order for us to do so we need to be able to make the right decisions, and good decisions can’t be made without the data to back them up.

One of our key focuses over the last couple of years has been to become a data-driven company. This post covers that journey and the lessons we learned along the way. They may sound familiar to you and I’m hoping our learnings can guide you.

Designed by Freepik

As part of this focus, we needed to ensure that each department had the data that they needed to make informed decisions.

• We found that data wasn’t easily accessible and that there was a huge constraint on only the product engineering team being able to access that data.
• As we work in sprints(2 week cycles) and only the engineers had access to the data, we had to either derail sprint scope and switch to the data requests or delay returning the data to the team. Neither of which were good.
• As the engineers had to pull the stats manually there was no data dictionary. We’d often see different versions of the same stats depending on the product engineer providing the data (e.g. what constitutes an “active customer”).
• We have separate regions for our infrastructure, broken down into multiple services with many database instances. We typically had to aggregate and join data manually to get a single picture view/statistic.
• System design — our CRM system never had a unique account ID associated with our salons on the platform, so we couldn’t join our data easily. This lead us to a big engineering effort to clean up and centralise our data, giving us a source of truth.

Build vs Buy

We realised quickly we needed a BI data tool, Google Analytics wasn’t going to cut it. We needed one that could handle SQL as config and we could point to our databases. We knew that by doing that we could open up the potential for our people to build their own dashboard.

Designed by Rawpixel.com

We thought about building this ourselves and dropped that idea fast when we saw the complexities around building ETL process, maintaining a Redshift cluster and the operational overhead involved. These are the vendor requirements we had when we began looking at the market.

• Different DBs — Need to support multiple database types across different regions
• Joins — It needed to be capable of joining multiple databases across regions
• Security — We needed to be able to tightly control the data that the tool could access
• Pivot data functionality — allowing us to pivot, split and search on aggregated data
• Views/snippets — by having snippets we could create common queries and definitions for our the organisation, essentially becoming our data dictionary and ensuring consistency between dashes.

Tool choices

These tools are really expensive, you need to ensure you are going to get the ROI on them. They vary on pricing but the standard is a license fee, per user cost or per number of data row (or a combination of all 3). Some of the tools we evaluated were

• AWS QuickSight
• Tableau
• Klip
• Periscope Data
• Looker

After lots of discussions and trials, we finally selected Periscope Data. They ticked all of the boxed for us around our requirements. We also found their API, embedded dashboard functionally, alerting and email of dashboards to be extremely useful for us.

Getting data to our teams

Now that we had this amazing tool we needed to get data into the teams hands. Each team needed to figure out what numbers and metrics do they actually need to track to know they’re doing a good job.

Designed by Pikisuperstar

This is a really broad thing to nail down so we started with the teams having internal talks with their leaders to start to think it through and then lined it up. The product team would then have sessions with each team and get the 5–10 critical metrics lined up on paper (to be sketched on a board later in a mock dashboard). After that, we took that work into the engineering team to implement the dashes.

After the teams had the data they needed, we needed some data champions. One person from customer support took on this role and became an integral part of helping others learn SQL and create their own dashboards. She ran out of hours SQL workshops and helped organise an external lecturer to teach SQL to those who didn’t understand it.

People in other teams who knew SQL started creating their own dashboards. Quite soon after that, product engineering wasn’t needed to be involved in creating the dashboards.

Results from rolling our the BI tool

One of our core values is seirbhís go hiontoch, which means delivering an amazing customer service. By giving our customer support teams pivots and the ability to grab their own data, they were able to answer customer questions and give our customers insights they needed immediately, without having to come to engineering to ask for bespoke work to be done or an ad-hoc database query to be run.

Designed by Freepik

Our data migrations team have a set of tools which helps import from competitors. This new tool helped the migrations team get insights on their imported data, test queries and generally helped them become more efficient in their job.

We previously had built an internal tool to view and search our SMS and Email send history, really important again for customer support. But we realised by pointing the BI tool at the table we could remove that tool — therefore removing an engineering, maintenance, and infrastructure overhead.

We also seen a growth in our capabilities, from starting off with basic daily stats we discovered we could take on ambitious projects like cohort analysis, setting company goals with shared dashboards and building an export tools for our financials though the BI tool.

Challenges along the way

We certainly felt some teething pains along the way to enabling the company to be data-driven.

• Engineers sometimes upgraded database schemas without thinking about the queries in our BI tool and therefore broke them, leading to frustrations from the people who built or used the dashboard affected.
• Our caching strategy and tuning of data refreshes was sometimes not correct and needed tuning, if not correct it would mean people were seeing stale data. We typically do an incremental update every hr based on the updated column.
• Something we had not been on top of but caught us out were challenges with stale data that wasn’t needed or used anymore, in turn slowing down cache updates and query time.

Looking back

We took on a strategic initiative to become a truly data-driven organisation and it’s safe to say we have achieved a lot with Periscope. Driven by many departments and individuals in the company we now have 756 dashboards, 8,674 charts and 105 monthly active users.

The customer support person I mentioned, that helped champion our move to being data-driven has now moved into our product engineering group working as a full-time data analyst, we would not have achieved so much without her. We are now expanding that team as data is such a vital part of our company.

We’re currently on the lookout for new people to join the Product Engineering team, if you are interested please get in touch by emailing careers@phorest.com.