Don't Forget About Documentation

Don’t get me wrong, writing documentation is tedious and can be very boring. Unfortunately, it also can get pushed to the bototm of the to do list. Everyone’s busy, so in the long list of to-dos this is constantly pushed to the bottom. A former colleague once told me “the code is self explanatory, we don’t need to do documentation”. Ugh. While it may not seem valuable now, you’ll be thankful a couple months from now when you come back to this work.

Documentation Isn’t Only for Developers

Good documentation isn’t only the developers and engineers. It is beneficial for the analysts, business users, and even executives! Being able to understand, at a high level, what is going on can help connect the purpose of a project to how that project works.

A business user may not care what tool you are using to move data to the data warehouse, but they do care what data you are bringing in to the warehouse.

Good documentation helps describe the business logic and explains how it is implemented. dbt does a great job of column level documentation, empowering developers to explain key business logic in language that the business will understand. This is valuable for the business users to leverage it so they can ensure the data is correct. Correctness can ensure bad data is handled properly, but also helps to make sure people trust the final output.

Trusting the Data

It’s crucial for people to trust what is happening within the data pipeline. While some people won’t understand the technical complexities of your pipeline, it’s imperative that they are able to understand at a high-level what is happening to the data. If you are implementing business logic within the pipeline, by showing how that is implemented you ensure that the final output will be trusted by all because they can see how the data makes it way to its final destination.

At a high-level, people want to understand where that data is coming from, what transformations are made, and how it ends up in their hands. For example, if the marketing team wants to know about their engagement on a recent campaign, they will want to understand how you are getting the data. If they clearly understand where their numbers come from, they will be empowered to use the data in their analysis.

Moving Away from the Black Box

Nothing ensures that the data is not trusted like a black box. One time, someone told me that “the pipeline just runs, we aren’t really sure what is happening”. Not only does that cause confusion, but no one is quite sure the data arrived and they aren’t sure if they can trust it.

It’s important to first identify what could be a black box in your data. Some things to look for:

• Legacy software that no one maintains.

• Business calculations that no one can explain.

• Data sources without clear ownership.

• Stale data that no one understands.

To move away from this, identify the black boxes in your current system. Next, ensure that the documentation is easily understood by the correct parties. This will also require you to identify the persona of who will be consuming the documentation.

How to Create Useful Documentation

Just because you have documented everything doesn’t mean it’s going to be used. You need to ensure that your documentation is going to be used by the rest of the business, both engineers and business-users, to ensure success. To build good documentation, ensure that you follow these best practices:

• Keep it simple enough that it’s readable, but not too simple that it’s useless.

• Connected to the business. Show how it affects a specific product, business process or KPI.

• Identify who can benefit from the documentation as well as who is the owner. Not as in-depth as a RACI chart, but enough to answer the question “who can I ask if I want to know more information?”

What are some good examples of documentation that you have seen? Let me know your thoughts!

Share this with someone who is working on an analytics project! As a token of my appreciation, you will receive a free questionnaire to help you refine your next project!

CONTACT US

We are here to make the data world more accessible for everyone, one bite at a time. Please reach out if you have questions or want to collaborate on any data projects.