Blog.

This post will be talking about the data process you need in managing your data team. There are so many different factors that go into building a successful team, but there are five data processes that represent the most important in my experience. This article is the first in a series of posts on Building an Analytics Platform; make sure to read up on the other four articles for a comprehensive guide to your data architecture. There are five main areas we are going to examine:

• Team structure
• Delivery management
• DevOps
• Meetings
• Tools

Culture

Before we dig into the five main topics, it’s worth talking about culture for a second; so much of what makes a good team is tied up in their culture. However, this is a tricky thing because you can’t force it.

Culture grows organically from the people you bring into the team, and the way the leadership manages things; it can be greatly influenced by how the company as a whole operates. While I’m far from an expert on creating business cultures, you should get your team together, take an afternoon out somewhere, and agree as a group what you want the team to be.

Team Structure

In my last post, I listed out the people that go into making a great team. The structure I propose here is just one of the many ways you can organise, but it’s one that works well.

It’s worth noting this is a delivery structure rather than a line management structure, so you don’t have to have people reporting to each other if it isn’t suitable for your business.

There’s nothing particularly revolutionary here. The structure of the team is kept quite flat in order to avoid a load of unnecessary middle-management and to drive a sense of responsibility across the board. Nearly everyone in the team is doing some coding or development maximising output.

The structure above represents a full complement of team members for a more mature team, rather than one just starting up. If you’re starting with a much leaner group of people, then avoid the hierarchy until it becomes necessary. As ever, you should adapt to your circumstances with this, and make sure you have something that works for everyone.

When it comes to organising around a specific project or deliverable, you can use squads. Squads take a cross-section of the team and put them together for a while to get a piece of work done. Normally 2-4 people in size, try to bring together a cross-section of skills, including design and development.

One thing I see people doing is trying to align the data engineers to a specific subject area (in retail, this might include stock, logistics, e-commerce, or finance). I’ve tried this before, and it didn’t work. People get bored being pigeon-holed into one area, and it stifles personal development for people in the team. While it is useful to get people building up a deeper understanding of a certain business area, you should balance this with avoiding over-reliance on a single person for any one thing.

Agility, not agile

I’m going to keep this section very short. There’s a lot of material out there on Agile methodology and I’m certainly not the right person to tell you about it.

What I will say is that when it comes to delivery, it goes without saying that agile is king at the moment. I certainly advocate it, but the focus should be on adapting it to the needs of the team rather than living by the book.

Having a product backlog, organising into sprints and having scrums will help you get things done, but don’t get too worried about doing it all perfectly. Much better to find a process that works for the whole team and more importantly one that people can stick to. As you grow, you’ll likely find you need more rigidity and structure about how you do things so keep reviewing your process.

Defining DevOps

Running the team using a DevOps model is a great way to take greater control of your delivery, turn around work faster and provide a better level of support to your customers.

Recently, I was trying to hire a DevOps software engineer, and I kept hearing the same thing from the recruiter. When she talked to candidates about being a DevOps software engineer, they said, “I don’t want to do the other software engineers dirty work”.

I found this strange. While a big part of DevOps is certainly around automation of environment and code deployment and enabling a more rapid release cycle, for me DevOps was about having a development and operations team that were one and the same thing. This means that anything you build, you support and if anything was broken you would prioritise the fix to that.

This is a very effective way to grow and scale the team when you have a limited number of headcount and don’t want team members solely dedicated to operations. As the team expands and you have more live code to support you will likely end up bringing in dedicated operations staff.

I call these OpsDev. The difference between the two being the DevOps focus on development first and ops second and OpsDev focus on ops task first and Development second. This way, when everything is working well, your operations team are working through the development backlog and speeding up delivery.

I’m a big advocate of having support activities handled within the team rather than handing it over to a centralised ops team. No one is going to have more expertise on the platform than the team that built it and it avoids wasting time on incomplete handovers. All in, incidents are resolved faster and become less of an issue.

Meetings, Meetings, Meetings

I think we can all agree that most businesses have way too many meetings. When did we all start to think that’s how work gets done? Of course, some are necessary. Here’s my selection of meetings you’re going to need.

• Scrums – If you’re even loosely following Agile methodology, you’re going to need scrums. Keep them short and sweet. I’ve always preferred them at the end of the day.
• Sprint planning – Like scrum, this is a necessary part of agile. Do it on a Monday at the start of your sprint and get the team to come prepared. Tasks should already be in your tool of choice and up to date before the session. To save time, you can always bundle your retrospective and planning into a single session.
• Show and tell – While this might seem like something you could do without, I highly recommend having a weekly show and tell. It helps unite the whole team and gives someone the opportunity to run through what they’ve been working on. Throwing in the occasional “lean coffee” session is another great way to get the team involved.
• Team leads – Once a week, the team leads should get together and catch up on the status of everything. This should be an action-focused meeting rather than a general chit-chat. My recommendation is to do it over a beer on a Friday afternoon!

And that’s it. That’s enough planned contact points to run the team without spending your life in meetings. The team leads should be doing whatever they can to help keep the development team out of meetings so they can focus on their tasks. Coding requires unbroken concentration and having a meeting every other hour won’t aid this.

Tools

To finish off, I wanted to recommend a couple of my favourite tools that support the team. These are far from the only options, they’re just ones I have experience using and have found to be highly effective.

• Confluence – If you haven’t heard of it before, Confluence is a wiki-style documentation tool from Atlassian. I will never, ever go back to doing documentation any other way. Having a living document that is easily searchable and available to everyone is incredible. You can put everything from user guides and data dictionaries to solution architecture in there and nobody ever worries about if they’re looking at the latest version.
• Jira – If you’re doing agile, you’re going to need somewhere to manage your stories and sprints. Jira’s integration with Confluence and other tools is handy and it’s one of the big players in this space. Your delivery lead will spend a lot of time in here, so let them have a say in what you use.
• Wunderlist – Wunderlist is a to-do list that has now been acquired by Microsoft (image above). I live and die by this tool now and if the things I need to do aren’t on there they may as well not exist. When used within a team you can share lists, allocate tasks and set due dates. This is really handy for things that aren’t really worth putting in Jira but still need to be done (like making sure someone brings doughnuts to the next show and tell).
• Slack – Slack is an excellent messaging tool with a million extensions. I highly encourage you to ditch emails within the team and move all comms to slack. You’ll find it boosts productivity and communication and if your team is geographically spread it’ll help people still feel part of the action. If you’ve ever tried dealing with a production incident over email instead of on slack, you’ll know exactly what I’m talking about.

Summary

There is so much more we could talk about in this space, but I’ve avoided laying out rigid delivery patterns for you to follow as it’s all about finding the right fit for the team. In fact, if there’s one thing to take away from this it’s that you need to find something that works for you. The key things to remember are:

• Get in place an effective, reasonably flat team structure that empowers everyone to make decisions is where you’ll need to start.
• Be agile, but do so in a way that people will stick to without complaining the whole time, and that gives you the minimal level of governance necessary to be effective.
• DevOps can be a great way to get things done and make sure that your team is responsible for the end-to-end customer experience of using the platform, but it means more than automation.
• Avoid lots of meetings and just get the bare minimum in the diary to free up time for delivery instead.
• Find some great tools that make communication easier, whether that be documentation of what you’ve done or deciding which pub to get lunch in.

If you found this article valuable, you might be interested in our next webinar: Considerations for a Successful Data Platform. This hour long session is led by James Lupton, coaching leaders in business, data and tech on how to build a data platform.