Careers @ Hudya

Four Lessons After Our First Year of Building The Hudya Platform

1298 days ago in building a tech companyNew

As I promised in an earlier post, I will here share more about our learnings going from zero to three countries and three product verticals in a bit more than a year. When we started, we used every known trick (and then invented some) to get high velocity, and as I wrote in my blog post on how we approched things from the beginning, “Phase 2 is about hiring, re-design and re-factoring“. Well, I was right about that one…

If you read my blog post with more details on how we launched electricity services in eight weeks and how we used scaffolding and Minimum Workable Product as methods, you could probably literally see technical debt pile up!  We knew we would end up with a lot of code that needed to be re-factored, so we actually worked on a version 2 of our architecture and core components in parallel. However, we had another problem that caused more waste and re-factoring than anything else: we lacked non-technical people savvy in development who were able to translate a business problem into the correct technical deliverable.  So, with a large team of developers, we constantly had a bottleneck in product ownership/product management and developers had to try to figure out what might deliver on the ask themselves. That resulted in a lot of wasted work that had to be re-done.

So, lesson number 1: in a startup where you don’t really know which problems you need to solve and how to solve them, it is better to have good product people work (maybe with a good prototyper) on exploring what to build. You should not start up the dev team too early.

As of today and 19 months after the first code lines were written, we have have re-factored about 80% of the entire code base and 100% of the most used components.

Lesson number 2: when building a microservice architecture, you either need to understand the end goal incredibly well, or you are better off building your core business logic in a monolith (prepared for modularisation) and then later split it up. Why? If you don’t understand what your core business logic is, it is very hard to make the right microservices and APIs between them. Also, microservices (and the fact that different people typically own each) slows down development time way too much when you are in an explorative stage.

Hudya’s tech platform consists of three things:

  1. A multi-vertical product-platform that offers customer subscriptions, my pages, management of service options, usage, invoicing and payments. This platform integrates with Hudya’s product partners in each vertical
  2. A flexible and secure core with multi-level, multi-role access control that differentiates access based on many different factors
  3. A customer engagement platform establishing and managing our on-going conversations with our customer across any channel and product

In each of these, there are quite a few things that are new and where we need to explore how to reach our goals, for example building insanely good customer experiences across multiple products as diverse as mobile, insurance, and banking (which, by the way, is not something you get after the first shot…) Our customer engagement platform basically needs customer relationship management (CRM) functionality built into the product platform (and not as a separate external system).

Lesson number 3: building a robust technical solution to a business problem that is not well known requires a mix of expertise, and you should be really careful not trying to tackle more than one/too many at the time (dependent on your team). Consistently exploring, improving, innovating, and building require a certain amount of “organisational willpower”. As an example, we were trying to build version 2 of our core product platform at the same time as we did a project named “auth v2”. Although the team at the time was around 25 engineers, the skill set and focus needed on both projects was too much for us to really be able to get good progress on both projects at the same time.

Lesson number 4: don’t underestimate the effort required to establish practices for everything from code style to API design and CI/CD (Continuous Integration/Continuous Deployment). Even if you have done it before and even if you communicate clear guidance and follow up, there is a need to lead by example. This means that you don’t get better designs and processes than what your lead developer(s) actually do, no matter what you ask for!

Some Best Practices

With these lessons, we also arrived at some best practices that gave results. Here is a quick summary of a few of them:

  • Establish a real benchmark of what is expected. As an example, we made a microservice implementing feature toggles. As a fairly standalone piece of software, we used this component to establish how to do Docker packaging, design-driven API development, end to end API test coverage, full OpenAPI documentation, local debugging inside the Docker container, as well as Jenkins test and deploy pipeline.
  • Prioritize the developer experience. A build that takes 15 minutes and that is necessary to go through repeatedly to debug certain problems can kill productivity. Lack of good test data can result in bad quality code. Spend enough time on ensuring that all developers are productive.
  • Make re-factoring a continous activity. Many developers like to “complete” a task by crossing the t’s and dotting the i’s. With devops and micro services, all code should be in constant re-factoring and “polishing” everything is a waste of time. Know what is good enough until the next time you (or somebody else) make an iteration on the code.
  • Live the pain. It can be tempting to let the most senior people solve the hard problems/do prototyping and let the more junior developers deliver some of the perceived simpler deliverables and/or complete the prototypes. Don’t! Force the senior developers to use the same dev environment and pipeline, and make them complete the work end to end, at least for some of what they do. If not, a slow build or test pipeline, bad debugging setup, tedious manual tasks, and repeated errors are never fixed.
  • Only allow shared code in narrow-purposed libs. Make sure that the libs have an open-source contribution style and that a lib never has more than one purpose. If not, you basically end up with a monolith.
  • Get the shared structures right quickly and without democracy. Your approach to APIs, API design, testing, your CI/CD pipeline, shared libs, code style, logging, monitoring, tracing, frameworks/key libraries, documentation, code reviews, and approvals are key to productivity. How to make decisions on the best approach for you depends on your organisation, but it’s definitely not a democracy or something that each developer can decide!
  • Standardize and only allow deviation if there are real reasons to do so. Don’t buy the myth that microservices are standalone and it doesn’t matter what is inside. For productivity and business reasons, you will want feature teams that are capable of touching all the code bases necessary to deliver a feature. Unless you are big enough to put together feature scrum teams with one participant from each team owning a code base, you have a HUGE productivity gain in making sure that most code bases are similar in the basic aspects mentioned in the bullet above. This is especially true for your tech stacks. If there are real productivity or technical challenges that require you to choose e.g. another tool chain, database, support system, or programming language, evaluate again, and only do it if the benefit is at least one order of magnitude bigger.

All in all, some of these things we got right fairly quickly, others gave us some real headaches and where the above lessons could have been applied earlier than we did. There are several deep dives we hope to be able to share here soon; from how we build our Docker images (which turns out to be a key thing to get right) to how we do design-first APIs and end to end API testing.

Leave a Reply

Your email address will not be published. Required fields are marked *