The Chronicles of a Mechanic

Over the past years, I have discussed my journey as an entrepreneur trying to sever ties from capitalism while ironically trying to build my version of it. Frankly, it’s been an exhilarating experience, won distinctively and failed gracefully. While the conviction to get humans to Mars in my extraordinary approach persists, this tale, however, is a far cry from the daring exploits of a Peter Thiel escapade or a Shola Akinlade odyssey. Here, I will run you through my other experience, which traverses a realm closer to the endeavors of Elon Musk or, perhaps Ezra Olubi. No, I didn’t design the control system that caught a heavy booster using chopsticks (though I want to see that design architecture someday), and neither did I build the reliable system that processes over a hundred billion dollars annually.

This fable is a reflection of my journey as a Software Engineer, which reached new heights of intensity this year. I encountered numerous challenges that pushed me out of my comfort zone and into the fray, navigating a series of adventures — from mitigating DDoS attacks to dealing with missing production log files, constructing security systems, and developing exciting new features. It’s been a thrilling ride, to say the least!

Being accepted into SHiiP was a highlight in my journey from “Hello, world” to real-life problems. In January 2022, I received my offer to join the company as a full-stack developer. When I joined the engineering team, I became the ligament between the backend and frontend, as I dabbled between sprints. On the frontend, I mostly worked on landing page styling and a bit of logic. Back then, I believe one of the most challenging tasks was developing a bulk upload feature, where businesses could use CSV or Excel files to upload their addresses, and we would give them rates based on their individual pickup and delivery locations. It was dreadful working on that, while my major focus on the backend was courier integrations.

I explored a lot of poor documentation and some very interestingly straightforward ones, gaining an incredible amount of knowledge along the way. These were mostly the tasks I worked on in the engineering team, with some other cool tickets I can’t mention to avoid being served a subpoena. I have had the best teammates: Ife, the bedrock of Engineering, overseeing our design systems and architecture (which I believe the HR team would call a Site Reliability Engineer); Temi and Abdulrahman, our Frontend Engineers; and Mary, the Mobile App Developer. Working with these folks kept the team flowing with creative energy. We solved complex problems without hassle because it was a very open team, and everyone was willing to help when needed. Temi was my go-to person initially before he left the team, and I gained a lot of insight working with him. Abdulrahman, on the other hand, is crazy! This brother could do anything in JavaScript and had the best coding structure. And Mary? Flutter should be her middle name.

As my focus shifted fully towards the backend, I became very close to Ife. He became a senior brother to me on the team, always available to help resolve my blockers. For some reason, he solved problems so fast, it fascinated me every time. I remember one time I helped him solve a bug and told him I would forever brag about it.

i, subtly bragging!

Ife had a knack for speedy execution — that’s something I learned from him, and he was quite brilliant. You might be wondering why I’m using past tense when referring to Ife. No, he didn’t kick the bucket; he actually left the team. How dare Ife? After everything we’ve been through!

reality check!

Ife leaving the team gave me a quick reality check. All responsibility fell on me instantly. Ife had done a great job automating many processes and creating a lot of abstraction, so there was no immediate need to hire a new Senior SRE. Moreover, SHiiP worked lean and only hired when absolutely necessary. But this didn’t make the dose of reality any easier. I had to roll up my sleeves and get into the nitty-gritty. I had a few protocol meetings with Ife, and the offboarding was smooth yet gloomy.

The beginning of my saga: SHiiP partnered with one of the Leading Telcos Companies, and our system with this company is built in such a way that, after each shipment has been delivered , our ever-capable operations team updates the shipment status, which is then sent to them via webhook. Without this, they can’t validate an order that has been completed, which, in turn, would pend our payment. The first month after Ife left, they complained they were not receiving status updates from our end, and payment was due. About 20 shipments had a pending status, but our operations team had completed those shipments, and customers had received their packages. I opposed those complaints because they were contradictory to the scheduler we had put in place, which had been running smoothly ever since. I also suggested we get on a call so I could share the log files with the timestamps of our events.

We got on a call with their engineering team, and they first shared their screen, showing that the event had stopped a few days ago. This was strange because, as I mentioned, our scheduler hadn’t stopped running in the past three months. It was my turn to share my screen and show the log files. Guess what? It so happened that I had been debugging a hotfix and accidentally wiped the log files for the past five days. You can imagine how tense the meeting was, and I couldn’t back up my position when it was time to do so.

define grace!

Ridwan handled the situation gracefully and gave me an hour to fix it. That was one of the most productive hours of my life. I ran a backtrace and did a quick server recovery on the log files. That wasn’t enough, though. To ensure this doesn’t happen again, I built a tracker to save events on a temporary database, as well as a monitoring system that reports exceptions in case our events are being bounced by the recipient (which, as it turned out, was the case here). Within the hour, I shared the log files and also updated the scheduler to run both synchronously and asynchronously. It was a crazy day, and I learned that while debugging in production is necessary sometimes, however, you should never downplay an effective engineering process in the name of speedy execution.

Just when I thought I’d had my fill of challenges, another stressful situation came up. Sprints usually end on Thursdays, and features are promoted on Fridays. On a call that blissful evening with Ridwan, as we were reviewing blockers and completed tasks, he suddenly stopped talking and asked if we were running a background test. I wondered why he was asking that because, honestly, I was exhausted. I didn’t even have the energy to do anything, let alone run background tests.

He then shared a screenshot of an email alert with the group. We were being DDoS attacked. The attack had already created over 700 dummy accounts in our database. I quickly checked the log files and saw that our system was fighting back against an IP address, but whoever was behind this attack was skilled. The attack came in intervals, sometimes using different IP addresses. Why didn’t I notice this earlier? And why wasn’t Cloudflare doing its job? I was thankful that Ridwan caught it. What could I have done if our servers had crashed? Ridwan, being connected to our email service, gets all sorts of alerts. It was a hectic day, and there was still so much work to do.

Confused, as this was my first experience with such an issue, I called Ife. He quickly gave me leads on how to temporarily ban the attack, though he was busy at the time. I also called my cousin, who is quite senior with deep experience, and he provided additional suggestions. Jake chipped in as well. The first thing I did was write an SQL query to wipe all the accounts created by the bot. Next, I implemented a second rate limiter on our API endpoints. These measures temporarily stopped the attack. I then moved to Cloudflare and discovered that our base URL wasn’t proxied — it had been turned off a few months ago because it acted as a blocker. I re-enabled proxying for the URLs, wrote new firewall rules, and implemented other security updates.

Additional updates included working on a honeypot feature, restricting IP addresses based on location and frequency, adding rate limiters, and implementing redis caching on account creations, among other measures. That evening didn’t end until around 2 a.m.

Over the past nine months, I have overseen some exciting features and made several technical improvements. Here are a few:

Features & Improvements

1. **Dual-Environment
   **This feature provided our users with access to multiple versions of our applications. It was built to ease the process of going live after integrating with our API.
2. **Data-Storage Switch
   **I improved our rate systems, which previously stored rates in a separate database. Each request queried this database, which was heavy on our servers due to the high number and frequency of requests. I transitioned from database storage to converting our rates into JSON files, stored remotely, and called only once before being saved to cache. This approach drastically reduced request times and significantly improved our processes.
3. **API Documentation Revamp
   **I rewrote and updated the API documentation, transforming it into a more modern and user-friendly version.
4. **API Limits & Analytics
   **This improvement enhanced an existing feature that tracked API usage. Previously, we did not enforce limits based on criteria, which led to a 300% surge in our Google resource usage costs within a month. I addressed this by developing better analytics to monitor API usage. Once a user exceeds their limit, our middleware temporarily restricts their access to the API for a specific period.

This year has been a remarkable milestone in my software engineering journey. I’ve tackled challenges that once seemed insurmountable, and I’m proud of the growth I’ve experienced since joining SHiiP. I didn’t achieve this alone; I’d like to acknowledge the support and collaboration of Ife, Abdulrahman, and Jake — a seasoned engineer who’s always willing to brainstorm and share his expertise. In fact, Jake and I built the backend for Copnow together, and I look forward to writing about the design systems and architecture we employed.

There are few folks I look up to Tochukwu, Julian, Prosper, Femi, and Ezra. Their articles and problem-solving approaches have inspired me, and I aspire to follow in their footsteps.

Reflecting on my three years and three months of professional experience, I’d love to hear from others: am I heading in the right direction? What areas should I focus on improving or exploring further?

Thus far, I’ve primarily worked with PHP/Laravel and JavaScript/Nuxt, and I’ve also obtained certifications in Cloud DevOps and gained experience with cloud services. Currently, I’m venturing into Golang to build distributed systems, which aligns with my future aspirations. Alternatively, AI remains a fascinating field I’m eager to explore.

As I look to the future, I’m excited to see what’s in store. With the clock ticking five steps behind and new opportunities emerging, I’m ready to embrace the next chapter in my journey.

Ciao.