2023 in review

Feeling grateful about my job#

I’m incredibly proud of what I’ve done at my day job. I’m now working as an engineer in a squad in a product team. This year, I’ve led 3 projects and contributed to several projects in the team. All 3 projects come with their own challenges and satisfactions on their own. Here are some highlights:

• I made a big bet and re-architected a considerable part of the codebase that solves memory leak issues and reliability issues & supports new requirements from the stakeholders. After delivering this project, we reduced our CPU & memory capacity by >60%.
• I worked closely with the Engineering Manager (EM), Business Analyst (BA), and >9 engineers who contributed to the project. We had many conflicts (in a good way), including prioritization, asking for more staffing from other squads/teams, and many others. I learned a bit about project management and stakeholder management.
• We maintain our tests (both integration & unit tests) coverage at almost 100%, which gives us confidence in continuously updating & deploying our changes. When we first started the project, we bootstraped a new repository. After many months and hundreds of commits, it grew too sophisticated for it to fit in anyone’s mind. Pushing changes is extremely risky without good test coverage, and downtime can cause reputation damage and revenue loss.
• It was satisfying seeing my team present the project demo to the leadership team and see how this could impact our business. After months of hard work, everything has become a whole unit, and the customers started using it. I sometimes take a look at our Datadog dashboard to see the usage of what we’ve built.
• The projects took more than 6 months to complete.

I’m happy that I’m working with highly skilled people. Without them, none of these can be achieved.

Reflections#

How did I end up leading the projects? Sometimes this year, my team’s (acting) staff engineer went on a month-long vacation. The other lead engineer moved to a new team many months ago. As the next senior engineer in the team, I was naturally selected to take over and lead the project.

How do I feel after leading these projects? Honestly, I’m terrible at this, especially the people part. But this is outside my comfort zone and means I’m learning something.

Do I like it? I feel so satisfied after many months of work and finally seeing all the pieces assembled, together with all the learnings and the connections with the people involved. I couldn’t describe my feelings when we presented the project to the leadership team. Working with just code feels boring to me now.

My takeaway I’m forever grateful for the incredible people I worked with this year. Among my most significant leverages was working closely with the team’s Business Analyst (BA) and Engineering Manager (EM). From there, we accomplished a lot as a team.

As a project lead, I only have a few chances to deliver my work throughout the year. Assuming each project takes 2 quarters (or 6 months), I can only deliver 2 tangible projects in a year. This concludes that, as an engineer, our most valuable resource is, TIME. Spend your time wisely :)

Previously, as a junior and mid-level engineer, I could jump on any tasks, write code, and feel productive about my day. Now, my productive day looks different.

In my team, there are many talented engineers. Channeling the expertise and effort to build the right thing is essential. Otherwise, we might make something people do not want at the expense of opportunity and personnel costs. In my opinion, this is where EM, PM, BA, and lead engineer or project lead play essential roles.

Art of Balancing#

I struggle a lot with figuring out the right amount of balance between everything. For example:

• Balance between time/scope vs cost vs quality (Project management triangle)
• Balance between the team resources/expertise vs requirements from stakeholders
• Balance between training (investing time in) a new joiner in the team vs rushing to complete the projects
• Balance between the old & stable tools vs shiny & edge-cutting tools
• Balance between fast/optimized but unreadable code vs slow & maintainable code
• Balance between tasks/bugs priorities
• Balance between saying yes and no

I’m honestly still struggling with these questions. There is often no right answer to these questions (that’s why I wrote this as an art), only wrong answers (jk). Engineers usually think in binary (e.g., right or wrong, correct or incorrect), but this doesn’t apply here. Usually, you seek a good enough (ideally reversible) answer to those questions based on your context, and a perfect answer is typically impossible. Your decision today may differ for next quarter when your next project has to support new requirements in the same codebase. Nothing is wrong with that; it’s a natural evolution of your codebase. It’s fun if you’re staying on the same team long enough; you’ll witness the decisions you made last year turn out to be good, or you’ll learn some lessons from them.

Previously, this was a solved issue for me. Someone higher than me (EM, Tech lead) made the decisions (sometimes the right one, the other time the wrong one), and I have to implement, sometimes question them, sometimes offer better suggestions, but rarely make the decision.

Regrets#

Although I’m grateful for being given the trust to lead the projects and learned a lot from that, there are some areas that I wish to be better at:

• Get more involved in the recruitment process. I wanted to learn more about the interview process here, but I have yet to get a chance.
• Make full use of the mentorship program in the company. Until recently, towards the end of the year, I asked one of the engineers I admire to be my mentor, specifically about my career. It took me so long to think of someone to be my mentor. After several catch-ups, I better understand how the leveling works in the company.

Since I joined the company#

When I first joined the company, I was:

• Not confident speaking English with Australian colleagues - Before joining the company, I was comfortable speaking English with a Manglish accent. But working with Australian colleagues, my accent suddenly sounds awkward. And for those who don’t know, Australian accent differs from British or American English. I enrolled in English & communication classes to improve my communication skills. The only key here is to talk in complete sentences and understand the local accent arvo. Training my ear to adapt to their accent took me a while.
• Not confident in Javascript/Typescript - I used to work with many programming languages, primarily Python. Still, I used to code for professional work in Ruby, Java, Lua, and Go for the backend. Although I can quickly grasp the concepts between the languages, writing idiomatic code in the respective language took me some time. Not to mention getting used to the ecosystem. After reading and reviewing hundreds of PRs, I can write a decent Typescript code. Sometimes, my Typescript code still smells Pythonic, but it’s improving.
• Not used to big corporate setup - I came from startup companies. I learned a wide range of technologies, but since most startups don’t have formal processes (for example, promotion), it took me some time to finally get used to them. Big tech corp have different priorities compared to startup companies.

Read some books#

I’m actually not an avid book reader. But at this point, reading books is the only way to learn valuable lessons and advance my career. I’m not learning much anymore from reading/watching tutorials.

This year, I read (actually, I listened to audiobooks) Staff Engineer’s Path by Tanya Reilly, which taught me a lot about executing a project and navigating the organization.

Next year, I wish to read any of these:

• Data Intensive Application by Martin Kleppman
• Tidy First by Kent Beck
• The Software Engineer’s Guidebook by Gergely Orosz

Besides this, I still enjoy watching conference talks, especially from Lead Dev, NDC, Strange Loop, USENIX, @Scale, Devoxx, YOW!, GOTO Conferences, InfoQ QCon, etc. At the moment, I have over 1000 videos in my Watch Later playlist on YouTube :p

Poor time management#

Unlike last year, I have made some progress in improving my focus, although there is still much room for improvement. If I could focus and have better clarity, I would have achieved more success in my life.

The tech world is moving too fast, and so many things interest me. However, protecting my time and choosing what’s important to me is also important. Time is finite, and you can’t have everything in this world.

I used to spend many hours daily keeping up with the latest trends and developments and engineering blogs from Shopify, Pinterest, Meta/Facebook, etc. Sadly, I can no longer spend as much time reading them, but it’s a choice I have to make.

I blame Twitter as well in some ways. It’s tough to quit Twitter.

Side Projects#

Simpan Cukai#

This project was made to satisfy both my routine need to keep up with my tax filing & my urge to actually build something with Django & HTMX.

This is the second time building the same app but with a different approach. Previously, I wrote a similar app using JAMstack: NextJS & Django DRF. However, I’m not too fond of the Developer Experience when working with it, especially when doing it alone. I have to write the serializer in Django DRF, which spits JSON. Then, serialize the data again in the NextJS frontend. The app feels snappy, but the effort to do so is not worth it, especially in my situation. After I rewrote it in Django + HTMX + AlpineJS, the codebase is much cleaner and concise, and I can move much quicker. No need to write serializer anymore, just plain HTML! Luckily, I found linters like djlint that help lint & format my HTML code and Ruff that allows lint & format my Django code every time before I commit to Github, now my code looks less ugly. The developer experience is much improved now!

Mukmin Life#

A simple app for prayer time. I built this app with a friend, but I mainly worked on the backend of the app. It’s been a while since I wrote Go, so I wanted to write in Go again to refresh my memory. The backend is just a simple HTTP server with a database connection and accepts REST requests. Along the way, I tried several serverless databases but ended up with Cockroach DB again, like many of my personal projects.

A friend worked on a mobile app built using Expo & React Native.

You can download the app on Google Play Store.

The backend is built in Go, sqlc, Go Fiber, Google Cloud Run Cloudflare CDN, Cloudflare Pages, & CockroachDB.

Satisfying my curiosity#

Playing and breaking self-hosted services#

In the past, I used to work in a company where we self-host most of our services (database, CI/CD pipeline, message broker, and other fun stuff) on bare-metal servers on colocation sites. I don’t really enjoy doing that, especially after seeing friends using AWS Lambda, DynamoDB, EKS, Terraform, etc., and not having to think about the operation tasks. But then, it’s funny that the pendulum has swung: Basecamp and OneUptime are moving out of the cloud and going bare metal now.

To satisfy my itchiness, I’ve created a Github repository distributed-system-labs to learn and break distributed systems. I enjoyed playing with them but still need more personal time to hack them. I mostly spend time hacking that software on VMs instead of containers or Kubernetes for 2 reasons: it’s easier to break things in a VM, like injecting network failures, and some companies still operate stateful services on VMs instead of Kubernetes. I wanted to see how things break in VMs, like reaching CPU saturation, OOM, network failures, etc. In Kubernetes, those pods were likely to get replaced before I could SSH into and look at the logs.

I can’t thank Orbstack enough for its magical software. Spinning up and down containers, VMs & Kubernetes clusters is so fast that it rekindles the fun of playing with these toys again. Unlike using Virtualbox in the past, it takes minutes to spin up a new machine and configure everything. With Orbstack, everything is set up in under a second, with all the networking plumbings, local DNS setup, HTTPS certs, and dir mount done on my behalf! If I broke anything, just delete the VM and create a new one.

Database inventions#

Many new databases are being developed to satisfy unique and specialized needs: JunoDB, Temporal Cloud Data Store, KalDB, Venice DB, Talaria DB, New Relic Database (NRDB), Datadog Husky. It’s great to see companies bending the rules and thinking with the first principle to design something foundational, which sometimes contradicts the best practices in the industry. Not to mention the innovation in open source databases: Quickwit and CacheGrand.

One database technology I just learned recently was the Apache Pinot & Star tree index, which is fantastic for high throughput analytical queries.

In 2023, I’m excited to see many databases evolve to become cloud-native using S3 as their primary storage media: Clickhouse Cloud and Warpstream. During the AWS Reinvent event, they announced the AWS S3 Express class, and I’m excited to see how the industry adopts (or rejects) this.

These are innovations that excite me.

Programming trends#

These are some of the exciting things I learned in 2023 but don’t actually use in production:

• Service Weaver. The idea is that you can write software that looks like a monolith but deploys like microservice software. I hope this gets adopted more widely in the future and gets more mature.
• MS Orleans (Virtual Actor model). Learning this paradigm has changed the way I see software development a lot. Unfortunately, as someone who never writes C#, this MS Orleans is not something I quickly pick up hands-on. I know there are a lot of talks out there explaining how MS Orleans is used to build the backend for Halo4 & other tech in Microsoft, but this presentation impressed me a lot. You can write single-threaded software (so you don’t have to worry about race conditions & many other potential issues) and persist the data asynchronously on the data store outside the hot request path. I hope this technology gets more adoption, especially in Go.
• Remix. This could be a hot take, but I like how they design the API around Remix. Everything they built, especially the idea, impressed me and aligned with my values. One advantage of using Remix over HTMX is that it is an isomorphic framework, so I can do optimistic operations.
• HTMX. As frontend illiterate, HTMX is bliss when paired with AlpineJS. This is the stack that I used to build the front end for Simpan Cukai. Rendering HTML is fun again!
• Temporal. After seeing talks on how companies (especially Hashicorp, Datadog & Redpanda) uses Temporal to run & automate their infrastructure operations, really inspires me. I think this software has so much potential.

Again, I want to emphasize that I don’t use any of these technologies in my day job (except HTMX for Simpan Cukai), but I would love to use them if there’s an opportunity to do so.

Personal life#

I’m blessed with a son early this year. My days are now busy with my beloved family.

September: The baby was admitted to the hospital for a bacterial infection. The bacteria is multi-resistant, so the doctor took time to determine the right antibiotic for him.

October: The baby was admitted to ICU due to Bocavirus infection. The baby had to take oxygen support from a ventilator machine, and it was a really sorrowful moment for us.

Our experience in the hospital was really humbling.

Like everybody else, my life is full of ups and downs. There were setback moments where I questioned my life choices & what kind of life I’m living right now. In short, my life in 2023 was quite emotionally and physically exhausting. I believe any working parent with a young baby would agree with this statement.

My focus in 2024#

Tbh, I’m still not too sure atm. I have a few things on my mind but still need more clarity.

Aside from the projects I led this year, only a little interesting happened in my work. Next year, I’d like to grab more chances to work on more exciting projects.

Conclusion#

I’m grateful for everything I have today: my family, job, and friends. Alhamdulillah, for all the blessings. I pray for a better year in 2024 for everyone :)