Five Web Project Lessons

Last month, I finally launched my collaborative map-based urban planning web application: Plan Together. It was a lot of work, but I must say I am proud of the result; it is my most feature-complete project to date. It’s quite an experience having to do 100% of the work for product development, system architecture design, programming, and user interface/experience design. It definitely increases one’s respect for the work of their co-workers in different departments. I highly recommend it.

Here are some lessons I learned along the way from a software developer’s point of view:

1. Consider HTTP request Overhead

As a C++ Back-End Web Service Developer I work mainly with a in-house binary transmission protocol which composes and parses the bare minimum of bytes per message. Sending data which is not directly relevant is considered an affront. During the initial design and prototype phase, my driving philosophy was to send many small requests, retrieving only what was immediately necessary. However, during trials, I had to confront the fact that each individual HTTP request is padded with a rather large amount of overhead due to Headers. I was forced to reduce the number of requests and increase their size, which is the logical approach from a web developer’s point of view. Lesson learned.

2. Remember Database Access Overhead

Theoretically, this is an obvious one. But nothing hits you as hard as actually having to design data shape and retrieval processes yourself. Just as with HTTP requests, my initial approach had numerous small SELECTs, in an effort preventing retrieval of data that might not be needed. Originally, one SELECT would retrieve a collection of ids (defining relevant records), and other SELECTs would retrieve other data columns as needed. I was being too clever. Maybe my approach would work if database access was free, but that is far from the case. It became very clear; it is better to perform wider queries, selecting more information than needed. Go in, grab a chunk of mostly relevant data, and exit. Finally, cache it. Disturbing the database should be your last resort.

3. Good deployment pipelines are hard to implement (But Worth It)

As a developer at a large company, automated build setup, production deployment and configuration are parts of the software world that I don’t need to touch. I just focus on my teams specific modules, write code that fulfills requirements, and follow existing guidelines. After code is reviewed and tested, it is out of my hands. I do not directly take the code and make it do live work for users. Since I was the only person working on Plan Together, I obviously had to do it myself. Sometimes, configuration is pure hell. Figuring out the details of how GitHub Actions are triggered, which cloud platforms are best for your Front-End and Back-End, and how to best deploy your application takes a while and feels very unproductive. It was hard to let go of the mindset that if I was not working on the code or the concept, I was wasting time. Now, I am happy I spent that time. I can easily deploy changes just by merging feature-branches into the production branch and all deployment is automatic.

4. Most External APIs are easy to integrate

We are really living in a Golden Age of integration. Adding external features to your product is easier than ever. Finally writing something myself proved this directly. The Plan Together application automatically posts certain Map Items to twitter, mentioning relevant accounts: City Of Calgary Tweet Example. I was surprised how easy this was to implement. I applied for a Twitter Developer Account, integrated the relevant library into the code-base and was easily able to start tweeting map images.

5. Ruthlessly Prioritize Features and Focus on Releasing Anything

It becomes very easy to delay a release. Feature and scope creep are real. And especially since you are the only person working on this project, your ego becomes tied up in it. You want it to be good enough, you start daydreaming of a big launch. So you keep adding more and more to the product. I caught myself doing this more than once during development. Now, after the initial release, I realized how important it is to get something out there into the real world. Not only do you get real feedback, but your self-pressure is taken off your shoulders. I have actually been able to accelerate development since release and added a handful new features: 9 New Features Added to Plan Together Since Launch

Thanks for reading my article! Feel free to checkout Plan Together, or email me with comments/questions at dg@davidgorski.ca.