![\"\"](\"/assets/guestbook.png\")
In this blog post, we’re implementing a no-JavaScript guestbook system for statically-generated websites with built-in spam protection. That’s right: no JavaScript involved. Just good old HTTP and HTML working together… with a bit of Netlify magic. The solution I’m presenting here is specific to Netlify and Jekyll, but I’m pretty sure you could port this to other service/tool combinations.
\n\nSometime ago, I set out to build a guestbook for this website. I love the idea of a shared space where all visitors can share their experience and connect with others. An internet guestbook is like a hotel guestbook on steroids. Here’s an excerpt from my own guestbook (view the live guestbook here!):
\n\n
I didn’t, however, want to move from the static website setup I had to a full-blown WordPress setup or anything that had a server-side backend that would unnecessarily make things more complicated and potentially lead to a worse experience. Additionally, I did not want the guestbook to rely on JavaScript, because I respect those who turn off JavaScript when browsing the Internet. The guestbook would need to work for everyone regardless of their web browsing preferences.
\n\nLucky for us, we already solved this problem in the early 1990s: HTTP has a POST verb that a server can handle to receive data from the client. However, we run into another problem when considering that this is a static website (which is effectively just a bunch of HTML files linked together). There’s no dynamic back-end of any kind serving those HTML files to you: at the time of writing this, it’s all served straight from Netlify’s CDN.
\n\nSo what do we do?
\n\nWell, this website was already off to a good start: it’s powered by Netlify, which offers an excellent Forms product, which provides an API. Once it’s set up, Netlify intercepts POST calls sent to a web page and runs logic in the background to store the contents of the form submission for later use via their API. In practice, this means that when someone goes on the guestbook and submits their comment using the form at the bottom of the page, Netlify intercepts the POST call and creates a database entry somewhere on their backend, which I can then query through their Forms API.
\n\nNote that when someone submits a comment on the guestbook, it doesn’t automatically show up on the guestbook page: this is because I manually moderate comments by pulling them from the Netlify Forms API myself and then re-generate the website.
\n\nThe beauty in this workflow is two-fold:
\n\nMake sense? Let’s see how to implement this. I’ll assume you already have a Netlify account and a site set up there. If not, do that, and come back when you’re done.
\n\nLet’s start with the entrypoint: the actual HTML form itself.
\n\nNetlify’s excellent “Forms setup” documentation page explains how to start using their Forms API: simply add an HTML form to your website, then publish it to the Internet. The first time someone fills out the form and submits it, you’ll get an email, and see your form created in the Forms section of your site in the Netlify admin panel. It’s that easy.
\n\nHere’s the HTML form I added on my own guestbook page (view source on GitHub):
\n\n<div>\n <h2>Sign the guestbook</h2>\n\n <form action=\"/submitted\" name=\"guestbook\" netlify netlify-honeypot=\"not-for-humans\">\n <p style=\"display: none;\">\n <label>Don’t fill this out if you're human: <input name=\"not-for-humans\" /></label>\n </p>\n <p>\n <label>Name<br><input placeholder=\"Your name...\" type=\"text\" name=\"name\" /></label>\n </p>\n <p>\n <label>Message<br><textarea name=\"message\" placeholder=\"Your message...\"></textarea></label>\n </p>\n <p>\n <button type=\"submit\">Submit</button>\n </p>\n </form>\n</div>\nNotice these few key points:
\nnetlify attribute in the form tag is what tells Netlify to actually care about this form. Without this, Netlify wouldn’t track submissions to this form.not-for-humans). This field is not shown to humans browsing the page, so it should always be empty. However, bots tend to fill out all fields, so Netlify knows that if this field is not empty, it’s almost certainly a spam submission, so it knows it can simply ignore it.name attribute (in this case, guestbook) is the identifier of the form, which you’ll see appear in the Netlify admin panel later when submissions start coming in.Once you have added this form to your site, publish it to the Internet, and submit the form on your live site to test it out. If all goes well, Netlify will give a success response. You can customize this success page by adding a Jekyll page at the /submitted permalink URL, similar to what I’m doing on this website.
We now have a way to receive submissions from people browsing our website. Let’s now see how to pull these submissions from the Netlify Forms API and into our Jekyll website.
\n\nI wrote a Ruby script (not very clean, PRs welcome!) that talks to the Netlify Forms API and pulls form submissions into a YAML file that Jekyll reads when generating the static site. You can download the script from my repository on GitHub.
\n\nYou’ll need to install the HTTParty gem (gem install httparty) and replace three values in the file (keep those values secret, don’t commit them to a public repository!):
NETLIFY_PERSONAL_ACCESS_TOKEN: you can generate a personal access token in your Netlify settings.NETLIFY_SITE_ID: you can find this ID by going in your Netlify site settings (Site information ➞ API ID).NETLIFY_FORM_ID: you can find this ID by going in the Forms section of your Netlify site, clicking on the form in the list, and copying the last part of the URL (/sites/<site_name>/forms/<FORM_ID>).Before running the script however, make sure to create a _data/ directory in your Jekyll site, and inside of it, an empty guestbook_messages.yml file; this is where the script will write form submissions to for Jekyll to use when generating the site.
Once you’re done, run the script with ruby update_guestbook.rb. This should pull form submissions from the Netlify Forms API and write the entries to your site’s _data/guestbook_messages.yml file. Here’s how the file looks like in my own repository.
At this point, there’s only one step left: showing the form submissions in the _data/guestbook_messages.yml on your site.
To show the entries from the YAML file on your site, add this snippet where you’d like to show the messages:
\n\n{% for message in site.data.guestbook_messages reversed %}\n <div>\n <div><span style=\"font-weight: bold;\">{{ message.author }}</span> ({{ message.timestamp | date: \"%Y-%m-%d\" }})</div>\n <div>{{ message.body }}</div>\n </div>\n{% endfor %}\nAt this point, your site should show the entries in the YAML file that the Ruby script wrote to after fetching the form submissions from the Netlify Forms API.
\n\nAll you need to do now is to generate your static site and upload it somewhere on the Internet.
\n\nThe next time someone sends a form submission on your site, you’ll get an email from Netlify, after which you can run the Ruby script to moderate the comment, and if it’s acceptable, commit the changes, then push them to Netlify to update your site. In the event that you don’t want to show a specific form submission on your site, log in to your Netlify admin panel, go to your site’s Forms section, open the form, then delete the entry.
\n\nA potential next step could be to extend this to every single blog post to effectively create a commenting system (rather than a single guestbook page)… but that’s for another time.
\n", "url": "https://maximevaillancourt.com/blog/building-fully-static-guestbook-jekyll-netlify", "summary": "No JavaScript. Just good old HTTP and HTML working together.", "date_published": "2021-08-08 00:00:00 +0000" }, { "id": "/blog/moving-away-from-google-services-8-years-in", "title": "Moving away from Google services, 8 years in", "content_html": "On July 1st, 2013, Google shut down Google Reader forever. On that day, I understood that my personal data was at the mercy of a company that might just shut down access to it whenever business incentives lined up.
\n\nMy digital life almost entirely used to be stored within Google services: emails, personal files, to do lists, photos, RSS subscriptions, contacts, passwords, music playlists, apps, etc. Since then, I’ve been on a journey to reduce my reliance on Google products and to put my eggs in more privacy-friendly baskets, and I’ve come to realize that having agency over my own data is really, really satisfying.
\n\nThis blog post covers how I migrated away from a dozen Google services to privacy-friendly (and sometimes self-hosted) services instead. Let’s go over those one-by-one and see how they all work (and how you can set them up yourself) Hint: it’s quite fun.
\n\nBefore we start, it’s worth pointing out my personal device setup:
\nOkay, with that out of the way, let’s jump in!
\n\nHere it is: the original trigger. When Google Reader shut down, I moved my subscriptions to Feedly, then to Feedbin, and eventually I decided to self-host Miniflux on the Raspberry Pi. Miniflux is super easy to install: it’s a single binary, and from that I simply created a service that runs the binary whenever the Raspberry Pi boots up.
\n\nThe nice thing about Miniflux is that it provides a Web UI, which I use when reading feeds from my laptop. I simply punch in the IP address of my Raspberry Pi, tack the port where the Miniflux service is running at, and log in. Easy!
\n\nFor Android however, I use a neat little app called Constaflux. Setting it up is easy too: punch in the IP address and port of your Miniflux service, username and password, and connect. From there, you have a native application that lets you read your RSS feeds on your Android device.
\n\nVery happy with this setup. It’s fast, there are no ads, and I greatly enjoy that my data is mine and mine only.
\n\nIf I was to stop relying on Google services, I needed to work on the one that contained most of my life: Gmail. I decided to go with ProtonMail for multiple reasons: their services are hosted in Switzerland, their products are open-source, and messages are stored with zero-access encryption, as any good online service provider should do. I’m very happy to pay for ProtonMail Plus. Only good experiences with it 4 years in.
\n\nGetting started is easy: set up a forwarding rule in Gmail settings so that any emails sent to your Gmail address get forwarded to your ProtonMail address. This way, no need to even have to monitor Gmail account anymore: it’s all going to the ProtonMail address.
\n\nThe tricky thing however is to change email addresses everywhere: online services, offline businesses, friends and family, etc. Patience is key for this one. I’m still waiting to delete my Gmail account because to this day I receive the odd email to the Gmail address every now and then, though this is becoming less frequent as I’m changing email addresses every time I get those odd emails directly to Gmail. I created a label in ProtonMail for emails that are sent to the Gmail address, so directly in the inbox I can tell when an email was sent directly to the Gmail address (meaning I should let the sender know I changed my email address).
\n\nAdditionally, if you’d like to migrate the old emails you have in your Gmail account to ProtonMail, you may use Google’s export service Google Takeout, which lets you download an .mbox file you can then import into ProtonMail.
Good. Gmail is out of the way. But the other big thing I relied on Google for was calendars and tasks. How does one move years of calendar data to another service, and most importantly, which one?
\n\nI decided to go with Radicale, a “Free and Open-Source CalDAV and CardDAV Server” I consider in the “set-and-forget” category. I genuinely never bothered with debugging Radicale even 3 years after setting it up. It’s that good.
\n\nRadicale is only the “server” part of the setup, meaning it provides a CalDAV API for clients to use to create/update/delete calendar events. To actually manage your calendar, you need to use a client to connect to your Radicale server.
\n\nI installed Radicale on my Raspberry Pi 3B+, which, as mentioned earlier, has a static IP on my local area network (LAN). This means that the Raspberry Pi is always available at the same IP address even when my router reboots. Thanks to this, I can point clients on my Android smartphone and on my Linux laptop to that IP address knowing they’ll always be able to talk to Radicale.
\n\nOn Android, I use DAVx⁵ (previously known as DAVdroid) to connect to the Radicale server (punch in the IP address, username and password, and you’re done), and Etar to actually interact with my calendar (add/update/delete events). I also use ICSx⁵ to read external calendar files (for PagerDuty on-call shifts, for example) so that they show up in Etar. For tasks, I use OpenTasks, which just like Etar also talks to DAVx⁵.
\n\nOn my laptop, I simply use Thunderbird with the Lightning extension. Doesn’t get any easier than that.
\n\nWhat this provides is a LAN-only set up, meaning my calendar and tasks data are only stored on my Raspberry Pi, Android phone, and Linux laptop. Given that I nearly always come back home (on my local network) at least once a day, I know my data will be synchronized daily. This setup might be a problem however if/when I leave home for an extended period of time with both my laptop and my smartphone, because they won’t have the Radicale server on the Raspberry Pi to communicate with anymore, leaving both my devices out of sync until I come back home. This hasn’t been a problem for me yet, but certainly something to keep in mind.
\n\nI never really used Google Drive for much, but I did have one very important file in there that I used across multiple devices: my KeePass .kdbx database (in an encrypted archive). I removed everything I had in Google Drive, opted for a local-first approach, and started using Syncthing to selectively share what I need. Thanks to Syncthing, I can directly share my KeePass database between my Android smartphone and Linux laptop without needing a third-party to sync anything. I find the simplicity of this setup quite elegant. To access and update my KeePass database, I use KeeWeb on my laptop and Keepass2Android on my Android device.
\n\nFor occasional file server requirements (say, downloading a PDF on my Kobo e-reader using its web browser), I have two options: I set up nginx on the Raspberry Pi to expose a specific directory as an HTTP file server, so I can copy a file to the Raspberry Pi in that directory to make it available to my LAN; alternatively, I sometimes use Termux on my Android device to spin up an HTTP server in a specific directory. Both approaches work nicely, though I have to say there’s something really fun in spinning up a Python SimpleHTTPServer in the command line on my Android device. :)
\n\nI never was one to upload my photographs to a third-party service (Flickr, Picasa, Google Photos, etc.), so I never actually did such a migration, but I want to point out that it’s a good idea to keep your photos to yourself. Never share more than necessary with untrusted third-parties.
\n\nI store photos and videos on a pair of external hard drives that I manually mirror 1:1 (your typical RAID 1 pair). This is a weak part of my data setup, because both drives are in the same physical location. If my apartment building burns down, so do all my photos and videos. I’m looking into creating a third copy to store in a remote location (e.g. a trusted friend/relative’s place) to reduce this risk. I’m not there yet, but the end goal is the 3-2-1 backup strategy: “have 3 copies of your data on two different media with one copy off-site for disaster recovery”.
\n\nMy use case for YouTube is simply sharing videos with friends and family. I’m not a YouTuber that shares content with the entire world, so I don’t need the large-scale capabilities of YouTube, nor its comment and rating system, etc. To power my use case with a privacy-friendly solution, I signed up with Linuxfabrik, a Nextcloud instance hosted in Switzerland (again, for its data privacy laws) with end-to-end encryption. This lets me upload videos and create shareable links with passwords to send to friends and family.
\n\nI started using Google Play Music soon after it launched, and eventually they shut it down and later turned it into YouTube Music, so I moved to Spotify. No regrets there: Spotify has a lot more music, great playlist recommendations, and feels much better than Google Play Music did.
\n\nI used to keep (pun unintended) notes and ideas and thoughts in Google Keep, which was easy to use on Android and okay on the web, but eventually discovered Standard Notes, a private and secure (and open-source!) notes app with excellent Web and mobile clients. In fact, I’m writing this blog post in Standard Notes on my ThinkPad X220 using the Standard Notes web UI.
\n\nAgain, I’m happy to pay for Standard Notes Extended because I know I’m encouraging a small team of people who care about privacy and long-term data integrity. It feels good to pay for something knowing the folks behind the scenes are doing the right thing.
\n\nWhile I could self-host Standard Notes (they have great documentation explaining how to do that), I enjoy having access to my notes from anywhere in the world, so I’m sticking to their hosted solution for now.
\n\nThis one, I have to say, I struggle to stick to. Google Maps is just that good. The search feature works really well, and time estimates (with traffic considered) are accurate. As much as I’d like to jump to OsmAnd full-time and delete Google Maps from my Android device, I find that OsmAnd’s search feature doesn’t cut it, and its UI is somewhat confusing. One day! For now, sticking to a combination of the two.
\n\nMost importantly: whatever app I use, I always keep location services off unless I actively need them.
\n\nAnother big one. While the Android distributions we run on our devices are based on the Android Open Source Project (AOSP)–which is open to everyone–Google still has a lot of influence on the direction of the operating system and how it connects to Google services. I previously used a custom ROM on my Android device without Google Play Services (though with microG), but I eventually went back to a stock ROM (with Google Play Services) to ensure full functionality with PagerDuty, which I rely on for work purposes. I had a hard time imagining myself justifying why I missed a page with “well I use a custom ROM on my Android phone and somehow the telephony firmware was messed up so I missed all the calls and notifications stopped working so this is why everything’s been down for 2 hours”.
\n\nStill, I don’t have a Google account connected on the phone, so no data being shared with Google there, and I uninstalled all Google-powered apps (except Google Maps as mentioned above). I use F-Droid to install apps, and Aurora Store for the occasional non-free app (such as PagerDuty).
\n\nDuckDuckGo. Enough said. :)
\n\nAll my two-factor authentication codes are currently stored in the Google Authenticator app on my Android device. I intend to move to another solution soon; just need to take the time to actually do it. Still not sure about which solution to pick yet.
\n\nThis one is quite obvious. Download Mozilla Firefox (on mobile too!), install, and enjoy. Consider reading this blog post from Asif Youssuff about privacy in Firefox to improve your experience, and make sure to install an ad content blocker such as uBlock Origin.
\n\nThis is the result of a few years of thinking about agency over my data, about long-term data integrity failure scenarios, and data privacy in general.
\n\nNext step: potentially introduce Tailscale into the mix to safely allow access to the devices on my LAN from anywhere in the world.
\n\nNext next step: finally delete that Google account. 🔥
\n", "url": "https://maximevaillancourt.com/blog/moving-away-from-google-services-8-years-in", "summary": "Self-hosted LAN-only services FTW!", "date_published": "2021-07-28 00:00:00 +0000" }, { "id": "/blog/canon-dslr-webcam-debian-ubuntu", "title": "Using a Canon DSLR as a webcam with Debian/Ubuntu", "content_html": "The following was tested with a Canon EOS 6D and a ThinkPad X220 running Debian 10 Buster on Linux Kernel 4.19.0-9-amd64. Your mileage may vary depending on your particular configuration, but I expect this to work on a wide range of Canon DSLRs and other Linux distributions.
\n\n1. Install and set up dependencies
\n\nFirst, we need to install a few packages.
\n\n$ sudo apt-get install gphoto2 v4l2loopback-utils v4l2loopback-dkms ffmpeg build-essential libelf-dev linux-headers-$(uname -r) unzip vlc\n2. Connect your camera
\n\nUse the Mini-USB to USB-A cable to connect the DSLR to your computer. Then, turn the camera on, and make sure it’s dialed to Photo mode (in Photo mode, I’m able to get ~20fps, but in Video mode, I barely get 4fps).
\n\n3. Start the capture
\n\nOne last step before we can start capturing: we need to enable the v4l2 kernel module.
\n\n$ sudo modprobe v4l2loopback\nNow, let’s start the capture using gphoto2 and ffmpeg.
\n\n$ gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video2\nYou may need to tweak the last argument to a different device depending on your configuration. Try using /dev/video0, /dev/video1, etc. if /dev/video2 doesn’t work.
4. Watch the feed
\n\nvlc v4l2:///dev/video2\n(again, make sure to use the appropriate device)
\n\nIf you see the image from your camera, you’re all set! You should now be able to use this video feed with video conference software (Jitsi Meet, Google Meet, etc.).
\n\nEnjoy!
\n", "url": "https://maximevaillancourt.com/blog/canon-dslr-webcam-debian-ubuntu", "summary": "No EOS Webcam Utility on Linux? No problem!", "date_published": "2021-01-04 00:00:00 +0000" }, { "id": "/blog/optimizing-server-side-storefront-rendering-shopify", "title": "Optimizing server-side storefront rendering at Shopify", "content_html": "This post originally appeared on the Shopify Engineering blog on December 10, 2020. I co-wrote it with my colleague Celso Dantas, Staff Developer at Shopify.
\n\n\n\nIn the previous post about Shopify's new storefront rendering engine, we described how we went about the rewrite process and smoothly transitioned to serve storefront requests with the new implementation. As a follow-up and based on readers’ comments and questions, this post dives deeper into the technical details of how we built the new storefront rendering engine to be faster than the previous implementation.
\nTo set the table, let’s see how the new storefront rendering engine performs:
\nThanks to the new storefront rendering engine, the average storefront response is nearly 5x faster than with the previous implementation. Of course, how fast the rendering engine is able to process a request and spit out a response depends on two key factors: the shop’s Liquid theme implementation, and the number of resources needed to process the request. To get a better idea of where the storefront rendering engine spends its time when processing a request, try using the Shopify Theme Inspector: this tool will help you identify potential bottlenecks so you can work on improving performance in those areas.
\n\n![\"A](\"https://cdn.shopify.com/s/files/1/0779/4361/files/Storefront-renderer-data-schema_c5f379b7-619f-4ddb-8064-d093550c4731.jpg?v=1607636250\")
Before we cover each topic, let’s briefly describe our application stack. As mentioned in the previous post, the new storefront rendering engine is a Ruby application. It talks to a sharded MySQL database and uses Redis to store and retrieve cached data.
\nOptimizing how we load all that data is extremely important. As one of our requirements was to improve rendering time for Storefront requests. Here are some of the approaches that we took to accomplish that.
\nTo reduce the number of network round trips to the database, we use MySQL’s multi-statement feature to allow sending multiple queries at once. With a single request to the database, we can load data from multiple tables at once. Here’s a simplified example:
\nThis request is especially useful to batch-load a lot of data very early in the response lifecycle based on the incoming request. After identifying the type of request, we trigger a single multi-statement query to fetch the data we need for that particular request in one go, which we’ll discuss later in this blog post. For example, for a request for a product page, we’ll load data for the product, its variants, its images, and other product-related resources in addition to information about the shop and the storefront theme, all in a single round-trip to MySQL.
\nAs shown above, the new storefront rendering engine uses handcrafted, optimized SQL queries. This allows us to easily write fine-tuned SQL queries to select only the columns we need for each resource and leverage JOINs and sub-SELECT statements to optimize data loading based on the resources to load which are sometimes less straightforward to implement with a full-service object-relational mapping (ORM) layer.
\nHowever, the main benefit of this approach is the tiny memory footprint of using a raw MySQL client compared to using an object-relational mapping (ORM) layer that’s unnecessarily complex for our needs. Since there’s no unnecessary abstraction, forgoing the use of an ORM drastically simplifies the flow of data. Once the raw rows come back from MySQL, we effectively use the simplest ORM possible: we create plain old Ruby objects from the raw rows to model the business domain. We then use these Ruby objects for the remainder of the request. Below is an example of how it’s done.
\n\n\n
\nOf course, not using an ORM layer comes with a cost: if implemented poorly, this approach can lead to more complexity leaking into the application code. Creating thin model abstractions using plain old Ruby objects prevents this from happening, and makes it easier to interact with resources while meeting our performance criteria. Of course, this approach isn’t particularly common and has the potential to cause panic in software engineers who aren’t heavily involved in performance work, instead worrying about schema migrations and compatibility issues. However, when speed is critical, we accept to take on that complexity.
\nAn HTTP request for a Shopify storefront may end up requiring many different resources from data stores to render properly. For example, a request for a product page could lead to requiring information about other products, images, variants, inventory information, and a whole lot of other data not loaded on multi-statement select. The first time the storefront rendering engine loads this page, it needs to query the database, sometimes making multiple requests, to retrieve all the information it needs. This usually happens during the request at any given time.
\n![\"A](\"https://cdn.shopify.com/s/files/1/0779/4361/files/flow-request-bookeeping-solution_adbd68eb-cc30-4be5-9bdf-104011224ad2.jpg?v=1607636269\")
As it retrieves this data for the first time, the storefront rendering engine keeps track of the queries it performed on the database for that particular product page and stores that list of queries in a key-value store for later use. When an HTTP request for the same product page comes in later (which it knows when the cache key matches), the rendering engine looks up the list of queries it performed throughout the previous request of the same type and performs those queries all at once, at the very beginning of the current request, because we’re pretty confident we’ll need them for this request (since they were used in the previous request).
\nThis book-keeping mechanism lets us eager-load data we’re pretty confident we’ll need. Of course, when a page changes, this may lead to over-fetching and/or under-fetching, which is expected, and the shape of the data we fetch stabilizes quickly over time as more requests come in.
\nOn the other side, some liquid models of Shopify’s storefronts are not accessed as frequently, and we don’t need to eager-load data related to them. If we did, we’d increase I/O wait time for something that we probably wouldn’t use very often. What the new rendering engine does instead is lazy-load this data by default. Unless the book-keeping mechanism described above eager-loads it, we’ll defer retrieving data to only load it if it’s needed for a particular request.
\nMuch like a CPU’s caching architecture, the new rendering engine implements multiple layers of caching to accelerate responses.
\nA critical aside before we jump into this section: adding caching should never be the first step towards building performance-oriented software. Start by building a solution that’s extremely fast from the get go, even without caching. Once this is achieved, then consider adding caching to reduce load on the various components on the system while accelerating frequent use cases. Caching is like a sharp knife and can introduce hard to detect bugs.
\n![\"A](\"https://cdn.shopify.com/s/files/1/0779/4361/files/Storefront-renderer-data-schema-in-memory-cache.jpg?v=1607636398\")
At the frontline of our caching system is an in-memory cache that you can essentially think of as a global hash that’s shared across requests within each web worker. Much like the majority of our caching mechanisms, this caching layer uses the LRU caching algorithm. As a result, we use this caching layer for data that’s accessed very often. This layer is especially useful in high throughput scenarios such as flash sales.
\nAs a second layer on top of the in-memory cache, the new rendering engine leverages a node-local Redis store that’s shared across all server workers on the same node. Since the database is available on the same machine as the rendering engine process itself, this node-local data transfer prevents network overhead and improves response times. As a result, multiple Ruby processes benefit from sharing cached data with one another.
\nOnce the rendering engine successfully renders a full storefront response for a particular type of request, we store the final output (most often an HTML or JSON string) into the local Redis for later retrieval for subsequent requests that match the same cache key. This full-page caching solution lets us prevent regenerating storefront responses if we can by using the output we previously computed.
\nIn a scenario where the full-page output cache, the in-memory cache, and the node-local cache doesn’t have a valid entry for a given request, we need to reach all the way to the database. Once we get a result back from MySQL, we transparently cache the results in Redis for later retrieval based on the queries and their parameters. As long as the cache keys don’t change, running the same database queries over and over always hit Redis instead of reaching all the way to the database.
\nThanks to the Liquid templating language, merchants and partners may build custom storefront themes. When loading a particular storefront page, it’s possible that the Liquid template to render includes multiple references to the same object. This is common on the product page for example, where the template will include many references to the product object: {{ product.title }}, {{ product.description }}, {{ product.featured_media }}, and others.
Of course, when each of these are executed, we don’t fetch the product over and over again from the database—we fetch it once, then keep it in memory for later use throughout the request lifecycle. This means that if the same product object is required multiple times at different locations during the render process, we’ll always use the same one and only instance of it throughout the entire request lifecycle.
\nThe Liquid object memoizer is especially useful when multiple different Liquid objects end up loading the same resource. For example, when loading multiple product objects on a collection page using and then referring to a particular product using {{ all_products['cowboy-hat'] }} on a collection page, with the Liquid object memoizer we’ll load it from an external data store once, then store it in memory and fetch it from there if it’s needed later. On average, across all Shopify storefronts, we see that the Liquid object memoizer prevents between 16 and 20 accesses to Redis and/or MySQL for every single storefront request, where we leverage the in-memory cache instead. In some extreme cases, we see that the memoizer prevents up to 4,000 calls to data stores per request.
Garbage collection execution is expensive. So we write code that doesn’t generate unnecessary objects. Use of methods and algorithms that modify objects in place, instead of generating a new object. For example:
\n#map! instead of #map when dealing with lists. It prevents a new Array object from being created.This may not seem like much, but consider this: using #map! instead of #map could reduce your memory usage significantly, even when simply looping over an array of integers to double the values.
Let’s set up an following array of 1000 integers from 1 to 1000:
\narray = (1..1000).to_a
Then, let’s double each number in the array with Array#map:
\narray.map { |i| i * 2 }
The line above leads to one object allocated in memory, for a total of 8040 bytes.
\nNow let’s do the same thing with Array#map! instead:
array.map! { |i| i * 2 }
The line above leads to zero object allocated in memory, for a total of 0 bytes.
\nEven with this tiny example, using map! instead of map saves ~8 kilobytes of allocated memory, and considering the sheer scale of the Shopify platform and the storefront traffic throughput it receives, every little bit of memory optimization counts to help the garbage collector run less often and for smaller periods of time, thus improving server response times.
\nWith that in mind, we use tracing and profiling tools extensively to dive deeper into areas in the rendering engine that are consuming too much memory and to make precise changes to reduce memory usage.
\n\nPost-publication addendum: the Rubocop team offers a set of performance-oriented cops that highlight potential performance improvements - I highly recommend trying them out!
\n\nTo prevent accidentally increasing memory allocations, we built a test helper method that lets us benchmark a method or a block to know many memory allocations and allocated bytes it triggers. Here’s how we use it:
\nThis benchmark test will succeed if calling Product.find_by_handle('cowboy-hat') matches the following criteria:
We allow a range of allocations because they’re not deterministic on every test run. This depends on the order in which tests run and the way data is cached, which can affect the final number of allocations.
\nAs such, these memory benchmarks help us keep an eye on memory usage for specific methods. In practice, they’ve prevented introducing inefficient third-party gems that bloat memory usage, and they’ve increased awareness of memory usage to developers when working on features.
\nWe covered three main ways to improve server-side performance: batching up calls to external data stores to reduce roundtrips, caching data in multiple layers for specific use cases, and simplifying the amount of work required to fulfill a task by reducing memory allocations. When they’re all combined, these approaches lead to big time performance gains for merchants on the platform—the average response time with the new rendering engine is 5x faster than with the previous implementation.
\nThose are just some of the techniques that we are using to make the new application faster. And we never stop exploring new ways to speed up merchant’s storefronts. Faster rendering times are in the DNA of our team!
\n\nThis blog post is not available under a Creative Commons license.
\n", "url": "https://maximevaillancourt.com/blog/optimizing-server-side-storefront-rendering-shopify", "summary": "Simplify, batch, and cache.", "date_published": "2020-12-20 00:00:00 +0000" }, { "id": "/blog/monitoring-a-grain-dryer-via-the-internet", "title": "Monitoring a grain dryer via the Internet", "content_html": "A relative of mine called a few months ago: “Hey, I have a grain dryer that I’d like to monitor from my phone. Is that something you could set up in time for harvest season?”
\n\nI have no idea how grain dryers work, especially not Internet-enabled grain dryers — because of course that’s a thing — but I figured I’d give it a shot anyway.
\n\nFor context, a grain dryer is literally a combination of a huge furnace and fans that, you guessed it, dries harvested grain for optimal (lack of) moisture.
\n\n![\"A](\"/assets/dryer/gsi-1220.jpg\")
\nSuch dryers typically have an embedded computer running some version of Windows and a touchscreen display that allows monitoring and controlling the dryer’s temperature, timers, and other parameters to get the perfect output.
\n\n![\"A](\"/assets/dryer/controller.png\")
\nHowever, because these things run for hours at a time, and because agriculture workers during harvest season usually end up in a sleep-deprived state that lasts for multiple weeks, having the ability to monitor these grain dryers remotely lets workers get more precious sleep time (instead of having to walk/drive up to the grain dryer to see if everything’s okay in the middle of the night).
\n\nThe desired outcome is to monitor the grain dryer remotely using the web interface:
\n\n![\"An](\"/assets/dryer/web-ui.jpg\")
\nIt’s now harvest season 2020, so I connected a grain dryer to the Internet yesterday. It was fun (and weird) and I learned a few things. Here’s how we did it.
\n\nHere’s how the network topology looks like:
\n\n (via public static IP)\n +----------+\n +--------------> Internet <--------------+\n | +----------+ |\n +-------v--------+ +-------v-------+\n | Cellular modem | | Mobile device |\n +-------^--------+ +---------------+\n |\n+--------v----------+\n| Grain dryer |\n| (Watchdog module) |\n+-------------------+\nEssentially, we’ll expose the grain dryer’s web interface to the Internet via a public static IP using a cellular modem. Once it’s exposed, the operator will be able to navigate to that static IP from their mobile device to access the grain dryer’s web interface.
\n\nI called various cellular carriers here in Canada to learn more about their data-only plans and the possibility of adding a public static IP option to the plan: the only carrier that seemed like they knew what they were talking about was Telus. Others either didn’t offer the public static IP option, or didn’t know what I was talking about.
\n\nWith that, we went ahead and signed up for a business account at Telus (required to get access to the public static IP option), bought a SIM card, and a data-only plan for the modem.
\n\nFirst up, we need to make sure we’re able to connect to the nearest cellular tower to connect to the Internet. The Netgear LB1120 is fairly easy to set up on paper: pop a SIM card in there and we’re done, right? Well, for our use case, not quite. There are a few things to do to set it up to use the static IP assigned to the data plan from the cellular carrier. More importantly, the modem needs to be set up in “bridge” mode (instead of “router” mode) to act as a simple bridge (rather than a router) to connect it directly to the grain dryer.
\n\n![\"Laptop,](\"/assets/dryer/setup-modem.jpg\")
\nDays before arriving on the premises, I read online that the firmware that ships out of the box is broken: “bridge” mode doesn’t function at all, and an update is required to fix it. This made me a bit nervous because I didn’t have the modem with me, and wasn’t sure if I was going to be able to get the update to work properly. Thankfully, upon inserting the SIM card, turning the modem on, plugging it into the computer via an Ethernet cable, and accessing the modem’s setup interface at http://192.168.5.1, the modem’s web UI suggests downloading the latest firmware over the air. Neat!
From what I read, you must use a firmware more recent than NTG9X07C_12.09.05.27 to make sure “bridge” mode works fine. In our case, we updated to NTG9X07C_12.09.05.30, so we’re good.
![\"Updating](\"/assets/dryer/firmware-update.jpg\")
\nOnce the update completes and the modem reboots, we can go in the settings and change the operation mode to “bridge”:
\n\n![\"Changing](\"/assets/dryer/bridge-mode.png\")
\nDoing so “turns off the router function of the device and assigns the network IP address directly to the attached host”, which is exactly what we need to connect the modem directly to the dryer’s Watchdog module. Save and let the modem reboot.
\n\nNow, the SIM is in the modem, which has the latest firmware and is in “bridge” mode, but there’s one last thing to fix: the cellular carrier is assigning a dynamic IP to the modem, which isn’t what we want: we should be getting the static IP for the data plan we’re paying for. To resolve this issue, we need to tweak the modem’s access point name (APN).
\n\nI got stuck on this issue for around 30 minutes before remembering that APNs are a thing and that I could possibly tweak it in the modem settings. I’m glad I remembered!
\n\nBy default, the modem auto-detects the APN from the cellular carrier. For most cases, this works fine, but in this particular case, we want to use a different APN that allows us to use the static IP assigned to our data plan. APN settings vary by cellular carrier, and I ended up searching for Telus’ APN settings while on premise. I found USAT Corp’s website to include various APN settings, so I highly recommend trying those values out for your particular cellular carrier.
\n\nIn our specific case, we needed to use the following settings to be able to get the static IP assigned to the modem:
\n\nstaticipeast.telus.com (we’re on the east coast)![\"Configuring](\"/assets/dryer/apn-settings.png\")
\nThe LB1120 has a setting to automatically connect to the Internet upon booting: I recommend turning this on to have it connect automatically in the case of a power failure.
\n\nLet’s reboot the modem one more time. At this point, the modem should be using the static IP assigned to your data plan by the cellular carrier.
\n\n![\"Signal](\"/assets/dryer/2-bars.jpg\")
\nA tip to know if the modem is configured correctly: with the computer (device A) connected to the modem via Ethernet, use another device (device B) to access the expected static IP address (I used my smartphone). On device A, serve something (anything) on port 80 (I did sudo python -m SimpleHTTPServer 80), and see if you get that on device B when visiting the static IP address. If you do, then the modem is all set up and ready to go. If not, something’s wrong (invalid APN, incorrect data plan, no public static IP configured by the cellular carrier, some sort of NAT that prevents direct access, etc.).
Now that the modem is configured properly, let’s move on to configuring the grain dryer itself.
\n\nThe process to configure the dryer is fairly similar to configuring the modem: we essentially connect the dryer to the laptop via Ethernet to configure it via a web interface. But first, let’s look inside the dryer’s computer compartment.
\n\nIt all peels up like an onion: the outermost layer is the protective door to prevent water and debris from hitting switches and the touchscreen, and the middle layer is the computer and touchscreen itself along with a PLC. The Watchdog module is fixed inside the box itself.
\n\n![\"The](\"/assets/dryer/onion.jpg\")
\nWhen looking inside the box, the Watchdog module is fixed inside and connected to the rest of the PLC via an RS232 serial port. There are two Ethernet ports: one WAN port (the one on the left in the picture), and a LAN port (the one on the right).
\n\n![\"The](\"/assets/dryer/watchdog-module.jpg\")
\nTo configure the dryer’s network settings, we need to connect the Watchdog’s LAN port to the computer via Ethernet and access the web interface at http://10.0.0.1/setup. On this screen, you’ll need to click “Configure”, which will bring up the possibility to use DHCP or a static IP address. In our case, we select the “static IP” option.
![\"The](\"/assets/dryer/network-setup.png\")
\nOn the next screen, we fill out the following fields:
\n\nAssuming that the static IP address provided by the cellular carrier is 241.2.31.59, we’d fill out the fields as such:
241.2.31.59 (static IP from the cellular carrier)255.255.255.0 (pretty standard stuff)241.2.31.1 (same as static IP, but last component is 1)241.2.31.1 (we use the gateway as the DNS provider)8.8.8.8 (this is Google DNS, because why not)Once that’s done, save the network configuration, and turn off the grain dryer.
\n\nYou may now connect the modem to the Watchdog module’s WAN port (as pictured a few paragraphs above).
\n\nNow, the moment of truth: turn on the dryer. You may need to wait up to 3 minutes for the dryer and Watchdog to boot completely. A red LED should turn on near the WAN port once the Watchdog module is ready.
\n\nAt this point, try accessing the static IP from your favourite device. If you did everything right, you should see the Watchdog web interface, and you’re done! 🎉
\n\nQ: Why didn’t you just run a cable from the home to the dryer?
\n\nA: Because the dryer’s way too far from any building with Internet access. If only it had been so simple!
\n\nQ: Why pay for a static IP? Why not dynamic DNS or IPv6? Why not TeamViewer?
\n\nA: Telus (and others) do carrier-grade network address translation (CG-NAT), so there’s like a double layer of NAT going on. Using a public static IP bypasses this issue. What’s more, the modem itself doesn’t support dynamic DNS. Additionally, the dryer’s computer runs Windows CE .NET 4.2, so no IPv6 nor TeamViewer support. Heck, I even thought about installing ngrok on that thing, but Windows CE being what it is, I couldn’t.
\n\nLet’s wrap this up with a pair of nostalgia-inducing visuals: it appears that these dryers run on Windows CE .NET 4.2, which was released in April 2003. Time flies!
\n\n![\"Welcome](\"/assets/dryer/ui.jpg\")
![\"Windows](\"/assets/dryer/windows-ce.jpg\")
👉 Looking for actionables and screenshots? Scroll to the bottom of this post.
\n\nI opened my smartphone’s “screen time” app for the first time recently and was somewhat troubled to discover that on a particular day a few weeks ago:
\nMy first thoughts upon seeing these numbers: how much of this time could I consider as “time well spent”? How much of it could I redirect towards healthier activities?
\n\nLet’s open up an imaginary toolbox for a moment. Imagine a screwdriver.
\n\nA screwdriver is a tool in its purest form. It does not ask for anything, does not require maintenance, and doesn’t distract from the task at hand. It just sits there, patiently waiting to be picked up when needed.
\n\nA tool does its job, then gets out of the way.
\n\nThis is the kind of relationship I want with my smartphone. It should be tool, and nothing else. With that in mind, I set out to turn my smartphone into a tool again. This “tool, not distraction” mentality is useful for all kinds of modern technology, not just smartphones. Cal Newport’s “Digital Minimalism” covers this idea pretty well too (view my reading notes).
\n\nPicking up my smartphone happens in one of two mental modes: mindfully, or mindlessly. There are no other possible modes of operation when picking up a smartphone.
\n\nMindful use is the best case scenario. Ideally, every single time I’d pick up my smartphone, it would be to resolve a problem, and I’d put it down the second I’d be done. That’s entirely reasonable, though it’s not always what happens. What’s more, it’s not very realistic.
\n\nLet’s focus on improving the “mindless” part: I noticed that when I mindlessly pick up my phone, it’s either because a notification came in, or because I’m bored. Knowing this, I now have a few points of leverage I can use:
\nTo reduce incoming notifications:
\n\nTo reduce the number of mindless unlocks:
\n\nTo spend quality time on the device once it’s unlocked:
\n\nAll in all, the goal is to make it easier to use the device in a thoughtful, productive fashion than it is to use it for distracting purposes.
\n\nAfter these changes, screen time data for this week shows that on average I unlocked my device 11 times per day (~6x less than before), received 52 notifications per day (~3x less than before), and spent 70 minutes on the device per day (~3x less than before). ✨
\n\n![\"Lock](\"/assets/lockscreen.jpg\")
\n \n Lock screen\n \n ![\"Home](\"/assets/homescreen.jpg\")
\n \n Home screen\n \n ![\"App](\"/assets/app-drawer.jpg\")
\n \n App drawer\n \n A fair share of my waking hours involves communicating with other people on GitHub to make sure we’re solving the right problems, and that we’re solving them the right way.
\n\nAs a result, I receive many email notifications about various things that happen on there: direct requests to review a particular piece of code, feedback on pull requests I’ve opened, pull requests merged by their authors, people directly mentioning our username in a comment, issues closed by their authors, etc. I receive hundreds of emails every single week.
\n\nNow here’s the thing: some of these emails are more time-sensitive and/or actionable than others. We should probably address direct requests to review a particular piece of code first, since someone is explicitly asking for our attention and not responding promptly would likely prevent them from shipping something swiftly. Conversely, I’ll want to delete (or at least deprioritize) email notifications communicating that a given pull request was merged by its author - there’s no actionable to derive from it, so in the bin it goes.
\n\nHowever, by default, all these email notifications arrive in our inboxes with the same perceived level of importance, which makes it difficult to identify what I should address next.
\n\nThis post presents a solution to this problem: using Gmail filters, we can automatically add labels to GitHub notification emails based on their content. This solution takes less than 10 minutes to implement, and the long-term return on investment is quite appreciable.
\n\nHere’s how my inbox looks like with automatic labeling set up:
\n\n![\"\"](\"/assets/github-labels.png\")
Pretty neat, right? Thanks to these labels, I’m able to quickly parse through emails and reach inbox zero every day.
\n\nLet’s see how to implement this solution with Gmail.
\n\nStart by saving the following XML filters template to a file on your device:
\n\n<?xml version='1.0' encoding='UTF-8'?>\n<feed xmlns='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'>\n <title>GitHub filters</title>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='Merged into'/>\n <apps:property name='label' value='Merged'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='"@<YOUR_GITHUB_USERNAME_HERE>"'/>\n <apps:property name='label' value='Mention'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='because you authored the thread'/>\n <apps:property name='label' value='Author'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='modified the open/close state'/>\n <apps:property name='label' value='Reopened'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='"requested review from @<YOUR_GITHUB_TEAM_NAME_HERE>"'/>\n <apps:property name='label' value='Team review request'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='"because you are on a team that was mentioned" AND "You can view, comment on, or merge this pull request online at"'/>\n <apps:property name='label' value='Team review request'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='from' value='github'/>\n <apps:property name='hasTheWord' value='"Closed \\#" "You are receiving this because"'/>\n <apps:property name='doesNotHaveTheWord' value='dependabot'/>\n <apps:property name='label' value='Closed'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n <entry>\n <category term='filter'></category>\n <content></content>\n <apps:property name='hasTheWord' value='"requested your review on"'/>\n <apps:property name='label' value='Direct review request'/>\n <apps:property name='sizeOperator' value='s_sl'/>\n <apps:property name='sizeUnit' value='s_smb'/>\n </entry>\n</feed>\nIn the template, replace the following string:
\n\n@<YOUR_GITHUB_USERNAME_HERE> (in my case that would be @maximevaillancourt)If you’re part of a GitHub team that other people mention in issues and pull requests, also replace the following string:
\n\n@<YOUR_GITHUB_TEAM_NAME_HERE> (for example, @my-organization/best-team-ever)If you’re not part of a GitHub team, remove the <entry> node that looks for this pattern.
Finally, save the XML file.
\n\nNow that your template is ready, it’s time to import it in your Gmail account.
\n\nOpen up your Gmail settings, and click on the “Filters” tab. Once you’re there, find the “Import filters” link near the bottom of the pane. Select the XML file you just modified and upload it. Review the filters and and confirm the import.
\n\nThat’s it - you’re done. At this point, your emails should be labeled automatically. Feel free to change the colors of the labels via the “Labels” tab in your Gmail settings. I find it helpful to make direct review requests red as this means “urgent and important” to me, and the label for merged pull requests is green because it means “success, everything’s good” to me.
\n\nExperiment to find what works best for you.
\n", "url": "https://maximevaillancourt.com/blog/github-email-notifications-gmail-filters", "summary": "Cut through the noise and identify what’s important.", "date_published": "2020-10-15 00:00:00 +0000" }, { "id": "/blog/shopify-storefront-renderer", "title": "How Shopify reduced storefront response times with a rewrite", "content_html": "This post originally appeared on the Shopify Engineering blog on August 20, 2020.
\n\n![](\"https://cdn.shopify.com/s/files/1/0779/4361/articles/sewing-digital-product_1000x400_crop_top.jpg\")
In January 2019, we set out to rewrite the critical software that powers all online storefronts on Shopify’s platform to offer the fastest online shopping experience possible, entirely from scratch and without downtime.
\nThe Storefront Renderer is a server-side application that loads a Shopify merchant's storefront Liquid theme, along with the data required to serve the request (for example product data, collection data, inventory information, and images), and returns the HTML response back to your browser. Shaving milliseconds off response time leads to big results for merchants on the platform as buyers increasingly expect pages to load quickly, and failing to deliver on performance can hinder sales, not to mention other important signals like SEO.
\nThe previous storefront implementation‘s development, started over 15 years ago when Tobi launched Snowdevil, lived within Shopify’s Ruby on Rails monolith. Over the years, we realized that the “storefront” part of Shopify is quite different from the other parts of the monolith: it has much stricter performance requirements and can accept more complexity implementation-wise to improve performance, whereas other components (such as payment processing) need to favour correctness and readability.
\nIn addition to this difference in paradigm, storefront requests progressively became slower to compute as we saw more storefront traffic on the platform. This performance decline led to a direct impact on our merchant storefronts’ performance, where time-to-first-byte metrics from Shopify servers slowly crept up as time went on.
\nHere’s how the previous architecture looked:
\n![\"Old](\"//cdn.shopify.com/s/files/1/0779/4361/files/OldArchitecture.jpg?v=1597931630\")
Old Storefront Implementation
Before, the Rails monolith handled almost all kinds of traffic: checkout, admin, APIs, and storefront.
\nWith the new implementation, traffic routing looks like this:
\n![\"New](\"//cdn.shopify.com/s/files/1/0779/4361/files/NewArchitecture.jpg?v=1597931738\")
New Storefront Implementation
The Rails monolith still handles checkout, admin, and API traffic, but storefront traffic is handled by the new implementation.
\nDesigning the new storefront implementation from the ground up allowed us to think about the guarantees we could provide: we took the opportunity of this evergreen project to set us up on strong primitives that can be extended in the future, which would have been much more difficult to retrofit in the legacy implementation. An example of these foundations is the decision to design the new implementation on top of an active-active replication setup. As a result, the new implementation always reads from dedicated read replicas, improving performance and reducing load on the primary writers.
\nSimilarly, by rebuilding and extracting the storefront-related code in a dedicated application, we took the opportunity to think about building the best developer experience possible: great debugging tools, simple onboarding setup, welcoming documentation, and so on.
\nFinally, with improving performance as a priority, we work to increase resilience and capacity in high load scenarios (think flash sales: events where a large number of buyers suddenly start shopping on a specific online storefront), and invest in the future of storefront development at Shopify. The end result is a fast, resilient, single-purpose application that serves high-throughput online storefront traffic for merchants on the Shopify platform as quickly as possible.
\nOnce we clearly outlined the problem we’re trying to solve and scoped out the project, we defined three main success criteria:
\nBefore building the new implementation, we needed a way to make sure that whatever we built would behave the same way as the existing implementation. So, we built a verifier mechanism that compares the output of both implementations and returns a positive or negative result depending on the outcome of the comparison.
\nThis verification mechanism runs on storefront traffic in production, and it keeps track of verification results so we can identify differences in output that need fixing. Running the verifier mechanism on production traffic (in addition to comparing the implementations locally through a formal specification and a test suite) lets us identify the most impactful areas to work on when fixing issues, and keeps us focused on the prize: reaching feature parity as quickly as possible. It’s desirable for multiple reasons:
\nThere are two parts to the entire verifier mechanism implementation:
\nThe following diagram shows how each part interacts with the rest of the architecture:
\n![\"Legacy](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_1A.jpg?v=1597932504\")
Legacy implementation and new implementation at the same conceptual layer
The legacy implementation (the Rails monolith) still exists, and the new implementation (including the Verifier service) is introduced at the same conceptual layer. Both implementations are placed behind a custom routing module that decides where to route traffic based on the request attributes and the verification data for this request type. Let’s look at an example.
\nWhen a buyer’s device sends an initial request for a given storefront page (for example, a product page from shop XYZ), the request is sent to Shopify’s infrastructure, at which point an nginx instance handles it. The routing module considers the request attributes to determine if other shop XYZ product page requests have previously passed verification.
\n![\"First](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_1.jpg?v=1597932239\")
First request routed to Legacy implementation
Since this is the first request of this kind in our example, the routing module sends the request to the legacy implementation to get a baseline reference that it will use for subsequent shop XYZ product page requests.
\n![\"Routing](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_2.jpg?v=1597932398\")
Routing module sends original request and legacy implementation’s response to the new implementation
Once the response comes back from the legacy implementation, the Lua routing module sends that response to the buyer. In the background, the Lua routing module also sends both the original request and the legacy implementation’s response to the new implementation. The new implementation computes a response to the original request and feeds both its response and the forwarded legacy implementation’s response to the verifier service. This is done asynchronously to make sure we’re not adding latency to responses we send to buyers, who don’t notice anything different.
\nAt this point, the verifier service received the responses from both the legacy and new implementations and is ready to compare them. Of course, the legacy implementation is assumed to be correct as it’s been running in production for years now (it acts as our reference point). We keep track of differences between the two implementations’ responses so we can debug and fix them later. The verifier service looks at both responses’ status code, headers, and body, ensuring they’re equivalent. This lets us identify any differences in the responses so we make sure our new implementation behaves like the legacy one.
\nTime-related and randomness-related exceptions make it impossible to have exactly byte-equal responses, so we ignore certain patterns in the verifier service to relax the equivalence criteria. The verifier service uses a fixed time value during the comparison process and sets any random values to a known value so we reliably compare the outputs containing time-based and randomness-based differences.
\n![\"The](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_3.jpg?v=1597933710\")
The verifier service sends comparison result back to the Lua module
The verifier service sends the outcome of the comparison back to the Lua module, which keeps track of that comparison outcome for subsequent requests of the same kind.
\nOnce we had verified our new approach, we tested rendering a page using the new implementation instead of the legacy one. We iterated upon our verification mechanism to allow us to route traffic to the new implementation after a given number of successful verifications. Here’s how it works.
\nJust like when we only verified traffic, a request arrives from a client device and hits Shopify’s architecture. The request is sent to both implementations, and both outputs are forwarded to the verifier service for comparison. The comparison result is sent back to the Lua routing module, which keeps track of it for future requests.
\nWhen a subsequent storefront request arrives from a buyer and reaches the Lua routing module, it decides where to send it based on the previous verification results for requests similar to the current one (based on the request attributes
\n![\"For](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_4.jpg?v=1597933966\")
For subsequent storefront requests, the Lua routing module decides where to send it
If the request was verified multiple times in the past, and nearly all outcomes from the verifier service were “Pass”, then we consider the request safe to be served by the new implementation.
\n![\"If](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_5.jpg?v=1597934257\")
If most verifier service results are “Pass”, then it uses the new implementation
If, on the other hand, some verifications failed for this kind of request, we’ll play it safe and send the request to the legacy implementation.
\n![\"If](\"//cdn.shopify.com/s/files/1/0779/4361/files/VerifierMechansim_6.jpg?v=1597934545\")
If most verifier service results are “Fail”, then it uses the old implementation
With the verifier mechanism and the dynamic router in place, our first goal was to render one of the simplest storefront pages that exists on the Shopify platform: the password page that protects a storefront before the merchant makes it available to the public.
\nOnce we reached full parity for a single shop’s password page, we tested our implementation in production (for the first time) by routing traffic for this password page to the new implementation for a couple of minutes to test it out.
\nSuccess! The new implementation worked in production. It was time to start implementing everything else.
\nAfter our success with the password page, we tackled the most frequently accessed storefront pages on the platform (product pages, collection pages, etc). Diff by diff, endpoint by endpoint, we slowly increased the parity rate between the legacy and new implementations.
\nHaving both implementations running at the same time gave us a safety net to work with so that if we introduced a regression, requests would easily be routed to the legacy implementation instead. Conversely, whenever we shipped a change to the new implementation that would fix a gap in feature parity, the verifier service starts to report verification successes, and our custom routing module in nginx automatically starts sending traffic to the new implementation after a predetermined time threshold.
\nWe collected Apdex (Application Performance Index) scores on server-side processing time for both the new and legacy implementations to compare them.
\nTo calculate Apdex scores, we defined a parameter for a satisfactory threshold response time (this is the Apdex’s “T” parameter). Our threshold response time to define a frustrating experience would then be “above 4T” (defined by Apdex).
\nWe defined our “T” parameter as 200ms, which lines up with Google’s PageSpeed Insights recommendation for server response times. We consider server processing time below 200ms as satisfying and a server processing time of 800ms or more as frustrating. Anything in between is tolerated.
\nFrom there, calculating the Apdex score for a given implementation consists of setting a time frame, and counting three values:
\nThen, we calculate the Apdex score:
\n\n\napdexScore = (s + t/2) / n\nBy calculating Apdex scores for both the legacy and new implementations using the same t parameter, we had common ground to compare their performance.
\nWe want all Shopify storefronts to be fast, and this new implementation aims to speed up what a performance-conscious theme developer can’t by optimizing data access patterns, reducing memory allocations, and implementing efficient caching layers.
\nThe new implementation uses optimized, handcrafted SQL multi-select statements maximizing the amount of data transferred in a single round trip. We carefully vet what we eager-load depending on the type of request and we optimize towards reducing instances of N+1 queries.
\nWe reduce the number of memory allocations as much as possible so Ruby spends less time in garbage collection. We use methods that apply modifications in place (such as #map!) rather than those that allocate more memory space (like #map). This kind of performance-oriented Ruby paradigm sometimes leads to code that’s not as simple as idiomatic Ruby, but paired with proper testing and verification, this tradeoff provides big performance gains. It may not seem like much, but those memory allocations add up quickly, and considering the amount of storefront traffic Shopify handles, every optimization counts.
\nWe implemented various layers of caching throughout the application to reduce expensive calls. Frequent database queries are partitioned and cached to optimize for subsequent reads in a key-value store, and in the case of extremely frequent queries, those are cached directly in application memory to reduce I/O latency. Finally, the results of full page renders are cached too, so we can simply serve a full HTTP response directly from cache if possible.
\nOnce we could measure the performance of both implementations and reach a high enough level of verified feature parity, we started migrating merchant shops. Here are some of the improvements we’re seeing with our new implementation:
\nSo another success! We improved store performance in production.
\nNow how do we make sure this translates to our third success criteria?
\nWhile working on the new implementation, the Verifier service identified potential parity gaps, which helped tremendously. However, a few times we shipped code to production that broke in exceedingly rare edge cases that it couldn’t catch.
\nAs a safety mechanism, we made it so that whenever the new implementation would fail to successfully render a given request, we’d fall back to the legacy implementation. The response would be slower, but at least it was working properly. We used circuit breakers in our custom nginx routing module so that we’d open the circuit and start sending traffic to the legacy implementation if the new implementation was having trouble responding successfully. Read more on tuning circuit breakers in this blog post by my teammate Damian Polan.
\nTo ensure that the new implementation responds well to flash sales, we implemented and tweaked two mechanisms. The first one is an automatic scaling mechanism that adds or remove computing capacity in response to the amount of load on the current swarm of computers that serve traffic. If load increases as a result of an increase in traffic, the autoscaler will detect this increase and start provisioning more compute capacity to handle it.
\nAdditionally, we introduced in-memory cache to reduce load on external data stores for storefronts that put a lot of pressure on the platform’s resources. This provides a buffer that reduces load on very-high traffic shops.
\nWhen an external data store isn’t available, we don’t want to serve buyers an error page. If possible, we’ll try to gracefully fall back to a safe way to serve the request. It may not be as fast, or as complete as a normal, healthy response, but it’s definitely better than serving a sad error page.
\nWe implemented circuit breakers on external datastores using Semian, a Shopify-developed Ruby gem that controls access to slow or unresponsive external services, avoiding cascading failures and making the new implementation more resilient to failure.
\nSimilarly, if a cache store isn’t available, we’ll quickly consider the timeout as a cache miss, so instead of failing the entire request because the cache store wasn’t available, we’ll simply fetch the data from the canonical data store instead. It may take longer, but at least there’s a successful response to serve back to the buyer.
\nFinally, as a way to identify potential resilience issues, the new implementation uses Toxiproxy to generate test cases where various resources are made available or not, on demand, to generate problematic scenarios.
\nAs we put these resilience and capacity mechanisms in place, we regularly ran load tests using internal tooling to see how the new implementation behaves in the face of a large amount of traffic. As time went on, we increased the new implementation’s resilience and capacity significantly, removing errors and exceptions almost completely even in high-load scenarios. With BFCM 2020 coming soon (which we consider as an organic, large-scale load test), we’re excited to see how the new implementation behaves.
\nWe’re currently in the process of rolling out the new implementation to all online storefronts on the platform. This process happens automatically, without the need for any intervention from Shopify merchants. While we do this, we’re adding more features to the new implementation to bring it to full parity with the legacy implementation. The new implementation is currently at 90%+ feature parity with the legacy one, and we’re increasing that figure every day with the goal of reaching 100% parity to retire the legacy implementation.
\nAs we roll out the new implementation to storefronts we are continuing to see and measure performance improvements as well. On average, server response times for the new implementation are 4x faster than the legacy implementation. Rhone Apparel, a Shopify Plus merchant, started using the new implementation in April 2020 and saw dramatic improvements in server-side performance over the previous month.
\nWe learned a lot during the process of rewriting this critical piece of software. The strong foundations of this new implementation make it possible to deploy it around the world, closer to buyers everywhere, to reduce network latency involved in cross-continental networking, and we continue to explore ways to make it even faster while providing the best developer experience possible to set us up for the future.
\n\nThis blog post is not available under a Creative Commons license.
\n", "url": "https://maximevaillancourt.com/blog/shopify-storefront-renderer", "summary": "Making server response times 4x faster than before.", "date_published": "2020-09-03 00:00:00 +0000" }, { "id": "/blog/debugging-ruby-casecmp-bug-using-gdb", "title": "Two Ruby apps, same code, different output", "content_html": "I noticed something odd today while working on two different Ruby codebases. This simple line of Ruby behaved differently in both applications:
\n\n\"luck\".casecmp(\"L`auguste\")\nExecuting \"luck\".casecmp(\"L`auguste\") in application A returned -1, while executing it in application B returned 1.
“Did the alphabet change at some point and I didn’t get the memo?”, I thought.
\n\n\n\n\nAside
\n\n\n\n
String#casecmpis a built-in Ruby method that returns-1,0,1, ornildepending on whether the object on which it’s called is less than, equal to, or greater than the function argument, and it does so in case-insensitive fashion. Here are a few simple examples of how it behaves:\n\n\"aBcDeF\".casecmp(\"abcde\") #=> 1\n\"aBcDeF\".casecmp(\"abcdef\") #=> 0\n\"aBcDeF\".casecmp(\"abcdefg\") #=> -1\n\"abcdef\".casecmp(\"ABCDEF\") #=> 0\n
Seeing as one of the applications is built on top of Ruby of Rails and the other isn’t, my first thought was that maybe there was a Rails and/or ActiveSupport patch on String#casecmp that would change the behavior of this line in one of the applications. However, I didn’t find anything that pointed to this. I kept digging, hoping to maybe find a patch in the other application that could explain this difference in behavior. Again, I didn’t find anything. 🙈
Eventually, after exploring a bit more, I realized that both applications ran on different versions of Ruby: application A was on Ruby 2.6, while application B was using Ruby 2.7.
\n\nRunning the same command on both versions of Ruby indeed gives us different results:
\n\n$ ~/.rubies/ruby-2.6.6/bin/ruby -e 'puts \"luck\".casecmp(\"L`Auguste\")'\n-1\n\n$ ~/.rubies/ruby-2.7.0/bin/ruby -e 'puts \"luck\".casecmp(\"L`Auguste\")'\n1\nAh ha! We’re getting closer. While I could have called it a day here and simply updated application B to Ruby 2.7 to resolve the issue, I wanted to understand: what causes it?
\n\nbinding.pry\nI then started to comb through Ruby changelogs, trying to find if anything changed between Ruby 2.6 and Ruby 2.7 for String#casecmp, or anything somehow related to string comparison. I didn’t find anything.
Of course, it would be nice to debug this using binding.pry or other similar Ruby-level debugging tools by stepping into the String#casecmp call to see what’s going on inside. However, this doesn’t get us very far, as trying to use Ruby’s Tracer or binding.pry doesn’t really help.
Running this:
\n\n$ ruby -r tracer -e '\"luck\".casecmp(\"L`Auguste\")'\nreturns this output:
\n\n#0:-e:1::-: \"luck\".casecmp(\"L`Auguste\")\nand not much else. That’s because String#casecmp is implemented in C, directly inside MRI’s string.c, so there’s no actual Ruby code underneath String#casecmp that we can step into using Ruby-level debugging tools.
Here comes the GDB part: because we’re essentially dealing with C code at this point, we can use GDB to understand what happens inside the call to String#casecmp. So with that, I fired up GDB for the first time in years (I typically work with Ruby, so GDB is not something I commonly use).
Let’s see how to use GDB to understand why both Ruby 2.6 and Ruby 2.7 behave differently with the same input to String#casecmp.
I first prepared a simple Ruby file containing the source that replicates the issue:
\n\n# ~/casecmp.rb\nputs \"luck\".casecmp(\"L`Auguste\")\nNotice that the second character in the input to casecmp is a backtick (`), which has ASCII code 96. This is relevant for paragraphs below.
Let’s start by firing up GDB with a self-compiled version of Ruby 2.7.0:
\n\n$ sudo gdb /Users/maximevaillancourt/.rubies/ruby-2.7.0/bin/ruby\nReading symbols from /Users/maximevaillancourt/.rubies/ruby-2.7.0/bin/ruby...\nThen, we add a breakpoint on the str_casecmp function so execution pauses once we reach it:
(gdb) break str_casecmp\nBreakpoint 1 at 0x1001fa766: file string.c, line 3371.\nPerfect. We’re now ready to run the casecmp.rb Ruby script from above.
(gdb) run casecmp.rb\nStarting program: /Users/maximevaillancourt/.rubies/ruby-2.7.0/bin/ruby casecmp.rb\nWe eventually hit the breakpoint we just set:
\n\nThread 2 hit Breakpoint 1, str_casecmp (str1=4329352680, str2=4329352640) at string.c:3371\n3371\t enc = rb_enc_compatible(str1, str2);\n\n\n\nAside
\n\nInternally,
\nString#str_casecmpis quite simple: it iterates over each character in both inputs by index starting from the first character, converting both characters to the same case so that the function behaves in a case-insensitive way, and returns early if the two currently considered characters from each input are different. In doing so, it determines which character is “bigger” than the other using the character code (an ASCII code table is a useful asset to have nearby for the rest of this blog post).
In Ruby 2.7.0, notice that the case conversion converts both inputs to lowercase using TOLOWER:
while (p1 < p1end && p2 < p2end) {\n if (*p1 != *p2) {\n unsigned int c1 = TOLOWER(*p1 & 0xff);\n unsigned int c2 = TOLOWER(*p2 & 0xff);\n if (c1 != c2)\n return INT2FIX(c1 < c2 ? -1 : 1);\n }\n p1++;\n p2++;\n}\nAfter navigating in str_casecmp using next a few times, we enter the loop and arrive at a point where we can print c1 and c2, which are the codes for the characters at the current index for both inputs:
3382\t unsigned int c2 = TOLOWER(*p2 & 0xff);\n(gdb) print c1\n$11 = 108\n(gdb) next\n3383\t if (c1 != c2)\n(gdb) print c2\n$12 = 108\nHere’s a visual representation of the buffers:
\n\nc1\n ↓\n108 ? ? ?\n l u c k\n\n108 ? ? ? ? ? ? ? ?\n l ` a u g u s t e\n ↑\nc2\n108 is the decimal ASCII character code representation for the first letter of both inputs: l (lowercase “L”), so the loop continues to the next iteration because c1 and c2 are the same.
On the second iteration of the loop (on the second character of both inputs), we get the following results:
\n\n3382\t unsigned int c2 = TOLOWER(*p2 & 0xff);\n(gdb) print c1\n$14 = 117\n(gdb) next\n3383\t if (c1 != c2)\n(gdb) print c2\n$16 = 96\nHere’s a visual representation of the buffers:
\n\n c1\n ↓\n108 117 ? ?\n l u c k\n\n108 96 ? ? ? ? ? ? ?\n l ` a u g u s t e\n ↑\n c2\nc1 contains 117, which is the decimal ASCII character code representation for u, while 96 (in c2) is the character code for a backtick (`). We then enter the if (c1 != c2) conditional, and the return value is 1 because c1 > c2 (117 > 96).
Okay. So far so good. This lines up with the initial observation of the issue. How are things different in Ruby 2.6.6?
\n\nWe do almost the same setup as above (same one-line Ruby script to replicate the issue, same breakpoint on str_casecmp), but we fire up GDB with Ruby 2.6.6:
$ sudo gdb /Users/maximevaillancourt/.rubies/ruby-2.6.6/bin/ruby\nReading symbols from /Users/maximevaillancourt/.rubies/ruby-2.6.6/bin/ruby...\n\n(gdb) break str_casecmp\n...\n\n(gdb) run casecmp.rb\nStarting program: /Users/maximevaillancourt/.rubies/ruby-2.6.6/bin/ruby casecmp.rb\n\nThread 2 hit Breakpoint 1, str_casecmp ...\nLet’s look at the loop we presented above in Ruby 2.7.0, but in Ruby 2.6.6 this time:
\n\nwhile (p1 < p1end && p2 < p2end) {\n if (*p1 != *p2) {\n unsigned int c1 = TOUPPER(*p1 & 0xff);\n unsigned int c2 = TOUPPER(*p2 & 0xff);\n if (c1 != c2)\n return INT2FIX(c1 < c2 ? -1 : 1);\n }\n p1++;\n p2++;\n}\nNotice that instead of using TOLOWER as in Ruby 2.7.0, Ruby 2.6.6 uses TOUPPER. Interesting.
Let’s fast-forward to the part where we get to c1 and c2 for the second character in the input:
3414\t\t\tunsigned int c2 = TOUPPER(*p2 & 0xff);\n(gdb) next\n3415\t if (c1 != c2)\n(gdb) print c1\n$5 = 85\n(gdb) print c2\n$6 = 96\nHere’s a visual representation of the buffers:
\n\n c1\n ↓\n108 85 ? ?\n L U C K\n\n108 96 ? ? ? ? ? ? ?\n L ` A U G U S T E\n ↑\n c2\nc1 is 85, which is the character code for U, and c2 is 96 (just like in Ruby 2.7.0), which is the character code for a backtick (`).
This time though, the comparison result is different, because c1 < c2 (85 < 96), so str_casecmp returns -1.
There it is: because Ruby 2.6 uses TOUPPER and Ruby 2.7 uses TOLOWER before comparing the inputs, and because one of the characters to compare is a backtick (`, which can’t be converted to uppercase or lowercase in any way), the other character’s code “moves” differently around the “fixed” backtick character code, affecting the result of the String#casecmp function.
To summarize, the root cause of the issue is that String#casecmp was updated in Ruby 2.7 to lowercase the two inputs before comparing them, while Ruby 2.6 used to uppercase the two inputs before comparing them. This is the commit where this change was introduced.
Fun debugging session. :)
\n", "url": "https://maximevaillancourt.com/blog/debugging-ruby-casecmp-bug-using-gdb", "summary": "Debugging a weird Ruby string sorting issue with GDB.", "date_published": "2020-08-13 00:00:00 +0000" }, { "id": "/blog/code-walkthroughs", "title": "Accelerating software onboarding with code walkthroughs", "content_html": "By the end of this article, you’ll have a new tool in your toolbox to help newcomers quickly discover how your software project works, while reducing the time you explicitly spend introducing new folks to your project.
\n\nRamping up on a new software project is hard, both for the person ramping up and for the team that supports this person’s onboarding process. It’s especially worse when multiple people start onboarding the same project at the same time, which happened to my team recently: our project’s area of influence has grown quickly recently, and as a result, we’re seeing dozens of developers starting to contribute to the project.
\n\nThis sudden spike in interest lead to what I call “onboarding load” for our core team, as we suddenly faced pressure to help newcomers learn and navigate the codebase, while simulteanously trying to keep up with our day-to-day tasks.
\n\nTo reduce this pressure on our core team, I recently introduced a “code walkthrough” in the project’s codebase. It’s essentially a collection of high-quality, high-context code comments “tagged” with a [docs/walkthrough] line comment. When newcomers come our way and want to learn more about our project, I point them to a GitHub search for that special identifier in the project’s codebase to pull up all those great code comments to learn about the project. (They could also clone the code locally and search for that identifier using their code editor, but a GitHub search is available to everyone.)
Here are some examples of such high-context comments and the different purposes they serve:
\n\nNavigation and flow
\n\n![](\"https://images.unsplash.com/photo-1573166613605-3b4dfcbf1268?ixlib=rb-1.2.1&auto=format&fit=crop&w=1498&h=500&q=80\")
# [docs/walkthrough]\n# This module is responsible for generating the content to insert in the <head>\n# HTML tag across all pages.\n# [docs/walkthrough]\n# This class is the entrypoint to all output generation. From here, content\n# flows down into controllers, then into serializers and formatters.\nPerformance
\n\n# [docs/walkthrough]\n# This is an example of an implementation that doesn't feel like it's the\n# right way to do it, but after looking at profiles and traces, we know\n# that this implementation performs better than a SQL-only solution.\n#\n# There's currently no good index on the `xyz` table to quickly search\n# for nested values, so it ends up being faster to filter through all\n# values in Ruby than it is to filter those out using a `WHERE`\n# clause at the database level. We're looking into remodeling this data.\nTrivia
\n\n# [docs/walkthrough]\n# Even though this class usually handles objects of type X, this one\n# is an exception because it can be overridden by people manually creating\n# resource of the same name.\nResiliency
\n\n# [docs/walkthrough]\n# Historically, we kept track of the resource count by incrementing a value in a\n# key-value store whenever a resource was created. However, because of scaling\n# constraints, we started tracking this value using a proper data ETL process.\n# We now simply read the count from that data store and cache it aggressively.\nImplementation
\n\n# [docs/walkthrough]\n# By default, this class (including its subclasses) doesn't expose any method\n# to the outside world. To do so, methods must be marked with the \"world_public\" \n# identifier. This is a safe-by-default way to avoid allocating wrapper objects,\n# while still allowing public methods to be created for internal use.\nThe benefit of this approach is that the documentation is built into the code, so the risk that it becomes stale or outdated is much lower than if that documentation lived outside of the code itself.
\n\nOne thing you can play around with is moving up and down levels of abstraction throughout the comments. Some of them may explain why a specific function is implemented this way, while others may explain the business domain history behind this legacy module.
\n\nAs a starting point, identify existing high-quality comments in your project, and tag them with the special identifier of your choice. Then, look for areas in the codebase that would benefit from having a little bit more context and documentation, and take some time to write clear, helpful comments that you can add that special tag to. Finally, once you have a generous collection of high-quality comments with that special identifier, start pointing newcomers to search for this identifier, and let them soak up all that context.
\n\nAs the documentation shepherd on your team, always be on the lookout for opportunities to write more of those high-context code comments: you’ll be rewarded for it in the future, either by your colleagues thanking you for writing those comments that helped them step up quickly, or by your boss for reducing the “onboarding load” I mentioned earlier on the rest of the team, who instead were able to focus on the task at hand.
\n\nGreat software documentation is like a flywheel. It’s hard to write at first, and the return on investment is not immediately visible. But with time, it compounds, and eventually, you’ll wonder how you managed to live without it.
\n", "url": "https://maximevaillancourt.com/blog/code-walkthroughs", "summary": "Welcoming newcomers with high-quality, high-context code comments.", "date_published": "2020-07-28 00:00:00 +0000" }, { "id": "/blog/setting-up-your-own-digital-garden-with-jekyll", "title": "Setting up your own digital garden with Jekyll", "content_html": "Eager to try the demo of the template? 👉 digital-garden-jekyll-template.netlify.app
\n\nDigital gardens and public note-taking spaces are all the rage these days, as they’re a great way to foster an environment where ideas mesh together and others can take inspiration from. You can set up a digital garden of your own in a few minutes, and have your own personal corner of the Internet where you’ll seed and grow ideas.
\n\nIf you’re familiar with Markdown and/or HTML, you’ll be right at home, as that’s how your notes will be formatted.
\n\nThe end result will look similar to this:
\n\n![\"\"](\"https://user-images.githubusercontent.com/8457808/82400515-7d026d80-9a25-11ea-83f1-3b9cb8347e07.png\")
If you happen to use Obsidian to work on your notes, you’ll likely want to open this guide from Mike and read through it in parallel with this one.
\n\nAlternatively, if you use Roam and would like to automatically convert your Roam Research backup to a garden using this template, take a look at this repository.
\n\nWithout further ado, let’s get started!
\n\nFor this tutorial, we’ll need to install a few things on your machine (you may have some of these already). Following the instructions on each website to install them.
\n\n\n\nYou’ll also need to create accounts on the following services:
\n\nOnce everything is set up, let’s start creating your own digital garden.
\n\nTo simplify things, I provide the template shown in the image above to get started. You can always tweak this template to your taste later.
\n\nVisit the GitHub page for my template repository (maximevaillancourt/digital-garden-jekyll-template), and fork it to your account using the Fork button:
![\"\"](\"https://help.github.com/assets/images/help/repository/fork_button.jpg\")
Once the forking process is complete, you should have a fork (essentially a copy) of my template in your own GitHub account. On the GitHub page for your repository, click on the green “Clone or download” button, and copy the URL: we’ll need it for the next step.
\n\nNext, we want to download the files from your GitHub repository onto your local machine. To do this, replace <YOUR_COPIED_URL_HERE> in the command below with the URL you copied in the previous step, then execute this command:
$ git clone <YOUR_COPIED_URL_HERE> my-digital-garden\nAs a reference point, this is how it looks like for me (the difference is likely just the GitHub username):
\n\n$ git clone git@github.com:maximevaillancourt/digital-garden-jekyll-template.git my-digital-garden\nThen, navigate into the directory that was just created:
\n\n$ cd my-digital-garden\nSweet! You now have your repository’s source code on your machine. Within the my-digital-garden directory, run the following command to install the necessary dependencies like Jekyll:
$ bundle\nOnce that’s done, ask Jekyll to start serving the site locally:
\n\n$ bundle exec jekyll serve\nThen, open up http://localhost:4000 in your browser.
If everything’s done correctly, you should now see the home page of your digital garden. 🎉
\n\nKeep in mind that this site is only available locally (notice the localhost part of the URL), so if we want it to be available on the Internet for everyone to enjoy, we need to deploy it to the Internet: we’ll use Netlify for that in the next step.
Netlify lets you automatically deploy your digital garden on to the Internet when you update your GitHub repository. To do this, we need to connect your GitHub repository to Netlify:
\n\nThat was easy! We’re almost done.
\n\nWait a couple of minutes for the initial deploy to complete.
\n\nOnce that’s done, your digital garden should be available on the Internet via a generic Netlify URL, which you can change to a custom domain later if you’d like.
\n\nNow the cool thing is this: whenever you push an update to your GitHub repository, Netlify will automatically deploy your updates to the Internet.
\n\nAt this point, you can start updating the files on your machine (in the my-digital-garden folder) to change your digital garden to your liking: update the copy, add some notes, tweak the layout, customize the colors, etc.
Once you have something you’re happy with, push your changes to your GitHub repository with the following commands:
\n\n$ git add --all\n$ git commit -m 'Update content'\n$ git push origin master\nIf that command succeeds and the rest of the tutorial was done correctly, in a couple of minutes, you should see your changes live on your Netlify website. 🚀
\n\nAnd we’re done! You now have your own digital garden. Take care of your mind and the rest will follow. 🍃
\n", "url": "https://maximevaillancourt.com/blog/setting-up-your-own-digital-garden-with-jekyll", "summary": "Carve out your own space where you’ll seed, cross-pollinate, and grow ideas.", "date_published": "2020-05-20 00:00:00 +0000" }, { "id": "/blog/why-i-use-a-thinkpad-x220-in-2021", "title": "Why I still use a ThinkPad X220 in 2021", "content_html": "Update (2020-05-18): I’ve since switched to a more powerful desktop computer for my photography business. I still use the X220 as a dedicated machine to connect with Zwift, a social cycling app. 🚴
\n\nUpdate (2021-01-17): … and I’m back on the X220 full-time! I sold the desktop computer mentioned above. I much prefer the X220’s portability compared to the desktop’s raw power.
\n\nMy personal machine is a 8-year-old Lenovo ThinkPad X220 running Ubuntu 18.04 with i3wm.
\n\nIt does not have a single USB-C port. It sports a 1366x768 TN display panel (in case you’re wondering, you can see each individual pixel with your bare eyes). The battery life of the 4-cell battery is horrendous (I barely get 2 hours out of this thing). The trackpad is so small, one could even wonder if it’s a trackpad for ants (no need to say that I never use it). The Wi-Fi card only supports 2.4GHz networks, so hopes for blazingly fast Wi-Fi are to be pushed aside. There’s even a bit of gaffer tape on the bottom left corner of the body to hold the cracked plastic together.
\n\n![\"Lenovo](\"/assets/x220-full.png\")
\nAt my day job, I’m lucky enough to work on a top-spec MacBook Pro provided by my employer. It has a glorious Retina display, 16GB of RAM, a modern Core i7 CPU, and a huge trackpad to boot. All in all, it’s a pretty fancy machine, one that many people would love to use as a daily driver.
\n\nI mean, let’s just compare the trackpads for a second. It’s almost funny at that point.
\n\n![\"Comparison](\"/assets/x220-mbp-trackpads.png\")
\nSurprisingly though, out of the two laptops, my favourite is not the MacBook Pro. When I’m at home, I tuck the aluminium slab away and take out the magnesium brick that is the ThinkPad X220.
\n\nIt’s not pretty. It’s not particularly fast. But it does everything I need, and it’s always ready for everything I throw at it. Could it be a bit of nostalgia for old school hardware? Maybe.
\n\nI strongly believe a Lenovo ThinkPad X220 is still a terrific laptop to use in 2021 and beyond. It’s not for everyone, but the X220 definitely sparks joy. Plus, its accessible price point makes it almost impossible to ignore. I got mine second-hand (third-hand? fourth-hand? I don’t even know) in great condition for less than 200$ in Canada.
\n\nIn practical terms: the X220 plays 1080p videos from YouTube wonderfully, renders Portal 2 quite happily (albeit with lower graphics quality than what you may usually enjoy on higher-end machines), and is a perfect machine to dual boot Windows on for maximum value. I spend most of my time in a Web browser or CLI tools, so it’s not like I’m running complex simulations, but still.
\n\nThe X220, just like any other classic ThinkPad, is extensible, sturdy, reliable, and provides everything I could ask for in a laptop.
\n\nThe classic ThinkPad laptops have “extensibility” written all over them. Most components are user-replaceable. In “ship of Thesus” fashion, if you individually replace every single component of the X220 one at a time, is it still the same X220?
\n\nSeriously though, just look at this list:
\n\nThat’s a list many laptop owners can only dream of. Laptops are increasingly shut tight deliberately, preventing users from fixing and/or upgrading their devices themselves. Not with a classic ThinkPad though.
\n\nI previously owned a ThinkPad X230, which many consider to be part of the last generation of “classic” ThinkPads. Multiple components of that X230 had been upgraded: IPS display, SSD storage, additional RAM, backlit keyboard in my native language, new 9-cell battery, etc. It was a dream machine, and the X220, just like other classic ThinkPads, offers the same extensibility and user-friendly servicing. I eventually bricked the X230 by spilling water in the underside RAM slot (weird accident, don’t ask).
\n\nAfter bricking the X230, I purchased a second-hand ThinkPad X250 on eBay, only to sell it a few weeks later as it’s a huge step backwards compared to the X220/X230: there’s only one user-accessible RAM slot in the X250 (instead of two). The rest of the RAM is soldered to the board. The keyboard is user-replaceable, but to do so you need to take the entire computer apart (instead of just replacing the keyboard directly as with the X220/X230). Like, what? Who thought that was a good idea?
\n\nBeing a 2011 laptop, it also features various ports that some modern laptops users may only have heard of. At work, where everyone uses a top-of-the-line MacBook Pro, it’s like some sort of utopia where everything is wireless, and we don’t ever need to use the USB-C ports for anything other than charging the laptops or connecting to a giant 4K display.
\n\nIn the real world, however, you’d need a handful of adapters with a MacBook Pro to connect to the rest of the world. The X220 provides everything you could practically ask for here:
\n![\"Left](\"/assets/x220-ports.png\")
\nPlus, there are other goodies about this machine:
\nI like to think that a classic ThinkPad is akin to a Toyota Corolla, one of the most (if not the most) reliable production cars ever produced. Give it a good and thorough clean up once a year, change the oil at regular intervals, keep your software up-to-date, and you’ll enjoy this machine for a long time.
\n\nClassic ThinkPads just feel like business. They won’t let you down. The /r/thinkpad sub-reddit is full of classic ThinkPads (some of them I would even call “retro” instead of “classic”), and these things just keep on running, decades after the initial release date.
\n\nThe laptop’s shell is made out of magnesium instead of plastic, making it extra sturdy. The keyboard feels great. Not your typical cheap keyboard from your run-of-the-mill HP laptop. The display hinges are solid.
\n\nAgain, going back to the X250 I used for a couple of weeks: it felt cheap compared to the X220/X230. The shell was made out of plastic, the display seemed fragile, and the trackpoint buttons felt flimsy. Not a great experience coming from a X230.
\n\nThat’s when I knew I’d go for the X220, and stay for a while.
\n\nIf you’re looking to purchase a second-hand classic ThinkPad, do it. Buy the thing. Slap GNU/Linux distribution on there (or *BSD, if you’re into that sort of thing), and have fun.
\n", "url": "https://maximevaillancourt.com/blog/why-i-use-a-thinkpad-x220-in-2021", "summary": "Extensibility, compatibility, and reliability.", "date_published": "2019-11-24 00:00:00 +0000" }, { "id": "/blog/sleep-activation-energy", "title": "On sleepiness, activation energy, and flow", "content_html": "I noticed something noteworthy a few weeks ago.
\n\nIt seems that if I sleep a little less than what is usually recommended, say approximately 6 hours instead of the usual 7-9 hours, I will find it easier to get started on tasks and keep the ball rolling throughout the day. In other words, sleeping less makes me more productive.
\n\nTo be clear, this observation does not fit in the overall picture painted by modern science, which notes that adequate sleep leads to improved health and productivity.
\n\nNow, what I observe is more than likely to be a false impression—a mere feeling that I’m more productive when really I’m just as productive as usual or even less so—but it’s something I’ve been able to reproduce often enough to feel like there must be something going on here, as it’s something I can temporarily leverage and benefit from. It’s also worth pointing out that this very biased, non-scientific experiment on a sample size of 1 should not be taken seriously. I’m simply documenting what I’ve observed.
\n\nIn chemistry, there’s a concept called “activation energy”, which is the quantity of energy that must be provided to a system in order to generate a reaction. Unless a system is supplied enough energy to pass the threshold required to get things moving, the system won’t budge, and will remain inactive.
\n\nImagine standing at the base of a hill with a ball at your feet (this is the system). You want to kick the ball to your friend over on the other side on the mountain (this is the reaction). If you kick the ball gently, it will roll up the mountain for a few meters, then roll back down to your feet—insufficient activation energy to trigger the reaction. Only if you kick the ball strong enough will it reach the top of mountain and roll over to the other side.
\n\n![](\"/assets/energy.svg\")
\n\nDrawing a parallel with psychology and human behavior, we can use activation energy as a mental model for motivation and procrastination.
\n\nWhen faced with a given task, the human brain must come up with the required activation energy to get started and get the ball rolling onto the other side of the mountain. Without it, we remain still, unable to get started for a long period of time.
\n\nAnalysis paralysis. Indecision & inaction. Overthinking and not moving. These are all related.
\n\nI sometimes get stuck and paralyzed by overthinking and overanalyzing a system. This indecision leads to inaction, why means I’m stuck at square one where I continue analyzing the situation, hoping eventually I’ll have enough information to get started.
\n\nIn my experience, getting less sleep leads to a natural ability to short the above circuit and get moving immediately, while also reducing the time to reaching a state of flow.
\n\nIf I dig in a little more, it seems to have to do with a desire to get it over with whatever task is at hand, and a drive to move on to the next thing quickly. This, combined with a willingness to be more scrappy resourceful as time passes, leads to a practical sense of acceleration that I haven’t been able to replicate in any other way.
One day, when I was in elementary school (mid-2000s), I decided I would create a personal website to share games, news, and other cool tidbits of my life with my friends and family. I also wanted it to be password protected so that only my friends would see it, and to prevent big bad internet strangers from seeing private information (oh how times have changed).
\n\nHowever, I didn’t know anything about HTTP authentication, HTML forms, databases, or security best practices. I couldn’t care less, and figured I would find a way. I was, after all, in elementary school. Like, 10 years old maybe.
\n\nSo I set out to build that website, and used the one technology I knew could achieve this: Flash.
\n\nThat’s right, good old .swf files and all.
I first implemented the password protection this way: within the Flash view, there was a login screen with a single field where my friends would type in the password. Naturally it was a hardcoded password, straight in the .swf, because why the hell not, I’m 10 years old, this is fine.
\n\nWhat is this sound I hear? Ah, yes, that’s the sound of security engineers from around the world screaming in unison. Yes. I know. Again, I was 10 years old.
\n\nI encountered a problem pretty early though: what if I wanted to show different things to different people? Hmm.
\n\nInstead of sharing the protecting the website with the same password for all my friends and family members, I would need them to each have their own password (because duh, security), and on top of that, surface different things to different people. So…
\n\nAgain, with the infinite creativity of a 10-year-old who has no idea what they’re doing, I found a brilliant solution:
\n\nCreate a separate .swf file for each of my friends, with a different hardcoded password in every of them, and serve them all as different files on my ISP-provided web server.
\n\nYES. I KNOW.
\n\nAmazing, isn’t it?
\n\nI just didn’t know any better.
\n", "url": "https://maximevaillancourt.com/blog/flash", "summary": "A security horror story starring 11-year-old me… and Flash.", "date_published": "2019-02-28 00:00:00 +0000" }, { "id": "/blog/usb-drop-olevba", "title": "Inspecting a “USB Drop” Attack Using olevba.py", "content_html": "![\"\"](\"https://cdn-images-1.medium.com/max/800/1*r8-nLE9X_-HqKsiQ3w6u1w.png\")
TL;DR: Never plug USB keys you find laying around in your computer. They may\ncontain malware that silently deploys as you plug it in, stealing sensitive\ninformation and downloading viruses in the background.
\n\nFor those of you who may know me, I’m the kind of guy who runs Linux on his\ncomputer full-time (except for the occasional music production or video editing\nsession). This means no Windows vulnerability to worry about, no random update\nthat starts while I’m working… and also no Microsoft Office suite. This is\nimportant to the story, so keep it in mind.
\n\nIt all started with a regular old bright orange USB key that was left laying\naround on a meeting room table.
\n\nThe key itself wasn’t identified and did not appear to give away any information\nabout its owner, so I thought I would just do my good deed of the day and try to\nfind some sort of IF_FOUND.txt file to identify its owner and give it back\nto them. So I proceeded to boot up my Linux partition (you can never be too\ncautious!), then plugged in the key.
At first glance, the key contained seemingly important (and sensitive!)\ninformation about the company’s assets and strategic planning. I first\nwondered why anyone would think it safe to carry such valuable documents on a\nfriggin’ unencrypted USB key (and most importantly, forgetting it in a meeting\nroom). Then, in my attempt to identify the owner of the key, I opened a random\nfolder and started browsing through the files.
\n\nA few seconds in, two oddidities made me realize that something wasn’t quite\nright.
\n\nFirst, all the documents on the key had the .docm extension, which are files\nknown as “Office Word Documents with Macros”, basically meaning that they’re\nWord documents with Visual Basic for Applications (VBA) code baked in. In Excel\nspreadsheets and Access databases especially, VBA Macros allow for improved\nfunctionality and useful sequences of actions that can be used to automate\notherwise mundane tasks.
The other thing that ticked me off was the Date modified attribute of the\nfiles on the key: they were all set to the exact same date. I mean, yes, the\nfiles could all have been copied together and their Date modified fields\nresetted to the same moment, but still. It just didn’t feel right.
\n\nThe combination of the way the key was left on the table AND the two oddities I\nfound led me to think about a completely different scenario:
\n\nThis wasn’t a regular old “normal” USB key. Somebody was trying to fool me,\nand I needed to delve deeper.
\n\nI took the risk of opening one of the Word documents using LibreOffice (with\nmacros disabled, of course), but was left disappointed when I saw that the\ndocument was completely empty. I tried opening another file, hoping to see\nsomething in there, but alas, it was completely empty as well.
\n\nThe rabbit hole was deepening yet again.
\n\nThinking about the .docm extension, I quickly understood that the documents\nprobably contained malicious VBA code, but I did not know exactly what kind of\nstuff I was dealing with. I figured I would just keep digging until I would find\nsomething. After a quick Google search, I stumbled upon the excellent\nolevba.py tool, part of the\noletools collection, which parses\nMS Office documents to detect VBA macros then extract their source code.
Starting with MS Office 2007, Word documents are really just OpenXML files\nwrapped in .doc/.docm/ .docx extensions, so after extracting one of the\ndocuments and browsing through the extracted files, I found an interesting\nbinary file which was significantly larger than the others, appropriately named\nvbaProject.bin. I was pretty confident that this was the file that contained\nmalicious VBA code, probably downloading malware from a remote server and\nexecuting it on the victim’s machine (in this case, my computer, but since I was\nnot on Windows, the chances of being attacked were pretty slim).
However, extracting the source of the vbaProject.bin file using the\nolevba.py tool gave me a completely different answer:
Huh. So this was not malware, after all. This Word document was simply sending\nthe hostname and username to a remote server to identify who had plugged in the\nrogue USB key and opened one of its files. I then realized that this was an\ninfosec effort to identify imprudent employees who used the (intentionally left\nbehind) USB keys (and who should never do that, unless they absolutely know\nand trust the USB key).
\n\nI was pleasantly surprised to see that the subroutine actually used MsgBox to\ncreate a popup announcing to the imprudent user that “the file is corrupted and\ncannot be opened”.
Long story short: only use USB keys you know and trust.
\n", "url": "https://maximevaillancourt.com/blog/usb-drop-olevba", "summary": "Or, “why you shouldn’t pick up random USB keys”.", "date_published": "2017-04-07 00:00:00 +0000" }, { "id": "/blog/kodekpl", "title": "“Who is this KodekPL pushing commits to my GitHub repo?”", "content_html": "That's what I thought when I saw \"KodekPL\" as the author of a commit in my repo... A commit I had just pushed myself seconds before. But why KodekPL and not someone else? Who is this guy?
\n\nI’m willing to bet you play Minecraft. A lot of Minecraft. Enough to warrant the need of creating a server. Perhaps one day you downloaded Spigot. “Wait a second here, why are we talking about Minecraft? I’m talking about my GitHub repo here. Someone is in my repo! What the heck!?”, I hear you say.
\n\nTurns out, there’s a link between Spigot and KodekPL on GitHub.
\n\nUpon further investigation, KodekPL’s GitHub email address is “unconfigured@null.spigotmc.org”. Git, for some reason, picked that up as your email address, hence the wrong author on GitHub. Setting the global variables fixes this.
\n\ngit config --global user.email your@email.com\ngit config --global user.name \"Your Name\"That’s it. Push something to GitHub and enjoy!
\n", "url": "https://maximevaillancourt.com/blog/kodekpl", "summary": "Seriously, who’s that guy, and how is he committing to my repository?", "date_published": "2015-12-22 00:00:00 +0000" } ] }