Thinking Clearly

Writing is the most powerful tool for thinking clearly that I know of. ¹

Writing things out wonderfully complements my cognitive shortcomings:

It allows me to work around the hard upper limit on the number of things I can keep in my head at one time.
It enables endless meta-cognition; enabling constant critique and refinement.

At the same time, I'm not used to writing: I have countless pages of notes, plans, half-baked pieces of communication for work; but it's not a tool I intuitively reach for.

Words are hard. Editing is harder.

To that end, I'm aiming towards building a habit by

writing every day, for the next 100 days²;
at least for half an hour, without interruptions;
mostly about programming, but I'll allow myself detours.

Obviously I won't be able to publish one article every day; thoughtful writing takes time. Instead, I'll publish drafts and keep iterating on them (including this article).

Tentative Topics

This is a very optimistic list of what I would like to write about; I expect to touch very few of these over the next 100 days.

Deep dives into programming

Motivated by areas I want to learn more about, and tend to be relevant to me professionally.

A deep exploration of asyncio in Python, and other event loop implementations in general.
HTTP/3, Quic.
GDB.
Jupyter Internals.
Observable programs
Concepts in machine learning ³

Tools for Programmers

I've spent a significant portion of my career building tools for engineers; even when that isn't my primary goal I find it extremely valuable to spend at least 10% of my time speeding up the work I'll do.

Emacs, themes, plugins, and extensions
The Suckless toolkit, particularly dwm and st
Building customizable tools

My approach to software engineering

Explicitly writing out how I approach several aspects of Software Engineering. Mainly to consciously review and refine what I do, to become a better engineer, but also to share and get feedback.

These are things I do, and not necessarily things I'm good at: any recommendations and approaches should be evaluated in that context.

Reviewing code
Debugging problems
Ramping up on a project
Designing and structuring a program
Designing and structuring a programming notebook
Organizing and planning work
Building feedback loops

Books

The one aspect of my life I have managed to write about somewhat consistently is the books I've read. The most valuable notes I've collected are not notes on a single book, but collections of notes around a single topic collected from many sources.

Books on Programming Languages
Books on Software Engineering
Books on Visualization
Books on Photography
Books on Writing ⁴
Books on Learning
Books on Thinking
Books I would like to read

Retrospectives

I'm close to completing a decade as a professional Software Engineer, I probably know something worth writing about.
Side projects – past, current, and ideas for the future; these might be closer to developer logs for a given project.
Various attempts at building productivity systems, and things that stuck – and didn't.

Review

28 days

I've managed to consistently keep writing every day for the past month: it's a good time to take a look at what's working and what's not.

As part of this project, I've "published" four notes (excluding this one), and written a lot more on asyncio. Three of the notes have been on relatively soft – based on opinion and experience – and one was an attempt at re-explaining something valuable. I've also created my own Python tracer on the side, which I hope to open source soon.

I've also captured several notes from books and other resources as an attempt at syntopical reading: my very own wiki / zettelkasten / simple hyperlinked web page. These give me hope towards building a better knowledge-base for myself, as I engage with and write about areas I care about.

It's occasionally been a very exhausting experience, and rewarding when I see readers engage with the contents and reach out to me. That said, I started this exercise as a way to think more clearly, and I can't say I've accomplished that.

Writing regularly has also made it somewhat smoother: words flow more easily, even if I do need to constantly revise them to be readable. I also have a somewhat reasonable workflow around writing and publishing new notes.

Ideally I'd prefer to write about topics I have a hard time understanding, and re-phrasing in my own words makes it more accessible to me (see also: Feynman's methods of study), so those are the topics I'd like to start focusing on after wrapping up the series on asyncio.

60 days

I'm two-thirds of the way through, and I can't say I'm getting what I'd hoped for out of writing every day: building the muscles is good, but I'm not very satisfied with the quality of my output. Which, in turn, becomes disappointing enough to make me less excited about writing the next day, and so on with a somewhat negative cycle.

With around ~40 days to go I'm going to choose the notes I'd like to see published explicitly:

A cleaned up series on Asyncio, including a description on event loops, as well as internals.
Completed notes on A Random Walk, and In Place Permutations, with fancy diagrams.
Syntopical notes on books I've been reading; particularly Introduction to Mathematics, Fooled by Randomness and How Not To Be Wrong.
Carefully edit, re-write, and clean-up my existing notes; a side effect of writing every day is a lot of late-night typing in delirium with all the mistakes imaginable.
Write about debugger-based-tracing, and my new tracer once I can open source it.
Start building software from "No Ideas but In Things", and maintain a developer's log that I can then publish as a post later.
Redesign expLog: the current iteration has been consistently maintained as an MVP, but the sloppiness in the website encourages me to be just as sloppy with my writing.

I also need to have a little more clarity around what I want to improve on: so far, I've been muddling between –

Writing to build habits and muscles: Working on forming cohesive, clear sentences, choosing the right words; building habits to write out things.
Writing to learn: writing to capture what I've learned in my own words, as a mechanism to both strengthen my knowledge and identify gaps.
Writing to teach: writing something useful that can be grasped comfortably by others; this requires one giant step forward from simply writing to understand.
Writing to think: taking and shaping an unformed idea through long form writing; constantly refining and improving on it, and using the extra available memory to observe my thoughts and fix flaws.

Looking at the origin of this post, I value writing to think of all the types of writing I've been attempting; at the same time I've spent most of my time meandering between the other three types of writing. This also explains my general frustration with the quality of results so far.

Editing posts aggressively, and perhaps introducing numbers (at the very least, fermi approximations) to any opinion based posts should help me here.

On the positive side, I've been finding a lot of value with my short posts around specific parts of programming acting as a quick reference that I know very well (because I wrote it). Particularly the post on sys.settrace.

Finally, I have much more respect for people who write good, well structured books: and I can easily understand why something like TAOCP where every single exercise is meticulously crafted consumes decades of work.

90 days

I've recently been losing steam with writing every day, mostly because I haven't been finding it as valuable as I had hoped. Instead of beating myself up about it, I'm going to revisit the premise of this exercise and figure out how best to make it work for me.

The Plan

My original plan (right at the top in this document? diary-entry?) was to build a habit of writing every day for at least 30 minutes. The underlying motivation was to build a habit of writing to think.

Parts that went well

I feel much more comfortable with the mechanics of writing; I have a reasonable publishing flow⁵; sketching out an outline, writing, and rewriting are no longer the dragons they used to be. Both my writing setup: emacs configuration and my publishing setup: explog have leveled up since I started.

I think I see an improvement in my clarity of thought and expression, particularly when comparing the introduction, the 28-day review and the 60-day review.

The content of what I've been writing about proved useful:

Collecting notes across books on a single topic has been a very promising idea; my very own simple zettelkasten. Re-documenting code I often look up has also paid off.
Explicitly thinking through and writing through subconscious processes like debugging refined the them. Having the processes written out also acts as a convenient checklist when I need it.
My several incomplete attempts at writing about asyncio helped me build a deep understanding around how it works and it came to be. I've been able to consistently apply that knowledge.
Feedback on posts – and requests for RSS feeds – have been encouraging and valuable.

Parts that didn't

I spent far too much time on the meta: it's a character flaw that I've never really been able to manage well. Time spent reading about writing, writing about writing, setting up tools for writing, setting up tools for reading, reflecting on writing easily exceeds time spent writing. There's probably no better example than this page.

The need to write every day resulted in a lot of sloppy writing. The exercise devolved into checking off a box instead of actually building muscle and thinking on several days. Over time, that became fairly demotivating because I was no longer fulfilling the prime directive.

Context switching into whatever I'm writing about that day has also been costly; particularly on days I've been working on harder things. I need to ramp up significantly on Machine Learning to be effective at work, but I end up returning to asyncio because I've been writing about it instead.

The quality of what I've published so far hasn't made me particularly happy either: re-reading what I've written makes me urgently want to re-write it and edit it. Nothing of what I've published so far seems to deserve the time and effort to make it better.

Shaking things up

The goals I would still like to accomplish include:

writing something I'm actually proud of – perhaps the note on asyncio;
writing through something complex that requires a significant amount of effort to learn well: normalized entropy, other observations on machine learning, etc.
verifying what I'm writing about with Maths instead of opinions; particularly Fermi numbers and Monte-carlo simulations
building out my collection of syntopical notes on different subjects as a personal reference

On reflection, the biggest change I need to make is to focus my writing and avoid taking fanciful strolls as frequently as I do to avoid the dissatisfaction and unnecessary context switches.

I'll keep updating this post till I complete a full 100 instances of meaningful writing – ignoring the days when I check a box. Beyond that, I'll simply continue writing and posting.

Consistency

(2021 Note: As part of cleaning up for new years I've removed the half-completed notes out, so several links below will be broken; hopefully to be replaced with real articles in the future.)

Updates

2020-05-16: Set up this post: brainstorming and listing out ideas to write about.
2020-05-17: Started iterating on a post about asyncio, figuring out the outline and how to approach it, both to learn and to write.
2020-05-18: Updated the post on asyncio. I finally understand how asyncio.sleep works!
2020-05-19: Dug deeper into the event loop implementation in asyncio, reaching and stracing epoll. Also submitted a typo fix to cpython.
2020-05-20: Explained where the ioctl calls were coming from (hint: signals). Read about epoll's implementation.
2020-05-21: Started a new parallel article about exploring the Linux Kernel, to allow me to dig into epoll. Set up bcc.
2020-05-22: Dug into the disassembly of the python program, and briefly documented what I found.
2020-05-23: Took a break and wrote about my workflow for planning.
2020-05-24: Forked a new note about setting up and using python-gdb to explore python execution.
2020-05-25: Started writing about tracing asyncio tasks, my secret motivation to start on this series of notes.
2020-05-26: My copy of The Art of Science and Engineering arrived. Taking notes chapter by chapter to take a break from programming after work.
2020-05-27: I started writing about how I approach debugging and realized that I have a lot to say. I should be able to publish this soon.
2020-05-28: Completed a first draft of the post on debugging; I think I have the broader pieces in place but there's still a lot to do.
2020-05-29: Revising, editing and tweaking the post on debugging.
2020-05-30: Revised and edited the post on debugging, created a diagram and added it to the index.
2020-05-31: Wrote about David Sedaris's book. Worked on asyncio tracing, which I'll write about in the future.
2020-06-01: Writing for myself today; about what's going on in the world and my place in it.
2020-06-02: Added a small section to the note on asyncio with the output of my tracer on the minimal async example.
2020-06-03: Spent quite a bit of time scratching my head at the differences between async generators and coroutines.
2020-06-04: Read another chapter of The Art of Doing Science and Engineering and added to my notes.
2020-06-05: For some reason I couldn't stop thinking about learning to use tools, so it seemed appropriate to write about it.
2020-06-06: Edited and published my heuristics for learning to use tools.
2020-06-07: Back to asyncio tracing: explored how to connect async tasks to their actual execution.
2020-06-08: Slowly working through tracing python code, and intercepting task creation to connect it to execution.
2020-06-09: Learned and briefly described the power of sys.settrace. Wrote a summary of Systemantics.
2020-06-10: Split out a new note on sys.settrace.
2020-06-11: Filled in the new note on sys.settrace, and significantly improved my tracing code as I learned more about it.
2020-06-12: Continued iterating on sys.settrace to add examples; I still need to reorganize the content for readability.
2020-06-13: Published the note on sys.settrace. Added brief notes from books on writing.
2020-06-14: Started reviewing this exercise, and my results with it so far.
2020-06-15: Wrapped up the 4 weeks review. Also dug into coroutine creation without actively writing about it.
2020-06-16: Greatly improved my understanding of generators, coroutines, async generators, etc. Added an example to terms.
2020-06-17: Started writing about event loops in general.
2020-06-18: Doing a private retrospective around how I have created – and would like to create value at work.
2020-06-19: Read another chapter of Hamming and took brief notes.
2020-06-20: Wrote a little bit more about event loops. Still struggling with organizing my notes on asyncio.
2020-06-21: I wanted to observe on-cpu time for python code as part of my tracer, and dove into ftrace to figure it out.
2020-06-22: Simulated a random walk.
2020-06-23: Continued iterating on the post on a random walk, realizing I've forgotten a lot of math.
2020-06-24: Added an analytical solutionto determine the root mean square distance, but struggling a little with latex.
2020-06-25: Learned some mathjax and fixed up the solution, iterating on the graphs.
2020-06-26: A few brief notes on books on mathematical thinking that I've been reading lately.
2020-06-27: Some personal notes on the projects I've been working on, why I've been working on them; and others I should start.
2020-06-28: Expanded on coroutines and functions a little bit. Stumbled across PEP 3156.
2020-06-29: Added brief notes from PEP 3156; I wish I'd read this much earlier.
2020-06-30: Started sketching out what a coherent series of posts on asyncio should look like, extracted from the pile of notes I've accumulated.
2020-07-01: Brief addition to my notes on mathematical thinking.
2020-07-02: Added a few more notes from the second chapter on mathematical thinking.
2020-07-03: Continued studying and summarizing introduction to mathematical thinking.
2020-07-04: Worked on a README for my python tracer: panopticon.
2020-07-05: Started another note on asyncio: on the invisible scheduler.
2020-07-06: Continued iterating on the scheduler and getting fairly confused about how Task works.
2020-07-07: Wrote a brief description around how I use tmux.
2020-07-08: Fleshed out my notes on how I use tmux. I should return to my series on asyncio tomorrow, I can finally see the shape in my head.
2020-07-09: Sketched out an outline and structure for to make my somewhat copious notes on asyncio coherent.
2020-07-10: Nerd-sniped myself while digging into ways to shuffle an array in place while programming Rust. I'll write about it.
2020-07-11: Experimenting with another approach at describing event loops, but not feeling very comfortable with it.
2020-07-12: "No Ideas but in Things", or how I'm now approaching learning.
2020-07-13: Started documenting the rabbit hole I fell into a few days ago. Coincidentally, I fell down another one starting the post.
2020-07-14: Continued filling in the different approaches to the problem.
2020-07-15: Expanded on the rabbit hole; thinking about the best way to make diagrams.
2020-07-16: Started adding notes from an old paper on permutations.
2020-07-17: 2 months in, updated this post with another review and my plans for the last ~40 days of this exercise.
2020-07-18: I spent another hour writing about writing, this time in private as I figure out the best way to make the most of my time.
2020-07-19: Continued writing about writing for thinking: what to write, and contrasting alternatives like sketching, modeling, etc.
2020-07-20: Spent some time sketching out how I should re-write all my notes on asyncio so that they can be understood by others.
2020-07-21: Self-review season; spent some time writing about what I did well, and what I didn't as well as planning the next half.
2020-07-22: Spent some time looking forward and what I want the rest of this year to look like professionally.
2020-07-23*: Started writing about normalized cross entropy. entropy looks, but fell asleep before spending a full 30 minutes.
2020-07-24: Wrote about my writing setup using Emacs.
2020-07-25: Updated my notes on books on thinking.
2020-07-26: Yet another attempt at writing about asyncio, this time with a simpler structure in mind.
2020-07-27*: Too tired, and I ended up missing a second day.
2020-07-28*: And the bad streak continued, with 2 days in a row skipped.
2020-07-29: I finally completed all the formalities for opensourcing my python tracer: Panopticon! Cleaned up the README.
2020-07-30: Wrote a lot for Panopticon, particularly a walk through along with actually open sourcing it.
2020-07-31: Continued polishing the walk through before I share the project more widely.
2020-08-01: Completed the introduction to Panopticon.
2020-08-02: Edited and tweaked the introduction to be more readable.
2020-08-03: Tweaked Panopticon's source, comments, etc.
2020-08-04: Wrote about the habits I want to develop over the next several months.
2020-08-05: Back to my old friend, asyncio. I learned and wrote a little about the Python parser and compiler today.
2020-08-06: I finally understood and wrote about how coroutines get distinguished from standard old functions.
2020-08-07*: Fell asleep too soon again, I was feeling surprisingly tired.
2020-08-08: Started writing a proposal for my office work that I'm surprisingly excited about.
2020-08-09: Added an introduction to this blog in the first entry in my shiny new RSS feed.
2020-08-10: Updated my books list with some new books. Added a new syntopical page on making photographs.
2020-08-11*: Too tired, I've been having trouble clearing my head to write after work recently.
2020-08-12*: Same as above.
2020-08-13: Transcribed some of my older notes on reading.
2020-08-14*: Skipped.
2020-08-15*: Skipped.
2020-08-16*: Skipped.
2020-08-17: Reviewed my progress so far, and my recent lack of motivation. Wrote some elisp and python to generate a diagram.
2020-09-07: Re-started writing half-an-hour a day, because I wasn't making any progress otherwise. Focusing on asyncio in depth.

Comments or feedback?

Reach out @kunalbhalla, or drop me an email.

Footnotes:

My experience with increasingly stronger doses of coffee – bullet-proof or other variants, tea and matcha only suggests an increase in speed, and not the quality or clarity of thought.

Yes, this is a #100daysofwriting challenge for myself. I've never considered attempting 100 days of anything – except perhaps programming or reading – but a man can dream.

I can already hear you groan, but it's something I would like to build depth in, particularly for work. You should feel grateful that I don't have yet another tutorial on Monads in this list.

⁴

Passing up opportunities to be meta is physically painful for me.

⁵

To no-one's surprise, I spent a significant amount of time tweaking my writing setup, and my article about it has been the most popular one yet.