#+TITLE: Thinking Clearly

Writing is the most powerful tool for thinking clearly that I know
of. [fn:1]

Writing things out wonderfully complements my cognitive
- It allows me to work around the hard upper limit on the number of
  things I can keep in my head at one time. 
- It enables endless meta-cognition; enabling constant critique and

At the same time, I'm not used to writing: I have countless pages of
notes, plans, half-baked pieces of communication for work; but it's
not a tool I intuitively reach for. 

Words are hard. Editing is harder.

To that end, I'm aiming towards building a habit by 
- writing every day, for the next 100 days[fn:2];
- at least for half an hour, without interruptions;
- mostly about programming, but I'll allow myself detours.

Obviously I won't be able to publish one article every day;
thoughtful writing takes time. Instead, I'll publish drafts
and keep iterating on them (including this article).

[fn:1] My experience with increasingly stronger doses of coffee --
bullet-proof or other variants, tea and matcha only suggests an
increase in speed, and not the quality or clarity of thought.

[fn:2] Yes, this is a #100daysofwriting challenge for
myself. I've never considered attempting 100 days of anything --
except perhaps programming or reading -- but a man can dream.

* Tentative Topics
This is a very optimistic list of what I would like to write about; I
expect to touch very few of these over the next 100 days.

** Deep dives into programming
Motivated by areas I want to learn more about, and tend to be relevant
to me professionally.
- A deep exploration of asyncio in Python, and other event loop
  implementations in general.
- HTTP/3, Quic.
- GDB.
- Jupyter Internals.
- Observable programs
- Concepts in machine learning [fn:3]

[fn:3] I can already hear you groan, but it's something I would like
to build depth in, particularly for work. You should feel grateful
that I don't have yet another tutorial on Monads in this list.

** Tools for Programmers
 I've spent a significant portion of my career building tools for
 engineers; even when that isn't my primary goal I find it extremely
 valuable to spend at least 10% of my time speeding up the work I'll
- Emacs, themes, plugins, and extensions
- The Suckless toolkit, particularly dwm and st
- Building customizable tools

** My approach to software engineering
Explicitly writing out how I approach several aspects of Software
Engineering. Mainly to consciously review and refine what I do, to
become a better engineer, but also to share and get feedback.

These are things I do, and not necessarily things I'm good at:
any recommendations and approaches should be evaluated in that

- Reviewing code
- Debugging problems
- Ramping up on a project
- Designing and structuring a program
- Designing and structuring a programming notebook
- Organizing and planning work
- Building feedback loops

** Books
The one aspect of my life I have managed to write about somewhat
consistently is the books I've read. The most valuable notes I've
collected are not notes on a single book, but collections of notes around
a single topic collected from many sources.
- Books on Programming Languages
- Books on Software Engineering
- Books on Visualization
- Books on Photography
- Books on Writing [fn:4]
- Books on Learning
- Books on Thinking
- Books I would like to read

[fn:4] Passing up opportunities to be meta is physically painful for
** Retrospectives
- I'm close to completing a decade as a professional Software
  Engineer, I probably know something worth writing about.
- Side projects -- past, current, and ideas for the future; these
  might be closer to developer logs for a given project.
- Various attempts at building productivity systems, and things that
  stuck -- and didn't.
* Review
** 28 days
I've managed to consistently keep writing every day for the past
month: it's a good time to take a look at what's working and what's

As part of this project, I've "published" four notes (excluding this
one), and written a lot more on asyncio. Three of the notes have been on
relatively soft -- based on opinion and experience -- and one was an
attempt at re-explaining something valuable. I've also created my own
Python tracer on the side, which I hope to open source soon.

I've also captured several notes from books and other resources as an
attempt at syntopical reading: my very own wiki / zettelkasten /
simple hyperlinked web page. These give me hope towards building a
better knowledge-base for myself, as I engage with and write about
areas I care about.

It's occasionally been a very exhausting experience, and rewarding
when I see readers engage with the contents and reach out to me. That
said, I started this exercise as a way to think more clearly, and I
can't say I've accomplished that. 

Writing regularly has also made it somewhat smoother: words flow more
easily, even if I do need to constantly revise them to be
readable. I also have a somewhat reasonable workflow around writing
and publishing new notes. 

Ideally I'd prefer to write about topics I have a hard time
understanding, and re-phrasing in my own words makes it more
accessible to me (see also: Feynman's methods of study), so those are
the topics I'd like to start focusing on after wrapping up the series
on asyncio.

** 60 days
I'm two-thirds of the way through, and I can't say I'm getting what
I'd hoped for out of writing every day: building the muscles is good,
but I'm not very satisfied with the quality of my output. Which, in
turn, becomes disappointing enough to make me less excited about
writing the next day, and so on with a somewhat negative cycle.

With around ~40 days to go I'm going to choose the notes I'd like to
see published explicitly:
- A cleaned up series on Asyncio, including a description on event
  loops, as well as internals.
- Completed notes on A Random Walk, and In Place Permutations, with
  fancy diagrams.
- Syntopical notes on books I've been reading; particularly
  Introduction to Mathematics, Fooled by Randomness and How Not To Be
- Carefully edit, re-write, and clean-up my existing notes; a side
  effect of writing every day is a lot of late-night typing in
  delirium with all the mistakes imaginable.
- Write about debugger-based-tracing, and my new tracer once I can
  open source it.
- Start building software from "No Ideas but In Things", and maintain
  a developer's log that I can then publish as a post later.
- Redesign expLog: the current iteration has been consistently
  maintained as an MVP, but the sloppiness in the website encourages
  me to be just as sloppy with my writing.
I also need to have a little more clarity around what I want to
improve on: so far, I've been muddling between --
- Writing to build habits and muscles: Working on forming cohesive,
  clear sentences, choosing the right words; building habits to write
  out things.
- Writing to learn: writing to capture what I've learned in my
  own words, as a mechanism to both strengthen my knowledge and
  identify gaps.
- Writing to teach: writing something useful that can be grasped
  comfortably by others; this requires one giant step forward from
  simply writing to understand.
- Writing to think: taking and shaping an unformed idea through long
  form writing; constantly refining and improving on it, and using
  the extra available memory to observe my thoughts and fix flaws.

Looking at the origin of this post, I value writing to think of all
the types of writing I've been attempting; at the same time I've spent
most of my time meandering between the other three types of
writing. This also explains my general frustration with the quality of
results so far.

Editing posts aggressively, and perhaps introducing numbers (at the
very least, fermi approximations) to any opinion based posts should
help me here.

On the positive side, I've been finding a lot of value with my short
posts around specific parts of programming acting as a quick reference
that I know very well (because I wrote it). Particularly the post on

Finally, I have much more respect for people who write good, well
structured books: and I can easily understand why something like TAOCP
where every single exercise is meticulously crafted consumes decades
of work.

** 90 days
I've recently been losing steam with writing every day, mostly because
I haven't been finding it as valuable as I had hoped. Instead of
beating myself up about it, I'm going to revisit the premise of this
exercise and figure out how best to make it work for me.

*** The Plan
My original plan (right at the top in this document? diary-entry?) was
to build a habit of writing every day for at least 30 minutes. The
underlying motivation was to build a habit of writing to think.

*** Parts that went well
I feel much more comfortable with the mechanics of writing; I have a
reasonable publishing flow[fn:writingsetup]; sketching out an outline, 
writing, and rewriting are no longer the dragons they used to be. Both
my writing setup: emacs configuration and my publishing setup: explog
have leveled up since I started.

I think I see an improvement in my clarity of thought and expression,
particularly when comparing the introduction, the 28-day review and
the 60-day review.

The content of what I've been writing about proved useful:
- Collecting notes across books on a single topic has been a very
  promising idea; my very own simple zettelkasten. Re-documenting code
  I often look up has also paid off.
- Explicitly thinking through and writing through subconscious processes
  like debugging refined the them. Having the processes written out
  also acts as a convenient checklist when I need it.
- My several incomplete attempts at writing about asyncio helped
  me build a deep understanding around how it works and it came to
  be. I've been able to consistently apply that knowledge.
- Feedback on posts -- and requests for RSS feeds -- have been
  encouraging and valuable.
*** Parts that didn't
I spent far too much time on the meta: it's a character flaw that I've
never really been able to manage well. Time spent reading about
writing, writing about writing, setting up tools for writing, setting
up tools for reading, reflecting on writing easily exceeds time spent
writing. There's probably no better example than this page.

The need to write every day resulted in a lot of sloppy writing.
The exercise devolved into checking off a box instead of actually
building muscle and thinking on several days. Over time, that became
fairly demotivating because I was no longer fulfilling the prime

Context switching into whatever I'm writing about that day has also
been costly; particularly on days I've been working on harder
things. I need to ramp up significantly on Machine Learning to be
effective at work, but I end up returning to asyncio because I've been
writing about it instead.

The quality of what I've published so far hasn't made me particularly
happy either: re-reading what I've written makes me urgently want to
re-write it and edit it. Nothing of what I've published so far seems
to deserve the time and effort to make it better.

*** Shaking things up
The goals I would still like to accomplish include: 
- writing something I'm actually proud of -- perhaps the note on
- writing through something complex that requires a significant amount
  of effort to learn well: normalized entropy, other observations on
  machine learning, etc.
- verifying what I'm writing about with Maths instead of opinions;
  particularly Fermi numbers and Monte-carlo simulations
- building out my collection of syntopical notes on different subjects
  as a personal reference
On reflection, the biggest change I need to make is to focus my
writing and avoid taking fanciful strolls as frequently as I do to
avoid the dissatisfaction and unnecessary context switches.

I'll keep updating this post till I complete a full 100 instances of
meaningful writing -- ignoring the days when I check a box. Beyond
that, I'll simply continue writing and posting.

[fn:writingsetup]  To no-one's surprise, I spent a significant amount
of time tweaking my writing setup, and my article about it has been
the most popular one yet.

* Consistency
#+name: updates_contents 
#+begin_src emacs-lisp :results raw :exports none
(let* ((heading (save-excursion
                  (org-link-search "updates")
       (begin (org-element-property :contents-begin heading))
       (end (org-element-property :contents-end heading)))
  (buffer-substring-no-properties begin end))

#+RESULTS: updates_contents...

#+begin_src python :var updates_contents=updates_contents :results output :exports none
import datetime
import matplotlib.pyplot as plt
import re
import seaborn as sns

dates_regex = re.compile("- (\d{4}-\d{2}-\d{2}):")
writing_dates_str = dates_regex.findall(updates_contents)
writing_dates = [datetime.datetime.strptime(x, "%Y-%m-%d")
                 for x in writing_dates_str]
offsets = [(x - writing_dates[0]).days for x in writing_dates]

stats = f"{len(writing_dates)}/{(writing_dates[-1] - writing_dates[0]).days}"
plt.figure(figsize=(8, 3))
    left=0.1, bottom=.2, right=.9,
    top=.9, wspace=0, hspace=0

sns.rugplot(offsets, height=1, color='black')
plt.xlabel(f"Days since start ({stats})")


* <<updates>> Updates
- 2020-05-16: Set up this post: brainstorming and listing out ideas to
  write about. 
- 2020-05-17: Started iterating on a post about asyncio, figuring out
  the outline and how to approach it, both to learn and to write.
- 2020-05-18: Updated the post on asyncio. I finally understand how
  asyncio.sleep works! 
- 2020-05-19: Dug deeper into the event loop implementation in asyncio, reaching
  and stracing epoll. Also submitted a typo fix to cpython.
- 2020-05-20: Explained where the ioctl calls were coming from (hint:
  signals). Read about epoll's implementation.
- 2020-05-21: Started a new parallel article about exploring the Linux
  Kernel, to allow me to dig into epoll. Set up bcc. 
- 2020-05-22: Dug into the disassembly of the python program, and
  briefly documented what I found.
- 2020-05-23: Took a break and wrote about my workflow for planning.
- 2020-05-24: Forked a new note about setting up and using python-gdb
  to explore python execution.
- 2020-05-25: Started writing about tracing asyncio tasks, my secret
  motivation to start on this series of notes.
- 2020-05-26: My copy of The Art of Science and Engineering
  arrived. Taking notes chapter by chapter to take a break from
  programming after work.
- 2020-05-27: I started writing about how I approach debugging and
  realized that I have a lot to say. I should be able to publish this soon.
- 2020-05-28: Completed a first draft of the post on debugging; I
  think I have the broader pieces in place but there's still a lot to do.
- 2020-05-29: Revising, editing and tweaking the post on debugging.
- 2020-05-30: Revised and edited the post on debugging, created a
  diagram and added it to the index.
- 2020-05-31: Wrote about David Sedaris's book. Worked on asyncio
  tracing, which I'll write about in the future.
- 2020-06-01: Writing for myself today; about what's going on in the
  world and my place in it.
- 2020-06-02: Added a small section to the note on asyncio with
  the output of my tracer on the minimal async example.
- 2020-06-03: Spent quite a bit of time scratching my head at the
  differences between async generators and coroutines.
- 2020-06-04: Read another chapter of The Art of Doing Science and
  Engineering and added to my notes.
- 2020-06-05:  For some reason I couldn't stop thinking about learning
  to use tools, so it seemed appropriate to write about it.
- 2020-06-06:  Edited and published my heuristics for learning to use
- 2020-06-07: Back to asyncio tracing: explored how to connect async
  tasks to their actual execution. 
- 2020-06-08: Slowly working through tracing python code, and
  intercepting task creation to connect it to execution.
- 2020-06-09: Learned and briefly described the power of
  sys.settrace. Wrote a summary of Systemantics.
- 2020-06-10: Split out a new note on sys.settrace.
- 2020-06-11: Filled in the new note on sys.settrace, and
  significantly improved my tracing code as I learned more about it.
- 2020-06-12: Continued iterating on =sys.settrace= to add examples; I
  still need to reorganize the content for readability.
- 2020-06-13: Published the note on =sys.settrace=. Added brief notes
  from books on writing.
- 2020-06-14: Started reviewing this exercise, and my results with it
  so far.
- 2020-06-15: Wrapped up the 4 weeks review. Also dug into coroutine
  creation without actively writing about it.
- 2020-06-16: Greatly improved my understanding of generators,
  coroutines, async generators, etc. Added an example to terms.
- 2020-06-17: Started writing about event loops in general.
- 2020-06-18: Doing a private retrospective around how I have created
  -- and would like to create value at work.
- 2020-06-19: Read another chapter of Hamming and took brief notes.
- 2020-06-20: Wrote a little bit more about event loops. Still
  struggling with organizing my notes on asyncio.
- 2020-06-21: I wanted to observe on-cpu time for python code as part
  of my tracer, and dove into ftrace to figure it out.
- 2020-06-22: Simulated a random walk.
- 2020-06-23: Continued iterating on the post on a random walk,
  realizing I've forgotten a lot of math.
- 2020-06-24: Added an analytical solutionto determine the root mean
  square distance, but struggling a little with latex.
- 2020-06-25: Learned some mathjax and fixed up the solution,
  iterating on the graphs.
- 2020-06-26: A few brief notes on books on mathematical thinking that
  I've been reading lately.
- 2020-06-27: Some personal notes on the projects I've been working
  on, why I've been working on them; and others I should start.  
- 2020-06-28: Expanded on coroutines and functions a little
  bit. Stumbled across PEP 3156.
- 2020-06-29: Added brief notes from PEP 3156; I wish I'd read this
  much earlier.
- 2020-06-30: Started sketching out what a coherent series of posts on
  asyncio should look like, extracted from the pile of notes I've accumulated.
- 2020-07-01: Brief addition to my notes on mathematical thinking.
- 2020-07-02: Added a few more notes from the second chapter on
  mathematical thinking.  
- 2020-07-03: Continued studying and summarizing introduction to
  mathematical thinking.
- 2020-07-04: Worked on a README for my python tracer: panopticon.
- 2020-07-05: Started another note on asyncio: on the invisible scheduler.
- 2020-07-06: Continued iterating on the scheduler and getting fairly
  confused about how Task works.
- 2020-07-07: Wrote a brief description around how I use tmux.
- 2020-07-08: Fleshed out my notes on how I use tmux. I should return
  to my series on asyncio tomorrow, I can finally see the shape in my head.
- 2020-07-09: Sketched out an outline and structure for to make my
  somewhat copious notes on asyncio coherent.
- 2020-07-10: Nerd-sniped myself while digging into ways to shuffle an
  array in place while programming Rust. I'll write about it.
- 2020-07-11: Experimenting with another approach at describing event
  loops, but not feeling very comfortable with it.
- 2020-07-12: "No Ideas but in Things", or how I'm now approaching
- 2020-07-13: Started documenting the rabbit hole I fell into a few
  days ago. Coincidentally, I fell down another one starting the post.
- 2020-07-14: Continued filling in the different approaches to the
- 2020-07-15: Expanded on the rabbit hole; thinking about the best way
  to make diagrams.
- 2020-07-16: Started adding notes from an old paper on permutations.
- 2020-07-17: 2 months in, updated this post with another review and
  my plans for the last ~40 days of this exercise.
- 2020-07-18: I spent another hour writing about writing, this time in
  private as I figure out the best way to make the most of my time.
- 2020-07-19: Continued writing about writing for thinking: what to
  write, and contrasting alternatives like sketching, modeling, etc.
- 2020-07-20: Spent some time sketching out how I should re-write all
  my notes on asyncio so that they can be understood by others.
- 2020-07-21: Self-review season; spent some time writing about what
  I did well, and what I didn't as well as planning the next half.
- 2020-07-22: Spent some time looking forward and what I want the rest
  of this year to look like professionally. 
- 2020-07-23*: Started writing about normalized cross entropy.
  entropy looks, but fell asleep before spending a full 30 minutes.
- 2020-07-24: Wrote about my writing setup using Emacs.
- 2020-07-25: Updated my notes on books on thinking.
- 2020-07-26: Yet another attempt at writing about asyncio, this time
  with a simpler structure in mind.
- 2020-07-27*: Too tired, and I ended up missing a second day.
- 2020-07-28*: And the bad streak continued, with 2 days in a row skipped.
- 2020-07-29: I finally completed all the formalities for opensourcing
  my python tracer: Panopticon! Cleaned up the README.
- 2020-07-30: Wrote a lot for Panopticon, particularly a walk through
  along with actually open sourcing it.
- 2020-07-31: Continued polishing the walk through before I share the
  project more widely.
- 2020-08-01: Completed the introduction to Panopticon.
- 2020-08-02: Edited and tweaked the introduction to be more readable.
- 2020-08-03: Tweaked Panopticon's source, comments, etc.
- 2020-08-04: Wrote about the habits I want to develop over the next
  several months.
- 2020-08-05: Back to my old friend, asyncio. I learned and wrote a
  little about the Python parser and compiler today. 
- 2020-08-06: I finally understood  and wrote about how coroutines get
  distinguished from standard old functions.
- 2020-08-07*: Fell asleep too soon again, I was feeling surprisingly
- 2020-08-08: Started writing a proposal for my office work that I'm
  surprisingly excited about.
- 2020-08-09: Added an introduction to this blog in the first entry in
  my shiny new RSS feed.
- 2020-08-10: Updated my books list with some new books. Added a new
  syntopical page on making photographs.
- 2020-08-11*: Too tired, I've been having trouble clearing my head to
  write after work recently.
- 2020-08-12*: Same as above.
- 2020-08-13: Transcribed some of my older notes on reading.
- 2020-08-14*: Skipped.
- 2020-08-15*: Skipped.
- 2020-08-16*: Skipped.
- 2020-08-17: Reviewed my progress so far, and my recent lack of
  motivation. Wrote some elisp and python to generate a diagram.
- 2020-09-07: Re-started writing half-an-hour a day, because I wasn't
  making any progress otherwise. Focusing on asyncio in depth.

* Comments or feedback?
Reach out @kunalbhalla, or drop me an email.