Keep logging simple<\/p>\n
For years, I used Weights & Biases (W&B)* as my go-to experiment logger. In fact, I have once been in the top 5% of all active users. The stats in below figure tell me that, at that time, I\u2019ve trained close to 25000 models, used a cumulative 5000 hours of compute, and did more than 500 hyperparameter searches. I used it for papers, for big projects like weather prediction with large datasets, and for tracking countless small-scale experiments.<\/p>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/08\/Bildschirm\u00adfoto-2023-01-03-um-12.07.50-1024x746.png\")
My once upon a time stats of using W&B for experiment logging. Image by the author.<\/p>\n
And W&B really is a great tool: if you want beautiful dashboards and are collaborating** with a team, W&B shines. And, until recently, while reconstructing data from trained neural networks, I ran multiple hyperparameter sweeps and W&B\u2019s visualization capabilities were invaluable. I could directly compare reconstructions across runs.<\/p>\n
But I realized that for most of my research projects, W&B was overkill. I rarely revisited individual runs, and once a project was done, the logs just sat there, and I did nothing with them ever after. When I then refactored the mentioned data reconstruction project, I thus explicitly removed the W&B integration. Not because anything was wrong with it, but because it wasn\u2019t necessary.<\/p>\n
Now, my setup is much simpler. I just log selected metrics to CSV and text files, writing directly to disk. For hyperparameter searches, I rely on Optuna. Not even the distributed version with a central server \u2014 just local Optuna, saving study states to a pickle file. If something crashes, I reload and continue. Pragmatic and sufficient (for my use cases).<\/p>\n
The key insight here is this:\u00a0logging is not the work. It\u2019s a support system. Spending 99% of your time deciding on what you want to log \u2014 gradients? weights? distributions? and at which frequency? \u2014 can easily distract you from the actual research. For me, simple, local logging covers all needs, with minimal setup effort.<\/p>\n
Maintain experimental lab notebooks<\/p>\n
In December 1939, William Shockley wrote down an idea into his lab notebook: replace vacuum tubes with semiconductors. Roughly 20 years later, Shockley and two colleagues at Bell Labs were awarded Nobel Prizes for the invention of the modern transistor.<\/p>\n
While most of us aren\u2019t writing Nobel-worthy entries into our notebooks, we can still learn from the principle. Granted, in machine learning, our laboraties don\u2019t have chemicals or test tubes, as we all envision when we think about a laboratory. Instead, our labs often are our computers; the same device that I use to write these lines has trained countless models over the years. And these labs are inherently portably, especially when we are developing remotely on high-performance compute clusters. Even better, thanks to highly-skilled administrative stuff, these clusters are running 24\/7 \u2014 so there\u2019s always time to run an experiment!<\/p>\n
But, the question is, which experiment? Here, a former colleague introduced me to the idea of mainting a lab notebook, and lately I\u2019ve returned to it in the simplest form possible. Before starting long-running experiments, I write down:<\/p>\n
what I\u2019m testing, and why I\u2019m testing it.<\/p>\n
Then, when I come back later \u2014 usually the next morning \u2014 I can immediately see which results are ready and what I had hoped to learn. It\u2019s simple, but it changes the workflow. Instead of just \u201crerun until it works,\u201d these dedicated experiments become part of a documented feedback loop. Failures are easier to interpret. Successes are easier to replicate.<\/p>\n
Run experiments overnight<\/p>\n
That\u2019s a small, but painful lessons that I (re-)learned this month.<\/p>\n
On a Friday evening, I discovered a bug that might affect my experiment results. I patched it and reran the experiments to validate. By Saturday morning, the runs had finished \u2014 but when I inspected the results, I realized I had forgotten to include a key ablation. Which meant \u2026 another full day of waiting.<\/p>\n
In ML, overnight time is precious. For us programmers, it\u2019s rest. For our experiments, it\u2019s work. If we don\u2019t have an experiment running while we sleep, we\u2019re effectively wasting free compute cycles.<\/p>\n
That doesn\u2019t mean you should run experiments just for the sake of it. But whenever there is a meaningful one to launch, starting them in the evening is the perfect time. Clusters are often under-utilized and resources are more quickly available, and \u2014 most importantly \u2014 you will have results to analyse the next morning.<\/p>\n
A simple trick is to plan this deliberately. As Cal Newport mentions in\u00a0his book \u201cDeep Work\u201d, good workdays start the night before. If you know tomorrow\u2019s tasks today, you can set up the right experiments in time.<\/p>\n
* That ain\u2019t bashing W&B (it would have been the same with, e.g., MLFlow), but rather asking users to evaluate what their project goals are, and then spend the majority of time on pursuing that goals with utmost focus.<\/p>\n
** Footnote: mere collaborating is in my eyes not enough to warrant using such shared dashboards. You need to gain more insights from such shared tools than the time spent setting them up.<\/p>\n","protected":false},"excerpt":{"rendered":"in machine learning are the same. Coding, waiting for results, interpreting them, returning back to coding. Plus, some…\n","protected":false},"author":2,"featured_media":36241,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[261],"tags":[291,289,290,28154,6668,18,28155,19,17,1814,28156,82],"class_list":{"0":"post-36240","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-career-insights","12":"tag-data-science","13":"tag-eire","14":"tag-experiment-design","15":"tag-ie","16":"tag-ireland","17":"tag-notebook","18":"tag-producitivity","19":"tag-technology"},"share_on_mastodon":{"url":"","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/36240","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/comments?post=36240"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/posts\/36240\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media\/36241"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/media?parent=36240"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/categories?post=36240"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ie\/wp-json\/wp\/v2\/tags?post=36240"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}