The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
But you didn’t spend hundreds of hours on it, so when it did happen to be useful it seemed like an outsized benefit.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
Like anything else, I think it comes down to having a good use case.
I've gotten deep into weightlifting/bodybuilding over the past couple of years, and that's the kind of hobby where micro-optimizations and data tracking can have a pretty big impact on results (and sort of necessary, you can't fly blind with things like diet, especially)
E.g. I track and weigh everything I eat, take body measuraments on a weekly basis, Dexa scans every few months, etc - for me it's worth it because I know what I want to do with the data. If I didn't have a goal, all that tracking would clearly be overkill.
How long have you been tracking? Can you share an insight you've had from your data?
I've been weight lighting for ten years and initially tried to track things (down to how many reps I did of which exercise, with how much weight) and quickly came to the conclusion that is want worth it for me.
The goal in bodybuilding during a gaining phase is to be in a very slight calories surplus (200-300 calories above maintenance, at most) to maximize the amount of time you're building muscle before you need to cut again (bring calories back to a deficit to shed body fat).
Tracking scale weight is difficult because shifts in water weight and hydration can swing the scale 5+ pounds in either direction without any change in body fat. So I pair scale weight with a 7-point skin caliper measurements taken on a weekly basis, along with waist circumference, in order to infer whether body fat is trending up or down. And also take weekly progress photos of 6 angles/poses with consistent lighting, which I share with a coach.
And then you pair that with weighing and logging everything you eat, and you can make small adjustments to your meal plan on a monthly basis to try to stay in that 200-300 calorie per day surplus for as long as possible. (Although most bodybuilding coaches adjust diet based purely on how your physique is changing in weekly check-in photos without the need for measurements, but I like extra data)
> down to how many reps I did of which exercise, with how much weight)
I also do this. Track every exercise, every weight, number of reps. It's necessary for knowing whether you're progressively overloading over long periods of time. Progressive overload becomes harder to measure once you're past newbie gains because you can't increase weight every week, so some weeks the goal is just to squeeze out an extra couple of reps. Which adds up over time
This is obviously excessive for 99% of people. But I enjoy doing it as a hobby. I would absolutely not recommend this level of tracking for health reasons (not necessary) - I find enjoyment in the process.
I track the reps weights of every exercise (in my own app). But the historical values are only useful up to last couple of weeks just to now if the general trends go up and what is stalled. Unless your goals are the numbers themselves and not health, I don’t think there is a reason to track everything. But it is fun.
Right, but you don't necessarily need to spend hundreds of hours to capture most of this; the data is already out there. If there were a tool that could collect it all in one place and give you insights with minimal effort that would be pretty neat.
> Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
Not sure if in your case the data was critical, since the doctor likely would have just had you wear a monitor for a while after to come to the same conclusion.
I never spent as much time as OP on this but I did collect a lot of data during peak quantified self. I liked having automatic data collection and being able to see trends, for example. I stopped because of a few issues, not related to time:
* Hardware companies went out of business, stopped supporting devices, etc. It became obvious that there was no long term commitment to make good quality hardware that lasted a long time.
* Many devices and/or data collection was consolidated big, data hungry companies Google and Apple. Competitors have similar anti-consumer uses of data. I don't want any of these companies to have my data.
* Related to the last one, limited to no offline or local only data collection.
It is very hard to gather most of this data with off the shelf hardware and keep your data private.
You're equating two different things here: "building a custom web application" with "quantified self". These are two entirely different things.
Felix's statement isn't a condemnation of quantified self. I think that sometimes, when you're applying algorithms that aren't well-studied, you get pointless or bogus data. I feel much this way about sleep tracking algorithms from the likes of Apple and Whoop. Vitals, too, although it seems remarkably good at detecting when I'm about to get sick.
As a person in the older demographic of HN, having an Apple Watch quietly collect useful data over years and store it in a database has been immensely useful in helping resolve medical issues.
I used to really be into QS, and if I'm training for a marathon, I'm still studying graphs and drawing conclusions. I'm still into QS, but now I just silently log data to Apple Health and use it for medical histories or to identify certain trends (vo2max, cardio fitness, blood pressure, etc) a couple times a year.
I had a similar epiphany a few years back when I started wearing a step-tracker/sleep monitor.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
Eh, I found several interesting things from various tracking tools. Take a nap? Sleep is destroyed this night. Exercise in the evening? Same. Not something I’d pay attention to without noticing the chart afterwards.
There’s also the motivation factor. I’m not sure of the total %, but I certainly did some exercising just to fill the daily goal. Nothing life-changing, but for the price of a cheapo apple watch se once every 5 years or so, more than worth it.
It’s not unlike simplistic time tracking on my iphone. I spent a lot of time on bullshit websites. Obviously I knew it was happening, but the sheer magnitude was surprising. It’s akin to acute pain letting you know there’s a health problem vs something brewing in the background that you are vaguely aware of, but have no motivation to truly care about - one is far more noticeable than the other
Same. I had a Garmin for about 6 months and I eventually just stopped wearing it and sold it. Knowing how many steps I took today, checking it several times a day to see if I was meeting my goal, knowing how many vertical feet I skied.. none of this data ended up meaning anything to me.
The best use I got from my Apple Watch was to use the companion app of my gym routine tracker (to track current loads and personal best) and play music so I didn't have to bring the phone to the gym.
That was it, I got extremely annoyed by notifications so over time just disabled them. Also for some reason the heart rate monitor glitched a couple times, got alerts about my BPM at 180+ while I was sitting on the couch.
Eventually I just stopped using it and now sits in some drawer.
Being aware, and being aware that you are aware are very similar things that are subtly different.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
> it did not justify the hundreds of hours I invested in this project.
I agree with this but minimizing the cost changes the ROI.
Personally, I've discovered useful insights tracking various life metrics. But I also found quickly diminishing returns after a few weeks or months -- if an association isn't obvious within that timeframe it's either too much effort to isolate or too slow or small to matter.
At various points I've tracked calories, macronutrients, weight, allergens, supplements, sleep, exercise volume, exercise timing, nighttime screen use, spending/budget, air quality, and mood. Now I know what kind of cooking wrecks the air quality in my house, what foods I don't digest well, what various protein/carb/fat ratios look like on my plate, how much effort it takes to improve fitness, that exercise in the morning or early afternoon improves my sleep while exercise in the evening harms it, and that any alcohol or caffeine wreck my sleep while screens at night have no measurable effect. But once I understand the associations I can alter my behavior and move on.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I would agree that continuing to track metrics every day long after they've stopped yielding new insights is often compulsive behavior. But I think that's an argument for time-boxing experiments, not necessarily avoiding them altogether.
Agreed, I have tendency to do this to “fix” something in my life, after doing it for 2 weeks I already know, just the consciously pay attention to something even without recording make it very obvious what’s going on.
For me it's about managing ADHD-like (never diagnosed, and I don't care to assume) patterns, coupled with self-accountability. I have a self-improving dashboard (it kicks off Claude to propose additions based my positive/negative feedback on past attempts, and then builds them) that feels quite helpful. E.g. one of the things it added was fitbit integration that shoves my step count and resting heart rate in my face every day, and it's helped me drive my step count back up towards where I want it to be.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
It's important to tell the readers how long you've been doing this - especially to those that also manage ADHD or ADHD-like symptoms.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
I have data going back many years, but this recent effort is a few months old at this point. It's however notably an iteration that has reduced the amount of time I spend on collating and reviewing data, by automating away most of my previous manual effort, including most of the coding, and so I do suspect I'll stick with this for a very long time. A significant part of the prompt to the Claude part of it is to focus a substantial portion of plans on how to automate little things that costs me time, and it's doing a decent job at that.
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
> however retrospectively, it did not justify the hundreds of hours I invested in this project.
Trying to extrapolate this conclusion to the entire "quantified self" movement is not correct. The issue is the time cost, not the act itself. If a trusted company came along (as if...) that sucked up this much data to allow you to answer these questions with minimal effort, I'm sure this would be a different story.
Anything at the fringes of tech with no tried and true solution requires hundreds of hours of effort. The author's conclusions are also personal, there are other styles of living and conclusions to be drawn that change the calculus on whether to do it or not.
I ended up doing something similar to this project a year or so ago: for nearly an entire year, I tracked every single thing I ate or drank.
Who knows how many hours I spent scanning nutritional facts on the backs of boxes, estimating amounts of liquids, and even tracking sips of water. And weighing myself! Thank goodness I used a "smart" scale at least, and I didn't have to worry about carefully inputting my weight to an app each time.
But the whole project was an exercise in perfectionism. "I have to remember this sandwich and log it the next time I'm alone" made me anxious, but once I logged it, I felt a sense of completion. The database and my personal history are now at nirvana. Everything is complete.
All for me to learn things every human alive knows today: eat more food and you'll gain weight; eat less food and you'll lose weight. Yes, I can now tell you the exact average difference in calories I'll eat, statistically speaking, on a day that I have adderall in the morning vs a day that I don't. Yes, there's a similar (but much smaller) difference in average calories per day if I have caffeine in the morning as well. And I can tell you that I generally eat an additional 200-400 calories per day on a Fri-Sun than on a Mon-Thu. Wow, groundbreaking.
I've always had a lot of water, but matching foodanddrink.csv to my HealthKit data showed that I have more water on days that I walk more steps. Mildly interesting to see it written out for me? I guess. But was it worth cataloguing every cup of water? Absolutely not.
Was any of [gestures broadly at me pulling out my phone and cataloguing each item I ate] necessary to learn that? Of course not. It gave me a chance to look back at the database and say "Wowee! I did that! Every day for a year, wow, I'm so cool!" and not much else.
It's pure armchair psychology, but this type of project always makes me think about anxiety. Who really needs this level of self observation and control? At the same time, I really enjoy reading about it and I find the window into somebody else's world intriguing.
extensive tracking of self-related metrics to improve ones health is the equivalent to reading tons of self-help books to improve ones life/social skills/...
We already mostly know what makes people happy/healthy: personal connections, physical activity, healthy diet and some sort of purpose/goal in life that goes beyond day-to-day activities.
The problem is that these things generally require (hard) work and can be unpleasant sometimes, so humans do what humans do and spend unreasonable amounts of time doing the more pleasant things such as reading and gathering info rather than applying these and what they already know.
(That's not to say that a project like this can't be fun or lead to insights, especially across longer time spans, but i feel like all of the questions in the first paragraph have fairly obvious answers if you know yourself at all, that don't require extensive tracking of stats to get)
I have built a much more limited version of this that combines my journals, sketches, photos, geolocation etc. I found it useful for noticing patterns in my behaviour and in my irrationality. Journaling helped the most, because recording my feelings means acknowledging them and reviewing them.
Above all, it's just interesting. I enjoy reading about the day-by-day progression of a crush or my brutally honest feelings about a trip that produced stunning pictures. It weaves nuance into my history.
I too learned this the hard way. I still haven't figured out why this is the case - like, is the inherent incidental irreducible complexity of human life too high, so it dominates majority of our actions?
yep, I do a simple version of this in Google Sheets. Very useful to be able to "Ctrl-F" your life, especially when combined with Google Maps location history.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
Quantified Self, at least in the intended form, is focused on testing specific theories, not on collecting large amounts of data and trying to find something interesting in them.
If you are not sick, tracking your life like this is, like he said, is useless. Enjoy your life!
On the other hand, if you are sick like me, charting your long term heath data from doctors visits and photographing skin issues can lead to great discoveries. I have been diagnosed with Erythrocytosis and a susceptibility to mycobacteria infection which caused pulmonary nodules and skin lesions. Only after showing my data collection to my doctor. Since I have mental illness they constantly over looked my physical issues so I needed hard data to convince them of my ideas.
For those curious, I have an minor IL12B deficiency and a partial immune deficiency leading to mildly elevated levels of DexoyATP which is partially corrected with zinc supplements.
This is disappointing.... Last year i noticed large chunks of my life were being monitored via many spreadsheets and i endeavoured to bring them all together in to one Oracle DB. My plan was to eventually put some ui and graphing on top very similar to OP but seeing this is making me think twice.
A simple back of the envelope calculation shows that Felix causes between 70 and 110 tonnes of CO2 emissions per year just from flying.
Paris accord says 1.5t per person per year, from all activities, Felix's flying alonre is ~10-15x current European yearly per person emissions and ~50-75x those compatible with +1.5C.
Most people don't spend a lot of time wistfully considering their CO2 usage, myself included. The religious zeal by which people feel the need to tell me about the ever-warming planet is honestly more off-putting than most actual religions.
Of the very few "f*cks" I can give in my life, I prefer to spend mine as I choose rather than being scolded for not giving mine to the pressing issues that others deem important.
If you want to reduce air travel for environmental reasons, then tax it more.
Shaming individuals doesn't seem to be productive or helpful.
Air travel works for people if the benefits outweigh the costs. The only thing that changes behavior is to change the costs.
And even if costs were 10x there are still plenty of people who will fly tons, because it would still be economically productive. There are always going to be people who fly 10x more than others, because certain jobs and roles simply require it.
You wrote one of the solutions as if it conflicts with the other one.
Let's raise the tax on an activity according to its negative side effects, while pointing out individuals that do a lot of it and dont take personal responsibility.
> Shaming individuals doesn't seem to be productive or helpful.
I don't see how much support from history for that viewpoint. Some examples of positive societal change driven in part by shaming individuals: drink-driving, civil rights, sexual harassment, automobile safety, the slave trade, McCarthyism.
All those cases also have huge penalties or effective costs associated with them. Is there an accurate "shame first, then penalties came later" stand point?
Automobile safety in my life has only changed after fines.
Sexual harrassment still happens and doesn't seem to be helped by shaming someone as much as firing them. Though we often don't have the guts or legal backing to publically shame someone.
There is a section comparing flight emissions to US citizen average total emissions. This might make him feel good, but only 30% of an average American's emissions come from transportation and just a sliver from flying, so it's very likely his total emissions are much higher.
Exactly, I stopped reading when I saw the flying stats. There are people who still haven't clocked where our climate is headed.
I get that you may have to see family abroad or maybe indulge for a holiday, but this is "I'm using an airplane to commute" kind of level.
And here I am trying to book my train tickets to go to London instead of flying even though it costs three times as much just to avoid a few kg of CO2 (among other things), it's making me angry.
Depends where you take the train to London, it is a much nicer experience anyhow than going to airports and people should consider that as well (ignoring climate stuff)!
On the price, the very annoying thing is that fuel for planes is not taxed! Changing this would require quite some effort (falls under some specific laws, that are old and nobody wants to touch, etc.) but I think everybody should just ask "honest tax on fuels!" as this will make less people say (or thin) "but climate change is a hoax". Planes are just unfair competition to other transport due to taxes!
I agree re: fuel taxes, but it’s a complete nonstarter: passengers would be voting against their wallets, and airlines would lobby against it since it’s a vote against their business model.
Eh, other people throw litter on the floor and rob elderly folks in their homes. Those people hardly ever get caught, but neither you nor I are are going to start copying their actions.
Isn't this a drop in the ocean? Why would any 'normal' person forgo flying? How much CO2 emissions have 'world leaders' produced going to summits, or Taylor Swift and her fans flaying to concerts or war flights?
Couple of things: 1) NO ONE is suggesting any one forego flying altogether, or skipping their once-a-year overseas vacation or periodic family visit. 2) THIS level of flying is not normal and is exactly the kind of harmful behavior people have in mind when they complain about frequent flyers. 3) Whinging about summits and Taylor Swift is just a bad faith red herring argument. Obviously less flying is better, no matter by who. To the extent it's related to the topic at all, it bolsters the case for less air travel.
Everything any one person does is a 'drop in the ocean'. Thankfully, we organize and do things collectively very well - it's in our fundamental nature going back to non-human ancestors, and there is a long, rich history of how much we accomplish. Alsmost nothing that has ever been accomplished has been done without a lot of people doing it together.
The general concern around Taylor Swift's emissions has always struck me as shortsighted. Her Eras tour is estimated to have generated around $5bn in economic uplift in the US, at an estimated 10,000 tonnes CO2e for her personal travel. Even if the total footprint is higher, that is thousands of times lower than the emissions intensity of an industry like fast fashion. From an environmental point of view, attending a Taylor Swift show is a much less carbon-intensive way to spend your money than ordering from Temu.
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
Strongly agree with this. I’ve been using Apple’s “mood” log for about two years now, and it is extremely helpful for me to have a concrete view of the history of my general affect.
“This entire month I’ve been feeling good, I want to pinpoint why,” or “it’s clear since stressor X entered my life, my affect is lower; how can I resolve this?”
These long term trends are harder for me to track without data. It might be easy for others, but not me!
As someone with Schizoaffective Disorder Bipolar Type, if you are not diagnosed with a mood disorder, tracking "swings in you mood" when you have no clinical disorder seems like a disorder of its own.
I have had people tell me they were "manic". Then I showed them videos I took when I was manic and they see what I mean when I tell them they are not manic.
We have come to a place where we do not want even normal fluctuation in mood, and that is a illness of its own, but it is a cultural illness.
Maybe for some it's a lot more extreme than for others, but even if it's not so dramatic as to be categorized as a mental illness wouldn't you want to know if, say, there were a direct correlation between whether you went for your morning run and your mood later in the day?
Is this something that needs to be tracked to bring into your awareness? We have a memory storage device sitting on top of our spine. When I drink I feel drunk. Easy. If the change is noticeable you will notice it and remember it.
I am just trying to save you time and escape the cycle of "optimizations" which is where all this data logging leads.
Yes? As I wrote, the mere act of writing down your feelings forces you to acknowledge them and see patterns in them. Sometimes while writing something down I realise "wait, I've been through this before" or "every time this person is around, I feel this way". It helped me be more self-aware, for my own good.
It turns out that our memory storage device uses a very lossy form of compression. Memories get simplified and distorted over time. Heck, I can't even remember when something started hurting, so how should I notice a year-long pattern of thinking around a certain topic?
As someone who dislikes being tracked, I find it disconcerting to have this level of data stored in 3rd party services, but joining up multiple services to give one coherent picture is cool. How cool is questionable, trying to correlate health data with location data is going to give a strange picture, and I question how relating health data to weather is useful. (Do I have lower blood pressure on rainy days?)
Forgive me for I am being sceptical. The might be some insight here I have not considered, but I'd feel a lot more comfortable if it was all self-hosted / self-collected data.
Flying stats dashboards always amuse me. I get it, for the non-pilot it's kind of like a status thing, "Well I traveled X times in 2025!". As a pilot though, I have gobs of stats I could put up there but from flying for 15 years I realize there's not really anything meaningful in there other than "Gee whiz, I flew a little more/less than last year." I know some other professional pilots do track some of their stats a little closer as they try to optimize for hours flown:hours paid but I've never cared to hyper optimize my schedule in that way.
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
The hardest part of this kind of personal data system is retrieval not storage. At some point you have more data than fits in a prompt, so you need to decide what's relevant per query. Did you build any ranking or filtering logic, or do you query specific tables directly?
> Probably obvious for many, but I didn't realize ACs don't transport any air into the room, but just moves it around
I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
The value rarely shows up where you expected it to.
I kept a rough log of my sleep and mood for about a year with no specific goal. Mostly forgot about it. Then I had a weirdly bad few months and went back to look — turns out there was a pretty clear pattern I would've never noticed in the moment.
Maybe the framing of "was it worth it" is the wrong question. It's less like an investment with a return and more like keeping receipts. Useless 99% of the time, then suddenly you really need one.
Capturing data was the main bottleneck I ran into years ago when I tried something similar with a little MS Access database. I spent lots of time making it pretty and defining fields and data types, but actually typing stuff in? I lost interest quickly.
It's possible the friction could be reduced here by having some kind of Generative AI try to help capture data, but then you'd have to verify that it was being done correctly... honestly, I think it's simply not practical for most people to do this.
Real life is messy. How much time you want to spend on recordkeeping to make it seem less messy or make you feel like you have more control is up to you. But sometimes it's better to embrace the mess and let go of control, in my opinion. Chances are, no one's going to care about whatever you do here in 100 years. YMMV as always.
> Back in 2019, I started collecting all kinds of metrics about my life. Every single day for the last 3 years I tracked over 100 different data types - ranging from fitness & nutrition to social life, computer usage and weather.
I know this is the type of person i would not like.
You'll get crap here for it, but I agree. This is someone that has so little to worry about that he has to look for things. Then you look at his travel habits and jobs and you realize he's just one more rich bored guy showing off as best as possible.
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
"GitHub Open Source Contributions" aside, everything else falls into the consumption/intake/internal categories; thin data on production/output/external ones.
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew.
If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
This is fascinating. One question I always have with large personal datasets like this: at what point did you start getting genuinely surprising insights versus confirming things you already suspected?
380k datapoints sounds incredible but I imagine the real challenge is turning that into decisions that actually change behavior
Wow, this sounds amazing! I am a statistics and chart lover, I love tracking various types of data, but this? I can't even imagine how much time it must have taken to input all the data. Huge respect to you! Keep going.
Waiting for a neckband/choker wearable that can take transcription from my vocal cord nerve, so I can operate a chatbot silently. Yes, scientists have made impulse-to-speech lab equipment.
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
I had an idea similar to this where you could add information about yourself and answer daily questions and get paid by companies who access this data. This could be an ethical way to share information benefitting all parties.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
/edit: quantified, not qualified
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
I've gotten deep into weightlifting/bodybuilding over the past couple of years, and that's the kind of hobby where micro-optimizations and data tracking can have a pretty big impact on results (and sort of necessary, you can't fly blind with things like diet, especially)
E.g. I track and weigh everything I eat, take body measuraments on a weekly basis, Dexa scans every few months, etc - for me it's worth it because I know what I want to do with the data. If I didn't have a goal, all that tracking would clearly be overkill.
I've been weight lighting for ten years and initially tried to track things (down to how many reps I did of which exercise, with how much weight) and quickly came to the conclusion that is want worth it for me.
Tracking scale weight is difficult because shifts in water weight and hydration can swing the scale 5+ pounds in either direction without any change in body fat. So I pair scale weight with a 7-point skin caliper measurements taken on a weekly basis, along with waist circumference, in order to infer whether body fat is trending up or down. And also take weekly progress photos of 6 angles/poses with consistent lighting, which I share with a coach.
And then you pair that with weighing and logging everything you eat, and you can make small adjustments to your meal plan on a monthly basis to try to stay in that 200-300 calorie per day surplus for as long as possible. (Although most bodybuilding coaches adjust diet based purely on how your physique is changing in weekly check-in photos without the need for measurements, but I like extra data)
> down to how many reps I did of which exercise, with how much weight)
I also do this. Track every exercise, every weight, number of reps. It's necessary for knowing whether you're progressively overloading over long periods of time. Progressive overload becomes harder to measure once you're past newbie gains because you can't increase weight every week, so some weeks the goal is just to squeeze out an extra couple of reps. Which adds up over time
This is obviously excessive for 99% of people. But I enjoy doing it as a hobby. I would absolutely not recommend this level of tracking for health reasons (not necessary) - I find enjoyment in the process.
Not sure if in your case the data was critical, since the doctor likely would have just had you wear a monitor for a while after to come to the same conclusion.
* Hardware companies went out of business, stopped supporting devices, etc. It became obvious that there was no long term commitment to make good quality hardware that lasted a long time.
* Many devices and/or data collection was consolidated big, data hungry companies Google and Apple. Competitors have similar anti-consumer uses of data. I don't want any of these companies to have my data.
* Related to the last one, limited to no offline or local only data collection.
It is very hard to gather most of this data with off the shelf hardware and keep your data private.
Felix's statement isn't a condemnation of quantified self. I think that sometimes, when you're applying algorithms that aren't well-studied, you get pointless or bogus data. I feel much this way about sleep tracking algorithms from the likes of Apple and Whoop. Vitals, too, although it seems remarkably good at detecting when I'm about to get sick.
As a person in the older demographic of HN, having an Apple Watch quietly collect useful data over years and store it in a database has been immensely useful in helping resolve medical issues.
I used to really be into QS, and if I'm training for a marathon, I'm still studying graphs and drawing conclusions. I'm still into QS, but now I just silently log data to Apple Health and use it for medical histories or to identify certain trends (vo2max, cardio fitness, blood pressure, etc) a couple times a year.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
There’s also the motivation factor. I’m not sure of the total %, but I certainly did some exercising just to fill the daily goal. Nothing life-changing, but for the price of a cheapo apple watch se once every 5 years or so, more than worth it.
It’s not unlike simplistic time tracking on my iphone. I spent a lot of time on bullshit websites. Obviously I knew it was happening, but the sheer magnitude was surprising. It’s akin to acute pain letting you know there’s a health problem vs something brewing in the background that you are vaguely aware of, but have no motivation to truly care about - one is far more noticeable than the other
That was it, I got extremely annoyed by notifications so over time just disabled them. Also for some reason the heart rate monitor glitched a couple times, got alerts about my BPM at 180+ while I was sitting on the couch.
Eventually I just stopped using it and now sits in some drawer.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
If you could feel it why beed the stats?
I agree with this but minimizing the cost changes the ROI.
Personally, I've discovered useful insights tracking various life metrics. But I also found quickly diminishing returns after a few weeks or months -- if an association isn't obvious within that timeframe it's either too much effort to isolate or too slow or small to matter.
At various points I've tracked calories, macronutrients, weight, allergens, supplements, sleep, exercise volume, exercise timing, nighttime screen use, spending/budget, air quality, and mood. Now I know what kind of cooking wrecks the air quality in my house, what foods I don't digest well, what various protein/carb/fat ratios look like on my plate, how much effort it takes to improve fitness, that exercise in the morning or early afternoon improves my sleep while exercise in the evening harms it, and that any alcohol or caffeine wreck my sleep while screens at night have no measurable effect. But once I understand the associations I can alter my behavior and move on.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I would agree that continuing to track metrics every day long after they've stopped yielding new insights is often compulsive behavior. But I think that's an argument for time-boxing experiments, not necessarily avoiding them altogether.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
Trying to extrapolate this conclusion to the entire "quantified self" movement is not correct. The issue is the time cost, not the act itself. If a trusted company came along (as if...) that sucked up this much data to allow you to answer these questions with minimal effort, I'm sure this would be a different story.
Anything at the fringes of tech with no tried and true solution requires hundreds of hours of effort. The author's conclusions are also personal, there are other styles of living and conclusions to be drawn that change the calculus on whether to do it or not.
Who knows how many hours I spent scanning nutritional facts on the backs of boxes, estimating amounts of liquids, and even tracking sips of water. And weighing myself! Thank goodness I used a "smart" scale at least, and I didn't have to worry about carefully inputting my weight to an app each time.
But the whole project was an exercise in perfectionism. "I have to remember this sandwich and log it the next time I'm alone" made me anxious, but once I logged it, I felt a sense of completion. The database and my personal history are now at nirvana. Everything is complete.
All for me to learn things every human alive knows today: eat more food and you'll gain weight; eat less food and you'll lose weight. Yes, I can now tell you the exact average difference in calories I'll eat, statistically speaking, on a day that I have adderall in the morning vs a day that I don't. Yes, there's a similar (but much smaller) difference in average calories per day if I have caffeine in the morning as well. And I can tell you that I generally eat an additional 200-400 calories per day on a Fri-Sun than on a Mon-Thu. Wow, groundbreaking.
I've always had a lot of water, but matching foodanddrink.csv to my HealthKit data showed that I have more water on days that I walk more steps. Mildly interesting to see it written out for me? I guess. But was it worth cataloguing every cup of water? Absolutely not.
Was any of [gestures broadly at me pulling out my phone and cataloguing each item I ate] necessary to learn that? Of course not. It gave me a chance to look back at the database and say "Wowee! I did that! Every day for a year, wow, I'm so cool!" and not much else.
We already mostly know what makes people happy/healthy: personal connections, physical activity, healthy diet and some sort of purpose/goal in life that goes beyond day-to-day activities. The problem is that these things generally require (hard) work and can be unpleasant sometimes, so humans do what humans do and spend unreasonable amounts of time doing the more pleasant things such as reading and gathering info rather than applying these and what they already know. (That's not to say that a project like this can't be fun or lead to insights, especially across longer time spans, but i feel like all of the questions in the first paragraph have fairly obvious answers if you know yourself at all, that don't require extensive tracking of stats to get)
Above all, it's just interesting. I enjoy reading about the day-by-day progression of a crush or my brutally honest feelings about a trip that produced stunning pictures. It weaves nuance into my history.
A good thing.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
Quantified Self, at least in the intended form, is focused on testing specific theories, not on collecting large amounts of data and trying to find something interesting in them.
See, for example:
https://gwern.net/zeo/zeo#what-qs-is-not-just-data-gathering
On the other hand, if you are sick like me, charting your long term heath data from doctors visits and photographing skin issues can lead to great discoveries. I have been diagnosed with Erythrocytosis and a susceptibility to mycobacteria infection which caused pulmonary nodules and skin lesions. Only after showing my data collection to my doctor. Since I have mental illness they constantly over looked my physical issues so I needed hard data to convince them of my ideas.
For those curious, I have an minor IL12B deficiency and a partial immune deficiency leading to mildly elevated levels of DexoyATP which is partially corrected with zinc supplements.
Paris accord says 1.5t per person per year, from all activities, Felix's flying alonre is ~10-15x current European yearly per person emissions and ~50-75x those compatible with +1.5C.
Of the very few "f*cks" I can give in my life, I prefer to spend mine as I choose rather than being scolded for not giving mine to the pressing issues that others deem important.
Shaming individuals doesn't seem to be productive or helpful.
Air travel works for people if the benefits outweigh the costs. The only thing that changes behavior is to change the costs.
And even if costs were 10x there are still plenty of people who will fly tons, because it would still be economically productive. There are always going to be people who fly 10x more than others, because certain jobs and roles simply require it.
Let's raise the tax on an activity according to its negative side effects, while pointing out individuals that do a lot of it and dont take personal responsibility.
I don't see how much support from history for that viewpoint. Some examples of positive societal change driven in part by shaming individuals: drink-driving, civil rights, sexual harassment, automobile safety, the slave trade, McCarthyism.
Automobile safety in my life has only changed after fines. Sexual harrassment still happens and doesn't seem to be helped by shaming someone as much as firing them. Though we often don't have the guts or legal backing to publically shame someone.
I get that you may have to see family abroad or maybe indulge for a holiday, but this is "I'm using an airplane to commute" kind of level.
And here I am trying to book my train tickets to go to London instead of flying even though it costs three times as much just to avoid a few kg of CO2 (among other things), it's making me angry.
On the price, the very annoying thing is that fuel for planes is not taxed! Changing this would require quite some effort (falls under some specific laws, that are old and nobody wants to touch, etc.) but I think everybody should just ask "honest tax on fuels!" as this will make less people say (or thin) "but climate change is a hoax". Planes are just unfair competition to other transport due to taxes!
Reminds me of the soggy straw memes floating around now. I've been having those why bother? thoughts as well.
What does this have to do with Felix?
Unrelated link: https://xcancel.com/Ryanair/status/776292730179682304
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
“This entire month I’ve been feeling good, I want to pinpoint why,” or “it’s clear since stressor X entered my life, my affect is lower; how can I resolve this?”
These long term trends are harder for me to track without data. It might be easy for others, but not me!
I have had people tell me they were "manic". Then I showed them videos I took when I was manic and they see what I mean when I tell them they are not manic.
We have come to a place where we do not want even normal fluctuation in mood, and that is a illness of its own, but it is a cultural illness.
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
I am just trying to save you time and escape the cycle of "optimizations" which is where all this data logging leads.
It turns out that our memory storage device uses a very lossy form of compression. Memories get simplified and distorted over time. Heck, I can't even remember when something started hurting, so how should I notice a year-long pattern of thinking around a certain topic?
Forgive me for I am being sceptical. The might be some insight here I have not considered, but I'd feel a lot more comfortable if it was all self-hosted / self-collected data.
I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
I kept a rough log of my sleep and mood for about a year with no specific goal. Mostly forgot about it. Then I had a weirdly bad few months and went back to look — turns out there was a pretty clear pattern I would've never noticed in the moment.
Maybe the framing of "was it worth it" is the wrong question. It's less like an investment with a return and more like keeping receipts. Useless 99% of the time, then suddenly you really need one.
It's possible the friction could be reduced here by having some kind of Generative AI try to help capture data, but then you'd have to verify that it was being done correctly... honestly, I think it's simply not practical for most people to do this.
Real life is messy. How much time you want to spend on recordkeeping to make it seem less messy or make you feel like you have more control is up to you. But sometimes it's better to embrace the mess and let go of control, in my opinion. Chances are, no one's going to care about whatever you do here in 100 years. YMMV as always.
https://edwardbetts.com/agenda/trip/past https://edwardbetts.com/agenda/trip/stats
I know this is the type of person i would not like.
The database is the data, collected into a coherent structure.
The database management system is (optionally) software. :)
380k datapoints sounds incredible but I imagine the real challenge is turning that into decisions that actually change behavior
Don’t ask how I know…