How I find problems to solve as a staff engineer

(lalitm.com)

144 points | by vanpra 2 hours ago

32 comments

  • wpasc 1 hour ago
    The author notes:

    > One caveat: my experience comes mainly from working on infrastructure and developer tools at large companies, on teams where engineers have a lot of bottom-up autonomy to influence their roadmaps. In a more top-down environment, there may simply be less room to work this way.

    I wonder if the overall trend in tech is that engineers are experiencing less bottom-up autonomy and more top-down controlled environments. I would be curious to see how many tech companies (or the average engineer's experience) have changed from being tech led where engineers have autonomy to being more product management led. My suspicion without evidence is that the overall engineering autonomy has decreased over the years as the culture of tech has (in my opinion) shifted away from tech focus to more business, management, product focus with engineers just as the widgets who are tasked with fulfilling the goals of business, management, product.

    all hypothesis, only anecdata

    • geodel 46 minutes ago
      This sounds about right. At least I feel it acutely in my career. And the more experienced I got instead of more autonomy it has decreased. So to me it seems autonomy is decreasing at a rather fast clip.

      Even outside projects and general work condition it is worse. 10 years back I could just decide when to work from home, or do a few interesting projects show to managers get approval afterwards. Today I have to explain myself to managers and get proper approvals for same thing. And I have been in same place for quite some time so it is not even the case that I have to prove to my new employers first.

      Being not much ambitious I use to think after these many years and multiple successful project my win is to be a kind of made guy in that same mid-level position. Supporting , fixing project which I delivered over years and just take it easy, start day late, leave early until retirement.

      Turns out things I assumed, don't exist. And even if they do they are above my level. Not worked in big tech, no massive RSU based compensation and with AI flattening difference (in employer's view ) between 2 vs 20 years of experience, story is not going to end great for people like me.

      • majormajor 10 minutes ago
        >And I have been in same place for quite some time so it is not even the case that I have to prove to my new employers first.

        Any individual company is likely to atrophy because continual success breeds fear of change, fear of breaking what's already working. Especially if management rotates and new management didn't actually create the success in the first place.

      • cube00 31 minutes ago
        >Supporting , fixing project which I delivered over years and just take it easy, start day late, leave early until retirement.

        My experience has been you'll be regularly moved around to mitigate the "bus factor"

        If you're seen as highy capable you'll be moved on to firefighting duty.

    • hankbond 1 hour ago
      Without providing too much detail, at my current place of employment I find that to be the case.

      In most of my career, especially when in a Staff role, it has been up to me to propose the direction and priority of work within my scope. Usually, this is based on my estimation of the impact to the business (possible new features, security) or operations (devx, efficiency $$$). Obviously I still have to work with product to get things scheduled as they have needs that must be met, but it was as an equal partner. As a very creative person thats usually the space I enjoy the most.

    • zug_zug 1 hour ago
      I agree with this and think that when your company isn't profitable and is running out of runway (due to the end of the zirp) and can't get more, moving toward features that more directly could land sales is a natural top-down move (and totally correct in the abstract).

      However I still see a lot of "work theater" where directors couldn't care at all about changing the bottom line but just care that work that sounds plausible enough is executed in a predictable way that makes them look good (in fact they get pretty disappointed if something happens TOO fast [like half the estimate] as it illustrates they are out of touch the work they are claiming to represent). What I'm saying is that "do more top-down-product" in theory makes sense as a direction, but in practice I usually see it as wasted energy.

    • mcv 27 minutes ago
      I've certainly seen that shift in some places. 8 years ago I started on a project as a freelance software engineer, to build something they couldn't really envision yet. I had a ton of freedom, built a prototype, chose my own tech stack, designed every part of it. A team was built around it, and I migrated to a leadership role where I decided the direction, addressed issues I saw, redesigned parts that needed improvement. We had an enormous amount of freedom, and used it to build great things fast.

      A bit over a year ago, I joined that same company again, as lead of what's technically the same team, this time as an employee, but everything had changed. It was very hierarchical, very top-down, very little freedom, lots of red tape and office politics, and a tedious, slow pace.

    • austin-cheney 1 hour ago
      Most of my career has been spent in JavaScript and then MuleSoft. I have only ever seen top-down authority in about 20 years of doing this, except for when I was at Travelocity early in my career.

      The general perception is that JavaScript work in the corporate world is for young people who have no idea what they are doing and lack the maturity to write original solutions. This perception is common from developers who do other work, management, and developers who primarily perform JavaScript work. If I want to be creative or have autonomy to do anything more ambitious than putting text to screen I have to save it for personal side projects or do unrelated work.

      MuleSoft work was just more of the same with all decisions funneled to strict silos of a select few and everybody else was just a keyboard pressing stooge.

      Yes, that does sound dreadful, but what's worse is that it amplifies the personalities of people who will throw others under the bus for attention and potential rock stars who perform their most diligent work at hiding from that attention.

      • cube00 26 minutes ago
        > more ambitious than putting text to screen

        And you will be ordered exactly where to put that text down to the pixel.

        However your design system will be built around a twelve column bootstrap layout which your designer won't follow because "an experience can't be constrained by columns"

    • girvo 27 minutes ago
      This is my current world, at least. Bit of a shame. But they pay me well enough still and the work is interesting enough that I don’t mind too much.
    • sandeepkd 50 minutes ago
      Its a good thing that the Author pointed out the caveat cause most companies/teams/people do not operate in this mode. Subtle in the details is a fact hidden that the author is able to manage what he wants to do, most people do not get that kind of luxury. This either means that the author has earned enough political reputation based on past work or they are aligned with their management chain.

      tldr; this advice is probably impractical for a bigger chunk of industry.

      • geodel 37 minutes ago
        Agree. It is to author's credit that he shows self awareness about this kind of autonomy and privilege.
    • zmgsabst 56 minutes ago
      I think it was correlated with outsourcing and H1B replacements, because you can’t outsource the leadership required for bottom-up methods.

      Further, foreign business culture is significantly more too-down than American business culture.

    • pydry 1 hour ago
      I think the autonomy was stripped not that long ago and it's making software products far worse.

      It doesnt make sense that this would happen from an economic perspective at first glance - not unless you consider labor (devs), execs and investors to be three competing groups who are vying for power.

    • 0xbadcafebee 19 minutes ago
      It depends on company culture, and there's as many of those as there are kinds of people. I've worked at large, medium, and small companies, in many industries, and there's no common pattern. The people in charge set the tone, and the people are all different.

      If there's a pattern in the people, it's that more people in charge are less capable of doing the job, because they've grown up in a culture of ignorance. The past 10 years has been dominated by "StackOverflow Engineers" promoted to management, and people who read HN clickbait blog posts and believe it's good advice (it's not). They never had the time, training, or mentorship to learn from decades of experience. And Dunning-Kruger keeps them confident that they don't need to change.

      However, on this point:

      "the overall engineering autonomy has decreased over the years as the culture of tech has (in my opinion) shifted away from tech focus to more business, management, product focus with engineers just as the widgets who are tasked with fulfilling the goals of business, management, product"

      That's exactly what engineers are supposed to do: listen to and enable the business. A mechanical engineer does not dictate the shape of the car. The designer tells the mechanical engineer what the car will look like. It's up to the engineer to make it work. However, it's also up to the engineer to tell the business that you can't fit a 600hp engine in a Mazda Miata without seriously affecting handling. It's supposed to be a two-way street. Good management/leadership knows this and enables it.

  • 9dev 2 hours ago
    Funny people should have that problem. I have spent most of my career in the startup space, and my experience has consistently been that the amount of problems to solve is vastly larger than what I can reasonably achieve in my waking hours.

    So I don't find problems to solve, I try to assess which problems are most urgent, or which solution solves several of them at once. Learning to get that kind of prioritisation right to keep all teams and customers happy and productive is what I'm very proud of in my career.

    • sandeepkd 55 minutes ago
      I think this is more relevant reality. At Staff and Principal you have visibility into a lot of fires going around you. What helps is understanding the relative importance of what fire to douse and be at peace with the ones that you cannot control.
    • tikhonj 1 hour ago
      There are a lot of known problems... but there are also a lot of unknown, or at least unrecognized problems. Some of those unrecognized problems are going to be more useful and higher-leverage than recognized problems. The same "being a sponge" approach still helps.
    • Geof25 1 hour ago
      I am doing exactly same thing of creating a solution which can be applied to multiple problems at the same time.

      Sometimes coworkers does not like that because they don't see me working on the problem, but on a tool which will resolve the problem and similar problems from our backlog.

    • markus_zhang 1 hour ago
      You probably pivoted your career towards the startup space so you found it effortless. For others it is basically a castle with a 20m wall.
    • esafak 1 hour ago
      He says: That taught me to let potential problems pile up. Listening the way I do leaves me with far more of them than I could possibly solve, and not all deserve action. Most don’t need to turn into projects the first time I hear about them; waiting can be a superpower.
      • 9dev 1 hour ago
        Yes. He also says: “How do you find problems worth working on?” a senior engineer I mentor asked me recently. My point is that I'm working in a whole different environment, and the challenge - to me - is never finding interesting problems to solve, but identifying the most important problem out of a large pool of known problems.
      • hyperhello 1 hour ago
        “Can be a superpower” is so empty. What does that mean outside of trying to proliferate some cliche?
        • aleksiy123 1 hour ago
          Just means it can be powerful or useful technique that can feel “magical” when employed.

          I wouldn’t overthink this.

          • hyperhello 1 hour ago
            Not overthinking can be a superpower, but writing eloquently can be a superpower, and not trailing cliches behind your pen can be a superpower.
            • xandrius 6 minutes ago
              Now everyone is a bloody writing critic and everything is a piece of master literature that needs to be criticised as such.
    • devmor 1 hour ago
      It’s the same in a large corporate environment. I have a “personal projects” doc of ideas I’ve had to make development experiences better at my current role - it’s a couple hundred lines long.

      I managed to get a couple of smaller tools out recently thanks to having copilot available to churn on them while I spend my time on prescribed work, but it would consume all of my time to even make a significant dent in it.

    • underdeserver 1 hour ago
      This is not distinct.

      Even in a big corp there are always problems to solve. The point is to find problems that both:

      1) are hard enough that someone junior won't be able to solve them alone, and

      2) actually provide value to the company / users

  • ronnier 1 hour ago
    I think almost all of tech is bloated and massive layoffs wouldn’t phase most companies (though it would be harmful to people’s lives so it seems cruel to do this). Fewer people per teams means less context switching and devs own more. They don’t have to look for work, it will be in front of their face. In so many of these big tech companies I’ve worked at I’ve seen too many not having enough work to do. They end up creating meetings and other wasteful things (doc writing) to occupy their time. Managers and directors seem to want big bloated teams, the larger their head count the more they can demand and push for a promo.
    • dominotw 1 hour ago
      yea this was the case even before ai. fighting for scope is 80% of the job now. Ppl like OP dont really need to "Act like a sponge" and figure out "problems to work on" if there is natural demand for stuff they are building work will present itself.

      There is also crazy ladder climbing with all these titles and difference in payscale. So ppl like the author are trying to optimize what a role X is supposed to be doing. Everyone is just the same thing everywhere.

      Everywhere you go its the same ppl, same ladder , same tools same sucking up to boss blah blah. Cant wait to get out of this shit.

  • intoXbox 1 hour ago
    The part I find challenging in the senior-staff twilight is that having deep technical knowledge means I can solve short term problems, just as requests, fast and effectively. The author mentions that you should spend time understanding the frustrations from other teams, but that takes up loads of time and I don’t like being the person who talks and talks but doesn’t push code and ship features. I’d love to hear how others have experienced that
    • zbentley 1 hour ago
      I’ve experienced this. The critical realization for me was that most of those problems which I know I can solve quickly, without even having to write a Jira ticket or groom them into a sprint or whatever, are problems I should let someone else solve.

      Sure, I can do them faster than others and pretty well. But if they’re truly things I can knock out in a day or two, they’re things that other staff can knock out in a week or two, and learn from the experience, and at that size they likely don’t require the standard of quality that I delude myself that I hold myself to.

      At the lead/staff level, it’s far better to look for the kind of problems described in TFA: problems that are complex to identify, and for which the appropriate solutions aren’t always the obvious ones. My time is spent much better looking for those than being a 10x-speed senior engineer or whatever. There are actual senior engineers for that; I’m paid for the kind of work that they don’t or can’t do (yet, and to help model and mentor and train them to be able to do it), not to do the kind of work they already can do—whether I would do it faster/better doesn’t matter.

      It’s case-by-case of course; sometimes a simple issue is critical or obscure enough (or tempting enough to override my iffy-at-best self-discipline) that I’ll jump on it. But I generally try to remember that a lot of the stuff I could look heroic for fixing in a jiffy is probably both not worth my salary allocation in the eyes of my grandboss, not critical time-wise, and a potential learning/accomplishment opportunity for others.

    • jpitz 55 minutes ago
      I think that's a completely natural source of frustration for someone who is used to being able to put hands to keyboard and solve problems quickly, but IMO the higher you climb the ladder, the more you have the opportunity and responsibility to take a longer view and to delegate - which often means that other people are pushing the code.
  • punnerud 23 minutes ago
    “I start to see what’s really slowing people down and what my team or I can do about it.”

    I often tell engineers that employees will never go to an “innovation center” in a company to say that their job could be made obsolete. And by definition the workplace is not 100% effective as long as there is employees. Still there is most likely always more that can be done, so think that the existing people can be automated away and be used in new positions.

    A good way to find problem from top-down is to look for similar jobs done. Often a department with a lot of employees. 10% more efficient for 100 is better than 100% for two, unless those two indirectly slow the rest of the company down.

    And I like to think that when companies expand rapidly it’s easy to spot problems, the contrary is slow growth, then problems is often someone’s job and they will not complain if it’s not stressful. The last part is often solved over time with even more people.

  • CSMastermind 49 minutes ago
    This is all very good advice, but I would caution anyone asking the question at the start of the essay that they probably shouldn't be a Staff Eng. Unless you're at a company where that title is simply a rung on the ladder that doesn't have differentiated responsibilities (there are lots of those out there).

    Every person I've worked with who's been successful as a Staff+ Engineer, promoting them was normally more of a formality since they were already clearly doing the work. Every person I've seen 'rise to their level of incompetence' was striving to get the title/pay bump and looking to 'play the game' to get there.

    If your motivation for solving people's problems is that it will get you a promotion, instead of the fact that you like solving problems, then it's probably not the right job for you.

  • mcv 33 minutes ago
    Not a staff engineer, but this sounds very much like what I do and then am not allowed to follow up on. One particular company where I've been several times as senior or lead developer, I just keep stumbling over problems I would love to take on. I'd love to be a staff engineer there with the freedom to take on these sort of problems, but that's apparently just not how they work.
  • Neywiny 56 minutes ago
    It's interesting how engineer/developer levels are so meaningless across organizations. Where I am, somebody who can only do the work assigned to them isn't senior. That's borderline entry level. Doesn't mean they're a bad programmer or only have 3 hours of experience, just isn't at the higher level. Seems where this person works (Google?) that threshold is at staff
  • napo 1 hour ago
    I worked at the Staff level for a while, I think the key is to report to a director who manages managers. Every time I did I had a good time, projects were easy to define and people were easy to convince. I had the same level with a few managers who were managing other ICs, and that never worked. Other ICs try to compete on tasks, people don't come to you with their needs, you learn about different projects often too late, other teams are territorial and don't really want you to intervene.
  • rjbwork 44 minutes ago
    Also a staff engineer. This is pretty much exactly what I do.

    Often it is a result of suggesting an improvement to some PR for a program or system that someone is doing some work on, and discovering that for whatever reason it doesn't quite work. This tends to lead to a deep dive of really analyzing and understanding the problem and how the piece of the system fits into the overall picture. This analysis often leads to a more structured approach to a problem, placing it in the context of wider industry or CS theory, thus enabling us to leverage prior art and other people who have grappled with similar problems.

  • jofzar 22 minutes ago
    I'm surprised there was no comment here about asking the support team what is actually paper cutting the customer.
  • yipinwong 39 minutes ago
    People ask XY problems and your job is to find what issue they are having, not try to help them with the attempted solution.

    Basically a StackOverflow guidance on XY problem applies to the general problem solving as well.

  • hbarka 22 minutes ago
    It’s difficult to see the difference between a solution looking for a problem and vice versa.
  • cool-RR 57 minutes ago
    > “How do you find problems worth working on?”

    They usually find me.

  • NBJack 1 hour ago
    I think the first problem may be fixing the broken mobile experience. I'm getting black text on a very dark gray background.
    • lalitmaganti 1 hour ago
      Thanks for letting me know; I cannot reproduce this myself on my mobile (it should be white text on a dark grey background) but I pushed a speculative fix which might improve things. Please let me know if it helps or, if not, I would love to know more details so I can fix this!

      Thanks!

  • alexpotato 1 hour ago
    One thing that isn't often mentioned in these discussions:

    Making sure that the work was actually done.

    I've been a Staff Engineer and managed engineers and it's pretty shocking what people consider to be "done".

    A couple examples:

    - Doing a migration to using Tailscale and an engineer claims it's done even though there is just one giant ACL for the whole firm

    - Migrating from one monitoring system to another despite only 80% of the alerts have been migrated

    - etc

    Some of this is business folks creating bad incentives. Some of it is not creating good "success criteria" for projects. Either way, someone has to go through and make sure that both the details and the big picture deliverables landed correctly.

    This being HN, I'm sure someone will say something like "just hire better engineers". I've seen phenomenal engineers make bad choices here due to poor incentive design.

    A perfect example:

    - you reward people for hitting delivery deadlines

    - you punish people when there are outages

    you might think you're pretty smart until you realize the odds of getting yelled at if you miss a deadline is 100% but the odds of an outage are <100%. The EV+ outcome then becomes to hit the deadline even if you know the code isn't ready.

    Again, the job of senior engineers/engineering managers is to make sure these things don't happen by both double checking work and also pushing back on bad incentives.

  • 0xbadcafebee 21 minutes ago
    I have a different problem: I can find all the problems, but I can't solve them, because they aren't within my control. The people whose control they are within aren't interested in solving them (or letting others work on them). The political and psychological games required to get people to just let you solve problems seems like a second job.
  • theideaofcoffee 2 minutes ago
    The biggest problem of a sTaFF eNGInEeR can be solved right away: stop writing about this crap. All of these titles are so meaningless where one organization's staff is another's junior is another's CEO, it doesn't matter because all of the organizational insanity that lead you to your coveted title means jack shit anywhere else. I should write a big ol' think piece from my perspective as a Senior Staff Distinguished Principal! I will get lots of clicks.

    It's pathetic how hard people hold on to titles.

  • AlexMoffat 1 hour ago
    Listen to your customers but ignore what they say. “Treat your customers like a kindergarten class. If they’re all asking for a snack they’re hungry but perhaps they really need a nutritious lunch instead.”
    • alexpotato 1 hour ago
      I spent the early part of my career (mid 2000s) working on a registration system for sports tournaments.

      I had two jobs:

      1. Writing the actual software for the registration website

      2. Going to the events and managing the registration tent where people actually checked in

      Doing 2 gave me a VASTLY better understanding of who the users were and how they thought. e.g. I assumed it would be tech savvy, organized people like me in their mid 20s. It was actually a mix of 40 something team dads who weren't tech savvy at all (b/c mid 2000s) but very organized and teenagers who were the opposite.

      It meant effectively managing two completely different customer bases in the same product.

      Coupled with the fact that cable modems were a new thing but not evenly distributed meant that we had to design the site accordingly.

      At the time, I remember Joe Spolsky mentioning that at Microsoft they did two way mirror usability testing and it totally made sense. I wish more firms did that today.

  • zuzululu 1 hour ago
    My advice as someone working at staff engineer role for 3 different employers remotely and the "let problems accumulate" resonates but different context:

    - don't be too proactive in solving issues that signals you are not busy to your employers.

    - you are not hired to sit around 9-5, what you ship and how it impacts the business bottom line is far more important.

    Especially true when I have to juggle 3 different employers. I will be in a long standup meeting with company A while I am answering slack messages from company B or company C has a deadline that overlaps with another and I'd have to work on at the same time.

    Previously without LLMs this was very difficult but now its manageable, especially with openclaw and hermes doing a lot of lifting. This let me discover a lot of what's discussed in the article naturally.

  • johnbarron 49 minutes ago
    If you spent your whole career inside one corporate ecosystem...do not confuse your rank in that hierarchy, with your rank in the profession. By looking at the some comments here, many are doing that.

    https://www.youtube.com/shorts/FXPh2_BAZD8

  • syndacks 52 minutes ago
    “The shape” — smells like tokens. Each paragraph is too perfect.
  • zug_zug 1 hour ago
    Eh, I'm not going to say this is wrong, but pretty much like all advice, it's kinda cheap without data. Like Dale Carnegie may be very famous and have a book and say "Use people's names all the times and your conversations will be great" but who actually knows if that makes a difference without some controlled outside study.

    I see a lot of people really eager to tell other people how to staff, but a lot of them sure do seem to disagree, and most people I know get to staff anyways without any particular skill after a certain age (title inflation?) including myself.

    My smell test for an engineer is -- are they trying to quantify the size of everything in terms of impact, or are they just reacting to whatever customer knocks on their door?

  • pojzon 54 minutes ago
    TL;DR is that its extremely hard to show staff level expertise if you work at a wrong company. And the amount of companies that still need this expertise is ever decreasing (big corpos with a lot of autonomy)

    This is the real reason why we dont see natural path for more ppl to progress towards Staff level.

    Interestingly noone talks about this. “Be first or bust”

  • hna8hjbqzy 1 hour ago
    [dead]