Yvonne Bosquez’s Post

With the subject of the "Crowdstrike outage" all over the news today. It's good time to remind ourselves that mistakes in IT deployments can have pretty significant, wide ranging consequences. So, what can Project Managers and Delivery Teams do to help reduce and mitigate the impact of incidents? 🖥 Testing - never cut corners on your testing. This is the time where you assure the quality of your deployment. And quality is just as important as your schedules and costs. 🖥 Rollback Planning - before you deploy make sure you and your team have a "break glass plan". If worst comes to worst, can you back out the change and restore the previous environment? What are the steps to rollback, who needs to execute the steps and how long will it take? 🖥 Service Design - make sure your support teams are aware of the change you're making, and the risks. Make sure they know how to support any incidents and people are available to take the calls. If your change is happening overnight, can you use offshore resources, in different timezones?   If you need any support with deployments or your Project Management capability, get in touch with me directly or with Enfuse Group. #ProjectManagement #ITDeployment #Crowdstrike

🚀 Why Site Reliability Engineering is Not for the Faint of Heart 🚀 Being an SRE is a wild ride, full of challenges that keep us on our toes. Here’s why this role demands a special kind of resilience: Responsibility Overload: We’re the guardians of uptime, ensuring the digital world doesn’t crumble. No pressure! 🌐 Complexity Taming: Imagine a jigsaw puzzle thrown into a blender. We’re the ones who make sense of the chaos and put it all back together. 🧩 On-Call Adventures: While others dream peacefully, we’re ready to tackle 3 AM emergencies. Who needs uninterrupted sleep, right? 💤 Rapid Response: When systems falter, we’re the digital firefighters, dousing flames with expertise and maybe a gallon of coffee. ☕️🔥 Continuous Improvement: Just as we master one challenge, a new update rolls out. Back to square one! 📈 Versatility Required: We’re part coder, part sysadmin, part network genius, and part fortune teller. Predicting and preventing disasters is all in a day’s work. 🧙♂️ Balancing Act: Juggling new features, system maintenance, and infrastructure upgrades? It’s like performing a high-wire act while juggling flaming swords. 🎪 Incident Command: Coordinating during an incident is like herding cats—if those cats were also on fire and had conflicting advice. 😹🔥 Kudos to all SREs who thrive in this high-stakes environment, embrace the chaos, and keep the digital world running smoothly, one crisis at a time. You’re the unsung heroes of the tech world! 👏✨ #SRE #TechHeroes #DigitalGuardians #SiteReliabilityEngineering #TechLife #Availability #Reliability #Engineering

I had a wonderful discussion today with a team about what they were doing to improve Ops. Some of the highlights... They were addressing noisy alerts. Over time, we have a tendency to silence or ignore alerts that aren't meaningful or actionable. This is normalization of deviance and we run the risk of missing the real alert when something does go sideways. They were improving documentation to support future incident response. It's tough to triage and troubleshoot when our documentation isn't current or accurate -- especially for new team members. They were working to overcome the fear of being on-call. Improving alerting and documentation certainly help with this. So does creating a humane on-call rotation. So does pulling people out of the feature work while they're in that role. (Work on tech debt instead!) Progress made. More to do.

In times of major IT disruptions, it’s crucial to show empathy and support. Many dedicated professionals from IT specialists to corporate users are tackling significant challenges today. This could happen to any organization. While it’s easy to joke about the situation, let’s avoid quick judgments about the skills of the engineers involved. We may not know the full story. Kudos to all the engineering teams working tirelessly to resolve these issues, and to the support staff assisting users. This incident reminds us of the importance of preparing for unexpected scenarios and continuously improving our infrastructure resilience. #SupportYourEngineers #hugops #crowdstrike

I've read a Medium blog about one company that fired a software engineer because he was too good - https://lnkd.in/dhtWKz_v. I have a lot of examples from my experience where the team is not quite competent to follow ideas. They will do anything to figure out a WORKAROUND solution. Even though they could use this time to think about how to implement the best practices solution. And after that they will cry to CTO, that the engineer that does everything following the best practices is making decisions too fast... What is your experience with engineers, who follow the best practices? #engineer #incompetent #team

I was talking to a friend about my last job. After the conversation, I realised that the job of the on-call engineer is changing a lot. It used to be mostly about dealing with incidents, but now they're also handling lots of other urgent and important stuff that just can't wait until the next day. For example, in my team, our on-call engineer is usually also managing requests from different teams—things that weren't emergencies but still pretty urgent. They had to juggle between a lot of things, and literally be the human shield for the rest of the team to “protect” the team from distractions, letting them focus on the planned work. In some worlds, this kind of work is called “Keeping the lights on” or KTLO in short. It got me thinking: Is this just happening in pockets in a few companies or teams, or are other companies seeing the same shift? How are you all handling this change? Do your on-call folks have to deal with more than just emergencies now? Would love to hear how it's going in other places. Are there any tools, tricks or processes you're using to make things easier - either as an on-call engineer or an engineering manager/lead?

What does it mean to be a VP of Engineering, managing 80 people? How do you approach goal setting, professional growth, managing technical debt and increasing velocity? We are covering all of this in the interview with Greg Stewart, a VP of Eng at Flo Health. Listen it all here:

🔌 Dealing with a Large-Scale Outage: Lessons Learned 🔌 In the world of tech, we've all been there – facing a massive outage that sends shockwaves through our systems. 🌐 I recently had the opportunity to lead my team through such a challenge, and it taught me invaluable lessons: 1️⃣ **Preparation is Key**: Robust disaster recovery plans are a must. Ensure your team is ready to act swiftly. 2️⃣ **Transparency Matters**: Keep stakeholders informed every step of the way. Honesty builds trust. 3️⃣ **Collaboration is Crucial**: In times of crisis, teamwork shines. Foster a culture of unity. 4️⃣ **Learn & Adapt**: Outages are opportunities for growth. Analyze, adapt, and prevent recurrence. Let's turn challenges into opportunities. 💪 Share your thoughts and experiences in handling outages below! 👇 #TechLeadership #OutageManagement #LessonsLearned

The latest update for #FireHydrant includes "Now in beta: alerting for modern #DevOps teams" and "Captain's Log: Diving into our scheduling design". #IncidentManagement #IncidentResponse https://lnkd.in/dtMiNF8

🌙 Late-night outages: A nightmare for developers! Ever been jolted awake at 3 am by an outage alert, unsure if it's a minor hiccup or a major catastrophe? 🚨 Relying on one person in the dead of night adds to the already-high cognitive load. In our upcoming panel discussion with engineering leaders, we uncovered some key takeaways that can revolutionize how we handle incidents: Fostering a Culture of Incident Awareness: Discover how to instil a culture of incident awareness within your wider organization. 🚀 Learn the strategies to make every team member recognize the necessity of swift and effective incident response. This shared responsibility can be a game-changer. Mastering Incident Response: Understand what a well-documented, established incident response process looks like. Delve into the intricacies of a robust incident response playbook and how it can be your guiding light. Breaking Down Silos with DevOps: Learn how to adopt DevOps practices that truly break down silos. Discover the secrets to fostering collaboration and synergy among development, operations, and other teams, ensuring everyone is equipped to handle incidents seamlessly. Minimizing Impact on Productivity: Explore strategies to minimize the impact of incidents on productivity and performance. 📈 Understand how a well-prepared team can swiftly mitigate incidents without causing prolonged disruptions, keeping your systems running smoothly. Let's continue the discussion on October 18 with Mirella Batista, Cheng L., Chris Evans, and Spencer Norman on how these insights can transform late-night incidents and boost developer confidence! 💬 #Engineering #IncidentResponse #ProductivityTips #Culture