Monday 30 November 2015

Do we need to deploy clean up work if there is no additional functionality? i.e no (perceived) business benifit.

Background

We recently did some work to a number of services where all we were doing was removing some old functionality that is no longer used (obsolete code). Removing messages, handlers, classes, tests. I like deleting code, it makes things simpler, less logic to break, less places for bugs to hide. Once we finished, we wanted to get all of this released to prod ASAP. We tested all of the areas affected in a large system level test in the preprod environment, all of the services (and others that had not changed) working together.

But we had push back. 

Push back

The question was: Why do you need to release this clean-up in advance of any further work?
By doing so you are making this an active rather than passive deployment with associated extra risk and double the cost.
If you are removing unused code you can just deploy with the next addition/change to code because by testing that you are implicitly testing that absence of code. Even if the new code isn’t affected, then our deployment checklists cover that situation too – we have already double checked this removal/clean-up of code won’t have an impact on production

Rebuttal

I broke this down into a number of sections. A number of questions that I thought were being asked in the statement.

Question: Why do many releases instead of one, isn’t that more risky?
This is a question from the old skool of thought, releases are big bad things we need to do as few as possible.
Answer: I would say many small releases are inherently less risky.
If (very unlikely but) if something goes wrong it will be less clear what caused the issue. Was it new functionality, or the clean-up work that is the culprit?
If we release now, we know what to monitor over the coming days, and have to monitor less
If we don’t deploy all 8 things, someone else will (at some point and in some cases many months in the future). This poor soul will need to decide if the changes we did need testing, what the consequences of the changes are, and worry about if there are other dependent services to deploy.
Each service is push button, so deploy time is small.
Rollback is not hard if we need it with no business consequences at the moment. In future if we release with other functionality and the changes we have made break something we need to rollback new functionality too.

Question: If we are removing code why the need to release anything? There is no new functionality to release.
I guess this is a question about business value, no new business value no need to release.
Answer: That is true, its mostly removing old code and cleaning up. But its just as critical, almost more so to get this out in a small release sooner, as we may (again unlikely but possible, we are human) have removed something we should not have.

Question: Won't it be more to test doing it twice?
This assumes that we manually test everything on every release. where actually we only test what has changed manually in conjunction with automated testing for the rest.
Answer: We have already done good full end to end testing last week of the 8 things affected. If we wait until next week or the week after we will have to do the tests again as the versions of things to be released will all be different by then, so will need to test the 8 deployable things again full stack = extra 1 day

Other reasons for deployment.
The changes, what we did and why we did it, are still fresh in our minds. The longer we leave it the less sure we are that we will be doing the right things.
Its not critical, but I'd prefer not to do a partial deployment (service-X is going to get released soon) I'd like the rest of the clean-up to be deployed too.
Ideally prod and preprod are as same as possible (for environmental consistency and testing reasons). Any differences between the test system of preprod and prod invalidates other testing efforts. Because in prod services will be integrating with a different version of other services than that are in preprod making like-for-like testing impossible.

Conclusion

I maintain there is business benefit in doing the deployment now, and deploying all 8 services at that. To be fair businesses, managers, stakeholders, even developers (especially senior ones) have all seen their fair share of long deployments, failure and difficult rollbacks. Leading, ultimately to a fear of deployment. So it's natural to want to avoid the perceived risks. But perversely by restricting the number of deployments you are actually increasing the likelihood of future fail.
A core philosophy of the devops culture is to release early and often (continuous delivery). By doing the things you find painful more often you master them and make them trivial, there by improving your mean time to recovery.

The business benefit is ultimately one of developer productivity, testability and system up-time

Tuesday 17 November 2015

Splunk alerts to slack using powershell on windows

We use Splunk to aggregate all the logs across all our services and APIs on many different machines. It gives us an invaluable way to report on the interactions of our customers through new business creation on the back-end servers running on NServiceBus, to the day to day client interactions on the websites and mobile apps.

We have been investing in more monitoring recently as the number of services (I hesitate to use the buzzword micro, but yes they are small) is increasing. At present pace I'd say there is a new service or API created almost each week. Keeping on top of all these services and ensuring smooth running is turning into a challenge, which splunk is helping us to meet. When you add service control, pulse and insight from particular (makers of NServiceBus) we have all bases covered.

We have recently added alerts to splunk to give us notifications in slack when we get errors.

The Setup

We are sending alerts from splunk to slack using batch scripts and powershell.

Splunk Alerts

First set up an alert in splunk, This splunk video tells you how to create an alert from a search results. We are using a custom script which uses arguments as documented here. Our script consists of 2 steps a bat file and a powershell file. The batch file calls the powershell passing on the arguments.

SplunkSlackAlert.bat script in C:\Program Files\Splunk\bin\scripts
@echo off
powershell "C:\Program` Files\Splunk\bin\scripts\SplunkSlackAlert.ps1 -ScriptName '%SPLUNK_ARG_0%' -NEvents '%SPLUNK_ARG_1%' -TriggerReason '%SPLUNK_ARG_5%' -BrowserUrl '%SPLUNK_ARG_6%' -ReportName '%SPLUNK_ARG_4%'"

SplunkSlackAlert.ps1 lives alongside
param (
   [string]$ScriptName = "No script specified",
   [string]$NEvents = 0,
   [string]$TriggerReason = "No reason specified",
   [string]$BrowserUrl = "https://localhost:8000/",
   [string]$ReportName = "No name of report specified"
)

$body = @{
   text = "Test for a parameterized script `"$ScriptName`" `r`n This script retuned $NEvents and was triggered because $TriggerReason `r`n The Url to Splunk is $BrowserUrl `r`n The Report Name is $ReportName"
}

#Invoke-RestMethod -Uri https://hooks.slack.com/services/AAAAAAAAA/BBBBBBBBB/CCCCCCCCC -Method Post -Body (ConvertTo-Json $body)

Slack Integration

You can see the call to the slack API in the invoke-restmethod, the slack documentation for using the incoming web hook is here. there is quite a rich amount of customization that can be performed in the json payload, have a play.

Before you can actually use this you must first setup slack integration as documented here which requires you to have a slack account.

The fruits of our labor:


All the script code is given in my gist here.

Credits:

Thanks to my pair Ruben for helping on this, good work.

Tuesday 3 November 2015

Developer podcasts v2

A couple of years ago i wrote a blog post about podcasts for developers, this is a follow up as I've now got substantially more. That and a couple of my colleges have asked for my list recently.

Programming


.NET Rocks! : Feed Url http://www.pwop.com/feed.aspx?show=dotnetrocks&filetype=master
Adventures in Angular : Feed Url http://feeds.feedwrench.com/AdventuresInAngular.rss
All Chariot Podcasts : Feed Url http://chariotsolutions.com/podcasts/show/all-shows/feed/
All Things Pivotal : Feed Url http://pivotalsoftwarepodcast.libsyn.com/rss
Azure Friday - Channel 9 : Feed Url https://channel9.msdn.com/Shows/Azure-Friday/RSS
CodeChat (Audio) - Channel 9 : Feed Url https://channel9.msdn.com/Shows/codechat/feed/mp3
Coding Blocks | Software and Web Programming / Security / Best Practices / Microsoft .NET : Feed Url http://www.codingblocks.net/podcast-feed.xml
Debug : Feed Url http://feeds.feedburner.com/debugshow
Devnology Podcast : Feed Url http://feeds.devnology.nl/DevnologyPodcast
DevRadio - Channel 9 : Feed Url https://channel9.msdn.com/Blogs/DevRadio/RSS
Full Stack Radio : Feed Url https://simplecast.fm/podcasts/279/rss
Functional Geekery : Feed Url http://www.functionalgeekery.com/feed/mp3/
Hack && Heckle : Feed Url http://feeds.feedburner.com/HackAndHeckle
Hanselminutes : Feed Url http://feeds.feedburner.com/Hanselminutes
Herding Code : Feed Url http://herdingcode.com/feed
Javascript Jabber : Feed Url http://feeds.feedwrench.com/JavaScriptJabber.rss
Jesse Liberty - Silverlight Geek : Feed Url http://feeds.feedburner.com/JesseLiberty-SilverlightGeek
MS Dev Show : Feed Url http://msdevshow.libsyn.com/rss
NodeUp : Feed Url http://feeds.feedburner.com/NodeUp
PowerScripting Podcast : Feed Url http://feeds.feedburner.com/PowerScripting
Radio TFS : Feed Url http://feeds.feedburner.com/radiotfs
Ruby Rogues : Feed Url http://rubyrogues.com/podcast.rss
RunAs Radio : Feed Url http://www.pwop.com/feed.aspx?show=runasradio&filetype=master
Simple Programmer Podcast : Feed Url http://simpleprogrammer.libsyn.com/rss
Software Engineering Radio - the podcast for professional software developers : Feed Url http://www.se-radio.net/rss
STLTechTalk Podcast : Feed Url http://stltechtalk.libsyn.com/rss
The Azure Podcast : Feed Url http://feeds.feedburner.com/TheAzurePodcast
The Cognicast - Cognitect Blog : Feed Url http://feeds.feedburner.com/cognicast
The Java Posse : Feed Url http://feeds.feedburner.com/javaposse
The Static Void Podcast : Feed Url http://www.staticvoidpodcast.com/feed/podcast/
This Week On Channel 9 (MP4) - Channel 9 : Feed Url http://s.ch9.ms/shows/This+Week+On+Channel+9/feed/ipod
ThoughtWorks : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:94605026/sounds.rss
WebDevRadio : Feed Url http://webdevradio.com/feed/
Windows Weekly (MP3) : Feed Url http://feeds.twit.tv/ww.xml
YAPP: Yet Another Programming Podcast : Feed Url http://yapp.audio/feed/yapp

Dev-ops


Arrested DevOps : Feed Url http://feeds.podtrac.com/VGAulpN7MY1U
DevOps Cafe Podcast : Feed Url http://devopscafe.libsyn.com/rss
Devops Mastery : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:79143337/sounds.rss
DevOps.com : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:87329161/sounds.rss
Ops All The Things! : Feed Url http://opsallthethings.s3-website-us-east-1.amazonaws.com/podcast.xml
The Food Fight Show : Feed Url http://feeds.feedburner.com/TheFoodFightShow
The Ship Show : Feed Url http://theshipshow.com/podcast.xml

Developer related


Developer On Fire : Feed Url http://feeds.feedburner.com/developeronfire
Get up and CODE! : Feed Url http://feeds.feedblitz.com/getupandcode
Mastering Business Analysis : Feed Url http://masteringbusinessanalysis.com/feed/podcast
Programmer Vs World : Feed Url https://programmervsworld.wordpress.com/category/the-podcast/feed/
Startups For the Rest of Us » Episodes : Feed Url http://www.startupsfortherestofus.com/category/episodes/feed
The Security Influencer's Channel : Feed Url http://contrastsecurity.libsyn.com//rss

Agile


Agile Instructor - Coaching for Agile Methodologies such as Scrum and Kanban : Feed Url http://feeds.feedburner.com/AllThingsAgile
Agile NYC : Feed Url http://feeds.feedburner.com/AgileFM
Agile Weekly Podcast : Feed Url http://integrumtech.com/feed/podcast/
The Agile Coffee Podcast : Feed Url http://agilecoffee.com/feed/podcast/
This Agile Life : Feed Url http://feeds.feedburner.com/thisagilelife/podcast

Non tech


60-Second Mind : Feed Url http://www.scientificamerican.com/podcast/sciam_podcast_i_psych.xml
99% Invisible : Feed Url http://feeds.99percentinvisible.org/99percentinvisible
All items | LSE Public lectures and events | Audio : Feed Url http://www.lse.ac.uk/assets/richmedia/webFeeds/publicLecturesAndEvents_iTunesRssAudioOnlyAllitems.xml
Freakonomics Radio : Feed Url http://feeds.feedburner.com/freakonomicsradio
Friday Night Comedy from BBC Radio 4 : Feed Url http://www.bbc.co.uk/programmes/p02pc9pj/episodes/downloads.rss
Haute Couture Podcast - Claudia Cazacu : Feed Url http://claudiacazacu.podomatic.com/rss2.xml
Monstercat Podcast : Feed Url https://www.monstercat.com/podcast/feed.xml
NPR: Invisibilia Podcast : Feed Url http://www.npr.org/rss/podcast.php?id=510307
NPR: TED Radio Hour Podcast : Feed Url http://www.npr.org/rss/podcast.php?id=510298
Planet Money : NPR : Feed Url http://www.npr.org/templates/rss/podlayer.php?id=93559255
Radiolab from WNYC : Feed Url http://feeds.wnyc.org/radiolab
RI Blog : Feed Url http://www.rigb.org/blog.ajax
TEDTalks (audio) : Feed Url http://feeds2.feedburner.com/tedtalks_audio
TEDTalks (video) : Feed Url http://feeds.feedburner.com/tedtalks_videos

Notes


Bear in mind that some of these podcasts are no longer active. I've kept them in my list because i find past episodes very relevant to the here and now. you can search back to find relevent episodes on anything you care to think of, super useful.

Incidentally my current podcast player of choice is podcastaddict, variable speed playback and great control of how to download files, brilliant search functionality for local podcasts and its very easy to search for and add new podcasts. use it :-)

My opml file extracted from podcast addict is located here: https://gist.github.com/DamianStanger/9606fcf3fb09cc0bd87f