TLDR

TLDR: Consider adopting some of the JSOC communication rules for your organisations, and empower your highly skilled staff to communicate across the enterprise and make potentially disruptive decisions to prevent the situation worsening. Your organisation hired them for a reason, so trust them to make decisions without micromanagement. Your existing organisational processes are unlikely to be well suited to a fast-moving cyber incident unless you have an ongoing programme of real-world exercising and purple teaming and have worked through the friction.

Where possible, I’ve included some links to other sources, either as evidence for some of the points or for further reading. This isn’t aimed to be an academic article littered with footnotes and references – it’s aimed to provide food for thought to revamping and increasing the effectiveness of your incident response teams. Academia genuinely has its place, this article isn’t it.

I’ve Got 3 Minutes – Bulletpoints Only

• Get there if you can. Consider getting on a plane and be in the same room if you can. If not, use VTC as much as possible. Email has no place in fast moving incidents. Invest heavily in systems such as I3 Analysts Notebook or Jupyter notebooks (thanks to _@Nakomi at F-Secure Countercept for the pointer to this bit of software). Even Google Docs / O365 will do for written info sharing. Consider having dedicated Slack channels for each incident, with subteams being broken out below that workspace if absolutely essential due to noise. NB: Written before CV19 outbreak).
• Avoid producing written documents. Answer critical information requirements now. Everything else can follow later. Eliminate bureaucracy wherever possible. If it isn’t possible, only assign it the priority it deserves in relation to the situation or find a scribe. Whitepapers and beautiful PowerPoints can follow in the weeks after the incident. Teach and train the wider business to expect ‘less beautiful’ or polished documents during times of crisis, but they will be full of accurate, truly up to date information. Perfection is the enemy of timeliness.
• Empower the experts to speak. It doesn’t matter that they aren’t senior in the company. If they have better visibility of the corporate proxy or the EDR software, or intelligence on where they attacker will move next then let them speak. Regardless of the ‘seniority of the audience’. You hired them because they were good – let them do their job and build understanding across the team. You aren’t there to validate your existence by paraphasing what they told you – the audience cares about the effect your information has, not your efforts.
• Stop having meetings. Have discussions. Do not steal your colleagues time (and degrade your own credibility) through largely pointless meetings without concrete goals whilst an attacker is in your network. If you must have a meeting, introduce it with WHY you are meeting, and the outcome you expect to achieve. Ensure there are no silent attendees – break down barriers and ensure everyone contributes.
• Lean on your vendors. You pay a lot of money for their products, make sure you are getting the most out of them – hammer them with feature requests, implementation best practice and professional service requests.
• Bend out of shape. ‘We can’t do that because of $meaningless_reason and we have never done it was that before’ will not meet the standard.

Background and Further Context

The Joint Special Operations Command (JSOC) structure is a division of the US Special Operations Command (SOCOM), who you may have heard of through their (utterly excellent) work in hunting down terrorists overseas.

Perhaps also increasing JSOC’s fame was the publication of Lt Gen Stanley McCrystal’s excellent book Team of Teams, narrating his time within JSOC (Commander from 2003 – 2008) and the changes he made, as well as WHY and HOW those changes were made. JSOC underwent a significant organisational change in the early 2000s as a result of suboptimal success against insurgents across the world in the Zarqawi network.

This change was brought about due to the current structure and culture not being able to adapt fast enough to the rapidly changing situation, being able to deal with an exceptionally complex enemy network and struggling to fuse more information and intelligence sources than ever before. This situation maps ideally to reacting to cyber incidents. You can read more information on the catalyst for the change in McCrystal’s book (or read some of the key takeaways in a format similar to this), I don’t want to focus (or speculate – having never worked in JSOC) too much on the military aspects of the organisation.

Let’s focus on how the JSOC attitude to communicating effectively, building and sharing understanding across highly skilled and empowered subteams can be applied when reacting to cyber incidents in a modern enterprise (both in terms of size and/or technical complexity).

Security incidents are complex problems – working from the initial detection back to Patient / Device 0, determining the level of compromise, developing courses of action, containing the attacker, eradicating the attacker and remediation can all have tens of steps within each stage. This problem can be likened to the problem that JSOC had before their restructure – the Zarqawi network was very complex, geographically disparate and the information / intelligence available was initially sparse.

Don’t stay in your lane – a unique aspect of our fight is its breadth

Lt Gen Stanley McCrystal

Many organisations have security teams that are spread across multiple sites, or continents. This can allow for 24/7 coverage, but can also lead to a culture of communicating through emails, Jira tickets or emailed PowerPoint decks, as well as time zone differences. Effective communication and embracing technology can help lessen the impact of this.

The aim of the entire staff should be the sharing of information and intelligence (one view on the differences can be found here) and the building of understanding with both technical and non-technical staff. Non-technical staff are especially important as the technical issues are just one part of the wider incident strategy of the company or organisation. Your incident response plan is likely to involve several sections of legal, marketing, compliance, third party partner representatives or regulators who will all need to be brought up to speed and including in decision making. Effective external communications and engagement with government or regulators are equally as important as determining what technique the attacker used for lateral movement. These information sharing technologies can be in the form of always-on VTC in each situation room or pre-existing Teams or Slack channels to allow for face to face communication.

Never underestimate the power of being in the same room for building shared understanding and relationships. The cost of a short notice flight and hotel is likely to be significantly less than an uncontained attacker remaining in the network. Whichever technology you decide to opt for, make sure it is backed up by a solid SLA with reversionary (backup) communications, and use it for all training and simulation exercises so you know what works and what doesn’t for your organisation.

A key component to the JSOC message of empowerment and continual information sharing is the highly skilled members of the team. Military and partner members of JSOC are considered at the top of their specialisms and consummate professionals. It takes time to build up the credibility of a defensive team within an organisation, as well as careful nurture of relationships.

The security team is likely to be granted resources, disruption authority, manpower, budget and change authority if it’s value has been proven previously. In the commercial sector, it is unlikely that the wider business will tolerate a 5 day outage on an important system in exchange for a ‘guaranteed’ eradication of an attacker by a newly formed security team. However, through consistent exercising and high performance, it is possible that the business will be more tolerant of outages due to the accrued successes and credibility of the team.

Every action during ‘peacetime’ by the security team should be nuanced to strengthen relationships and credibility that have been formed previously by colleagues. If you aren’t exercising frequently (as realistically as possible, including ALL non-technical staff and partners) then you aren’t building credibility and future ‘favours in the bank’.

Teams will fail to the level of their training, not rise to their expectation during crisis events.

How to Communicate the JSOC Way

The JSOC rules for communication focus on style, process and tools. Most of this can be mapped directly across to cyber security response scenarios – both organisations deal in highly charged and time sensitive operations.

Style – ensure you are communicating quickly, don’t wait another few hours until you have the 100% picture. Communicate what you have now and add any caveats that are relevant or could affect decision making. Examples of this could be that we know exactly what credentials have been stolen by the attacker and where they are valid for lateral movement, but we don’t yet know the exact nature of the data on each of those affected systems.

Decimate any silos you find through exercising and continual improvement. Seniority has little meaning in a fast moving incident – if that person or other company can contribute, get them involved. Be precise – using words or phrases such as “We’d recommend that ABC does XYZ” can be interpreted by certain regulators as definitive calls to action that will be followed up on ruthlessly.

Process – Start any conversation with why you are there. What understanding do you want. What decision do you need to make? What resources do you need. Setting the context for the discussion (not meeting!) will allow everyone to get up to speed as fast as possible. Tell them why they are there – make sure they know why your information matters to them and vice versa.

Tools – Get on a plane where you can. Where you can’t – get on VTC or Skype/Teams with webcams. Where you can’t – use the phone. Slack and IM has it’s place for swapping information but you can’t build thorough understanding or relationships through it. When you open PowerPoint, realise that a big whiteboard and a webcam is probably just as good for discussion. If you don’t have webcams on your hardened and isolated incident response workstations, be empowered to go out and buy some.

The Synch

JSOC have a synch several times a week, with every member of the organisation attending. This can be adopted to fit with whatever rhythm your incident is working within – perhaps every few hours for a fast moving incident moving and then towards three times a week as the final stages of containment and eradication draw to a close. JSOC synchs are likely to be very wide ranging, with representatives from various intelligence agencies, local national forces, SIGINT feeds, HUMINT feeds, partner nation SF and industry. If you are part of JSOC, you will be at the synch. The complexity and wide ranging nature of a incident can map across to a JSOC synch, allowing us to take advantage of the methodology to improve understanding and analyse the second order effects of what we know.

The timings are dictated by the situation, and are aimed to allow for multiple viewpoints to be expressed and understanding to be gained. In an ideal world every member of the non-technical, executive, security and IT teams would be there – and that is what you should aim for. As a minimum an incident response synch could include all the staff from firewalls/IPS/IDS, AD, endpoints, trusted third parties, virtualisation experts, proxy experts, SIEM experts or analysts as well representatives from Microsoft. It’s not an exhaustive list – it’s more to highlight the value in getting everyone (not just the team leaders) from the subteams together to build understanding and inform onward decision making. It is this difference that can make a synch different from a ‘standup’ – a synch is where everyone is encouraged to contribute and share their knowledge, and find further opportunities to collaborate. A standup is usually a one or two way briefing.

Throughout the synch, make sure that things discussed are useful. This sounds exceptionally obvious but many briefs / conference calls are taken up with ‘summaries of what we have done’. Your existence doesn’t need justification – share new information and encourage subsequent collaboration. Don’t be afraid of putting your experts into the spotlight – the wider business / executives may have questions for them that need answering here and now. Your answer of ‘I’ll check with Joe Bloggs and get back to you’ just introduces a whole load of delay and stops collaboration. Joe Bloggs will be at the synch, so can answer any follow up questions authoritatively. Synchs can be fast paced and wide ranging, so make sure you pay attention. There’s unlikely to be any PowerPoint slides accompanying the synch, so it will require your full attention from start to finish – your Outlook isn’t going to light up – everyone involved in the incident will be at the synch, and normal BAU stuff can wait. The BAU stuff is unlikely to be as important as the incident you are currently running.

Discussions

Where you have no choice but to have a discussion (not meeting!), you can give consideration to running it like a JSOC discussion.

Explain why you are all there – do you need action, decision or more information? The chair of the discussion (who may not be the most senior person – but is likely to be the most suitable person due to the ‘fast and flat’ hierarchy within JSOC) is responsible for driving the agenda, keeping attendees on track and making no apologies for doing so. The attacker has a very small OODA loop and is unlikely to be making decisions via committee. Don’t deprive your colleagues of time and devalue their input elsewhere in the incident. Make sure that any materials to be referenced during the discussion are provided in advance – you are expecting people to turn up to inform decision making – so allow them time for analysis (look into the One Third / Two Thirds rule here if you want further reading) and warn them of any pre-actions – for example, a five minute sitrep on checking for known IOCs in the compromised workstation subnet.

Force collaboration at every point, each person should be an expert in their field and be able to contribute – that is why they are in the organisation. If there is some hidden barrier stopping them contributing (nerves, confidence or impostor syndrome) then support them to get through it. Don’t forget non-verbal messaging, and thank and encourage people throughout the discussion. When you think you need to record decisions in Microsoft Office, reach for a whiteboard and a digital camera. Assign a scribe and get them to share bullet points in Slack / IM with highlights, deadlines and any decisions rather than another Word document. Make sure to @ anyone who was there, as well as anyone else who should have been there. Oversharing is fine, undersharing is not.

Throughout the discussion, make sure that assumptions being made have been validated and challenged. Before assuming that you have blocked the malicious domain at the proxy, make sure you have checked with the satellite offices, as well as for when devices aren’t connected to the VPN. Did you block via IP or DNS too? Challenge your assumptions and validate your changes. If you are unsure whether contributors have grasped something, ask them to readback the content to you in their own words to check understanding. This is especially critical for action points. Follow up after an appropriate period of time to make sure what was agreed is happening. The whole point of the discussion was to inform others, build understanding and enable you to take action – make sure it happens.

Footnote: Incidentally, I noticed whilst writing this that F-Secure Countercept (very good EDR and supporting processes) have collaboration as a pillar of their product.