SOC are people. Downloading an Expert or How to Become a Level 20 Analyst

In a previous article, we talked about finding and training engineers for the first line of a cyber attack monitoring and response center. Today we’ll talk about finding and training for the second line - analysts who investigate atypical incidents and work with SIEM system content, as well as SZI maintenance engineers who are responsible for setting up security tools, analyzing attacks and developing custom signatures.

If you ask what requirements we have for candidates, the answer may seem very banal: certain technical competencies, analytical mindset, attentiveness ... However, how to check these qualities, what to rely on to reduce the influence of subjective assessment to zero? We tell what we pay attention to and what tasks we give to candidates.

For the vacancies of second-line analysts that are opening up for us, we also consider external candidates with specialized experience, but we always give priority to internal employees in translation. Thus, we increase the motivation of the guys: they have a large number of examples of colleagues who have gone from a first-line engineer to an expert analyst (see the article “ SOC for beginners. How to organize incident monitoring and response to attacks in 24x7 mode ”) or SZI administrator. Yes, and in the labor market there is a clear lack of competent information security specialists, and you will not find people with experience working in the SOC during the day with fire. It's a little easier to find a person with knowledge of the tools used in Solar JSOC, but here the choice is not rich.

Therefore, we not only encourage the desire for development in first-line engineers, but we also constantly educate children. For example, among other things, we have implemented the practice of case review - the second line daily devotes a certain amount of time to double-check part of the incidents that were examined by the first. Of course, this does not apply to all incidents - it makes no sense to simply duplicate someone else's work, and the flow of events is too large. First of all, we pay attention to incidents of high criticality, which were closed as false. The goal here is not only to identify possible errors, but also to give first-line analysts recommendations on how to analyze incidents and develop their professional skills.

With due regard to work, the engineer is potentially ready to move to the next level after a year of work on the first line. Of course, it all depends on the particular specialist - someone “matures” earlier, someone later, but it has been empirically determined that in about a year the engineer manages to join the team, gain insight into the essence of the work and gain practical skills in analyzing incidents - in general, become a full-fledged Solar JSOC combat unit.

Monitoring direction

The engineer of the first line can develop in one of two directions - monitoring or operation of SPI. In the first half, we will focus on monitoring specialists who are vigilantly monitoring the information security of Solar JSOC customers day and night. We tell you what qualities we are looking for in the candidates and how we check their availability.

Knowledge of tools

Firstly, the candidate for the second line must perfectly master the basic functionality of the SIEM system. The important thing is that from a certain moment SIEM ceases to be a program for an engineer that knows how to collect and filter certain events, and becomes a tool with which you can get the information you need - a kind of Palantir, a stone with which the SOC learns what is happening in customer-controlled information systems.

Interpretation of events
This implies a second important requirement. A second-line engineer should be able to interpret technical information into a human-readable description that is not understandable IT specialist. Sometimes for this you need to supplement it with relevant data from third-party sources. Compare, for example, two incident notification options that you can send to a customer:
На хосте NB-SIVANOV запущен процесс ««C:\Windows\System32\msiexec.exe» /i «Z:\Департамент разработки продуктов\Аналитический отдел\_Общая\ Enterprise_Architect\Sparx.Systems.Enterprise.Architect.13.0.1310.Corporate.Edition \easetupfull.msi» под учетной записью s.ivanov.
Старший аналитик группы анализа продуктов Иванов Сергей установил прикладное ПО для моделирования бизнес-процессов Sparx Systems Enterprise Architect из дистрибутива, расположенного в общем каталоге Z:\Департамент разработки продуктов\Аналитический отдел\_Общая\Enterprise_Architect.

Feel the difference? From a technical point of view, both options are correct, but, as always, the devil is in the details. The correlation rule worked to start the msiexec.exe process, which with a high degree of probability indicates the installation of new software, which for a critical workstation is a reason for analysis by an IS officer. In the first alert, the engineer copied the technical data from the most important fields of the 4688 event (host, process, ultrasound, process start parameters) and, in principle, did everything correctly in accordance with the instructions.

BUT the second alert is what we expect from an experienced engineer. The right technical information, coupled with business intelligence, provides the customer with much more useful information about the event.

To achieve such a level of immersion, an engineer must investigate more than one hundred tickets, study the events of the main infrastructure sources, realize in which direction to start spinning a tangle of investigation, and which event patterns are interesting in the first place.

The experience of conducting in-depth analytics The
look of a good first-line engineer should cling to non-standard things. And a second-line specialist, if such atypical incidents are detected, can deviate from the instructions if necessary.

For example, on a domain controller, the antivirus worked on the malware in the file \ tsclient \ C \ Users \ a.razumov \ Desktop \ shadow_miner_win.exe. Since this is an infection of a critical server, according to the instructions, such an incident should immediately escalate at any time of the day towards the second-line analyst, service manager and customer. However, if an engineer pays attention to the location of a potentially malicious file and conducts an in-depth investigation, he will understand that a user with a.razumov KM connected via RDP to a domain controller, and a local drive “C” with a user directory was forwarded to the RDP session and was checked by server antivirus. Such an incident, of course, also requires investigation, but its level of criticality is somewhat lower.

An experienced first-line engineer should remain alert. For every hundred registered IS events, 5-10 military incidents occur. It is clear that all responses are analyzed by living people, and it is human nature to make mistakes, but it is critically important not to miss these very military incidents, because for this the SOC services are purchased.

Here's an example: Solar JSOC has a pretty simple script to track the installation of new Windows services. Creating exceptions for this scenario is problematic, since the legitimacy of a service depends on the role of the server, startup rights, the context of its installation, etc. For this reason, the specialists of the first monitoring line have to look at a rather large number of operations of this scenario, especially in the case of connected workstations, and conduct an initial assessment of the legitimacy of the service. And here it is very important that the look of the engineer is not blurred.

More recently, there was a situation where a trainee engineer turned to a responsible analyst with a ticket in which he did not understand what process the freshly installed service starts. It turned out to be a PowerShell process with an argument in the form of an obfuscated script. Installing this service was a Kill Chain step, and its goal was to secure the malware that got to the host. The trainee’s attention allowed stopping the infection on the customer’s infrastructure.

Or here is another example related to geolocation of logins via VPN. By default, companies compile a white list of countries / cities from which they are allowed access for each VPN service user. One of the customers had compromised the domain ultrasound of an IT department specialist, and there was no two-factor authentication on the VPN gateway. On one “beautiful” night, the attackers connected via VPN under this KM from a proxy server located in Germany. According to the profile, logging in at any time of the day was considered legitimate for this employee, but only from IP addresses in Russia. Initially, it was agreed with the customer that we would notify him of this scenario by e-mail, even without duplication by telephone. However, the engineer decided to check the HR system and, seeing that this employee was not on vacation, escalated the incident, calling the service manager, who, in turn, notified the customer. The compromised ultrasound was immediately turned off, and the attackers did not even have time to really figure out where they got at all, not to mention consolidation in the infrastructure. Obviously, if the customer’s specialists had studied the alert only in the morning, having come to work, the story could have received a completely different development.

Experience with content
An experienced engineer should have an idea of ​​how the content works. Understanding the logic of the scenarios greatly simplifies the investigation process, as the specialist knows which chain of events the incident wound up on. False Positive criteria are also often tied to the technical nuances of the correlation mechanism, and to detect them, the engineer needs to be able to read the content.

Translation tasks
And so we found a great candidate for transferring to line 2. An engineer has a positive “credit history” according to the results of a case review (selective control of his findings on incidents), observes SLA when working with tickets, does not fall below a certain, stable high level during investigations and has all the qualities listed above in the article. What tasks do we give an engineer as a translator?

Usually we have several requests from customers to connect new types of sources. For this, it is necessary to analyze existing types of events, determine the optimal way to receive these events, and create a connector for collecting them. And this is a great task for an engineer to grow!

The second task is directly related to the first: after connecting a new type of source, it is necessary to integrate its events into existing scenarios. For example, after connecting anti-virus software as part of MS SCCM System Center Endpoint Protection, the event categorization was set up in such a way that existing virus activity scripts would work for this source. If this is a completely new type of source, the engineer analyzes what types of attacks we can detect by events from this source, and develops new scenarios.

The third task is usually the optimization of any content block of the SIEM system, i.e. correlation rules, filters, etc. Do not think that once written content remains unchanged. Either minor improvements are constantly taking place, or global rethinking of how it should be good :). Accordingly, solving this problem allows the engineer to dive deeper into the concept of writing Solar JSOC content.

And finally, the fourth task is related to the Threat Intelligence process. The second line is engaged in the primary processing of the incoming data, the analysis of their relevance, the addition of real-time monitoring, and the maintenance of indicators of compromise. The first line, in turn, is entrusted with the task of conducting a retrospective check on the indicators of compromise, and as part of the translation task, we want to hear from the candidate what pitfalls and weaknesses he sees in this process, which is quite complicated from a technical point of view, being a step in it below. So to speak, we ask you to give a person’s look “out of the field”.

Operation direction

So, congratulations: we have successfully “raised” a 2-line monitoring engineer. But in addition to monitoring, in Solar JSOC there is a direction of operation of information protection tools, on which 1 and 2 lines of specialists are also built. What heights does a first-line engineer have to reach for level up?

Immersion in the subject
First of all, an experienced administration engineer must clearly understand what this or that information protection system protects from. To do this, he must have an idea of ​​the main types of vulnerabilities of the protected system, the methods of their operation and the potential impact on the customer’s system.

The second-line specialist should know the basic methods for implementing IS threats - the most common types of attacks on protected systems. In particular, we are talking about attacks from the Internet on the customer’s protected web resources (which are primarily aimed at hackers' sting), as well as other resources accessible from outside.

Knowledge of tools

To successfully resist complex and sophisticated attacks, an engineer must own his tools 100%. The idea is not new and coincides with the statement from the section on monitoring, but this does not become less relevant. Know what you are working with. In Solar JSOC, engineers are armed with various IPS, WAF, AntiDDoS solutions, which, according to vendors, work well out of the box. But we strive to ensure that our engineers possess the full functionality of these products: they can read the default policies, edit the predefined signatures, understand what is hidden “under the hood” of a beautiful graphical interface.

A classic case for a WAF or IPS administrator is to develop a custom signature for a fixed long-term attack on a protected web resource. Our engineers recently faced just such a situation: monitoring tools recorded a long brute force of the accounts of the personal account of one well-known online store. The predefined signatures for brute force attacks did not work, since they did not take into account the specifics of a particular web application. On our own, we conducted an operational analysis of dangerous web requests and based on it created a WAF signature that blocks this activity. Hackers noticed changes in the protection system of the attacked resource and several times slightly changed the contents of the requests, trying to bypass the signature, but each time our engineers successfully tracked these manipulations and adjusted the protection policy. Вендоры средств защиты предоставляют подобный сервис, но зачастую им может не хватать оперативности и погруженности в специфику защищаемой системы.

Analysis of the retrospective
However, there are unpleasant situations when the attack is missed, and the attacker, having exploited the vulnerability, got to the target host. After some time, the penetration was detected, access was closed, the forensics team analyzed the compromised hosts and found artifacts on the file system that were traces of the attack. This is great, but what to do next? Running out indicators using a security scanner and scripts is a standard procedure, but forensics always has a place for street magic.

The administration engineer, with the help of his forensic colleagues, is trying to figure out how this attack could be detected and blocked “on the way” on the used protective equipment. To do this, attempts are made to reanimate the remains of a combat exploit to add its payload as a signature, the engineer notes the specific nuances of the attack that can transform into monitoring triggers, and software vulnerabilities successfully exploited as part of the attack are determined. In general, work is underway, without which it is impossible to learn from a successful hack. And this task requires the administration engineer to have a deep technical background and a sufficiently high qualification.

Architectural Approach

The engineer of the second line should be able to see two steps forward. Any operation of a complex product is inevitably associated with changes in its configuration. Taking into account the specifics of information security solutions, it may turn out that any changes threaten to lose the availability of not only the security tool itself, but also the protected system, and as a result, financial damage to the customer. That is why the IS administrator must have an architectural vision in preparing the RFC: be able to plan ongoing work, predict downtime, assess the risks and the impact of changes on other systems, and always keep in a safe place escape routes - reliable methods for rolling back the configuration to the operational state of the product.

Also, do not forget that the protected systems are usually not static, but are in a state of continuous change. And once a well-designed protection policy can stop working adequately within a few days after its deployment. Therefore, an experienced second-line engineer should be able to build a technical process consisting of the stages of profiling legitimate activity, testing and finalizing an adapted policy, launching this policy in a blocking mode in a productive way, etc. We will not dwell on this, this topic is described in great detail in the article “ The problem of continuous protection of web applications. View from the side of researchers and operators . ”

Instead of a conclusion

The keys of the letters of the word "engineer" are already beginning to sink on the keyboard, so we will round out :). Summing up, I would like to say that technical skills for a specialist in our field are very important, but first of all we pay attention to people with burning eyes who are “ill” with the IB topic, and in its “white” manifestation. Passion for their work allows a specialist to avoid stagnation and is the main driver for professional growth, which is important in our constantly changing industry. The first or second line is not so important. If a specialist enters this path with great desire, he is guaranteed success. Like cookies :).