SOC are people. How to assemble a team in conditions of staff shortage

It is said that in the 70s, Gennady Zaitsev, the first president of the legendary Leningrad rock club, formulated the principle of selecting musicians: “Finger fluency is a business. Would a good man . "

It is possible that if Gennady Borisovich and I decided to complain to each other about a difficulty in selecting personnel behind a glass of something fermenting, we would find many common points. When we assembled the first Solar JSOC team, the market for monitoring and counteracting cyber attacks did not actually exist, therefore, there were neither trained specialists for these tasks, nor even clear search criteria. We had to assemble a team of unique ones, but before that - to try new things many times, make mistakes and try again. It’s about the team today that we’ll talk about: how it was at the very beginning and what we came to as a result of.

I don’t know for certain how the idea of ​​creating JSOC was born - maybe just a coincidence, or maybe the guys gathered and firmly decided: “We are creating SOC for the commercial market of Russia!” Turn on rewinding and skip many exciting episodes about choosing a platform, working out the ideology of building content, and also about incredible adventures in the wild world of customer infrastructures.

Let's start from the moment when the JSOC from the synthetics of plans and estimations is ripe for entering the market and running in the worked out solutions in battle. And for this, there is not enough technology, ideas, ambitions and SIEM gurus and analytics. Here one can not do without the first line of analysts - a team that provides the front of the main SOC processes.

Our first line of monitoring is different from the first line in the classical sense. This is not a call center or girls with an application routing script. The role of the classic first line in our paradigm is assigned to SIEM with a pie of profiling, aggregating and enriching rules. And the task of the first line of JSOC is to conduct full-fledged investigations of IS incidents, including FP filtering, the formation of analytical information and recommendations on counteraction. A free speech is also welcomed, especially in situations where atypical events are seen in the vicinity of the incident.

"Any adventure must start somewhere ...
corny, but even here it is true."

Lewis Carroll

When the first version of the team was formed, we did not quite understand the exact profile of the analysts, since it is difficult to predict the development of events in a completely new market. Therefore, to compensate for the possible turns and vicissitudes of technological trends, we decided to look for guys with a wide technical background.

It turned out to assemble a team of such specialists rather quickly, and this greatly helped us at the initial stage. We felt ready for the winding and difficult way to bring SOC to the market.

However, after a while, we realized that we had miscalculated in some ways: most of the hired specialists were clearly overqualified for the position of first-line engineer, and the v.1.0 team turned out to be rather unstable.

Despite the high development dynamics of Solar JSOC and the constant emergence of new tasks, the children's ambitions were even higher, and the final list of responsibilities was significantly lower than their experience and expectations. Therefore, colleagues quickly grew out of the first line - someone within the JSOC, and someone, alas, outside.

However, the “virus” of information security obtained in JSOC turned out to be very persistent - despite the fact that the v.1.0 team was almost 100% come from pure IT, almost all of them continue to work on the information security field.

Nevertheless, this first attempt to assemble a team was very useful for us.
We “on the go” decided what we finally do and what we expect from the employees. We got used to the fact that in the new market, multiple growth and the attendant permanent training of personnel are the norm and routine. We found that the market cannot offer ready-made specialists to work on the first line of SOC. We realized that in these conditions it is simply utopia to pass by a frequent rake on a resume.

"To teach a scientist is only to spoil."

Russian proverb

So, we found ourselves in a situation in which there are practically no ready-made specialists on the market, and the number of customers begins to grow, which requires scaling up the team. At some point, we simply came into conflict with the capabilities of the region’s personnel market (the first Solar JSOC line is in Nizhny Novgorod).

The salvation for us was working with students. At this point, we had little experience of such work in the Moscow Solar Security team, but it was not aimed at working with the university as a platform organizing the flow of young specialists. We felt that in the case of Solar JSOC, a more systematic approach would be required.

And here our colleagues, graduates of the Nizhny Novgorod University, helped us a lot, and brought us to the student career center of NNSU, a peppy student organization interested in professional job placement for students. We began to regularly participate in Employer Days, to tell students and recent graduates of technical universities about the activities of Solar JSOC.

Then we launched an internship program. Initially, it was aimed at students of the IS specialty, but now it has been expanded to students of other IT specialties.

Looking ahead, I will say that this work yielded excellent results. For the third year now, young children come to us by gravity, ready to undergo testing, internship, study and become first-line engineers. And many graduates of our internships have truly found their first job at Solar JSOC.

- This is too difficult for my little brain ...
- It is not size that matters, but the ability to use.

From Folklore Solar JSOC

We understood what the ideal team of an ideal SOC should be in the end :) And we are ready to reveal this secret: a team should be primarily a Team, no matter how corny it sounds. And the phrase of Gennady Zaitsev was given at the beginning of the article not by chance. We approach staff recruitment, guided by the same principle: “Finger fluency is a business. Would a good man . " After repeated trial and error, we came to the conclusion that when searching for engineers of the first line, the main emphasis should not be on the technical background and previous work experience. (This, however, does not mean that the previous experience is completely unimportant. Of course, good knowledge of networks and linux is welcomed for a job seeker in the SZI technical maintenance group.) We are trying to find guys with “living brains”. Guys able to see a fan of judgment, able to reason, с хорошими аналитическими навыками.

When there was an influx of potential candidates for internships, and subsequently work, we were faced with the task of assessing the learning ability of each candidate. Therefore, at the interview, we try to bring the applicants to logical reasoning, give logical puzzles and change their conditions to see how a person adapts and selects the best solution. This approach helps us to select people who are pleasant to work with and who integrate very quickly into the team. Well, “finger fluency” is a business. About how we train her, a little lower.

As an entrance test for an internship, we offer children to deal with such tasks:
  • Почему UDP продолжает использоваться, когда TCP гарантирует доставку данных в неизменных виде, последовательности и без потерь?
  • Вам нужно написать ТЗ на keylogger. С чего начнете? По каким критериям вы бы стали агрегировать информацию для удобной работы с результатами?
  • Перечислите принципиальные отличия таких Remote Administration Tool, как Microsoft Remote Desktop Connection и TeamViewer, с точки зрения контроля за доступом к критичным хостам?
  • На каких из этапов APT-атаки можно встретить использование данных утилит, и как злоумышленник может их использовать?

The internship program includes:

  • Learning the basics of network technology.
  • Studying the features of IS events logging by various sources.
  • HPE ArcSight Skills
  • Analysis of simple test incidents at the training stand.
  • And much more.

During internships, the greatest attention is paid to test incidents, most of which are created based on real events from the practice of Solar JSOC. In addition to the template, automatically generated incidents, trainees regularly receive multilayer incidents with sub-marks for parsing. Incidents that require a special approach, because as a result of a template investigation, students, for example, conclude that malicious activity was initiated from their own hosts from under their own accounts.

Such challenges significantly increase the involvement in the training and investigation of incidents, learn to pay attention to various "little things", double-check your conclusions, make assumptions and prove or disprove them in the future.

Involvement in the investigation process is the most important at this stage. Having interested the guys in security and letting them “touch” incidents with a real story or incidents of which they themselves are allegedly participants, we motivate them to study a “boring” theory and develop non-standard approaches to solving the problem that are not described in the manuals.
At the end of the internship, we get guys half ready for work on the first line. Thus, we are creating a personnel reserve.

Regardless of whether we take a person to work as a result of an interview or after an internship, at the beginning of the professional career of a first-line engineer, Solar JSOC expects a training program for the completion of which is allocated from 2 to 4 months. It can be conditionally divided into the following blocks:

  • The theoretical block includes basic knowledge, starting with information about the company and its organizational structure and ending with an introduction to regulatory documentation, the latest trends in information security, APT, Threat Intelligence.
  • The practical block consolidates the acquired knowledge at a new level and teaches specific skills to work with the main tool of the first line - SIEM. In addition, the practice includes such mandatory areas as tracking IS incidents in the ticket system, the subtleties of communication with customers, reproducing and analyzing the most interesting incidents at the stand, and much more.
  • Further, newcomers are given gradually increasing access to a productive environment based on whitelist: an engineer is involved in investigating “combat” incidents according to the scenarios in which he successfully passed the exam and to which he received admission.
  • Since our company, like the information security industry itself, is developing rapidly, it is impossible to cover all the details with a training program. Therefore, at the final stage, each future engineer of the first line of monitoring is assigned a curator of experienced engineers - so-called "practicing surgeons" - to study the nuances and control the work performed on a productive environment.

Based on the assessment of the curator and the final control slice conducted by the commission, the engineer is allowed to "self-sail", night / weekend duty.
At first, most engineers work in the paradigm "For an incident of this type, I use such and such tools, they provide the necessary data about it." Such a methodology allows you to quickly train a person and bring to a level when he is able to bring benefits on the line.

As they gain experience, the guys realize the commonality of IS incidents, begin to better understand what information and in what context will be most useful to the customer for responding to a specific incident. Gradually they move on to work in the paradigm “I need such and such information, I can get it in many ways, in this situation, such and such is preferable.”

Such a restructuring of the approach, coupled with pumping the information security background and immersing yourself in SIEM content, prepares the children for the steps to the next steps, where they are waiting for in-depth investigations of highly critical incidents and information security incidents or diving deep into SIEM systems with all the variety of custom logic.

This ends the formalities and begins a full life on the first line: for administration - full of complex quests to satisfy the desires of customers and compliance with information security requirements; for monitoring - consisting of many fascinating stories reflected in the logs. Sometimes a day on the first line may resemble a puzzle on several boards with the length of the shift on duty: the collected pictures flicker in front of the engineer, changing not only their texture, but also the ticket number :)

But, although the theater begins with a hanger, and the operational life of SOC - with the first line, it does not end there. People grow, projects scale, and we have a great many roles and tasks inside. We greatly encourage the development desire in first-line engineers, therefore we have worked on several “professional elevators”. A certain percentage of engineers who are ready to move to other lines are even included in the KPI of team lines :) But the story of “boosting expertise” or further staff growth in Solar JSOC will become part of other articles. To be continued ...