Pwning the security engineer interview

Featured ~ allthingspwned ~ 4 Comments

Full Disclosure: I am a security engineer @ Google and the following are my own opinions.

Introduction:

I’m here to help you out – whether you are in school, a seasoned professional or just someone who is curious about the security engineer interviews, this guide is for you. Having struggled to find my own breakthrough in the industry and having received offers from security teams of companies like Google, Facebook and Yahoo, I have been in your shoes at some point and I am writing this guide to help you in your career. Interviewing is a nerve wracking process where you never know till the end whether you made it. It also requires an initial investment upfront to spend time preparing yourself, applying for relevant roles in companies and finally going through the process. Having gone through this several times in my career at the largest tech companies, I wish I had known a few things prior to the interview that would have helped me even more.

The focus of this post is on interviewing at large tech companies but the methods I describe do apply elsewhere to a certain degree. Information security is a vast and complicated field and there are a wide variety of topics that can appear in an interview. So instead of diving into each topic, I follow a “show and tell” approach in this post where I take a question as a running example and demonstrate a framework that can be handy to effectively answer the question.

In a security engineering interview, questions generally fall in one of a few several categories:

  1. Open ended questions
  2. Knowledge based questions
  3. Coding questions
  4. Soft skills or experience based questions

The scope of this post is limited to open ended questions. These are typically the ones that require a blend of knowledge and experience. They are also the ones where a response can be a never ending discussion. Due to the nature of such questions and how they are increasingly becoming more relevant in interviews, this post puts its emphasis on this question type. Knowledge based questions can be directly answered by knowing the subject matter. They are generally of the form of – “Describe X” or “What is Y?”. For coding questions, there are already tons of resources out there including “Cracking the Coding Interview”, online programming judges etc that are sufficient for preparing oneself for this aspect of the interview. The same applies to soft skills and experience based questions where existing resources available in the public space make it relatively easy to prepare oneself.

Framework for open ended questions:

The following is a framework that I have created for tackling open ended questions. In short I refer to it as the ACEDER framework.

Ask relevant questions:

Let’s say the interviewer asks you the following relatively open ended question:

How do you go about ensuring two parties can communicate without the possibility of eavesdropping on the network?

Just by reading this question, there can be several thoughts coming to you, for example:

  1. Is the communication one time or an ongoing one?
  2. Is the interviewer asking me indirectly to describe existing protocols like HTTPs? SSH? TLS?
  3. Should I be designing my own protocol?
  4. (Insert your additional thoughts here)

While having thoughts is normal, in my experience it is best to take a moment to organize the relevant ones and get clarity before jumping into an answer. So focus on the core elements of the question, think out loud and ask clarifying questions in a structured way without jumping immediately into potential solutions. Asking questions also stops you from making unnecessary assumptions which might not hold true and may even lead you to an unwanted path. For this particular question, we do know that we need to ensure two way communication and with the property of avoiding eavesdropping. Other than that, there are no specific details that the question mentions.

A response such as the following can be a good start:

I am aware of several different existing secure communication protocols like TLS, SSH and the like that operate on the application layer as well as others like IPSec which operate on the network layer. All of them satisfy the property of being secure against eavesdropping on the network. Do you want me to focus on an existing one or is the question asking me to design my own communication protocol?

Notice that the above structured response not only shows that you are considering different layers in the networking stack and mentioning relevant protocols that satisfy the criteria of the question, you also asked a clarifying question to determine the scope of the question. This provides you a clear direction to move forward. A word of caution on this: Asking too many questions without providing some answer can be detrimental. The clarifying questions should be reasonable in quality and quantity for them to be most effective.

Cover the basics

Once you have gained clarity on what the interviewer is looking for, it is useful to cover the basics first. To provide a concrete example, let’s say in the question for communication protocols, the interviewer responds and asks you to describe TLS.

Although it can be tempting to mention the awesomeness that TLS offers using crypto magic like generating the master secret from premaster secret, it might be a better idea to first cover the basics of the protocol.

To start off, you can formulate a response:

TLS stands for Transport layer security and operates on the application layer. TLS is used in conjunction with other protocols like HTTP, IMAP and others to provide a secure communication channel for them. It uses a sequence of high level steps as follows:

  1. Client and Server Hello
  2. Key negotiation and exchange
  3. Follow up data transfer

(Then proceed to give a overview of each step in some detail)

Such a response prevents the idea of “the candidate could not describe the basics” from popping up in the interviewer’s mind. However what is considered as basic really depends on the level of experience you have with that particular topic. It is definitely a good idea to keep asking questions and provide more information based on the response. It is best to also state your assumptions as you go through your answer to make them clear to the interviewer throughout the interview.

Expand the breadth

While asking questions and proceeding accordingly is definitely useful, having no informative and thoughtful content as follow up defeats its purpose. As an example, let’s say after hearing your response of the sequence of steps and its details, the interviewer asks you to contrast that with SSH. The idea might be to test your breadth of knowledge and to see if you can correlate different protocols.

An appropriate response here would be to again cover the basics of SSH and to highlight specific differences among the two. Something along the lines of would be considered a good response:

SSH stands for secure shell. It operates on port 22 and is used for remote login. It uses a sequence of steps to establish a secure communication channel. At a high level:

  1. ID Key exchange
  2. Algorithm negotiation
  3. Key Exchange
  4. Follow up data transfer

Then proceed to give a overview of each step in some detail and highlight the differences from TLS e.g. no ID key exchange in TLS

Expanding on the breadth is definitely a great idea as it shows your expansive knowledge base. Therefore doing so voluntarily as you answer the question automatically puts you in a good light if done without too much deviation from the original answer (e.g. jumping to another unrelated protocol).

Increase the depth

While the idea of expanding on the breath is great, we also want to ensure that details are mentioned to convey the depth of our knowledge. Demonstrating the depth of knowledge goes hand in hand as you expand the breadth. Imagine breadth as drawing a landscape for the interviewer to visualize and then you fill in the appropriate details like colors to make it a vivid picture.

As an example for the secure communication protocols questions, depth can be demonstrated when you describe the sequence of steps involved in detail, mention relevant steps both the client and the server take, the encryption and hashing algorithms that can be used and why which one is better etc. Similarly for SSH it would be best to mention why exchanging ID is important, the algorithms involved and the pros and cons of each. Calling out relevant attacks that can occur when weak protocols are chosen further demonstrates depth and provides an opportunity to discuss related pitfalls when implementing secure communications.

Add in experience

After the basics of the question have been covered, adding in real world experience is helpful. For new graduates, this can be a project you did at school or on your own on a weekend and fits right in with your answer. For an experienced professional, this can be a project you contributed to in your previous job or a project you led other members of the team through. Irrespective of the place the experience was acquired, meshing in your experience with your answer and including the takeaways does end up enhancing your responses. This also leads to a better impression in the mind of the interviewer in several ways – First it demonstrates that you can correlate information from the past and can build on that to guide your future work. Second it also conveys your practical experience to the interviewer which gives you bonus points. A structured way to communicate this is by using the well known STAR method (Situation, Task, Action, Result).

Going back to the communication protocols question, here are several ways experience can be meshed in:

  1. In graduate school, I was involved in a project to test TLS security guarantees. I was responsible for setting up the virtual infrastructure and conducting tests. I set up two virtual workstations as clients and servers and emulated an active attacker on the network. I experimented with both wireshark and tcpdump for capturing network traffic and observed that the encrypted traffic was gibberish. I also emulated performing a downgrade attack to weaken the strength of encryption. The result was that I was able to verify the security guarantees that TLS offered.
  2. In my previous job I was tasked with upgrading all website servers when the heart bleed vulnerability was announced. I spent considerable time reading and grasping the real issue at hand, scanned the company network for hosts that were affected and worked with our partner teams to deploy patches in a timely manner.

Notice how the depth is enhanced by adding in your experience with the appropriate tools, terminology and details in your answers.

Rinse and Repeat

The above process of asking questions, covering your basics, expanding your breadth and depth along with adding your experience can be repeated till the interviewer is satisfied or you feel that the answer is sufficient and complete. It is important to note however that generally interviews are time bound and in that spirit it is best to check in with the interviewer before diving way too deep into a particular concept, algorithm or idea.

Going back to our secure communication protocols question, the check in can be introduced at relevant points within your answer. As an example when you have described the protocol and its details with some of your experience, it can be worthwhile to check by asking – “Do you want me to go deeper into any particular aspect of the protocol or applicable attacks”?

Depending on the interviewer’s response, you can proceed accordingly in the anticipated direction. The interviewer also may have a series of questions to cover and they may interject if you deviate or go way too deep in your answer than what is required for them to know.

Security Engineer Interview: Conducting Empathetic Interviews

~ allthingspwned ~ Leave a comment

Being good at interviewing others is a skill. Let’s read that again, a little bit slowly this time. Being good at interviewing others is a skill. It is not about being able to ask one question after another to judge the candidate in front of you. It is a lot more than that. In this article I would like to talk about a technique I call “Empathetic Interviewing” which I came up with after conducting dozens of interviews. It is centered around the idea that people remember how you made them feel. It can be used in conjunction with the interviewer’s framework I described in my earlier post.

In a nutshell, empathetic interviewing is about the following:

  • Making the candidate comfortable
  • Recognizing the breadth and depth of responses
  • Changing and reframing questions to get the best answers possible from the candidate
  • Promoting a sense of equality by answering questions.

Making the candidate comfortable: This is the most important thing to start with. It helps establish an initial rapport. There are several ways of doing this:

  • Introducing yourself, what you do, how long have you been at the company.
  • Outlining how the interview is going to be structured.
  • Being open to answering questions during and after the interview.

The above are just a few examples but you get the idea.

Recognizing breadth and depth of responses: Once you have made the candidate comfortable and are in the middle of the interview, for each response from the candidate gauge the breadth and depth of the candidate’s answer. You may wonder how this falls under the empathetic interview umbrella. The key is “how” you accomplish this. One of the most effective ways I have found is to go from open ended questions to more specific ones as the candidate gives you more information. The point at which you feel that the detail of the response has significantly decreased, you can steer the candidate in another direction to expand the breadth. Repeating this will lead you to examine both breadth and depth of a response. For example, let’s take the question which I talked about in a previous blog post – “How do you go about ensuring two parties can communicate without the possibility of eavesdropping on the network?”

If the candidate starts describing how TLS works on a high level, you can increase the depth by asking specific questions around how key exchange would work? types of attacks possible? etc. Once you feel the candidate has reached their “knowledge” limit, you can pivot to another question e.g. what other protocols can be used to accomplish the same goal? In response to which the candidate might start describing an application level protocol like SSH. Rinsing and repeating this process would give you as an interviewer a really good idea of the strengths and weaknesses in the response.

Changing and reframing questions to get the best answers from the candidate: This is yet another important skill. Sometimes candidates get stuck or do not know how to answer a question. Recognizing “when” this happens and altering the question is the skill to develop. An effective technique is to reframe the question in a way that can still give you an idea of the candidate’s knowledge. Taking the same example as above about establishing a secure communication, if the candidate gets lost in the beginning, you can hint by being more specific and reframing – “how do you ensure confidentiality of data between a server and a client”. It is a similar question but slightly more specific which some candidates might pick up on. Note that both this and the above steps let you gauge a candidate even when it is challenging to do so.

Promoting a sense of equality by answering questions: At the end of the interview, finding time to answer the candidate’s question reflects well on you as an interviewer. Interviewing is a two way street where you are evaluating the candidate for the role and the candidate is evaluating the company as well. This is as important as any other part of the interview. The questions that a candidate asks also can show you the amount of research and thought they have put in.

In summary, the outcome of empathetic interviewing is a great experience for the candidate which should feel like a discussion with a teammate. At the same time you as an interviewer should be able to provide an accurate assessment of the candidate. If the candidate ends up working with you in the future you already have a good rapport with them. If not, they still carry a good impression of you as an interviewer. Win-win!

Challenges of security engineering at scale

~ allthingspwned ~ Leave a comment

As companies grow in size and work with massive amounts of data, the challenges faced by their security teams also grow manifold. This topic is of importance to not only the existing security community, practitioners and experts but also newcomers to the field who may not have experienced the same challenges themselves. In this post, I will be highlighting a few major challenges, sane practices and pointers on how to engineer security at scale. Note that these are not specific to any company but are rather generally applicable concepts that come into play automatically at scale. To bring everyone on the same page, when I say companies @ scale, I am referring to large tech companies who serve millions of users every day. The challenges I will be discussing are also therefore technical in nature. Lastly, this is not an exhaustive list but rather a starting point to a much broader and rather complex discussion.

Asset inventory: It is rightly said that you cannot protect what you don’t know about. Tracking a few hundred devices is very different than tracking over 100,000 machines. This not only means that you are aware that a device exists but also about its configuration, which user(s) it is assigned to, which operating system and patch level is it running, when was it last online etc. More importantly, it is critical that this inventory is kept up to date at ALL times instead of having a delay in it being updated. This is because this is one of the sources of truth that the security team (as well as other teams) rely upon.

Fleet management: This goes hand in hand with asset inventory and sometimes both of these are integrated into a single software solution. Once you know your assets, the fleet management is responsible for functionality like updating the host with the latest updates, deploying patches, gathering system information etc. Such data can be critical for instance say that application security team who is in a rush to deploy a patch to secure all hosts against the latest SSL bug. In an ideal world, the fleet management software should be able to reach the entire asset inventory and provide an easy way to manage them remotely.

Scalable Evidence Collection: In cases when a company is warding off an external or an internal attacker, having a sane, tamper proof evidence collection and storage mechanism is a life saver for a detection and response team. Evidence can be anything for example a file, the output of a command or some specific data from the system like memory dump, copy of the disk image etc. Evidence can also serve as an indicator of compromise. At scale, the challenge with evidence collection comes because your hosts can be in different states such as the following:

  • Offline
  • Stuck during the boot process
  • In the process of rebooting
  • Installing system updates before allowing the user to log in

The evidence collection mechanism should be able to handle data collection by issuing commands to effectively gather artifacts from systems when they are just not fully available (due to any of the above states). Needless to say since the collection is done remotely, having end to end encryption would be ideal. A fantastic piece of software that does this is GRR and best of all, it is open source. When hosts are unavailable, it can poll them and collect the desired artifacts when a machine comes back online. I highly recommend checking it out.

Speed vs storage: This is more of a metapoint but there will be challenges involving the decision between a faster solution that takes up more (in memory or disk) space vs a slower solution with less storage footprint. This is a typical software engineering challenge however I am bundling it up here as it is extremely common to also encounter when writing your own security software that handles petabyte scale data. It is ideal to foresee these key decision points BEFORE writing software and brainstorm them. The ideal place to document these designs are the design documents. Once those decisions are documented, are well thought of and agreed upon, building and maintaining software becomes much more effective.

Risk management: This is a high level point as everything in security revolves around managing risk however the important thing to note is that not all risk can be mitigated, especially when dealing with scale. There is where leadership and technical expertise comes together to decide which risks are accepted and which ones the team should mitigate by prevention, detection or transferring them out. For example, a company might decide to deploy a patch for a vulnerability (prevention), or by having its security team detect and respond to an attack when it happens (detection), or by outsourcing the risk to another vendor company and lastly accept the risk by making no change. This example shows a very simplistic case however risk management decisions are business decisions and different priorities and resources can result in different choices being made. At scale, these challenges just become more complex with competing needs and priorities.

Maintainable and readable code: This might appear like a best practice however it is certainly true when you have company owned internal software written out. When code is complex and hard to maintain, it leads to either engineers spending a lot of time on it (i.e. engineering resources) or it goes unmaintained and can lead to technical debt. Both of those situations are not ideal. When the size of your code base increases dramatically, these simplistic best practices become key.

Compliance: Given the large swath of country (or even state) specific laws that exist today for user’s security and privacy, this presents itself as a challenge for the security compliance team to effectively make changes to products and services to keep the company in compliance. At scale, tackling this challenge can involve making both technical (e.g. data retention policies) and non technical changes (e.g. advance notice to users, seeking consent etc). The second aspect of compliance is ensuring and proving that the company is compliant at all times. If you think about it, making a technical change may involve altering software on a certain set of hosts which is much easier to do if the existing code is easy to maintain and we have a good inventory and fleet management to know where to deploy the changes. In this example, everything comes together in a way.

Employee awareness and training: It is commonly said that security is everyone’s responsibility and this is most definitely true at scale. We can have the best defense systems and security team but all it takes sometimes is one click in a phishing email to do some serious damage. It is critical that employees are well educated to follow the best practices and adhere to internal security policies to ensure distributed protection.

Security Engineering Soft Skills: Communication

~ allthingspwned ~ Leave a comment

Full Disclosure: Karan and Raaghav are security engineers at Google and Dropbox respectively and the following are their own opinions.

The previous posts discussed the technical aspects of the security engineer interview. In this post, I collaborated with my close friend Raaghav to discuss different aspects of an important soft skill – communication.

It is a well-known fact that communication is a key skill for professionals in any domain, and not just within security. Furthermore, with the current pandemic situation and the shift to working from home, there has been a significant increase in the usage of specific modes of communication such as chat and video calls, which brings along a unique set of challenges. It is important to evolve your communication style to adapt to these changes. This served as an impetus for us to revisit the key aspects of effective communication.

Identifying Stakeholders

The first step is to identify who you have to communicate with and as easy as that sounds, it is important enough to state as a major area. This is best explained with some examples:

  • You are handling a security incident and you need to communicate its status to relevant parties. Generally, the team/points of contact responsible for fixing the root cause of the incident are obvious stakeholders. However also thinking about others who potentially need to be involved or are affected by the incident works in your favor e.g. executives who will make decisions based on the outcomes of the incident, the relevant legal teams if it concerns them, public relations, customer support, etc.
  • You have to communicate your project updates. Obvious stakeholders involve your team members and your management. Some other non-obvious stakeholders to think about maybe extended teams who are potential customers of the project who will benefit from the updates, executives who are sponsors of the project, etc.

Choosing the mode of communication

Once you have identified the stakeholders, the next thing to decide is the mode of communication. Here are some well-known modes:

  • Chat
  • Emails
  • Video Call/Conference meetings
  • Documents, Spreadsheets etc.

The key takeaway for the mode of communication is CHOOSING the right one. This depends on the situation, for example – If you have an urgent decision to be made that involves multiple stakeholders, relying solely on asynchronous mode of communication, like email, may not be the best idea. When urgency is involved, relying on a more real-time communication mode (and potentially a combination) may work in your favor. So in this case, it would be better to set up a group chat to get every stakeholder’s attention and then set up a follow-up meeting (if needed) to get the decision made on time.

Conversely, if you require stakeholder inputs on a project proposal on a non-urgent issue, writing up a document that informs them of the proposal and sharing it with them ahead of time (or a follow-up meeting to discuss it) would work better.

Creating Content

Now that you have identified your stakeholders and the mode of communication, let’s focus on the content. The content being communicated varies from situation to situation. Having said that, the following guiding principles are key for effectiveness:

  • Transparency: Ensure that the full knowledge, visibility, and context of the situation is evident in your content. The lack of context raises questions in the audience’s minds and can create confusion. In a remote situation, it is very easy for people to feel siloed, and adequate transparency is a way to eliminate the disconnectedness. Long term, a lack of transparency can negatively impact the morale of your team and will further disconnect them from the mission.
  • Clarity: Ensure that intent of the message is clear and any asks are well spelled out. In a remote situation, people no longer have the opportunity to simply swing by someone’s desk or talk in-person to get more clarity. So it is absolutely important to invest time to be extremely clear in what you are trying to communicate and what specific ask you might have from the recipient.
  • Brevity vs Verbosity: Ensure that the content has the right balance of conciseness and verbosity. In some situations, it is important to convey the point quickly and succinctly whereas in others it helps to be elaborate to explain the issue at hand.

Following up

Once you have communicated, it is important to decide if it requires a follow-up. Most communications are one-off, however, there will be several instances that require multiple follow-ups. The key aspect is to remember when to follow up and how often. Defining these two parameters ensures that you can achieve your deliverables on time and also adapt in the event of unplanned work or delay. Communicating early and often is key.

Overall everyone is putting in a lot of effort to adapt to the remote first work environment and as such, you should strive to demonstrate empathy at every step of your communication while keeping the discussion above in mind.

Go forth and communicate!

Security Engineer Interviews: Coding, Data Structures and Algorithms

~ allthingspwned ~ Leave a comment

Full Disclosure: I am a security engineer @ Google and the following are my own opinions.

I have received a ton of questions from students and professionals alike around coding, data structures and algorithms in security engineering interviews which has finally motivated me enough to write a dedicated post on this topic. This topic is a separate question type as compared to the open ended questions which I have already addressed in a previous post. In this post, I will be addressing the most common concerns around this topic in a way that makes sense, provide guidelines on how to gauge yourself as you develop your skills, and also answer a few frequently asked questions along the way. So with that, let’s dive right in.

Over time coding has become an increasingly important aspect of security engineering interviews and in my humble opinion, this will stay for time to come. Let me explain – Based on my recent observation from looking at security engineering job descriptions and my personal interview experiences, companies of pretty much all shapes and sizes are now explicitly specifying at least one coding/scripting based requirement directly in their job descriptions. The reason behind this is pretty straightforward. The security field is evolving faster than ever with increasingly complex problems being surfaced (e.g. can you deal with a million false positives and find that one needle in the haystack in your alerts?) and different concepts being applied in this field e.g. machine learning applications within security. Tackling this exponential growth and keeping up with business requirements require engineers who can write efficient tools and maintain them. Not only that but automation of repetitive tasks (e.g. your everyday alert triage process which you can now do in your sleep) has the potential to save an organization thousands of dollars in productivity hours every year. Therefore having engineers on board who possess the skill of writing, testing, and maintaining code has a good return on investment for organizations dealing with evolving challenges.

Now with that background, it should be a no brainer that security engineers should not only develop their coding skills but also hone them as much as a dedicated software engineer would. For this very reason, I tend to imagine security engineers as a hybrid mix of software engineers (who can produce and maintain high-quality efficient code), site reliability engineers (who can do operations work e.g. be on-call, perform investigations with a security mindset), and data analysts (who can make logical conclusions and perform analysis on large data sets). So the next obvious question is – Are you saying I need to be an expert at coding? The short answer is no, not every job requires that level, but in general the better you get at it, the stronger your chances of cracking the coding portion of the interview. I have come up with a way to give you the reader an approximate idea of levels by comparing this with a concept from psychology. Let’s first understand the Maslow hierarchy of needs which is a motivational theory in psychology comprising a five-tier model of human needs. It describes the five human needs in a pyramid form where the most fundamental physiological need is at the very bottom and the need for self-actualization and transcendence is at the very top. Moving to higher levels in the hierarchy corresponds to growth in personal development. I use this concept and apply it here to describe the “levels” as one “grows” their coding skills by using a similar pyramid for coding levels.

The physiological needs comprise food, water, warmth, shelter, etc. which I directly compare to the basic working knowledge of coding, data structures and algorithms. This means that you should be familiar with at least a language of your choice and know the basic data structures, common algorithms for operations on them, and their associated time and space complexities. The “basic” data structures I have in mind here start with stacks, queues, linked lists, maps, etc, and the list (pun intended) goes on. By basic algorithms I have searching, sorting, insertion/deletion etc in mind.

Going up the hierarchy comes the need for safety and security. I compare this with a proficiency in the basics along with practical knowledge and application of data structures that are considered beyond basics e.g. different types of trees, graphs, etc. At this level, you have developed a general intuition of runtime of a class of problems and be able to comfortably select the optimal data structures for a small to medium sized problem.

Further up the chain comes the requirement of love. I make this parallel with the coding level of advanced (you definitely love coding if you enter this level :)). This involves everything from the intermediate level along with grasping the advanced data structures e.g. Tries, AVL, B-Trees, etc as well as recognizing optimal patterns of programming e.g. whether dynamic programming will be a good solution here to this problem as it looks similar to another problem I have seen before? An engineer at this level can break down a large problem into various subproblems and select the optimal data structures and the applicable algorithm to solve each subset of problems thereby eventually solving the larger problem in the most optimal manner. e.g. Can I nest two data structures to solve this subproblem and integrate it with the other functions I have to solve the entire problem at hand?

Coming up another level up the chain, we progress from advanced to an expert in the coding hierarchy. At this level, you have an increasingly high confidence in your coding abilities, have seen and solved a broad swath of problems to quickly choose the most optimal solution, understands the micro-optimizations that apply to even rarely used data structures, may have or can compete in state, national and international level programming competitions and potentially have written out impactful code within an organization or outside.

The topmost level of Maslow’s hierarchy is self-actualization where a human attains their full potential. I compare this with the coding level of “Guru” where you are one of the limited few in the world who develop new data structures and/or programming languages, conduct breakthrough research, and come up with solutions to unsolved problems, better/faster solutions to an otherwise considered a complicated and slow class of problems or have contributed extensively to the world’s most used code bases. A few living legends that I consider for this level (and even beyond) are Donald Knuth, James Gosling, Linus Torvalds, Dennis Ritchie, Ken Thompson, and the like.

Another noteworthy aspect in the above description is that each level can have sub levels as well where you move through them progressively. This progression from one level to the next takes non linear time as you move higher up in the hierarchy i.e. moving from basic to intermediate is relatively faster than moving from intermediate to advanced etc. Security engineers should go beyond the basic level and into the intermediate and advanced territory for optimal performance. After that if you do find the time, the need on the job, or just have a burning desire to go up the hierarchy, please by all means go forth and conquer.

Security Engineer Interviews – An Interviewer’s Perspective

~ allthingspwned ~ Leave a comment

Full Disclosure: I am a security engineer @ Google and the following are my own opinions.

The last post focused on cracking the security engineer interview for a prospective candidate. In this post, we turn the tables and focus on the interviewer’s side and their perspective.

First and foremost, interviewing others is HARD. While it may appear that the interviewer is having an easy time asking questions, that is nothing farther than the reality. Interviewers are like the rest of us who have been in the interviewee’s shoes at some point, most probably have a developed skillset and it is part of their job to help in growing their team by hiring their peers. Just like the interviewee, an interviewer likely also spends time before the interview preparing the set of questions to ask, notes feedback during the interview, and participates in the interview follow up process depending on the company.

Every interviewer has their style and uniqueness in the way they go about conducting an interview. While this may vary depending on the interviewer and the type of interview (open-ended, coding, soft skills, etc.), there is an underlying purpose behind the interview round which boils down to getting an accurate sense of the candidate’s strengths, areas of improvement and the level of thereof. With the above in mind, I have developed a framework called “PCEGA” for interviewers that can serve as a guide to carry out productive interviews that lead to better candidate experience irrespective of the type of interview.

Prepare: This goes without saying but generally preparation prior to conducting an interview is beneficial. This might involve looking at the candidate’s background (and other data available), preparing a list of questions and follow up questions to ask. It is also a good idea to have an extra set of questions available as a backup in case the candidate goes through the list faster than expected.

Connect: This is an essential step when meeting the candidate. Taking some time to connect with the candidate generally offers a good experience for both. This might involve greeting them, making the candidate comfortable in the interview setting as much as possible, and giving them an overview of your role, the structure of the interview, etc. The idea behind connection should be to put the candidate at ease and aware for them to perform their best in the interview. This step will also greatly vary depending on the company, position, interviewer on the amount of information that is accepted “okay” to be shared but the central idea remains the same.

Explore: In this phase, the interviewer will go through the set of questions they prepared to ask to explore the candidate’s strengths and areas of improvement depending on their style and the type of interview. It is hard to comment on a guideline here as it is very specific to the role, position, company, and interviewer however the general idea here to be comprehensive and get enough “signals” to accurately comment on the candidate’s assessed skills. An informed decision with supporting data is better accepted than an uninformed opinion.

Guide: During the interview, an interviewer can also serve as a guide, helping the candidate get unstuck (if needed) and providing a general sense of direction if desired. The interviewer is also responsible for answering follow up questions from the candidate to provide more clarity on questions, especially open-ended ones. This step is highly subjective based on how the interviewee performs but the general idea is that the interviewer can be essential in unblocking the candidate and also steer the candidate in the right direction if the situation demands. Once the candidate has taken up the guidance, the exploration can continue (denoted by the dashed line in the depiction of the framework). The explore and guide steps can continue in a loop until enough signals have been received.

Answer: Finally, an interviewer can spend some time answering any questions a candidate might have. The idea here is to not only provide a balance in the interviewing process but also to clarify and inform the candidate about things that they might be curious about related to the job/company. This is fairly freeform but the interviewer can try their best to answer as much as possible (and permitted) to provide a smooth experience to the candidate.

The above framework provides a structure for interviewing candidates and provides them with a good interviewing experience.

Life of a forensics and incident response engineer

~ allthingspwned ~ Leave a comment

I have grown to never fully trust that I will be able to attend meetings scheduled in my calendar. As a forensics and incident response engineer for one of the top 10 alexa websites (As of 2017), things turn up quite unexpectedly and quite often. In this post, I wanted to highlight certain aspects of digital forensics, incident response, its pros and cons and what you should do to be sane or to keep growing. In the end, I also touch upon certain recommendations I have for people who plan to enter this field and be passionate about it.

Digital Forensics is based of science in which you apply knowledge of information security, computer systems, networks and programming to analyze 5 W’s (what, where, when, why and who) whereas incident response is an interdisciplinary field and lays out certain practices and procedures involving everything including digital forensics along with laws, crime, psychology, human factors and communication. The combination of the two is what makes life interesting. Fortunately, I get to practice all of this day in and day out. Since the number of fields I listed above are very broad and each field has a lot of depth, gaining complete understanding will take a lifetime (or even more) and as such, there is always something new to learn, apply and play with. As one peels off evidence (think hard drive or memory capture) layer by layer, it tells a story (or supports or refutes one you had in mind). In the end, there is utmost satisfaction in using every part of your working mind to analyze things that happened in the past, how to stop the bad guys before they get in, predict what they would do next and how to protect the organization from further harm.

Incident responders are like firefighters in the digital world, where the equivalent of a fire is a security loophole, an attack, vulnerability/risk or anything that can potentially harm an organization’s systems/networks. Recall the 2 obvious responses of people when they are faced with danger – “fight” OR “flight”. For an incident responder, there is NO flight! When one is always in constant “fight” mode, it builds certain skills which are extremely valuable for a lifetime. For instance, seasoned incident responders are calm, have great self control and take cool, collected decisions even when the world is burning down on them. They build up this unnatural stamina to face extremes. This is the positive side of incident response. On the other hand, the cons include burnouts which affects one’s physical and mental health. One has to strike a balance between fight mode and break mode (forced-flight) to lead a sane life.

If what I described above made you excited about a life filled with new puzzles every day and want to learn more about how to get started in this field, I would recommend starting off with solidifying the fundamentals of computer science (call this layer 0). This means understand every single bit and byte involved in network protocols, understand how operating systems works, learn to program (efficiently), learn data structures, compilers, how to build websites etc. Once you get to a stage where you know how the basic blocks of a computer work, jump to information security (layer 1) and repeat the same thing, however this time using your layer 0 as a foundation. Learn the basic principles, cryptography, ciphers, network security protocols, application security concepts, web security. When you reading about all this, always think how each and every concept ties back to the basic principles of computer science (layer 0). Now build layer 2 on top of information security which is digital forensics. This time learn about host based forensics and network forensics (all the the while thinking about how layer 1 and layer 0 are working underneath this). There will be a lot of “AHAs” and “Oh GOT IT!” during your journey of discovering the beauty of the field of digital forensics. The incident response layer goes hand in hand with the digital forensics layer where concepts run in parallel. I recommend starting out with reading the NIST guide to computer security incident handling, blue team handbook or taking a SANS class.  Additionally, learn the basics about laws in your country with regards to evidence handling. Hopefully by this time, you will be armed with the skills you need to get started off as a forensics and IR engineer.

 

Installing python scapy on MAC OSX Sierra

~ allthingspwned ~ Leave a comment

Python ships by default with OSX and resides in the path /usr/bin/python. Having run into a lot of problems with installing and using scapy and other packages with the correct version of python, pip and homebrew in the mix, I decided to write steps that anyone can follow to setup a clean and working python environment in OSX without running it in a virtual environment or fiddling with the default python installation.

Step 0: Check existing python version by the command `which python`. This should yield
$ /usr/bin/python

Step 1: Install homebrew
$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
(Reference: https://brew.sh/)

Step 2: Install python and pip using brew and pypcap using pip:
brew install python
pip install pypcap

Step 3: Setup your PATH variable correctly in ~/.bashrc to use the python installed by the above command in /usr/local/bin:
PATH=/usr/local:/usr/local/bin:$PATH
(add the above line at the end of your ~/.bashrc or ~/.zshrc, whichever shell you are using)

Step 4: Force updates to PATH variable by using source:
$ source ~/.bashrc

Step 5: At this point, if everything worked correctly, the output of `which python` and `which pip`should yield /usr/local/bin/python and /usr/local/bin/pip respectively

Step 6: Install scapy
$ pip install scapy

Step 7: Run scapy
$ scapy
If you get errors like “INFO: Can’t import packageXYZ. …”, see step 8 below to fix dependencies.

Step 8: Fix Dependencies
i) pyx
(If using python 2.7): $ pip install pyx==0.12.1 -I –no-cache
(If using python 3): $ pip install pyx

ii) pycrypto
$ pip install pycrypto

iii) ecdsa lib
$ pip install ecdsa

Step 9: Test scapy again
$ scapy
Welcome to Scapy (2.3.3)
>>>

The above shows a successful run of scapy with all dependencies correctly installed.

Wordlocks, really?

~ allthingspwned ~ Leave a comment

Recently one of my friends purchased a bike from walmart. At work, he was showing me the new beauty when i noticed that it was locked using a wordlock. Now these “wordlocks” are these fancy locks in which the required combination to unlock is formed by letters instead of traditional numbers or lock/key mechanism. They are designed to be “easy to remember” for the bike rider. As a sample, see the image of a wordlock below

(Image courtesy: wikimedia)

WordLock_word_lock_combination_lock_8286729624_o

These cost anywhere from $9 to as high as $20. These locks advertise a total of 10,000 possible combinations. In terms of their “security rating” which is between 1 and 5, 5 being the safest lock, they go up to level 2.

Curious to ride my friend’s new bike I was trying my luck at guessing his word combo. Smiling at my vain efforts to guess, he gave me 24 hours to get the right password. (24 hours is waaayyyy too much, but who says no to extra time?). I went back home, picked up a sample wordlock that had 4 character combinations with 10 “random” letters in each line making a total of 10^4 = 10,000 possible combinations. A quick python script to try out all combinations and checking if they are words in the english US dictionary using the enchant library is as follows:-

import enchant
import sys
dial1 = raw_input("Enter characters on first dial:")
dial2 = raw_input("Enter characters on second dial:")
dial3 = raw_input("Enter characters on third dial:")
dial4 = raw_input("Enter characters on fourth dial:")

if len(dial1) != len(dial2):
sys.exit("dials should have same number of characters")

if len(dial2) != len(dial3):
sys.exit("dials should have same number of characters")

if len(dial3) != len(dial4):
sys.exit("dials should have same number of characters")

d = enchant.Dict("en_US")
count = 0

for c1 in dial1:
    for c2 in dial2:
        for c3 in dial3:
            for c4 in dial4:
                tempstr = c1+c2+c3+c4
                if d.check(tempstr) is True:
                    count = count + 1
                    print tempstr
print "Total combinations:" + str(count)

I ran this script with the characters on a sample wordlock as shown below

 
wordlock-edited

This gives only 836 possible combinations. Now to try the combinations in reality, I set the first letter to a fixed character say ‘l’, the second to ‘e’, the third to ‘e’ and then manually go over every possible combination in dictionary that my program spitted out. Lets assume i take about 3 seconds to try each valid combination, that yields 836*3 seconds = 2508 seconds / 60 = 41.8 minutes (say 45 minutes approx). We know that brute force is not the smart way of breaking things but hey, it just works here. The whole idea of “wordlocks” making it easier to remember combinations just reduced the security by (1 – 836/10,000)*100 = 91%.

I do not feel safe leaving my bike anywhere more than 45 minutes with wordlocks. As they say in information security, you are only as secure as the weakest link in the security chain, wordlocks are “literally” the weakest chain in bike security. Pwned.