🚀 DevSecOps Mastery Guide 101

"Emirates Group" screening questions - Every Senior DevOps / Cloud Engineer should know

Master these fundamental DevOps concepts that separate noob engineers from senior ones. From Terraform state locking to idempotency principles, we'll break down complex topics into digestible explanations with real-world analogies.

---

Question 1

You are managing Terraform state files in an Azure Storage Account blob container to enable collaboration. To prevent state corruption during simultaneous runs by different CI/CD pipelines, you need to ensure state locking is active. Which specific Azure Storage feature does the Terraform backend use to acquire a lock on the state file?

• Azure Blob Soft Delete

• Azure Blob Lease

• Blob Versioning

• Immutable Storage Policies

✓ Answer: Azure Blob Lease

💡 Simple Explanation

Think of it like checking out a library book. When someone takes the book (acquires a lease), nobody else can take it until they return it. Terraform "leases" the state file so only one pipeline can modify it at a time. Others have to wait their turn.

🔒 Library Book Analogy: Just like you can't check out a book someone else has, pipelines can't modify a state file that's already "checked out" by another process.

---

Question 2

An application on a production Linux server is crashing, and the logs indicate a "Too many open files" error. You have verified that the system-wide limits are sufficient. Which command allows you to list all open files, network sockets, and pipes specifically associated with the crashing application's Process ID (PID)?

• netstat -plnt

• ps -aux | grep

• lsof -p

• top -H -p

✓ Answer: lsof -p

💡 Simple Explanation

lsof = List of Open Files

It's like asking "Hey, what files is this program currently using?" The "Too many open files" error means the app opened too many files/connections and forgot to close them. This command shows you exactly what it's holding onto.

# List all open files for process 1234
lsof -p 1234

# Example output shows:
# - Open files
# - Network connections
# - Pipes and sockets

---

Question 3

You are designing an automated pipeline where an Azure Virtual Machine (VM) needs to retrieve database credentials from an Azure Key Vault. You want to avoid storing any secrets (like Client IDs or Client Secrets) within the VM's configuration or code. What is the most secure, cloud-native method to authenticate the VM to the Key Vault?

• Create a Service Principal, generate a certificate, and install it on the VM.

• Enable a System-assigned Managed Identity on the VM and grant it access policies in Key Vault.

• Generate a Shared Access Signature (SAS) token for the Key Vault and bake it into the VM image.

• Use the Azure CLI with a dedicated user account and run az login in the startup script.

✓ Answer: Enable System-assigned Managed Identity on the VM and grant it access policies in Key Vault

💡 Simple Explanation

Instead of giving your VM a username/password (which can be stolen), Azure says "I know this VM belongs to you, I'll vouch for it automatically."

🎫 Employee Badge Analogy: It's like being an employee with a badge. You don't need to prove who you are every time — the building recognizes your badge automatically.

🔐 Deep Dive: Managed Identity Security

⚠️ What if the badge is stolen?

• The badge (Managed Identity) is tied to the VM itself — it cannot be copied or exported.

The credentials:

• Live only inside that specific VM

• Rotate automatically (Azure handles this)

• Are accessible only from that VM's internal metadata endpoint (169.254.169.254)

If someone steals the badge (compromises the VM), yes they can use it. But that means your VM itself is compromised — you have bigger problems.

✅ Mitigations:

• Principle of least privilege — give the identity only the permissions it absolutely needs

• Network restrictions — Key Vault firewall rules, private endpoints

• Monitoring — alert on suspicious access patterns

• Short-lived secrets — even if accessed, credentials expire quickly

AWS Equivalent:

Same concept:

• No hardcoded credentials

• Credentials fetched from instance metadata

• Automatically rotated by AWS

• Role is bound to the instance, not portable

---

Question 4

You have written a complex Bash script for a deployment process. You want to ensure that if any command within the pipeline fails (returns a non-zero exit code), the entire script stops execution immediately to prevent cascading errors. Which command should you place at the top of your script?

• set -e

• set -x

• exit 1

• trap "echo error" ERR

✓ Answer: set -e

💡 Simple Explanation

Normally, Bash scripts are stubborn — if something fails, they keep going anyway.

set -e tells the script: "If anything goes wrong, STOP immediately."

👨‍🍳 Chef Analogy: Like telling a chef: "If you burn the first dish, don't keep cooking the rest."

#!/bin/bash
set -e  # Exit on any error

# If this fails, script stops here
curl https://api.example.com/data -o data.json

# This won't run if curl failed
jq '.users' data.json

---

Question 5

A Junior DevOps engineer has written an automation script that appends a configuration line to a file: echo "config=true" >> /etc/app/config. As a Senior Engineer, you flag this code during review because it violates the principle of "Idempotency." Why?

• The script will fail if the file does not exist.

• The script does not verify if the user has root privileges.

• If the script is run multiple times, it will add the same line repeatedly, potentially breaking the configuration.

• The script does not use a variable for the configuration string.

✓ Answer: If the script is run multiple times, it will add the same line repeatedly, potentially breaking the configuration

💡 Simple Explanation

Idempotent = Running something 10 times should give the same result as running it once.

The >> append adds a line every single time. Run it 5 times? You get 5 identical lines. That's bad.

📝 Signup Sheet Analogy: It's like writing your name on a signup sheet every time you walk past it, instead of checking if your name is already there.

🛠️ Deep Dive: Fixing the Idempotency Problem

Why is idempotency needed?

In production:

• Scripts run multiple times (retries, CI/CD reruns, recovery)

• Automation tools like Ansible/Terraform run repeatedly

• Human error — someone accidentally clicks "deploy" twice

If your script isn't idempotent, repeated runs = corrupted system.

How to fix it:

# ❌ Bad (junior's code)
echo "config=true" >> /etc/app/config

# ✅ Good (idempotent)
grep -qxF "config=true" /etc/app/config || echo "config=true" >> /etc/app/config

What this does:

• grep -qxF checks if the exact line already exists

• || means "if NOT found, then add it"

• Run it 100 times — same result as running once

Even Better for Production:

CONFIG_FILE="/etc/app/config"
CONFIG_LINE="config=true"

# Ensure file exists
touch "$CONFIG_FILE"

# Idempotent append
if ! grep -qxF "$CONFIG_LINE" "$CONFIG_FILE"; then
    echo "$CONFIG_LINE" >> "$CONFIG_FILE"
    echo "Config added successfully"
else
    echo "Config already present, skipping"
fi

Or Use Proper Config Management:

• Ansible — lineinfile module (built-in idempotency)

• Puppet/Chef — declarative config management

• Terraform — for infrastructure

These tools handle idempotency automatically so you don't reinvent the wheel.

---

📚 Understanding grep -qxF

The command grep -qxF is a powerful, "quiet" way to check for the existence of an exact line in a file.

💡 Pro Tip:

For case-insensitive matching, add the -i flag: grep -qixF

---

🎯 Key Takeaways

• Always use proper locking mechanisms (Blob Lease) for shared state

• Master debugging tools like lsof for production troubleshooting

• Prefer Managed Identities over hardcoded credentials

• Use set -e to fail fast and prevent cascading errors

• Write idempotent scripts that can run multiple times safely

---

Happy DevSecOps Engineering! 🚀

Master these concepts in depth and level up your infrastructure game in a secured way.