Before you read:

• I write this post only for myself in devops & infrastructure context, not everyone’s else issues.
• And the whole thing could be very discrete, not closely related to each other.

FACTS

1. I’m obsessed with keeping infrastructure up and finding better ways for commitments everyday. My mental health turns bad because of seeing fucking services/ops that were built & done carelessly without any sufficiently exhausting efforts, not only from others engineers, but also from myself.

2. In Devops/SRE scope, trying to adapt incoming tasks, solve problems by providing commonplace solution is…easy, it’s get-shit-done inattentively, something is workaround, temporary. It’s EASY THINGS, come with useless, low impact also very breakable. But good things take time and efforts. And the hard things make big impact.

   As a good rule of thumb, proprietary technology must be at least 10 times better than its closest substitute in some important dimension to lead to a real monopolistic advantage. - Zero to One – Peter Thiel

Well… at some points in my work-life, I realize that issues are not from the context, not from others. Mostly issues are from me, from my bad habits, bad thingking, bad mindsets. It makes me feel shutting down, dumb, fucked-up also with getting lost. I went down the wrong path many times, also got many bad results in 2019.

So…

• AM I UNRELIABLE?
• what’s common failures that I was facing? Lesson learn?
• how can I make myself not to go down the wrong path?
• and how can I avoid stumbling blocks and feel better?

Clickable key points: [rtfm, asking-why, fancy-title, stick-in-the-mud, fearfulness]

Sending malicious HTTP requests (including overside header, meta char, method override header) to CDNs endpoint 
--> these kind of requests are not detected by caching system, are processed by intermediate CDN caching system 
--> then forward to the origin server 
--> origin return server-generated error page 
--> CDN stored error page on edge 
--> CDN serves the same error for normal requests from end-user

Main impact: AWS Cloudfront (FIXED)

Terraform acts so weird when mixing up between aws_security_group and aws_security_group_rule in extending AWS security group rules.

Context: that leads to not idempotent for each apply (without changing anything in IaC)

• Create one security group using aws_security_group with some predefined inlines
• Extending more rules (rows) in that secgroup by using aws_security_group_rule to add more discrete rules
• At each time executing terraform plan/apply, terraform re-creates rules lead to fcked up idempotent of result.

Sample code at:

• DO NOT USE - aws-secgroup-not-idempotence.tf
• BEST PRACTICE - aws-secgroup-best-practice-idempotence.tf

Being on-call means working/fixing live issues under high pressure and almost during midnight and/or weekend.

• Taking strong responsibility for what we work, for what we did, for what we build
• Uptime is an important metric, also critical to the success of our production.
• If our product is down, we acutely feel our customer’s pain
• Being on-call duty means we can learn a lot from it, for both technical knowledge & growing mindset.
• Must get high paid for each on-call duty shift, because of stressful.