cost

AWS Hardening

Posted by

0

Cloud Maturity Series — Posts 1–3 (LinkedIn Drafts)

Post 1: Start With the Account, Not the App

Most cloud problems don’t start in code.

They start in the AWS account itself.

 

Before VPCs.

Before CI/CD.

Before a single workload is deployed…

 

I lock down the account foundation:

• MFA on the root account

• Root access keys removed

• Break-glass strategy defined

• IAM users eliminated in favor of roles

• Clear ownership and access boundaries

 

Cloud maturity starts at the control plane, not the application layer.

 

#AWS #CloudArchitecture #IAM

#CloudSecurity #PlatformEngineering

#Infrastructure #OpenToWork

Post 2: Identity Is the Real Perimeter

Firewalls don’t protect cloud environments.

Identity does.

 

Once the account is secured, the next layer is IAM done right:

• Role-based access instead of long-lived users

• Least-privilege policies mapped to job function

• Explicit separation between human and workload identities

• MFA everywhere it makes sense

 

In AWS, IAM is the blast radius.

 

If identity is loose,

no amount of network segmentation will save you.

 

Get this layer right and everything downstream becomes safer:

deployments, automation, audits, and incident response.

 

Cloud security isn’t a tool problem.

It’s an identity design problem.

 

#AWSIAM #ZeroTrust #CloudSecurity

#DevSecOps #PlatformEngineering

#InfrastructureAsCode #OpenToWork

Post 3: Auditability Before Availability

Most cloud failures aren’t outages.

They’re untraceable changes.

 

Before scaling workloads or optimizing cost,

I establish control-plane observability.

 

That means always being able to answer:

• Who changed this?

• What changed?

• When did it happen?

• Can we prove it?

 

This layer is built with:

• CloudTrail — immutable API history

• AWS Config — resource state and drift detection

• GuardDuty — security signal, not just logs

 

Compliance benefits (SOC 2, ISO, PCI) are a side effect.

The real value is engineering control.

 

If you can’t reconstruct an incident,

you don’t truly control your environment.

 

Observability isn’t something you add later.

It’s something you start with.

 

#AWS #CloudEngineering #AuditLogging

#IncidentResponse #DevSecOps

#InfrastructureAsCode #OpenToWork

The Warehouse Sizing Paradox

Posted by

0

The Warehouse Sizing Paradox: Why I Sometimes Choose XL Over Small
“Always use the smallest warehouse possible to save money.”
I heard this advice constantly when I started with Snowflake. It sounds logical—smaller warehouses cost less per hour, so naturally they should be cheaper, right?
Except the math doesn’t always work that way.
Here’s what I’ve observed:
Snowflake charges by the second with a 60-second minimum. The cost difference between warehouse sizes is linear, but the performance difference often isn’t.
The actual formula is simple:
Total Cost = Credits per Second × Runtime in Seconds
A Small warehouse might be 1/4 the cost per second of an XL, but if it takes 5x longer to complete the same query, you’re paying more overall.
When I’ve seen this matter most:
Working with subscription and order data, certain query patterns consistently benefit from larger warehouses:
→ Customer lifetime value calculations across millions of subscribers
→ Daily cohort analysis with complex retention logic
→ Product affinity analysis joining order details with high SKU cardinality
→ Aggregating subscription events over multi-year periods
These workloads benefit dramatically from parallelization. An XL warehouse has 8x the compute resources of an XS, and for the right queries, it can complete them in less than 1/8th the time.
A simple experiment you can run:
— Test with Small warehouse
USE WAREHOUSE small_wh;
SELECT SYSTEM$START_QUERY_TIMER();

SELECT
    subscription_plan,
    customer_segment,
    COUNT(DISTINCT customer_id) as subscribers,
    SUM(order_value) as total_revenue,
    AVG(order_value) as avg_order_value,
    COUNT(DISTINCT order_id) as total_orders
FROM orders
WHERE order_date >= ‘2023-01-01’
GROUP BY subscription_plan, customer_segment
HAVING COUNT(DISTINCT customer_id) > 100;

— Note the execution time and credits used in query profile

— Now test with XL warehouse
USE WAREHOUSE xl_wh;
— Run the same query
Check the query profile for each:
Execution time
Credits consumed (Execution Time × Warehouse Size Credits/Second)
Total cost
The decision framework I use:
Size up when:
Query runtime > 2 minutes on current warehouse
Query profile shows high parallelization potential
You’re running the query repeatedly (daily pipelines)
Spillage to remote disk is occurring
Stay small when:
Queries are simple lookups or filters
Runtime is already under 30 seconds
Workload is highly sequential (limited parallelization)
It’s truly ad-hoc, one-time analysis
The nuance that surprised me:
It’s not just about individual query cost—it’s about total warehouse utilization. If your Small warehouse runs 10 queries in 100 minutes, but an XL runs them in 20 minutes, you’re paying for 80 fewer minutes of warehouse time. That matters when you’re paying for auto-suspend delays, concurrent users, or just opportunity cost.
My practical approach:
I start with Medium for most workloads. Then I profile:
Queries consistently taking 3+ minutes → test on Large or XL
Queries under 1 minute → consider downsizing to Small
Monitor credit consumption patterns weekly
The goal isn’t to find the “right” size—it’s to match warehouse size to workload characteristics.
Want to test this yourself?
Here’s a quick query to see your warehouse credit consumption:
SELECT
    warehouse_name,
    SUM(credits_used) as total_credits,
    COUNT(*) as query_count,
    AVG(execution_time)/1000 as avg_seconds,
    SUM(credits_used)/COUNT(*) as credits_per_query
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD(day, -7, CURRENT_TIMESTAMP())
    AND warehouse_name IS NOT NULL
GROUP BY warehouse_name
ORDER BY total_credits DESC;
This shows you which warehouses are consuming credits and whether you might benefit from right-sizing.
The counterintuitive truth:
The cheapest warehouse per hour isn’t always the cheapest warehouse per result. Sometimes spending more per second means spending less overall.
What’s been your experience with warehouse sizing? Have you found scenarios where bigger was actually cheaper?
#Snowflake #DataEngineering #CostOptimization #CloudDataWarehouse

**Temp tables, CTEs, or RESULT_SCAN? Here’s how I decide.**Every time I’m building a data transformation in Snowflake, I ask myself the same three questions. Getting these right has become one of the most practical ways I optimize both performance and cost.The problem? Most teams pick one pattern and use it everywhere. I did this too early on—defaulting to temp tables for everything because they felt “safe.”**Here’s the framework I’ve developed:****Question 1: How long do I actually need this data?**→ Just iterating on analysis right now?   **RESULT_SCAN**   – Reuse cached results from expensive queries   – Zero compute cost for subsequent filters/aggregations   – Perfect for exploration, but expires in 24 hours   – When: Refining a report, exploring segments, debugging→ Need it for multiple operations in this session?   **TEMP TABLES**   – Materialize once, reference many times   – Supports updates, joins, clustering   – Persists through the session   – When: Multi-step ETL, quality checks, complex workflows→ Just organizing logic within one query?   **CTEs**   – Maximum readability, zero storage   – Snowflake optimizes the entire plan together   – NOT materialized (this surprised me initially)   – When: Breaking down complex business logic**Question 2: Is this ad-hoc or production?**Ad-hoc analysis → Lean toward RESULT_SCAN and CTEsProduction pipeline → Temp tables for reliability and testability**Question 3: Am I reusing this computation?**If you’re filtering/joining the same expensive base query multiple ways, that’s your signal to materialize it somehow—either as a temp table or by leveraging RESULT_SCAN.**What this looks like in practice:**Imagine a daily reporting workflow:- Expensive aggregation across billions of rows → TEMP TABLE (computed once)- Logical transformations on that data → CTEs (readable, optimized)- Stakeholders request variations → RESULT_SCAN (free iterations)This hybrid approach combines reliability, readability, and cost efficiency.**The shift in mindset:**I stopped asking “which pattern should I use?” and started asking “what does this specific transformation actually need?”Snowflake gives us different tools for different jobs. The art is knowing when each one fits.**What decision-making frameworks have helped you optimize your Snowflake workflows?**#Snowflake #DataEngineering #CloudDataWarehouse #CostOptimization—

Posted by

0