Managing Cloud at Scale Doesn’t Have to Be Chaotic

Have you ever felt overwhelmed by the chaos of managing cloud environments?  If so, join the club. It’s a common challenge, and it’s why we set out to reimagine cloud configuration.

Enterprises managing cloud environments often face data sprawl, duplicate inventory processes, and inefficiencies caused by direct API dependencies. This was happening with the legacy inventory system at CMS’s Centers for Clinical Standard & Quality (CCSQ). So, we set out to find a way to make cloud inventory simpler, more reliable, and ready for analytics.

The result was a new platform that integrates AWS Config into a centralized S3 data lake. This transformation not only streamlines inventory visibility, but it also introduces an AWS Bedrock-powered front end for real-time insights and analytics.

Legacy Systems Need a Scalable, Automated, & Resilient Architecture

When our team first reviewed the old inventory process, we found inefficiencies that aren’t uncommon in legacy systems. Even though the system was still functional, it relied on a PowerShell script that queried AWS APIs directly across multiple accounts to gather resource details. 

This approach resulted in 4 main constraints that continued to create issues as the client’s cloud footprint grew:

1. High latency and API throttling due to repeated direct calls.
2. Data duplication (since AWS Config already captured similar information).
3. Manual maintenance whenever APIs changed or new resources were added.
4. Limited scalability and absence of a Dev/Test environment.

Our team knew that a more scalable, automated, and resilient architecture would make things run smoother and faster.

Reimaging Cloud Configuration with an S3-Based Data Lake

To give the client something faster, easier to maintain, and flexible enough to grow with their organization, we needed more than a quick fix.  So, we took a step back and then decided to completely reimagine the client’s cloud configuration.

We designed a cloud configuration platform built on an S3-based data lake with AWS Config as the primary data source. Here are the architecture highlights of our solution:

• Data Source: AWS Config continuously delivers configuration snapshots and changes history to S3.
• Data Lake Layers:
  • Bronze Layer – Stores raw AWS Config data.
  • Silver Layer – Standardizes organizational account data.
• SQL Views & Transformations: Predefined views parse and transform JSON data from AWS Config into tabular formats aligned with existing inventory reports.
• Front End: A modern web interface delivers intuitive visibility into AWS resources and configurations, enhanced with AI-driven querying through Bedrock Knowledge Bases.

Operational & Strategic Benefits

We delivered our solution, and it delivered the results.  The new system provided the client with both operational and strategic benefits like:

• Efficiency Gains: Eliminated redundant API-based data collection.
• Improved Performance: Queries against the S3 data lake are faster and more stable.
• Scalability: Seamlessly supports additional accounts and resource types as the CCSQ environment grows.
• Consistency: A single, authoritative data source reducing discrepancies between environments.
• Reduced Maintenance: Updates are handled via SQL view modifications instead of complex scripting.
• Enhanced Usability: Embedding generative AI and LLM capabilities into the user interface enables intuitive dashboards and search-driven insights.

As CMS’s multi-cloud footprint grows, the new design supports ingestion from all CSP data sources, ensuring a unified view of all cloud assets.

What We Learned from Our Solution

We firmly believe that taking a moment to assess our work, reflect on lessons learned and identify best practices is critical to achieving continued success. Specifically, we identified four best practices from this solution that we’ll use as we move forward:

• Automate data ingestion from native cloud services like AWS Config.
• Adopt layered data lake design (Bronze, Silver, Gold) for flexibility and reusability.
• Use SQL views for abstraction, reducing complexity, and decoupling logic from collection scripts.
• Design front-end systems with AI integration in mind.

Let Us Bring Our Best Practices to You

Ready to simplify your multi-cloud inventory management? Reach out to start a conversation about how our cloud configuration approach could unify your data into a single, intelligent data lake, powered by automation, analytics, and AI.