How AWS, Azure, and GCP Cloud Storage Integrates with DevOps: Use Cases and Services

Cloud storage forms the backbone of today's cloud computing landscape allowing companies to store and handle their data . The big cloud providers - AWS, Azure, and Google Cloud Platform (GCP) - each provide powerful storage options designed for different needs. This blog will delve into the cloud storage offerings from these three top providers, their applications, and how they work with DevOps methods.

1. What is Cloud Storage?

Cloud storage is a way to keep data on servers you can reach through the internet. It's more scalable, flexible, and cost-effective than storing data in-house. These cloud systems can handle all sorts of data tasks, from quick access needs to keeping files safe for a long time.

2. Cloud Storage Services Across Major Providers

AWS (Amazon Web Services)

• Amazon S3 (Simple Storage Service):

• • Use Cases: Web and mobile app data, backups, data archives, and big data analytics.

    • Tiers:

    • • Standard: To store often-accessed data with high durability.

        • Intelligent-Tiering: To move data between frequent and infrequent access tiers.

        • Glacier and Glacier Deep Archive: To keep long-term archives at low cost.

    • Integration with DevOps: AWS S3 stores artifacts, backups, and logs in CI/CD pipelines, and hosts static websites.

• Amazon EBS (Elastic Block Store):

• Use Cases: EC2 instances use block storage, which works well for databases and app data.

• • Integration with DevOps: Containerized apps managed by AWS ECS or EKS can use EBS volumes to store data long-term.

Azure (Microsoft Azure)

• Azure Blob Storage:

• • Use Cases: Unstructured data like text, images, and videos; data lakes; backup and archival.

    • Tiers:

    • • Hot: For data people access often.

        • Cool: For data people don't access much.

        • Archive: For long-term storage that costs less.

    • Integration with DevOps: Azure Blob Storage has an impact on storing artifacts and logs, backing up data, and hosting static websites in Azure Pipelines.

• Azure Disk Storage:

• • Use Cases: Persistent disks for VMs, databases, and apps.

• Integration with DevOps: Azure Disk Storage offers block storage for VMs, which comes in handy to meet persistent data needs in Azure Kubernetes Service (AKS) or other container-based setups.

GCP (Google Cloud Platform)

• Google Cloud Storage:

• • Use Cases: Storing and getting unstructured data, like media files, backups, and data for analysis.

    • Tiers:

    • • Standard: For data you access often.

        • Nearline: For data you access less than once a month.

        • Coldline: For data you access less than once a year.

        • Archive: For long-term storage at the lowest cost.

    • Integration with DevOps: Google Cloud Storage works with CI/CD pipelines. It serves as a place to keep build artifacts, logs, and packages for deployment.

• Google Persistent Disk:

• Use Cases: Compute Engine VMs, databases, and apps store their data on block storage.

• • Integration with DevOps: GCP's Kubernetes Engine (GKE) and other container setups rely on it to store data long-term.

3. Common Use Cases for Cloud Storage

• Backup and Recovery: Cloud storage keeps regular backups of vital data and systems, which helps businesses stay up and running.

• Data Archiving: Cloud storage provides a place to keep data for a long time even if people don't need to look at it often.

• Big Data Analytics: Cloud storage holds huge amounts of data so people can study and work with it.

• Web and Mobile Applications: Cloud storage hosts static files, media content, and app assets.

4. Integrating Cloud Storage with DevOps

1. Artifacts and Package Management:

• Keep build artifacts deployment packages, and Docker images in cloud storage services.

• Example: Use AWS S3 or Azure Blob Storage to keep artifacts from Jenkins or Azure Pipelines.

2. Logs and Monitoring:

• Collect and examine logs from applications and infrastructure.

• Example: Use Google Cloud Storage to gather logs from various sources for centralized monitoring.

3. Backup and Disaster Recovery:

• Put backup strategies and disaster recovery plans into action using cloud storage.

• Example: Keep daily backups of databases and application data in Azure Blob Storage's Cool or Archive tiers.

4. Static Website Hosting:

• Host static websites and deliver content .

• Example: Use AWS S3, Azure Blob Storage, or Google Cloud Storage to host static content for a website.

5. Data Lakes:

• Store and analyze big datasets in one central place.

• Example: Use AWS S3 or Google Cloud Storage to build a data lake to analyze big data.

5. Conclusion

Cloud storage options from AWS, Azure, and GCP give you many ways to handle your data needs. When you know what each service can do and how to use it, you can pick the right storage level to get the best performance at the right price. Using cloud storage with DevOps helps make things more automatic more productive, and more reliable when you manage and launch apps.

When you add cloud storage to your DevOps work, it makes things run smoother and helps you handle data well from start to finish. Whether you're dealing with app data, backups, or big data, using the right cloud storage can have a big impact on how quick and successful your company is.

- Intelligent-Tiering
- Glacier
- Glacier Deep Archive | - Hot
- Cool
- Archive | - Standard
- Nearline
- Coldline
- Archive | | Use Cases | - Web app data
- Backup and restore
- Data archiving
- Big data analytics | - Web and mobile app data
- Backup
- Data lakes
- Content storage | - Web app data
- Backup
- Long-term archival
- Big data analytics | | Block Storage | Amazon EBS (Elastic Block Store) | Azure Disk Storage | Google Persistent Disk | | Purpose | Provides persistent block storage for EC2 instances. | Provides persistent block storage for VMs. | Provides persistent block storage for VMs. | | Use Cases | - Databases
- Application data
- Filesystems | - Databases
- Applications
- Filesystems | - Databases
- Applications
- Filesystems | | Integration with DevOps | - Store build artifacts
- Backup data
- Host static websites | - Store artifacts
- Backup data
- Host static websites | - Store build artifacts
- Backup data
- Host static websites |

Key Points

• General Object Storage: This is used for storing unstructured data and offers various storage tiers based on access frequency and cost.

• Block Storage: This provides persistent storage that can be attached to virtual machines and is used for applications requiring low-latency access to data.

This table should help you compare similar cloud storage services across AWS, Azure, and GCP.

Azure Cloud Storage offers two main performance tiers: Standard and Premium. These tiers differ in terms of performance, use cases, and cost. Here’s an overview:

1. Standard Storage

• Performance: Optimized for general-purpose storage, suitable for workloads with varying performance needs.

• Use Cases:

  • Backup and restore

  • Data archiving

  • Large object storage (e.g., images, videos)

  • General-purpose file sharing

• Characteristics:

  • Stores data on hard disk drives (HDDs).

  • Offers consistent performance with a balance between cost and latency.

  • Supports all redundancy options (LRS, GRS, ZRS, GZRS).

  • Lower cost compared to Premium Storage.

• Scenarios: Best for applications with high capacity and lower performance requirements, such as file storage, backups, and content repositories.

2. Premium Storage

• Performance: Designed for low-latency, high-performance workloads with intensive I/O requirements.

• Use Cases:

  • Virtual machines (VMs) running performance-sensitive applications

  • Databases (SQL, NoSQL)

  • High-transactional workloads

• Characteristics:

  • Uses solid-state drives (SSDs) for faster read/write operations.

  • Higher throughput and IOPS (Input/Output Operations Per Second) compared to Standard Storage.

  • Supports only LRS for most types of storage accounts (other redundancy options are limited).

  • Higher cost but significantly better performance.

• Scenarios: Ideal for mission-critical applications, high-performance databases, and any workload requiring low latency and high IOPS.

Key Differences:

• Performance: Premium Storage offers much faster read/write performance compared to Standard Storage.

• Cost: Standard Storage is cost-efficient for general storage needs, while Premium Storage is more expensive due to its enhanced performance.

• Use Cases: Standard Storage is for everyday applications and archiving, while Premium Storage caters to high-transaction workloads requiring consistent low latency.

Here’s a detailed comparison of the "Hot," "Cool," and "Archive" storage tiers in Azure Blob Storage, including examples for each:

Examples:

1. Hot Storage Tier:

   • Scenario: A popular e-commerce website where user profiles, product images, and recent transactions are accessed frequently.

   • Reason: The Hot tier offers low latency and high performance, which is ideal for data that is actively used and frequently accessed.

2. Cool Storage Tier:

   • Scenario: A company that regularly backs up its data but does not access these backups on a daily basis. For example, weekly backups of application logs or monthly snapshots of a database.

   • Reason: The Cool tier is cost-effective for data that is infrequently accessed but still needs to be readily available.

3. Archive Storage Tier:

   • Scenario: An organization needs to store financial records for compliance reasons. These records are rarely accessed, and the organization is looking to minimize storage costs.

   • Reason: The Archive tier offers the lowest cost for storing data that is rarely accessed, though retrieval will take longer when needed.

This comparison should help in selecting the appropriate storage tier based on data access patterns and cost considerations.

In Azure, different types of redundancy options are available for cloud storage to ensure data durability and availability. Here's a breakdown of the common redundancy options: (LRS,GRS,ZRS,GZRS)

1. Locally Redundant Storage (LRS)

• Definition: LRS keeps multiple copies of your data within a single physical location in a region (i.e., one data center).

• Replication: Data is replicated 3 times within a single data center.

• Use Case: Suitable when your primary concern is to protect against local hardware failures but you don't require protection against regional disasters.

• Pros: Cost-effective and provides high availability within a data center.

• Cons: No protection against data center or regional outages.

2. Geo-Redundant Storage (GRS)

• Definition: GRS provides additional protection by replicating your data to a secondary region, hundreds of miles away from the primary region.

• Replication: Data is replicated 3 times within the primary region (LRS) and then asynchronously replicated to another data center in a different region (3 more copies).

• Use Case: Ideal for businesses that require high availability and disaster recovery even in the event of a regional outage.

• Pros: Provides cross-region redundancy, increasing data durability.

• Cons: More expensive than LRS and has higher latency for read operations from the secondary region.

3. Zone-Redundant Storage (ZRS)

• Definition: ZRS replicates your data synchronously across multiple availability zones within the same region.

• Replication: Data is replicated 3 times across separate availability zones within the region.

• Use Case: Ideal for applications requiring high availability and protection against data center-level failures within a region.

• Pros: Offers better fault tolerance by leveraging multiple availability zones. Instant failover without downtime.

• Cons: More expensive than LRS and requires a region with multiple availability zones.

4. Geo-Zone-Redundant Storage (GZRS)

• Definition: GZRS combines the benefits of ZRS and GRS by replicating data both across availability zones within the primary region and asynchronously to a secondary region.

• Replication: Data is replicated across 3 availability zones in the primary region and then asynchronously replicated to a secondary region.

• Use Case: Best suited for mission-critical applications requiring maximum durability and high availability across multiple regions and zones.

• Pros: Provides the highest level of resilience by protecting against both regional and zone-level failures.

• Cons: The most expensive redundancy option, and write latency may be higher due to cross-region replication.

Conclusion

Choosing the right storage redundancy option depends on your specific business needs and risk tolerance:

• For basic needs: LRS.

• For disaster recovery: GRS.

• For zone resilience within a region: ZRS.

• For ultimate protection and disaster recovery: GZRS.

Selecting the right redundancy option ensures your data is safe and accessible according to your requirements.

Unveiling Similar Cloud Services: A Comprehensive Comparison of AWS, Azure, and GCP

Les See a Example In AZURE

A Comprehensive Guide to Creating and Using Azure Blob Storage

Azure Blob Storage is a versatile solution for storing large amounts of unstructured data, such as text or binary data, images, videos, and backups. In this blog, we'll demonstrate how to create a storage account on Azure, set it up for use, and explore the advantages it provides.

Step 1: Creating an Azure Storage

Account Follow these steps to create your Azure Storage Account:

Step 1: Sign In

1. Log into Azure Portal Navigate to the Azure portal at portal.azure.com.

2. Sign in with your Azure credentials.

Step 2: Create a Storage Account

1. Click on “Create a resource” in the left-hand navigation pane.

2. Select “Storage” and then “Storage account.”

Step 3: Basics Tab

1. Subscription: Select Free Trial.

2. Resource group: Create a new resource group or select an existing one (e.g., ‘StorageCloud’)..

3. Storage account name: Enter a globally unique name for your storage account (e.g., ‘ccstorage0’).

4. Region: Select Central US.

5. Performance: Choose Performance.

6. Account kind: Select StorageV2 (general-purpose v2).

7. Replication: Choose Locally-redundant storage (LRS).

8. Access tier: Select Hot.

Step 4: Advanced Tab

1. Enable hierarchical namespace: Set to Disabled.

2. Enable SFTP: Set to Disabled.

3. Enable network file system v3: Set to Disabled.

4. Allow cross-tenant replication: Set to Disabled.

5. Enable large file shares: Set to Enabled.

Step 5: Security Tab

1. Secure transfer: Set to Enabled.

2. Blob anonymous access: Set to Enabled.

3. Allow storage account key access: Set to Enabled.

4. Default to Microsoft Entra authorization in the Azure portal: Set to Disabled.

5. Minimum TLS version: Select Version 1.2.

6. Permitted scope for copy operations (preview): Set to From any storage account.

Step 6: Networking Tab

1. Network connectivity: Select Public endpoint (all networks).

2. Default routing tier: Choose Internet routing.

Step 7: Data Protection Tab

1. Point-in-time restore: Set to Disabled.

2. Blob soft delete: Set to Enabled.

3. Blob retainment period in days: Enter 7.

4. Container soft delete: Set to Enabled.

5. Container retainment period in days: Enter 7.

6. File share soft delete: Set to Enabled.

7. File share retainment period in days: Enter 7.

Step 8: Versioning Tab

1. Versioning: Set to Disabled.

2. Blob change feed: Set to Disabled.

3. Version-level immutability support: Set to Disabled.

Step 9: Encryption Tab

1. Encryption type: Choose Microsoft-managed keys (MMK).

2. Enable support for customer-managed keys: Set to Blobs and files only.

3. Enable infrastructure encryption: Set to Disabled.

Step 10: Review and Create

1. Review all the settings to ensure they match the required configuration.

2. Click "Create" to deploy the storage account.

Live Screen Shots of Container And BOB After Creation

Step 2 Create a container

step 3 Container created successfully And now Add data

Step 4 Click on container Upload option and upload data

Step 5

Output of container data :-

Here’s a summary of the properties for the blob you described:

• URL: ccstorage0.blob.core.windows.net/container/..

• Last Modified: 25/8/2024, 1:00:52 AM

• Creation Time: 25/8/2024, 1:00:52 AM

• Version ID: Not applicable (no versioning enabled)

• Type: Block blob

• Size: 121.73 KiB

• Access Tier: Hot (Inferred from configuration)

• Access Tier Last Modified: Not applicable

• Archive Status: Not archived

• Rehydrate Priority: Not applicable

• Server Encrypted: Yes

• ETag: 0x8DCC4734430CBA6

• Version-Level Immutability Policy: Disabled

• Cache-Control: Not set

• Content-Type: image/jpeg

• Content-MD5: dEpkA2J0qeRTluUZOn0oQg==

• Content-Encoding: Not set

• Content-Language: Not set

• Content-Disposition: Not set

• Lease Status: Unlocked

• Lease State: Available

This blob is an image file with standard properties for a hot-tier, block blob in Azure Storage. If you have any specific questions or need further details, feel free to ask!

Pratik Lahamge
LinkedIn | Portfolio | GitHub | Email