This article will describe techniques that we use to manage configuration changes among various projects in a large distributed Azure-based solution called AzureWatch and keep settings in sync as various environments are targeted during development, testing, and debugging sessions.  AzureWatch is an auto-scaling and monitoring system for Windows Azure applications.  By its nature it utilizes multiple WCF services, a few Windows-based clients and services, a number of web-based projects.  As readers can imagine, keeping all configuration settings in sync and "shifting" them together on demand without making a mistake can be challenging.

Visual Studio technologies utilized

• Using Visual Studio Configurations to define target debugging environments
• Using ConfigSource to split configuration files
• Pre/Post build events
• Add existing files as Links to a Project
• Config Transformations

Approach at a high level

For every debugging environment, create Visual Studio Configuration.  Break out sections from every .Config file that vary by environment and include them back via ConfigSource settings.  Include these sections as unique files into a separate Config project that has folders for every Configuration.  Also, copy *.csdef and *.csccfg files from Azure-related projects into these folders as well. In each folder, customize the broken off partial configuration files (that are included via ConfigSource) and azure configuration files. Create Post or Pre-Build event of the Config project to xcopy the files from a folder that matches current Configuration environment to root folder of Config project.  Include files from root folder of Config project into other projects as needed via “Add as Link” command.

Utilize Config transformations to instrument Production (and other environments if needed) specific configuration changes that deal with security, debugging, tracing, etc.

Now, let's look at these steps in detail:

Step 1

Create as many Visual Studio Configurations as necessary to support number of different environments that developers must be able to debug from Visual Studio

Step 2

Create a separate empty project that is a part of the overall solution called Config.  Include into the project folders, one per matching Visual Studio Configuration environment that were created in Step 1

Step 3

Break-off Connection strings, end-point configuration, and other environment-specific sections from existing .config files into separate “include-only” config sections.

Source example web.config:

  <connectionStrings configSource="config\connectionStrings.config"/>

Broken off example connectionStrings.config:

  <connectionStrings>

    <add connectionString="..." name="db_connection"></add>

  </connectionStrings>

May want to make sure that web-projects have "bin" folder as a part of the relative path: "\bin\config\partialConfigFileName.config"

Step 4

Include separate customized partial .config files and Azure-specific .cscfg files under every sub-folder within Config project

Step 5

Customize Pre-Build event for Config project to copy config files for current configuration to root folder:

xcopy /Y /R "$(ProjectDir)$(ConfigurationName)\*.config" "$(ProjectDir)"

xcopy /Y /R "$(ProjectDir)$(ConfigurationName)\*.cscfg" "$(ProjectDir)"

Step 6

Add-as-Link partial .config files into main projects from the ROOT of the Config project.  To keep things organized, feel free to include partial .config files under Config sub-folders.  Do the same with Azure-specific .cscfg files

Files added as links have a shortcut icon in Solution Explorer

Step 7

Manually add project dependencies to Config project from the other projects in the solution, that need partial config files to function.  This will ensure that Config project is built before other projects.  Project Dependencies can be found inside Solution Properties window, under "Project Dependencies" tab.

Do not forget to specify that partial config files set "Copy to Output Directory" as "Copy if Newer"

Conclusion

Now, every time you debug and run, the pre-compile event from Config project will copy all the files from its particular environment folder into its root folder.  And all other projects that link to this root folder for their partial .config files will automatically get updated .configs switched to proper environment in unison.

Revisions

This blog was revised on 06/10/2012 to include the extra XCOPY step that keeps Azure cscfg/csdef files in sync with the Config project

A question frequently asked during architectural stages of a new Azure-based project: should we choose SQL Azure or Azure Table Storage to store our data?

Answer is typically: YES. Both of these storage technologies compliment rather than compete with each other.  If you are looking to gain maximum scalability, performance, and flexibility while at the same time paying the least amount of money, the trick is to understand the strong points of each technology and utilize both effectively.

Azure Table Services (ATS) is a new storage technology from Microsoft and is specifically designed to handle mega-scalability for applications and websites of Twitter/Facebook/Amazon's capacity.  Storage space is super cheap with ATS.  To offset the low cost and mega-scalability there are a few negative impacts that one must be aware of.  You are not only paying for storing the data, but also for accessing the data that lives in ATS.  There is also only limited support for transactions.  This is key to understand and design around.  ATS also is a new paradigm for developers to grasp.  Lastly, ATS forces your compute nodes to become mini-relational servers.  It simply does not do any of the relational processing we are all used to.  JOINs, GROUP BY's, ORDER BY's all have to be designed around or performed manually. 

I would say that ATS is best suited for large amounts of data that only rarely needs to be accessed or massaged.

On the other hand, SQL Azure is a fast, lighter-weight cloud-based relational storage.  Your developers will know how to code against it right away, because (barring a few small gotchas) coding for it simply like coding for any other SQL database.  On the negative side: SQL Azure is not meant to support huge applications like Facebook, Ebay, Twitter, etc.  Azure, stores your SQL Azure databases along with others in a multi-tenant environment on same servers and thus it has to throttle access should your application become too hot and impacts other databases on the same SQL Azure node.  SQL Azure is also somewhat pricey, at $10/gigabyte/month.  Having said, there is no cost to access the data stored in SQL Azure and there is plenty of CPU power dedicated to perform relational functionality on your data.  

I would recommend that SQL Azure for smaller amounts of frequently accessed data.

A few typical use-cases to illustrate the points:

Data in a banking system that stores customer account information along side a large amount of financial transactions would best divided across SQL Azure and ATS in the following way: Customer account information in SQL Azure, Financial Transactions in ATS.

Content for a large blog site should probably live in SQL Azure until it reaches a certain age and can be archived to ATS.