Steve Spencer's Blog

Blogging on Azure Stuff

Azure Key Vault Logging and Events with Log Analytics

Following on from my previous blog post (http://blogs.recneps.net/post/Setting-up-Azure-Key-Vault-with-Audit-logging) which explains how to set up Azure key vault with logging enabled, this post explains how to access the details of these logs and also to create an alert so you can see if someone is accessing the key vault from an unknown ip address (for example)

Open the Azure portal and navigate to the Resource Groups section and pick the resource group that we configured last time which contains the key vault and log analytics resources

image

Click your log analytics item, to open Log Analytics.

You can then select Log Search

image

This screen allows you to create your own query or select from existing ones.

image

Selecting “All Collected Logs” will show you the logs for the last day. I’ve highlighted the areas where you can change the time period, see the query and also click on Advanced Analytics to give a richer environment for analysing your logs.

image

If you want to query just for the Key Vault Audit logs then you can use the following query:

search * | where Category=="AuditEvent"

image

This will default to a list view, but clicking the Table button will format the data in an easier to read table.

image

You can sort and filter on the column headers. This can also be achieved using the order by clause as follows:

search * |where Category=="AuditEvent"  | order by TimeGenerated desc

A blog post discussing the query language can be found here

We are interested in all calls where someone has tried to access a Secret from the key vault. For that we are looking for an AuditEvent with an OperationName of SecretGet. If we also want to restrict the columns we retrieve then you can use “project” e.g.

search * | where Category=="AuditEvent"  and OperationName == "SecretGet"
| order by TimeGenerated desc
| project TimeGenerated, OperationName, CallerIPAddress, ResultSignature, requestUri_s

image

Now we are familiar with writing queries we can look at alerting. I’d like to set up an alert when the key vault is access from an IP Address other than the one where my application is running. This can be done as follows:

search * | where Category=="AuditEvent" and CallerIPAddress != "51.140.184.51"

This ip address is actually the Azure Portal and is shown when you view the resource group that contains the key vault.I’m using this ip address so that I will actually get an alert (at the wrong time) when my application runs

Click New Alert Rule

image

The following screen should appear

image

The Alert Target should be the Log Analytics we’ve been using and the Target Criteria (when clicked) should show the query we’ve just written

image

We need to configure the rule for when this alert should be triggered. I’m interested when at least 1 attempt has been made in the last 5 minutes to access the Key Vault from an unknown location, so I set the threshold to be zero and click Done. We’ve now configured the logic to determine when the event is fired. Now we need to say what we want to happen when it fires.Firstly we need to give the alert a name and description

image

Now we need to configure how we are alerted. For this you need to create an action group. An action group allows you to define a collection of activities that will happen when the alert is fired. Click New Action Group

image

Action Types can be any of the following:

image

An action group can have multiple actions and you can select both email and SMS in a single action.Once you have created your Action Group you need to select in then click “Create alert rule”

image

Your alert is now set up and running. You can view/edit alerts by selecting Monitor in the Azure Portal

image

then click Alerts (preview), you will be able to see the alerts that have fired.

image

Click Manage Rules to edit the alert.

When the alert is fired I will get an email containing the details of the alert.

Log analytics is a powerful tool and whilst this series of posts has been related to auditing of Key Vault we can use log analytics for a wide variety of log sources such as Application Insights. We can also use the same mechanism for alerting to these other log sources,

The next post is a video that shows you how to connect existing log files to log analytics

Setting up Azure Key Vault with Audit logging

Azure Key Vault is a good way to share secrets with your partners in a way that allows you to have control over the access to each of the assets in Azure. We also need to know who is accessing the resources and from where so that we can monitor for suspicious activity. This post will talk through setting up the key vault and then configuring logging to keep track of the audit information for your certificates, keys and secrets. For each application that you want to access your resources you will need to create some credentials that the application can use.

To allow an application to access key vault an App Registration needs to be added to Azure Active Directory (AAD). This effectively sets up a username and password that the application can use for credentials.

Open the azure portal (http://portal.azure.com) and navigate to Active Directory.

Click "App registrations"

clip_image002

Then "New application registration"

clip_image001

Name needs to be unique within your AD, select Web API/API and enter sign-on url. If you not building a website then enter anything in here. It might be useful to use a url related to your existing domain with application name appended. It doesn’t need to be a valid url. The click “Create”

Once created copy the Application ID as this is equivalent to a username to be used when calling the Key Vault in code. You now need to create the password.

Click Settings then Keys

clip_image002

clip_image004

clip_image006

Enter a name in the description field and select a duration, then click Save. The new key value will be displayed. You will need to copy this as it will not be visible again once you leave this page. This will be used as the password.

clip_image002[5]

Now create the Key Vault. To do that it is a good idea to put it in a specific resource group, especially if you are creating a set of resources that the key vault is going to access or if you are going to setup third party access. Once the Resource Group has been created, select it and add a Key Vault. When the Create Key Vault panel appears, click Access Policies, click "Add new"

clip_image004[5]

Pick the application you just created in AAD and select Get in Secret permissions, Save then go back to the main Key Vault pane and click Create

You have just given the application we created earlier access to just retrieving secrets. As you can see from the access policy you can give the application permissions to access a combination of Keys, Secrets and Certificates with the minimum access of Get. The Key Vault security is at the vault level and you cannot protect individual secrets at the user level. By granting only Get access on the Secret the application will not be able to list the Secrets available and will only be able to retrieve secrets it knows the names of.

Now the Key vault is set up and can be accessed, we want to know who is accessing the vault and from where. Out of the box this is not enabled and requires additional configuration and resources to allow us to be able to retrieve this audit information. This is achieved by enabling diagnostic logs in the Key Vault.

Before you can enable this you need to create a new storage account in this resource group to store the logs, then add Application Insights to the resource group

clip_image002[7]

Once these have been provisioned, navigate to the Key Vault you just created & click Diagnostic logs

clip_image004[7]

Click "Turn on diagnostics"

clip_image006[6]

Select “Archive to Storage Account” and Pick the storage account you’ve just created

Select “Send to Log Analytics” and Create a new OMS workspace in your resource group

clip_image008

Once created select this for Log Analytics

clip_image009

select the AuditEvent log and click Save. 

Now any changes to the Key Vault plus any access from your application will be logged and visible via log analytics. There’s a 10 – 15 minute delay between accessing the Key Vault and the log appearing.

To Add a Secret to the vault, Navigate to the vault, click Secrets then Add

clip_image010

Select Manual from the Upload options, enter a name and the secret

clip_image011

Remember the name you gave the Secret as you will need this in your code when accessing the key vault. This secret will now have a unique identifier that you will use. The one I’ve just created is:

https://recneps-vault.vault.azure.net/secrets/recnepssvsb-key

You should see in the logs this secret being created and also when it gets accessed.

Accessing the KeyVault in C# can be seen here: https://docs.microsoft.com/en-us/azure/key-vault/key-vault-use-from-web-application

The application in the example uses settings as defined below:

ClientID is the Application ID we created in the application registration in AD

ClientSecret is the key you created (that you had to save as it wasn’t visible again) as part of creating the application registration in AD.

Each Key, Secret and Certificate has a unique url which is used as the SecretURI e.g. https://recneps-vault.vault.azure.net/secrets/recnepssvsb-key

You now have your key vault set up with audit logging and are able to access it. My next blog post will talk you through how to access the logs and also how to set up alerting

Windows Azure Diagnostics

Diagnostics in any application is a necessity and Windows Azure applications are no different. You could remote desktop onto the instance and check the event logs and even run up debug view so that you can see your system diagnostic messages. There is however a mechanism provided to retrieve a whole load of diagnostic information. By enabling Windows Azure Diagnostics you can retrieve the diagnostic trace logs, windows event logs, performance counters and other useful data. Enabling Windows Azure Diagnostics can be done by setting the Enable Diagnostics check in the configuration of each Azure role. You also need to add a diagnostic connection string as follows:

image

This enables diagnostics but this alone will not provide you with any information. Windows Azure diagnostics works by capturing the data that you have requested and periodically transferring it to table or blob storage where you can view the information. The information can be requested by configuring the diagnostics you require either in code or in a config file. The config file resides in blob storage and can be changed at runtime so it make sense to use that configuration mechanism rather than code as it can be easily turned off when not in use.

Firstly you may want to retrieve your tracing information that is traced using System.Diagnostic.Trace. the easiest way to do this is to add a trace listener. The Windows Azure SDK contains one that can be used. this is called DiagnosticMonitorTraceListener. This can be added to the web.config file in the same way as other listeners or via code. When I added it to the web config file I had issue on certain projects where the Windows Azure Diagnostics assembly could not be found. Adding the configuration in code always seemed to work. As you will need to redeploy anyway to update your web config file in Windows Azure it makes little difference whether the configuration is in code or the web config file (except that you need to rebuild). In order to add it to code I added the following line to the  global.asx.cs file:

   1: void Application_Start(object sender, EventArgs e)
   2: {
   3:     // Code that runs on application startup
   4:     System.Diagnostics.Trace.Listeners.Add(new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener());
   5: }

You now need to modify the config file located in blob storage. The config file is located in blob storage inside the wad-control-container. There will be a config file in here for each deployment you have made with diagnostics enabled and for each instance. You will need to update each instance you wish to receive diagnostic information from.

Tracing can be enabled by modifying the following xml accordingly:

   1: <Logs>
   2:   <BufferQuotaInMB>1024</BufferQuotaInMB>
   3:   <ScheduledTransferPeriodInMinutes>5</ScheduledTransferPeriodInMinutes>
   4:   <ScheduledTransferLogLevelFilter>Verbose</ScheduledTransferLogLevelFilter>
   5: </Logs>

This will transfer the logs at 5 minute intervals at the highest log level. You can change the level to be Information or Error which represents whether you have used TraceInformation or TraceError whereas Verbose traces both and using Trace.WriteLine. The logs will be transferred to table storage in a table called “WADLogs”

Performance counters can be configured as follows:

   1: <PerformanceCounters>
   2:   <BufferQuotaInMB>0</BufferQuotaInMB>
   3:   <ScheduledTransferPeriodInMinutes>5</ScheduledTransferPeriodInMinutes>
   4:   <Subscriptions>
   5:     <PerformanceCounterConfiguration>
   6:       <CounterSpecifier>\Processor(_Total)\% Processor Time</CounterSpecifier>
   7:       <SampleRateInSeconds>30</SampleRateInSeconds>
   8:     </PerformanceCounterConfiguration>
   9:     <PerformanceCounterConfiguration>
  10:       <CounterSpecifier>\Memory\Available Bytes</CounterSpecifier>
  11:       <SampleRateInSeconds>30</SampleRateInSeconds>
  12:     </PerformanceCounterConfiguration>
  13:   </Subscriptions>
  14: </PerformanceCounters>

This will transfer the processor percentage and available bytes every 5 minutes with a 30 second sampling frequency. These are stored in table storage in a table called “WADPerformanceCountersTable”. Information about the performance counters that are available can be found here:

Event logs can be configured as follows:

   1: <WindowsEventLog bufferQuotaInMB="0"
   2:      scheduledTransferLogLevelFilter="Verbose"
   3:      scheduledTransferPeriod="1">
   4: <!-- The event log name is in the same format as the imperative 
   5:        diagnostics configuration API -->
   6:     <DataSource name="Application!*" />
   7:     <DataSource name="System!*" />
   8: </WindowsEventLog>

These are stored in a table called “WADWindowsEventLogsTable”.

Further information can be found at:

Example configuration file

Overview of Storing and Viewing Diagnostic Data in Windows Azure Storage

Windows Azure Roles Fail to run when deployed to Azure

Recently I was helping out at the Azure Bootcamp in London and during the labs a common theme kept occurring when the labs were deployed to a real Azure account. The roles failed to run and it appeared that the deployment was taking forever.  This is something I experienced first hand when I was starting out with Azure. There is a way to diagnosing these deployment errors and it is by using IntelliTrace. During deployment you can enable IntelliTrace as part of the publish dialog

image

The IntelliTrace option is only available if you have Visual Studio 2010 Ultimate. Once deployed to Azure the Roles will attempt to start and any errors during this phase will lead to the symptoms mentioned above. You can then connect to your Azure environment using the Server Explorer in Visual Studio to retrieve the IntelliTrace files which can be opened in Visual Studio and show any exceptions that may have been thrown. Further information can be found here. Once you have diagnosed your issue please ensure at you then disable the IntelliTrace by redeploying the fixed application as it will have a negative impact on performance if left enabled.

 

Getting back to the problem we have at the Bootcamp, the issues was that the deployed application was trying to writing information to Azure storage and the connection string was still pointing to Development storage. This was strange because none of the deployed applications had got to the Azure storage part of the lab so you would have thought that there was no need for a connection string. Luckily I had the exact same problem with one of my earlier deployments and it turns out that when a project is created the Diagnostic plug-in is automatically enabled. The diagnostic plug-in requires its own connection string to Azure storage so that the diagnostic information can be stored. Looking at the role configuration in Visual Studio you can see the Diagnostic plug-in configuration.

image

To fix the deployment issue click the button next to the connection string text box and enter the details of your Azure Storage account.

image

You will need to redeploy the application or upload the new ServiceConfiguration.cscfg to fix this issue. If this still does not resolve the issue then try disabling the Diagnostics plug-in and redeploy.

Logging with Log4Net

A while ago I was looking for an alternative to System.Diagnostics.Trace and found Log4Net. I know the patterns and practices group have created enterprise logging but I find Log4Net nice and easy to use. I created some documentation and samples on my website at http://www.recneps.co.uk/log4net.aspx.

The BlackMarble.Diagnostics.Logging libraries encapulate both System.Diagnostic.Trace and Log4Net so that we can use a standard way of logging and then configure at runtime one we want to use.