The Insecure Wire

a network engineers perspective.

Configuring F5 BIG-IP APM Webtop with Office 365

F5 Networks make a great application delivery controller called BIG-IP also known as a load balancer. A feature of this platform is Access Policy Manager or APM for short. A licensed APM module gives you the ability to use the F5 appliances as a SSL vpn, application portal and front end proxy for various authentication. It really is quite powerful.

In today’s enterprise IT environment, pretty much everyone has a Microsoft Office 365 or Azure subscription, cloud man! Office 365 is pretty feature rich and cool to use. When you setup your on premises Active Directory environment for single sign on, you are required to install MS Active Directory Federation Services. Essentially this allows a user to login to your Windows domain and when they browse to O365 it will log them in automatically. OK pretty standard stuff.

We use the F5 APM webtop feature as an web application single sign on portal. Users can go to a URL anywhere they have Internet access and login to the SSO portal. The portal gives users the ability to change expired passwords as well, such is the power of F5 APM. Recently I integrated this feature with on premises ADFS and Office 365. We have direct icon links for the 365 Portal, Email and One Drive. Since the user is already authenticated against AD, the session can be used for 365. Let me show you how;

Before we start please note: Using a webtop with ADFS does not require you to have ADFS presented with an iapp on the F5. It just has to be on-net and accessible by the F5 self IP.

1. Under Access Policy > SSO Configurations > NTLMv1 > Create a NTLMv1 SSO Configuration as below:

2. Under Access Policy > Portal Access > Portal Access List > Create a new Portal Access List:

Set the application URI as follows (update your ADFS domain name obviously it’s not contoso).

Office 365 Portal:
https://adfs.contoso.com/adfs/ls/?wa=wsignin1.0&wtrealm=urn:federation:MicrosoftOnline&wctx=MEST%3D0%26LoginOptions%3D2%26wa%3Dwsignin1.0%26rpsnv%3D2%26ver%3D6.1.6206.0%26wp%3DMCMBI%26wreply%3Dhttps:%252F%252Fportal.microsoftonline.com%252FDefault.aspx%26lc%3D1033%26id%3D271345

Email (Outlook Online):
https://adfs.contoso.com/adfs/ls/?wa=wsignin1.0&wtrealm=urn:federation:MicrosoftOnline&wctx=MEST%3D0%26LoginOptions%3D2%26wa%3Dwsignin1.0%26rpsnv%3D2%26ver%3D6.1.6206.0%26wp%3DMCMBI%26wreply%3Dhttps://outlook.office365.com

One Drive (edit the sharepoint URL to your own):
https://adfs.contoso.com/adfs/ls/?wa=wsignin1.0&wtrealm=urn:federation:MicrosoftOnline&wctx=MEST%3D0%26LoginOptions%3D2%26wa%3Dwsignin1.0%26rpsnv%3D2%26ver%3D6.1.6206.0%26wp%3DMCMBI%26wreply%3Dhttps://contsocom-my.sharepoint.com

3. Add your ADFS server as the resource item. We are just going to pass through authentication and the user will redirect to the O365 service. Click the Add button and configure as below:

Configure host as your ADFS host name i.e adfs.contoso.com and SSO configuration as the item you created in step 1.

4. Add your Portal resource to your access profile for webtop configuration. This is completed in Access Policy > Access Profiles > Edit the policy with Visual Policy Editor.

5. Portal icons can be customized in Access Policy > Customization > General

6. ADFS requires that TLS server name indicator is set on all requests. Since we are using this with the APM webtop feature. You need to create an SSL server profile that sets the ADFS server SNI. Browse to Local Traffic > Profiles > SSL > Server. Create a new profile and select advanced. In the SNI field set the name to the ADFS server such as adfs.contoso.com.
Edit the APM virtual server under Local Traffic > Virtual Servers > Virtual Server List. Add the ssl server profile to selected.

7. An irule also needs to be created that is attached to the VIP that selects the server profile when the ADFS is accessed. Browse to Local Traffic > iRules. Below is an example.

when HTTP_REQUEST {
set sslProfile [getfield [HTTP::uri] ":" 1] }

when SERVER_CONNECTED {
if { $sslProfile starts_with “/f5-w-68747470733a2f2f616466732e636f6e746f736f2e636f6d$$/” } { #log local0. “One $sslProfile”
SSL::profile adfs-contoso }
}

Apply this iRule as a resource to your APM portal virtual server.

8. You must add the adfs server URLs to your rewrite profile that is applied to the APM portal virtual server. Add both URLs – for example:
https://adfs.contoso.com/*
https://adfs.contoson.com:443/*

So there you have it; smart linking Office 365 apps with an F5 APM webtop via ADFS.

Error 29702 upgrading VMware VCenter 5.5 to 6.0

Like most enterprise IT shops, we have a significant VMware investment. One of the requirements of upgrading to VSphere 6 is to upgrade your VCenter out of band. Our VCenter is on Windows Server 2012 with Micosoft SQL Server on box. A Windows VCenter upgrade is fairly straight forward – make sure your SQL DB, SSL certs and SSO config are backed up prior to running the upgrade. Another gotcha is to make sure you know your VCenter DB username and password prior to any upgrade. If your VCenter is a VM – then here is where a snapshot is the perfect tonic.

During the installer pre-upgrade check i had the following error:

SQL Permissions Error

SQL Permissions Error

This was resolved by running an SQL query on the VCenter DB:
use master
go
grant VIEW SERVER STATE to [vCenter_database_user]
go
GRANT VIEW ANY DEFINITION TO [vCenter_database_user]
go

The KB below details this:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2114754

Once I ran the installer it started to remove the existing 5.5 version but then it failed with an error, dumping the logs to the system temp directory. Upon investigation the MSI error was 29702 – unable to configure log browser. Further inspection of the logs pointed to the VSphere Web Client. So the plan of action here was to remove all 5.5 components and then reinstall them cleanly, pointing to the existing database. All of the components would uninstall accept for the Web Client – it kept giving the above error.

Enter the Microsoft Fix It utility:
https://support.microsoft.com/en-us/help/17588/fix-problems-that-block-programs-from-being-installed-or-removed

This utility allowed me to force remove the Web Client (from the Windows registry) and then proceed to installing 5.5 fresh again. Pointing the 5.5 install to the existing DB – i now had a clean VCenter 5.5. This actually took several hours to find the fix and remove and reinstall 5.5. From this point running the 6.0 wizard was a breeze and took around 30 mins, this depends on the size of your VCenter DB.

CUCM – Report on all Call Forwards with SQL

Recently I had the task of integrating two separate CUCM clusters together. One of the tasks I had was to work out how many users had call forwards set on their directory numbers so we could either keep or remove them. Call forwards can cause a monetary or management issue if not looked after correctly. Specifically we didn’t want call forwards to a main campus number to be translated to a contact center queue.download

A simple way to achieve this was to run an SQL query on your CUCM Publisher via SSH.

From the CLI;

admin:run sql select dnorpattern,cfadestination,cfavoicemailenabled from CallForwardDynamic c, numplan n where c.fknumplan = n.pkid and (cfadestination != '' or cfavoicemailenabled ='t')

This SQL statement returns all the directory numbers that have call forwards to an extension or PSTN number or voicemail. Extensions or PSTN are denoted with an ‘f’. Voicemail forward is denoted with a ‘t’. You can copy the output into Excel for easy manipulation.

Pretty simple really but handy to know.

Using the Palo Alto API for Captive Portal Logout

It’s API time! Haven’t had a post in a while but in wanted to share this:

This year I had the opportunity to roll out a Palo Alto networks next-gen firewall appliance. We have many areas of our college that require specific access to deliver classes (think IT, engineering etc). These areas are separated by an L2 firewall. Since i was rolling out a full user-id implementation of Palo Alto, i needed a solution that allowed these isolated areas to be able to access the network.

Palo Alto Networks

This is where the captive portal is really handy. You can redirect HTTP requests to a login page where the user is authenticated against your directory source and at the same time User-ID on the PAN will map that user to the source IP. The user will then be able to hit layer 7 rules based on user or group. You will also get the full content-id tracking of which user did what during their session. The issue that I faced was how to remove stale sessions if a user left a PC before the timeout values kicked in.

A really cool feature of PAN-OS is its extensive RESTful XML API. You can browse PAN firewalls API by simply going to http(s)://hostname/api/ (where host name is the device name and domain or IP address). Using the XML API i was able to create a php logout script which when called would remove the user to IP mapping from both the data plane and the management plane. The API supports both http GET and POST methods. The guide is accessible here.

Below is the code for logout.php which we call with wget or similar (no proxy). I used PHP as it was easy with the global $_SERVER[‘REMOTE_ADDR’] variable – this is so we could get the true client address as some machines are behind nat or even worse dual nat! The php curl module is used to access the PAN API using the http GET method.

The variables needed are DNS name of the device and your API key. A prerequisite is that your host has the php5-curl package installed (for debian based /w apache it is ‘sudo apt-get install php5-curl’).
Hit the link to see the code! Continue reading →

Cisco UCCX Prompt Codec Format

This post outlines the Cisco UCCX Prompt Codec Format. At the college I work for we have the full Cisco Unified Communications stack. One of my day to day tasks is to manage the VOIP solution. This includes CUCM (Call Manager) and UCCX for Contact Center queues. One of the more mundane task that comes with administering UCCX is prompt recordings. These are your standard phone queuing IVR’s (Interactive Voice Recordings). UCCX uses the WAV container format for prompts. The most annoying issue here is that UCCX does not trans-code WAV codec formats. You must do this yourself before uploading your WAV file to the UCCX Appadmin management tool. If not the sound will be garbled, like white noise.

The accepted codec for UCCX WAV files is – CCITT u-Law, 8 kHz, 8 bit mono. This can be achieved using Audacity or NCH Switch sound converter. Both available on Windows OS.

Originally published as ‘Cisco UCCX Prompt Codec Format’

Cisco FXO Voice card disconnect issue

Recently i was working on setting up a branch office to use SRST PSTN failover as the site uses a SIP ITSP service via the WAN connection. So to give this small branch site some sort of redundancy the Cisco 2921 that i implemented came with x2 FXO VICs – giving me 4 ports total for PSTN failover when the router is in SRST mode.

One of the major issues i encountered was configuring the FXO card to disconnect the call when the calling party hangs up. Carrier tones are different worldwide so what i was seeing was when there was an inbound call on one of the 4 FXO ports and the call was flung to CUCM via a dial peer, the IP phone called would continue to ring if the called party had hung up.

After a bunch of research i found out that the voice router needs to know what tone to look for before it can tell CUCM that the call has disconnected. To do this you have to configure a custom cptone and apply it to the voice port like so:

voice class custom-cptone telstra
dualtone disconnect
frequency 425
cadence 375 375

voice-port 0/0/0
trunk-group FXO_TELSTRA_PSTN
supervisory disconnect dualtone mid-call
supervisory custom-cptone telstra
cptone AU
timeouts call-disconnect 2
timeouts wait-release 2
timing hookflash-out 250
connection plar 8200
impedance complex1
caller-id enable

These settings work really well for the Australian carrier Telstra. As you can see above the custom cptone ‘telstra’ is applied to fxo 0/0/0. I got the frequency and cadence settings our the Australian disconnect tone from the World PSTN Tone Database

The other command that is important on the voice card config is the – supervisory disconnect dualtone mid-call. If the fxo port is missing this line the router wont disconnect calls on this port based on tone. See the following Cisco URL for more information.

Originally published as ‘Cisco FXO Voice card disconnect issue’.

Riverbed Steelhead Winbindd bug in RiOS 8.6

Recently i noticed an alarm issue with our Steelhead fleet. At random times the Steelhead appliances were losing trust relationship with the Windows Domain they are members of. This can be really bad, when the main reason your have a product like Riverbed Steelhead is cache chatty protocols such as Microsoft CIFS/SMB. A further look into the logs on the Steelhead appliances – we have one at every site i was seeing the following log entry:

Jul 14 10:31:36 rb-flee sport[9207]: [domain_auth/trusted_domains.WARN] – {- -} Failed to update trusted domains : Failed to communicate with winbindd 

 

Further entries in the logs reported that CIFS and SMB optimizations will be disabled whilst the trust with the domain is down. This is very bad losing all application layer optimization for pretty much anything file transfer related as like many organisations we use Active Directory.

So the troubleshoot this issue, the first thing i did was make sure the steelheads could hit up the site local domain controller for kerberos authentication. I wanted to make sure there was no layer 2 or layer 3 connectivity issue. To do this you can issue a telnet command from the RiOS CLI:

SH > en
SH # telnet 10.10.10.1 88
If the connection establishes there is no connectivity issue between the steelhead and the Windows DC.
At this point i opened up a ticket with Riverbed Support and they informed me of bug id #167210 – a memory leak in the winbindd binary that causes such issues.A quick workaround for the solution was to run a cron job on the steelheads to restart the winbindd process nightly. See the below snippet:

SH # (config) job 1 command 1 “en”
SH # (config) job 1 command 2 “pm process winbind restart”
SH # (config) job 1 comment “This job will restart winbind service once every day”
SH # (config) job 1 date-time 00:00:00 2014/08/06
SH # (config) job 1 enable
SH # (config) job 1 name “Restart-Winbind”
SH # (config) job 1 recurring “86400”
With the daily restart of the winbind process i havent seen it hang on any of steelheads – so a daily restart seems to be a good figure. I’ve been informed by Riverbed TAC that this bug should be resolved in the next release.
Originally published as ‘Riverbed Steelhead Winbindd bug in RiOS 8.6’

Testing QoS with extended ping

Recently i have been working on a Cisco Telepresence Video roll out for my employer. One of the challenges i had was to configure QoS classes with the service provider – which manages our WAN. I provided to them what dscp (layer 3) tags the traffic would be marked as and how much bandwidth they should dedicate to video for these tags.

For Cisco Telepresence Video i used the following setup:
DSCP Marking – AF41 = ToS of 34
For our large bandwidth sites – 100mbit i reserved 16mbit in a QoS class based on a 720p video call @ 60fps equaling 4mbit of bandwidth. So 4 calls at 720p guaranteed over the WAN service.

I needed a way to test WAN configs on the router with purely sending traffic from Site A to Site B with the correct DSCP tag so it would be honored by the router. After a bit if research if found that the Linux networking stack still allows the ToS bit to be set with the inbuilt ping command like so:

ping -Q 34 10.1.1.1

The -Q option sets the Type of Service bit – which is a decimal value in IPv4 – if your using IPv6 it has to be a hex value. For this article im working with IPv4. How do you know which decimal value to use? Well they marry up to DSCP and CoS, i use a very useful table by the website tucny:

http://www.tucny.com/Home/dscp-tos

Really handy resource to work out what CoS, DSCP and ToS values you need for your QoS config. So when testing from a Linux host with the above command and at the same time running a shell session to the Cisco router – issuing the following command:

show policy-map interface gi0/0 (where gi0/0 is your inbound wan interface)

You should see the incremental qos class packet matches rise with every
re-iteration of the above command. Also FYI, Windows deprecated the use of the ToS bit in its ping command, so testing qos with extended ping can only be done from non-windows operating systems.

iKAT – Interactive Kiosk Attack Tool

I’d like to bring to your attention a tool that was posted to the full disclosure mailing list a while ago, but still useful. It is called iKAT (Interactive Kiosk Attack Tool) and allows you to exploit the local machine windows / linux from a browser web page full of nifty exploits and tools. Its very handy to browse to a site and bring up an unrestricted shell on a win32 box within seconds. There are now multiple flavors of iKAT; linux, portable, windows and photoKAT.

iKAT in Internet Explorer 8

It also works well on those internet kiosk machines, that are all locked down usually with windows based policies 😉

Kon-Boot – Windows password auditer and bypass – VMware support

We all have the Windows password bypass boot disks in our tool kits. I’d like to make mention of a new favourite of mine, it is called Kon-Boot which patches the loading kernel in memory and effectively bypasses the authentication mechanic. Kon-Boot supports both Linux and Windows operating systems, allow for root access on both OS’es in seconds and also privilege escalation scenarios on Windows. Kon-boot is a very nice piece of software so much so that a commercial version is available and comes in usb, floppy, cdrom based installations. It also supports VMware virtual machines.

Kon-Boot logo

OK so i may sound like an advertisement for Kon-Boot, but I’m not it’s just a really cool tool that all admins and techs should have in their tool kits. I purchased the commercial version and tested out privileged escalation in a Windows domain lab environment. You can actually impersonate users that have logged on to the system previously. This hack is really a bypass as you require physical access to the workstation and i feel it could be stopped if cached credentials are disabled on the domain.

Kon-Boot’s main purpose is to get you back into a Windows or Linux machine that you forget your password on. It does this in a neat way without the need for any injection or modification of the Operating System.