I should really be posting here more because I do something at work every day that could be posted as a tip of some sort. Usually it is something small that no one ever tells you which takes hours to figure out. Then once you know it, you'll probably never need it again so who cares?

Today I decided to split the internal Debian mirror at work off to its own machine. So I needed  install a new VM with Debian 6 (Squeeze) (my OS of choice btw). I wanted to have a small root partition of 10Gb and then use iSCSI to connect to an EqualLogic SAN volume of 150Gb. I currently mirror Etch, Lenny, and Squeeze. I also plan to move my Unattended install there as well. I just sync the main repo on Lenny and Squeeze and no sources. The Etch mirror has sources but no non-free. All of this takes up 111Gb.

Enough about all that. Created the volumes/datastores needed on the EqualLogic box. I installed Debian to the smaller volume and installed open-iscsi. I did a discovery right away just to see if everything was showing up correctly with 

iscsiadm -m discovery -t sendtargets -p <SAN ip>

A huge list of LUNs flew by because the SAN has multiple LUNS. I saw the one I needed in the list. I needed to configure CHAP so nano /etc/iscsi/iscsid.conf. I uncommented these lines and put in my CHAP info.

node.session.auth.authmethod = CHAP
node.session.auth.username = username
node.session.auth.password = password
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = username
discovery.sendtargets.auth.password = password

Now I may not need CHAP setup for discovery because I was able to discover just fine without CHAP. I was getting an authentication error when trying to connect so I added this. Ultimately I think my error might have been caused by an ESXi server connecting to the LUN.

I also set "node.startup = automatic" because I want this to connect on boot.

I tried to connect to the lun with

iscsiadm  -m node  --targetname <iqn> --portal <SAN ip> --login

I had some error messages as mentioned before. I made changes to the config file and restarted the open-iscsi daemon. When I did this open-iscsi started to connet to all the previous iqn's it found. Some of wich allow multiple connections (for vMotion to work). A quick fdisk -l showed that indeed I was connected to the volumes. I disconnected from all the volumes and edited the config again. I changed from automatic to manual. I restarted the daemon and BOOM all the nodes tried to connect again! So I'm thinking that the manual/automatic isn't working here.

I went into the /etc/iscsi/nodes folder and deleted all the nodes there. I then tried to figure out a way to do a discovery with only the node I needed to come up. I couldn't find any way to do that. So I just did another discovery. I then used this command in the /etc/iscsi/nodes folder to remove all the nodes discovered except the one I needed (which has the word he-storage).

ls | grep -v "he-storage" | xargs rm -Rf

This removed the many volumes I didn't need and left me with the one I wanted. So I then connected to that LUN successfully and proceded to partition, format, and mount the volume. I figured that now I should change back to automatic loading of the LUN. I edited the file and restated the daemon. It showed a successful disconnect but no re-connect. WTF? I double check the config and restart the daemon, same thing. So now I'm NOT getting the behavior I need when it was happening before.

I then remembered seeing a list of settings under the node folder. If you cat the iqn folder under the /etc/iscsi/nodes folder, you can tab complete out till you hit a file called "default". Once you cat "default" you see all the settings for that node. This basically looks like the config file without the comments. Right up toward the top was "node.startup = manual". It seems that the settings for the node are pulled from the config file when they are discovered, but if the config file is updated those changes won't take effect for any existing node. I edited "default" and restarted the daemon. The node came up automatically without issue.

The -R, --rescan for iscsiadm, may be a sollution to this problem, but I didn't look into it since I finally had everything working.

So there it is. The big discovery of the day was that under open-iscsi, nodes that you discover will take the settings from iscsid.conf at the time they are discovered and don't seem to update. So if you need to make changes that will effect these nodes you will need to edit the "default" file or dig more into the man page of iscsiadm.

I had a problem at work today where someone complained that they could search google and see all the results but when they clicked on one they got an error message. It turned out that the error message was from the Squid Proxy and they did not have access to the site. This was puzzling for 2 reasons. First this person was not supposed to be filtered through the whitelist proxy. Secondly, why was the whitelist proxy allowing a google search?

The problem with the user being on the wrong Proxy was easily fixed. What I then dug into was trying to determine what was going on with the whitelist to allow a google search.

So first the setup…

It’s pretty simple I’m running Squid Proxy 2.6 on a Debian 4 (Etch). I know I’m a few releases behind here but hey the system “just works” and it is a custom install on a Cobalt RAQ3. I’m not about to break it by trying to upgrade when it’s not needed. Any way, Squid is running a whitelist using this line in the config:

acl whitelist url_regex "/etc/squid/whitelist"

http_access allow all whitelist

So I’m never allowing www.google.com/* which is what would allow me to open up searching. The whitelist does have a few references to google.com but they are explicit for certain sites like:

maps.google.com/*

khm3.google.com/*

gg.google.com/*

ssl.google-analytics.com/*

Of course I set myself on the Squid Proxy and tried to go to http://www.google.com, https://www.google.com, as well as removing the www. In every case I was greeted with the “Access Denied” page. I then added google as a search provider in IE9. I searched and low and behold it returned a google search page. Matt helped out by sniffing packets and watching the logs. We could see where the Squid was allowing the google search page through. So what was going on here?

I started to take apart the string that was being used for the search:

http://www.google.com/search?q=dogs&sourceid=ie7&rls=com.microsoft:en-us:IE-Address&ie=&oe=

I cut everything out one at a time and found that this was the minimum to get through:

http://www.google.com/search?q=pregnant+mermaids&rls=com.microsoft:en-us

I’m not sure what the &rls means. I did a little searching but didn’t come up with anything except that it’s used other places like the default Firefox home page. We do whitelist microsoft.com. I commented that out of the whitelist and reloaded Squid with the same results. Matt changed microsoft to something else and that worked as well. After a bit it hit me what was happening.

We allow all .gov sites which sometimes include .us sites so .us/* is listed in the whitelist. I commented this out and the site was blocked. Finally success, but I need to allow those .us sites and additionally what was Squid recognizing –us as .us and allowing a search to go through to google.com which was not allowed?

Well I didn’t really waste time researching this. I’ve found the online documentation for Squid lacking. I’m not sure the book they sell would have helped. I admit that it could be a version issue but again, I’m not upgrading. So my solution was to add a blacklist and explicitly block google. As mentioned I need some google sites so I can’t just blacklist google.com. I added www.google.com/search to a file called /etc/squid/blacklist and added these lines to the squid.conf:

acl blacklist url_regex "/etc/squid/blacklist"
http_access deny blacklist

After reloading Squid, I tested a search and was given the “Access Denied” page. All was right with the world again. I hope this info will help someone else dealing with a similar issue.