About

Storage CH is a source for all Storage related topics. The postings on this site are my own.

Roger Luethy, Storage Geek, 30 years of Storage/Networking/IT experience and loves Sport,Travel,Photography,Blogging – works for a independent IT Solutions provider
in Switzerland

Questions ? Please DM me on Twitter, thank you.

P.S. : If you are into Photography please visit my Deviantart Page or my Tumblr Page

79 Comments

Guillaume
/ July 12, 2012

Hi Roger
Thanks for your blog I found interesting informations.
We have just migrated our storage from EMC to IBM (SVC + DS3500)
I have a question regarding multipath with IBM. I can’t find any GUI as we had with EMC powerpath monitor to see real time the state of the paths

For SVC it seems SDDSM has some command line “datapath query device” but no real time and for DS3500 I didn’t find any tool even command line to see the paths. (The same Windows server has Lun on SVC and DS3500)

Do you know if there is a way to see the paths on DS3500 or even better a tool like we had with EMC powerpath monitor ?

thanks

Reply
rogerluethy
/ July 13, 2012

Hi Guillaume

Thank you 🙂 Yes with SDDSM you can check on the paths (did you have a look at this document ->http://www-01.ibm.com/support/docview.wss?uid=ssg1S7000303&aid=11 Chapter 3 ?). For the DS3500 you may check ->http://download.boulder.ibm.com/ibmdl/pub/systems/support/system_x_pdf/00w0186.pdf
but i’m not aware of a status Monitor for the paths in the DS3500.

-Roger

Reply
Guillaume
/ July 16, 2012

Hi Roger

sorry to pollute your blog, but it didn’t find any other way to write you – You can delete these posts

in fact I found what I was looking for. I’m now using the HP MPIO manager (it is a snap-in in Storage / Device Manager) where you can see all your paths state in the GUI. It works better for DS3500 than SVC

It’s a pain, that we must use HP tool to get something. As you’re working at IBM if you know someone there, you can tell them that HP or even EMC(powerpath monitor) have nice tools IBM customer need.

thanks,
Guillaume

Reply
- rogerluethy
  / July 22, 2012
  
  Hi Guillaume
  
  Thank you for your response. I will bring this information to our Development.
  
  -Roger
  
  Reply
Brian Beeler
/ March 5, 2013

Roger – it’s come to our attention today via Seagate’s blog that you used our hybrid image that we created in one of your posts without giving attribution. It appears you deep linked too, so we can take it down, but I’d rather just a tip of the hat with a link to our site 🙂

Designing Hybrid Storage For The Enterprise

Reply
- rogerluethy
  / March 5, 2013
  
  Hi Brian
  
  Thank you. I will remove it immediately.
  
  -Roger
  
  Reply
Shortie
/ April 15, 2013

Hi, I was wondering if anyone else has encountered an issue with the loss of available partitions on an IBM 1818 DS5100 / DS5300 & DCS3700 when updating to CFW version 07.84 ?
I am aware of at least 2 situations where that appears to have happened but have so far been unable to find any threads out there.
There’s a ironic twist if this is the case because one of the benefits that V07.84 is designed to give is additional “base” partitions.
Very best regards
Shortie

Reply
John
/ April 26, 2013

Hi Roger,

I have a query regarding IBM SVC Global mirror . We have a global mirror configured between two sites, We are planning to shutdown the host connected to primary and breaks the global mirror after some time .Is it possible to get the exact same copy of data primary in secondary site . Is there any specific cooling peroid for copying data from primary to secondary site, can I verify this process.

Reply
- rogerluethy
  / April 26, 2013
  
  Hi John
  
  If I/O operations on the primary volume are paused for a small length of time, the secondary volume can become an exact match of the primary volume. You can find more details in the SVC Information Center ->http://pic.dhe.ibm.com/infocenter/svc/ic/index.jsp or in the Redbook ->http://publib-b.boulder.ibm.com/abstracts/sg247574.html Chapter 3 also 3.10
  
  Hope this helps
  
  -Roger
  
  Reply
  - John
    / April 26, 2013
    
    Thanks Roger for the quick reply. Does a consistent sychronised mirror means the primary and secondary are the same or is there any way I can verify this. In metro and global mirror relationship tab there is a row which shows the percentage of data transfer for each volume . But I never saw it in 100% even though the volume is in consistent synch state. May be this is because there might be write i/o happening in primary. If we stops the i/o to the primary volume will the data transfer rate in the metro and global mirror relationship tab will become 100% .
  - rogerluethy
    / April 26, 2013
    
    John,
    It’s because there are Write IOs to the Primary else (stopped IO on Primary) it should show 100%. A synchronous Mirror (Metro Mirror) has always the same (consistent) data on both side. On a asynchronous Mirror (Global Mirror) the Secondary side is lagging behind the Primary.
    
    -Roger
John
/ April 26, 2013

Thanks Roger , as we are stopping the I/O on primary volume the secondary should be in sync soon. However is there a way I can determine whether the primary and secondary volumes are consistant over a period of time after the I/O stops on primary.

Reply
- rogerluethy
  / April 26, 2013
  
  John,
  
  Sorry i’m a CLI guy. The best information will be showed by the lsrcrelationship command. Please see here ->http://pic.dhe.ibm.com/infocenter/svc/ic/topic/com.ibm.storage.svc.console.620.doc/svc_lsrcrelationship_21pdzi.html?resultof=%22%73%79%6e%63%22%20
  
  -Roger
  
  Reply
  - John
    / April 26, 2013
    
    Thanks Roger ,
    
    John
  - rogerluethy
    / April 26, 2013
    
    You’re welcome !
    
    -Roger
John
/ April 26, 2013

Sorry to bother once again , I am not getting any value for progress in lsrcrelationship, does that means it is in sync. I am attching the output .
name ecrp1dbp_sd19
master_cluster_id 0000020060414862
master_cluster_name UK-HFD-SVC1
master_vdisk_id 95
master_vdisk_name ecrp1dbp_sd19
aux_cluster_id 0000020060214A68
aux_cluster_name SP-HFD-SVC1
aux_vdisk_id 312
aux_vdisk_name ecrp1dbp_sd19
primary master
consistency_group_id 3
consistency_group_name ecrp1dbp_R
state consistent_synchronized
bg_copy_priority 50
progress
freeze_time
status online
sync
copy_type global

Reply
- rogerluethy
  / April 26, 2013
  
  Yes it’s in sync and consitent and of course online.
  
  Reply
John
/ April 26, 2013

Thanks Roger . One last thing regarding replication , while we are doing the activity we will stop the mirror and some activity will be performed at Site A (primary) . Due to some reason we dont require the changes which is happened at Site A (primary) Then we will reverse the replication, ie Site A will become secondary and Site B will become primary. My question is when we change the replication direction only the changes which is happened at Site A will be removed or else it will copy the entire data in Site B to Site A ie a full refresh.

Reply
- rogerluethy
  / April 26, 2013
  
  John,
  It has only to copy the delta until it’s in sync.
  
  -Roger
  
  Reply
  - John
    / April 26, 2013
    
    Thanks you so mmuch Roger for the help.
    
    Regards
    John
  - rogerluethy
    / April 26, 2013
    
    You’re welcome ! Happy to help.
    
    -Roger
Elena
/ July 16, 2013

Hi Roger,
I’d like to know is there an opportunity to publish a guest article/post on your blog?

Thank you.

Reply
- rogerluethy
  / July 16, 2013
  
  Hi Elena
  
  No thank you.
  
  -Roger
  
  Reply
John
/ September 13, 2013

Hi Roger,

I have a query regarding monitoring of an IBM DS3k device ,is there any native tool from IBM apart from IBM TPC.

Regards
John

Reply
- rogerluethy
  / September 13, 2013
  
  Hi John,
  
  The IBM Storage Manager which comes with DS3K can be used to
  do some monotoring tasks. Beside of this you can use IBM TPC.
  
  Regards,
  Roger
  
  Reply
John Mathew
/ September 13, 2013

HI Roger,

Is it possible to check the controller utilisation ,what willl be the default user name and password for DS3k.

Regards
John

Reply
- rogerluethy
  / September 13, 2013
  
  Hi John
  
  Yes you can check that. There is no default user name and password. If you start the IBM DS Storage Manager for the first time you need to set a user and a password. For more details please check ->http://www.redbooks.ibm.com/abstracts/sg247914.html
  
  Hope this help
  
  -Roger
  
  Reply
John Mathew
/ September 13, 2013

Thanks Roger , is there anything similar in SVC as well apart from TPC.

Regards
John

Reply
- rogerluethy
  / September 14, 2013
  
  Hi John
  
  No. With SVC if you want detailed information you need TPC or use the CLI in the SVC.
  
  Regards,
  Roger
  
  Reply
John
/ September 15, 2013

Thanks Roger , In CLI which command you mean .

Regards
John

Reply
- rogerluethy
  / September 16, 2013
  
  Hi John,
  
  Sorry there is an easier way. In the Management Console go to “Monitoring” and then “Performance” there you can see the CPU load.
  
  -Roger
  
  Reply
John
/ September 17, 2013

Thanks Roger , I saw that in Ds storage manager . but is there any such tool for SVC apart from TPC

Reply
- rogerluethy
  / September 17, 2013
  
  Hi John
  
  Maybe i wasn’t clear. In the SVC GUI go to “Monitoring” and then “Performance” there you can see the CPU load of the SVC node.
  
  -Roger
  
  Reply
John Mathew
/ September 17, 2013

Thanks Roger , Unfortunately I cannot see that may be because I was running in 5.1 code ..

Reply
- rogerluethy
  / September 17, 2013
  
  Hi John,
  Yes you must have v6. You may try in the CLI svcinfo lssystemstats -nohdr -history cpu_pc to get some Perf History
  
  Please see also ->http://aussiestorageblog.wordpress.com/2012/06/22/storwize-v7000-and-svc-performance-monitoring/
  
  -Roger
  
  Reply
Johnny
/ October 1, 2013

Hi
We help customer build a V7000 Globla Mirroring (without change volumes) with FCoIP between SAN Router (SAN60B-R). We have two lun Global Mirroring releationship to DR site.
And we do not have setup consistent group. Each lun releationship is independent syncing to DR site.

Here is the situaction we face….

Each time we start the Global Mirroring and wait the status become to “Consistent Synchronized ” , after 1 day or few days, the status become “Consistent Stopped” by itself. And then we have to start the Global Mirroring again.

And our setting is like these.
SAN router`s FCoIP channel QoS = 1Gbps (as 125 MB/s)
Both side`s V7000 partnership `s bandwidth = 100MB/s
Both side`s V7000 relationship`s bandwidth = 90MB/s

Any suggestion ? Thank you !

Reply
rogerluethy
/ October 1, 2013

Hi Johnny,
This can happen in the following cases. Either the secondary VDisks contain a consistent image, but it might be out-of-date with respect to the primary VDisks. This state can occur when a relationship was in the Consistent (Synchronized) state and experiences an error that forces a freeze of the consistency group. This state can also occur when a relationship is created with the CreateConsistentFlag set to TRUE.Did you receive any Error in the Log, like 1700 or else ?

-Roger

Reply
- Johnny
  / October 2, 2013
  
  Hi Roger
  Thank you !
  We found 1630 and 1720.
  
  And I check the Redbook
  “1720 error : In practice, the source of this error is most often a fabric problem or a problem the network path between your partners”
  
  and we will check SAN switch and Router`s log.
  
  Thank you !
  
  Reply
  - rogerluethy
    / October 2, 2013
    
    You’re welcome 🙂
John Mathew
/ October 4, 2013

Hi Roger,
I have a query regarding performance of SVC, how much will be the idle read , write i/ops for a 2 node cluster svc . backend is ds5300 ,disk are fc disks.
Also how about the allowable latency for read and write in SVC.

Regards
John

Reply
- rogerluethy
  / October 5, 2013
  
  Hi John,
  A single pair of SVC nodes reaches between 120-150K IOPS. The latency added by the SVC is very low and can be in 0.0x ms. In general as rule of thumb an SVC improves the given IOPS from the underlying array by 10-15%.
  
  -Roger
  
  Reply
John Mathew
/ October 6, 2013

Thanks Roger …

Regards
John

Reply
John Mathew
/ October 6, 2013

Hi Roger,

Just one more thing , What does 1630 error in SVC means, it show message number of device login reduced . Any specific reason for this.

Regards
John

Reply
rogerluethy
/ October 6, 2013

Hi John

A 1630 points to lost link to the backend Storage i.e. SVC has lost one or more links. This can happen if something is wrong with the SAN Zoning or you have a bad port in the SAN.

-Roger

Reply
John Mathew
/ October 6, 2013

Thanks Roger ..

Regards
John

Reply
- rogerluethy
  / October 6, 2013
  
  You’re welcome !
  
  Reply
John Mathew
/ October 16, 2013

Hi Roger ,

In an IBM SVC there is a global mirror configuration set. .Is it possible to check the bandwidth consumed by each consistency group ?

Regards
John

Reply
Jimmy
/ October 17, 2013

Hi Roger
Our Customer use 2 V7000, and Global Mirror to Synced a 4TB LUN to DR Site, and they use two (100Mb download / 40Mb upload Line) to build a Connection between Production & DR Site. And the FCIP Link lays in this connection.(The line is for personal use usually, they use it because of Budget,but not a leased line : http://www.cht.com.tw/personal/hinet.html)

Their environment is VMs store in V7000’s LUN
Our customer response that they sometimes found the file system on their Server is become read only, and they will have to reboot to fix it. And their end user feel laggy when use their application to input data, When stop Global Mirror ,it become normal again

It seems the Bandwidth affect the Storage to become laggy
May I ask a question about how to sizing a Bandwidth , Where I use v7000 Global Mirror?
Does Global Mirror cost the cache only when the bandwidth is not enough, and data is congested in the source side?

thank you very much

Jimmy Chien

Reply
rogerluethy
/ October 17, 2013

Hi John,

No you can’t see the consumption of bandwidth of a certain consistency group. You only can see the ‘global’ bandwidth of your global mirror configuration or on a single volume.

-Roger

Reply
- John Mathew
  / October 17, 2013
  
  Thanks Roger ..
  
  Reply
rogerluethy
/ October 17, 2013

Hi Jimmy

Do you use Global mirror with the change volume function or ‘standard’ Global mirror ? (you may check out ->https://www.ibm.com/developerworks/mydeveloperworks/blogs/869bac74-5fc2-4b94-81a2-6153890e029a/resource/ImplementationofLBGMandPerformanceMonitoring_v1.pdf

-Roger

Reply
- JimmyChien
  / October 17, 2013
  
  Hi Roger
  We use Standard Global Mirror.
  
  Reply
  - rogerluethy
    / October 17, 2013
    
    Hi Jimmy
    
    I stronge suggest that you use Global Mirror with Change Volumes.Also you should check the RTO i.e. how much your Target side lags behind the Primary. You may need to adjust the values there if the connection is unstable and can’t provide always the needed Bandwidth. Also you could check on the IP connection. Use QoS there to guarantee the Bandwidth needed for the Global Mirror connections.
    
    Hope this helps.
    
    -Roger
JimmyChien
/ October 17, 2013

Hi Roger,
The line is already dedicated to the replication .I am afraid that it is not enough even we use all the bandwidth.Because the disk is out of space now(Does it cost more disk space?),and not convenient to change to Global Mirror with Change Volume. is there any way I can do to improve ?. We could not determine how much Target behind the Primary. do you have any method to do that ?

Reply
- rogerluethy
  / October 17, 2013
  
  Hi Jimmy
  
  Ok. In your case you need to check gmlinktolerance feature which is described here -> http://tinyurl.com/qcgpoys. Did you check the logs ? Do you receive ‘1920’ errors ?
  
  -Roger
  
  Reply
JimmyChien
/ October 17, 2013

Hi Roger

1920 is the error code

I did adjust this parameter already

The default setting for gmlinktolerance is 300 seconds (5 minutes).
The default setting for gmmaxhostdelay is 5 milliseconds.

I changed gmlinktolerance to 600 seconds
Keep gmmaxhostdelay in 5ms.

Should I change gmmaxhostdelay ?

Reply
- rogerluethy
  / October 17, 2013
  
  Yes try to do that also.
  
  Reply
JimmyChien
/ October 17, 2013

Hi Roger

I will try to change gmmaxhostdelay.
But usually they said the gmmaxhostdelay is suggest not to more then 5ms
Because we have totally 4.6TB available disk space. and a 4TB LUN, (600MB used for Flash Copy for VMWare SRM ), someone suggest me to separate into two LUNs, then I can make good use of two controller. for one LUN,it use only one controller only.

Basically, I think a mater of bandwidth
if we add more bandwidth. the delay may be disappear, right?

Thanks for advice
Jimmy Chien

Reply
- rogerluethy
  / October 17, 2013
  
  Hi Jimmy
  
  At the end of the day it comes down to this. You need more bandwidth.
  
  -Roger
  
  Reply
John Mathew
/ November 21, 2013

Hi Roger,

Hope you are doing good. Thanks for the previous help and advises.
I had a small query regarding svctask chcluster -gminterdelaysimulation , will this setting delay the vdisk replication . How it is different from svctask chcluster -gmlinktolerance.
I am facing an issue with 1920 error. I am trying to investigate of the cause of this issue.
I am pretty sure that bandwidth was not the cause as it happens only on a particular day in a week (basically weekly basis). Rest all the days are fine . Any idea on the Algorithm of the replication as what time it works or basically how it decide on the time to replicate .

Regards
John

Reply
rogerluethy
/ November 21, 2013

Hi John,
Yes thank you hope you are fine too. Did you check this first ? ->http://tinyurl.com/kbmf8g8 on the 1920 error. An 1920 can be a interlink problem BUT i could also point to problem in secondary node. The chcluster -gminterdelaysimulation will simulate a Global Mirror roundtrip latency in milliseconds (0-100). The chcluster -gmlinktolerance is different – it can be used in case of not optimal interconnection to prevent a timeout in the GM relationship. (Seconds 10-400, default setting is 300).

Hope this helps

-Roger

Reply
John Mathew
/ November 21, 2013

Thanks Roger for the quick reply, I am doing good. I had gone through the link which you have provided . Also I was working with the networking team regarding this issue.They found some congestion in traffic between the wan switch and wan router , they checked the possibility to increase the timeout value that is why i checked you about gminterdelaysimulation and svctask chcluster -gmlinktolerance.

Regards
John

Reply
Kumar
/ November 29, 2013

Hi Roger,

What is the accepted packet loss on a WAN link of 300KM/100Mbps
In my case i do see consistently 10% loss for 1472 frame size for every 255 ping counts.
Basically when trying to check with tperf it reports very huge RTT values in medium priority, where as ping reports 30-40ms , but with these many packet loss. Background is 1920 error intermittently.

kumar

Reply
- rogerluethy
  / November 30, 2013
  
  Hi Kumar
  
  10% packet loss is way to high. There seems to be a problem with the WAN link quality. Packet loss should not exceed 2-5% in my opinion (but i’m not an Network expert)
  
  Roger
  
  Reply
Max Hogan (Maximum Midrange)
/ December 10, 2013

Hi Roger,
Our storage team put together a blog article that has a cross reference chart of part numbers and features for the drives in the Storwize line.
We thought your readers might find it useful.
The link is here
http://www.maximummidrange.com/blog/storwize-drive-comparison-chart/2854
if you want to post it.
Thanks,
Maximum Midrange

Reply
- rogerluethy
  / December 10, 2013
  
  Hi Max
  
  Thank you. Nice ! I will post it asap.
  
  -Roger
  
  Reply
Joe
/ January 15, 2014

Hope you have a great vacation! As a Storage CTSS in the states, I visit your site daily!

Reply
- rogerluethy
  / January 15, 2014
  
  Hi Joe ! Thanks a lot
  
  -Roger
  
  Reply
Alex
/ March 26, 2014

HI, my name is Alex. Sorry for my english, i am from Russia. My company’s MT-M 2810-a14 have a failure disk, and i must replace it, but i don’t know how. I try to find solution of this problem in internet, but all i can find is something about triggering “phase in” after disk replace. In your blog i can’t find solution too. I would be very appreciate for any answer or quick guide “how to replace failure disk in MT-M 2810-a14 step by step” )) if you can of course.

Reply
- rogerluethy
  / March 26, 2014
  
  Hi Alex
  
  You can’t replace a Disk on your own on a IBM XIV (2810-A14) System. This needs to be done by your local IBM Tech team. Please contact them and open up a ticket.
  
  -Roger
  
  Reply
Nikola
/ May 11, 2015

Hi,

my name is Nikola. I am contacting you and any reader here, regarding MPIO driver for IBM DS3500 series, and memory leak it causes. Read more here: http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005069

If anybody has some info how to solve this situation (without rebooting nodes) I would appreciate it. I need to reboot my cluster nodes every 2 weeks just to free memory.

Reply
- rogerluethy
  / May 12, 2015
  
  Hi Nikola, i’m afraid i can’t help you here. I don’t work for IBM anymore.
  
  Thanks,
  Roger
  
  Reply
Jens Janssen
/ December 21, 2015

Hi Roger,

I´ve been reading that you dont work for IBM anymore….
But perhaps u can still help me.
I got a DS3300 but I dont have the storage manager.
Any idea?

Have a nice christmas 🙂
greetings jens

Reply
- rogerluethy
  / December 22, 2015
  
  Hi Jens, yes i’m no longer with IBM. Did you try the Redbook ->http://www.redbooks.ibm.com/redbooks/pdfs/sg247065.pdf ? Have a nice christmas, too ! Cheers, Roger
  
  Reply
  - Jens Janssen
    / December 22, 2015
    
    Hi Roger,
    Yes… I did..
    Did I understand right? I should do it using the CLI?
    
    🙂 and again… have a nice christmas 🙂
  - rogerluethy
    / December 22, 2015
    
    Hi Jens, without Storage Manager i’m afraid you don’t have much choice then to use the CLI.
exa73961
/ April 10, 2016

hi
how can i monitor the replication status between storwize v3700 .

Reply
- rogerluethy
  / April 20, 2016
  
  Sorry i can’t help you here but you may check the V3700 Redbook -> http://www.redbooks.ibm.com/redbooks.nsf/RedbookAbstracts/sg248107.html
  
  Reply