Anonymous_User Absent Member.
Absent Member.
660 views

Cannot expand cluster pool

2 nodes in a cluster running OES 2015 SP1 connected via FC to a disk array.

I have a shared pool I'm trying to expand but the error given no matter
what I've tried is: "Error: There is no more disk space available. The
disk is full of active or deleted files. If salvage is enabled, purge
deleted files from the salvage area. Delete the files that are no longer
needed. Use move and split options to expand the storage area."

The pool in question is currently made up of 3 segments, and each
segment is 8TB for a total size of 24TB. I have another 8TB lun I need
to add to the pool, and it is from the same disk array as the other
working segments.

The new lun is set to GPT and shared.

Things I've tried:
re-initializing the new device

Rebooting all nodes in the cluster at the same time to fully restart the
cluster.

deleting the lun on the storage array and creating a new one shared as
the same lun #

deleting the lun on the storage array and creating a new one shared as a
different lun #.

Making sure the nodes were disconnected from the lun, connecting a
Windows machine to the lun, deleting the partition and creating an NTFS
filesystem on the lun, then deleting it, then reconnecting the cluster
nodes to the lun and initializing the disk to try to make sure any
remnants of the partition table that make it seem like it has files on
it are destroyed.

In between each of the things I've tried listed above I always run the
following on both nodes before trying to do the expansion:
rescan-scsi-bus.sh --forcerescan
nlvm rescan
multipath -v2
multipath -ll

I've tried different versions of iManager.

From either node I can create a non-cluster enabled new pool from the
lun no problem.

I've never had problems adding a lun to this pool before as evidenced by
the fact that it currently has 3 segments.

After everything I've tried I'm guessing the error message I'm getting
is a red herring of sorts, but I can't find any troubleshooting tips to
help deal with it.

All of the documentation I can find says to expand a cluster pool I need
to use iManager and be connected to the cluster object when making the
expansion. Can I use nlvm or nssmu connected to the master node
instead? If I have to use iManager, can I be connected to the master
node instead of the cluster object? I really don't want to screw up the
existing pool in the process, but I'm running out of space fast and need
to get more space added to it. OES 2015 has been a godsend since it
did away with the 8TB pool limit, and 32TB is nowhere near the current
limit, so any help would be appreciated.

-Mike
Labels (2)
0 Likes
6 Replies
Knowledge Partner
Knowledge Partner

Re: Cannot expand cluster pool

Well, not on OES 2015 here, so I can only give some pointers.

In the past, as long as you were on the master node (and that data resource/volume was ALSO on the master node), then you could use NSSMU to do what you needed.

So you're able to create a new LUN, and the server sees it in NSSMU under devices yes?
I've personally never spanned LUNs (it's just a "thing" i have where I don't like to do that, but then again our SAN can handle dynamic LUN expansion without issue), so I'm not sure what the Novell documentation says about that. They used to have two different doc sections (expanding existing LUN and adding LUN and expanding pool) in the OES11 docs.

Assuming that the new server sees the LUN (not sure if you have to mark as clusterable, I imagine you would), then I'd think you'd be able to just grow the pool by expanding it to that new device.

What's your device naming format? I think the default is the WWN (or as I like to call it, the big long hairy SCSI thingy). I know that you can optionally (although this may have changed in OES 2015) use "friendly" names like: dm-26 or something like that instead of the big long hairy SCSI thingy.

I think NSSMU has an option to enable debug logging, but not 100% sure (maybe that'd tell you why). I would think maybe something would show up in the /var/log/messages.

Is there a limit to volume segments? (I'm asking I don't know).

Massimo may wander along here and have more/brighter ideas.

--Kevin
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Cannot expand cluster pool

Hello,

have you read about chapter 13.18 of the
clus_admin_lx.pdf-documentation? There are two ways described of
expanding a clustered pool a) increasing the LUN of the pool b) adding
additonal space by a new (shared) LUN. Maybe these are the information
you are looking for.

Regards

Burkhard

OES Admin
Debeka Versicherungen Koblenz
Germany
0 Likes
Knowledge Partner
Knowledge Partner

Re: Cannot expand cluster pool

Hi.

Am 28.02.2017 um 17:22 schrieb Mike:
> 2 nodes in a cluster running OES 2015 SP1 connected via FC to a disk array.
>
> I have a shared pool I'm trying to expand but the error given no matter
> what I've tried is: "Error: There is no more disk space available. The
> disk is full of active or deleted files. If salvage is enabled, purge
> deleted files from the salvage area. Delete the files that are no longer
> needed. Use move and split options to expand the storage area."
>
> The pool in question is currently made up of 3 segments, and each
> segment is 8TB for a total size of 24TB. I have another 8TB lun I need
> to add to the pool, and it is from the same disk array as the other
> working segments.


Are you sure the total doesn't exceed 32TB by even one byte? Could you
live with expanding it by say only 7,5TB?

> All of the documentation I can find says to expand a cluster pool I need
> to use iManager and be connected to the cluster object when making the
> expansion.


Please ignore that and forget iManager even exists for managing storage
related stuff. Use NSSMU.

For starters, use it on each node to verify all servers see the same
things related to this pool and the new segment, especially the
shareable bit.

You may also consider migrating this pool to NSS64 and only one segment.
Or to split it up and use DFS instead. Or use DST. Personally, I don't
think a single 32TB pool is a very good idea, unless it only stores huge
files like disk images or some such. A poolrepair, should it ever become
necessary, would take ages for instance...

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de
0 Likes
Knowledge Partner
Knowledge Partner

Re: Cannot expand cluster pool

mrosen;2451951 wrote:
Hi.

Am 28.02.2017 um 17:22 schrieb Mike:
> 2 nodes in a cluster running OES 2015 SP1 connected via FC to a disk array.
>
> I have a shared pool I'm trying to expand but the error given no matter
> what I've tried is: "Error: There is no more disk space available. The
> disk is full of active or deleted files. If salvage is enabled, purge
> deleted files from the salvage area. Delete the files that are no longer
> needed. Use move and split options to expand the storage area."
>
> The pool in question is currently made up of 3 segments, and each
> segment is 8TB for a total size of 24TB. I have another 8TB lun I need
> to add to the pool, and it is from the same disk array as the other
> working segments.


Are you sure the total doesn't exceed 32TB by even one byte? Could you
live with expanding it by say only 7,5TB?

> All of the documentation I can find says to expand a cluster pool I need
> to use iManager and be connected to the cluster object when making the
> expansion.


Please ignore that and forget iManager even exists for managing storage
related stuff. Use NSSMU.

For starters, use it on each node to verify all servers see the same
things related to this pool and the new segment, especially the
shareable bit.

You may also consider migrating this pool to NSS64 and only one segment.
Or to split it up and use DFS instead. Or use DST. Personally, I don't
think a single 32TB pool is a very good idea, unless it only stores huge
files like disk images or some such. A poolrepair, should it ever become
necessary, would take ages for instance...

CU,
--
Massimo Rosen
Micro Focus Knowledge Partner
No emails please!
http://www.cfc-it.de


Excellent point about the pool repair. I forgot about that. I can vouch for using both DST and DFS. Mainly for us it was because tape backups were taking 10+ days to do a full backup, so we first split into DFS volumes and then eventually added DST to the split volumes. To the user it's like one big drive letter.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Cannot expand cluster pool

On 3/1/2017 3:43 AM, Massimo Rosen wrote:
> Hi.
>
> Am 28.02.2017 um 17:22 schrieb Mike:
>> 2 nodes in a cluster running OES 2015 SP1 connected via FC to a disk
>> array.
>>
>> I have a shared pool I'm trying to expand but the error given no matter
>> what I've tried is: "Error: There is no more disk space available. The
>> disk is full of active or deleted files. If salvage is enabled, purge
>> deleted files from the salvage area. Delete the files that are no longer
>> needed. Use move and split options to expand the storage area."
>>
>> The pool in question is currently made up of 3 segments, and each
>> segment is 8TB for a total size of 24TB. I have another 8TB lun I need
>> to add to the pool, and it is from the same disk array as the other
>> working segments.

>
> Are you sure the total doesn't exceed 32TB by even one byte? Could you
> live with expanding it by say only 7,5TB?
>


I did try different sized smaller luns with no difference, but since the
pool is already 24GB it's obviously NSS64 which has a 8 Exabyte limit,
so I'm not sure why 32TB (or just over) could possibly be an issue.

>> All of the documentation I can find says to expand a cluster pool I need
>> to use iManager and be connected to the cluster object when making the
>> expansion.

>
> Please ignore that and forget iManager even exists for managing storage
> related stuff. Use NSSMU.


Using NSSMU is what did the trick, and it did it so easily. I wasted a
ton of time trying to get it to work with iManager because usually OES
documentation will list the iManager way and also a command line way of
doing things. Not so with expanding a clustered pool. It only listed
the iManager way, so I was hesitant to try it in a way that wasn't
listed for fear of messing something up with the clustering. Thanks
for the tip, and all is good now, other then hating iManager even more.

-Mike



0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Cannot expand cluster pool

On Tue, 28 Feb 2017 16:22:23 +0000, Mike wrote:

> 2 nodes in a cluster running OES 2015 SP1 connected via FC to a disk
> array.
>
> I have a shared pool I'm trying to expand but the error given no matter
> what I've tried is: "Error: There is no more disk space available. The
> disk is full of active or deleted files. If salvage is enabled, purge
> deleted files from the salvage area. Delete the files that are no longer
> needed. Use move and split options to expand the storage area."
>
> The pool in question is currently made up of 3 segments, and each
> segment is 8TB for a total size of 24TB. I have another 8TB lun I need
> to add to the pool, and it is from the same disk array as the other
> working segments.


for the record,

there was a bug in the vfs interface, which caused this to not work in
iManager.

This is now resolved as per bug #1020941. So going forward, performing the
same using iManager should also work again.

Thanks
Hans
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.