jwhitson Absent Member.
Absent Member.
2129 views

Filr storage out of control

...or at least my control.

It could be that I am just missing the point but my Filr test setup has added 30GB of data in 24 hours. It looks like it's actually copying data from my OES/Linux file servers which is not something I want. I thought Filr was more of a gateway to all of that data and not another form of storage. Here is my scenario...

I have all my users/groups pulled in from LDAP- that was easy. Filr is reading the home directory attribute from eDirectory and giving every user that logs in a My Files/Home directory- easy. I setup 4 of the 7 OES/Linux file servers I have as "Net Folder Servers". I enabled "scheduled synchronization" to occur every 15-minutes on the Net Folder Servers thinking it would sync the information about the data stored there and not the actual data- was I wrong? I created 4 "Net Folders" that essentially point to a "Groups" directory on the NSS volumes that my users use as their "G:" drive here at the office. I then granted a group (that came in over LDAP) the "Access" right so they could see it when they login to Filr. These also have a "scheduled synchronization" setting of 15-minutes.

Where did I go wrong? It would be nice if Filr was just a gateway to data on my OES boxes but it looks like I need to plan for Filr to store a ton of redundant data. Or I need ot figure out how to control it. Can I even delete data from Filr? There doesn't seem to be a straightforward way of doing that.

So many questions...

Jason
0 Likes
9 Replies
jmarton2 Absent Member.
Absent Member.

Re: Filr storage out of control

jwhitson wrote:

> It could be that I am just missing the point but my Filr test setup
> has added 30GB of data in 24 hours. It looks like it's actually
> copying data from my OES/Linux file servers which is not something I
> want. I thought Filr was more of a gateway to all of that data and
> not another form of storage. Here is my scenario...
>
> I have all my users/groups pulled in from LDAP- that was easy. Filr is
> reading the home directory attribute from eDirectory and giving every
> user that logs in a My Files/Home directory- easy. I setup 4 of the 7
> OES/Linux file servers I have as "Net Folder Servers". I enabled
> "scheduled synchronization" to occur every 15-minutes on the Net
> Folder Servers thinking it would sync the information about the data
> stored there and not the actual data- was I wrong? I created 4 "Net
> Folders" that essentially point to a "Groups" directory on the NSS
> volumes that my users use as their "G:" drive here at the office. I
> then granted a group (that came in over LDAP) the "Access" right so
> they could see it when they login to Filr. These also have a
> "scheduled synchronization" setting of 15-minutes.
>
> Where did I go wrong? It would be nice if Filr was just a gateway to
> data on my OES boxes but it looks like I need to plan for Filr to
> store a ton of redundant data. Or I need ot figure out how to control
> it. Can I even delete data from Filr? There doesn't seem to be a
> straightforward way of doing that.


Filr doesn't copy the files locally. The only files stored locally are
files through personal storage if you've enabled the personal storage
option. If you have, then users have the ability to save files in My
Files outside of their network home directory and that additional
content is stored in /vastorage on the appliance. Outside of that, the
main storage usage is from the MySQL database and the Lucene search
index.

Exactly where do you see all of this additional storage is happening?

--
Your world is on the move. http://www.novell.com/mobility/
We know what your world looks like. http://www.novell.com/yourworld/

Joe Marton Emeritus Knowledge Partner
0 Likes
Knowledge Partner
Knowledge Partner

Re: Filr storage out of control

jmarton;2261843 wrote:
jwhitson wrote:

> It could be that I am just missing the point but my Filr test setup
> has added 30GB of data in 24 hours. It looks like it's actually
> copying data from my OES/Linux file servers which is not something I
> want. I thought Filr was more of a gateway to all of that data and
> not another form of storage. Here is my scenario...
>
> I have all my users/groups pulled in from LDAP- that was easy. Filr is
> reading the home directory attribute from eDirectory and giving every
> user that logs in a My Files/Home directory- easy. I setup 4 of the 7
> OES/Linux file servers I have as "Net Folder Servers". I enabled
> "scheduled synchronization" to occur every 15-minutes on the Net
> Folder Servers thinking it would sync the information about the data
> stored there and not the actual data- was I wrong? I created 4 "Net
> Folders" that essentially point to a "Groups" directory on the NSS
> volumes that my users use as their "G:" drive here at the office. I
> then granted a group (that came in over LDAP) the "Access" right so
> they could see it when they login to Filr. These also have a
> "scheduled synchronization" setting of 15-minutes.
>
> Where did I go wrong? It would be nice if Filr was just a gateway to
> data on my OES boxes but it looks like I need to plan for Filr to
> store a ton of redundant data. Or I need ot figure out how to control
> it. Can I even delete data from Filr? There doesn't seem to be a
> straightforward way of doing that.


Filr doesn't copy the files locally. The only files stored locally are
files through personal storage if you've enabled the personal storage
option. If you have, then users have the ability to save files in My
Files outside of their network home directory and that additional
content is stored in /vastorage on the appliance. Outside of that, the
main storage usage is from the MySQL database and the Lucene search
index.

Exactly where do you see all of this additional storage is happening?

--
Your world is on the move. Novell. Because Your World is on the Move.
We know what your world looks like. Novell. Because we know what your world looks like.


I think Joe missed the part where the OP said he enabled a scheduled sync on the netfolder servers ever 15 minutes. This causes FILR to troll the file systems and read all the file names, folders, locations, and metadata and put that into the MySQL database. So that's where the storage is probably being consumed.

For the OP:

If you want it more as a gateway, disable the scheduled syncs of the NetFolder SERVER and the NetFolder itself. But make sure you enable JITS (Just in time sync). Unfortunately I don't believe there is a way to stop a sync process once it's running (I'm not sure even a server reboot will do that). And, if you DO interrupt the sync process it has a nasty tendency to corrupt the MySQL database information (causes users to not see all their data) requiring that you delete the netfolder and/or server entries, wait for it to process the deletion entirely and re-add it back in.

Hopefully the OP has a small amount of servers/netfolders that he added otherwise I think he may run out of disk space or have to resort to deleting the netfolders/servers and waiting for that to finish (just because it goes away in the admin console doesn't mean it's really done with stuff) out of the database and re-create them with only JITS enabled.
0 Likes
jwhitson Absent Member.
Absent Member.

Re: Filr storage out of control

/vastorage was full. I tried a 'du -ch | grep total' but it took forever. That would tell me exactly what was eating up disk space.

Thanks for the tip about disabling the scheduled sync, I will give that a shot.

Speaking of disk space usage and managing it, there is a setting under LDAP that says, "When deleting users, delete associated user workspaces and content". I assume this ONLY deletes data form the Filr server and NOT on the Net Folder/Net Folder Servers?

Thanks,
Jason

Jason
0 Likes
Knowledge Partner
Knowledge Partner

Re: Filr storage out of control

jwhitson;2261859 wrote:
/vastorage was full. I tried a 'du -ch | grep total' but it took forever. That would tell me exactly what was eating up disk space.

Thanks for the tip about disabling the scheduled sync, I will give that a shot.

Speaking of disk space usage and managing it, there is a setting under LDAP that says, "When deleting users, delete associated user workspaces and content". I assume this ONLY deletes data form the Filr server and NOT on the Net Folder/Net Folder Servers?

Thanks,
Jason


Hi Jason,

Yes, that is correct. Deleting the users doesn't delete the content on the back-end storage.

I THINK the only time it may delete content is if it was stored on the Filr appliance itself (ie, an external user creates documents to share with you or something).
0 Likes
jmarton2 Absent Member.
Absent Member.

Re: Filr storage out of control

jwhitson wrote:

> Speaking of disk space usage and managing it, there is a setting under
> LDAP that says, "When deleting users, delete associated user
> workspaces and content". I assume this ONLY deletes data form the
> Filr server and NOT on the Net Folder/Net Folder Servers?


Correct.

--
Your world is on the move. http://www.novell.com/mobility/
We know what your world looks like. http://www.novell.com/yourworld/

Joe Marton Emeritus Knowledge Partner
0 Likes
jwhitson Absent Member.
Absent Member.

Re: Filr storage out of control

To remedy this problem I deleted all the net folders, waited, and then tried to delete the net folder servers. Currently, I have one net folder server that will not go away because it claims to be referenced by a net folder which does not exist. I suppose this is DB corruption.

How does one fix the DB?

Jason
0 Likes
jmarton2 Absent Member.
Absent Member.

Re: Filr storage out of control

jwhitson wrote:

>
> To remedy this problem I deleted all the net folders, waited, and then
> tried to delete the net folder servers. Currently, I have one net
> folder server that will not go away because it claims to be
> referenced by a net folder which does not exist. I suppose this is DB
> corruption.
>
> How does one fix the DB?


Any chance there are user home directories coming off of this NF
server? If so, then the message is correct in that the NF server is
still in use. You would have to delete the users who have home
directories on this guy before you can delete the server itself.

--
Your world is on the move. http://www.novell.com/mobility/
We know what your world looks like. http://www.novell.com/yourworld/

Joe Marton Emeritus Knowledge Partner
0 Likes
Knowledge Partner
Knowledge Partner

Re: Filr storage out of control

jwhitson;2262006 wrote:
To remedy this problem I deleted all the net folders, waited, and then tried to delete the net folder servers. Currently, I have one net folder server that will not go away because it claims to be referenced by a net folder which does not exist. I suppose this is DB corruption.

How does one fix the DB?


You can get that when the NetFolder hasn't fully deleted.

Even when it disappears in the admin console, it still hangs around.

For example, I had a NF that died mid-way through a 630 GB sync job.
It took 3 days for the NetFolder to disappear from the Admin Console, and another 7 days for it to finally purge out of the database before I could re-add it in.
0 Likes
jwhitson Absent Member.
Absent Member.

Re: Filr storage out of control

Ouch!

I've deleted and rebuilt the appliance 3 times now. It's tricky because when you add a Net Folder Server you're prompted to sync on the spot and if you do that your storage requirements go way up. From what I understand it only syncs the metadata but I wish I knew how much space was needed for that. For example, I have a 240GB "G: Drive" that I added as a Net Folder and it will fill up a 50GB disk with metadata very quick (less than 12-hours). I'm ignorant to how much meta data X MB/GB produces- I assume it varies. Also, it doesn't look like you can index unless you sync- which is understandable.

Right now I've got it setup with 4 Net Folder Servers, 4 Net Folders and about 100 Users. No Sync, No Index, No Desktop Sync outside of home directories. I am using 443MB on the Filr server to serve as a Gateway to ~10TB of data.


kjhurni;2262128 wrote:
You can get that when the NetFolder hasn't fully deleted.

Even when it disappears in the admin console, it still hangs around.

For example, I had a NF that died mid-way through a 630 GB sync job.
It took 3 days for the NetFolder to disappear from the Admin Console, and another 7 days for it to finally purge out of the database before I could re-add it in.

Jason
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.