Highlighted
Knowledge Partner
Knowledge Partner
4897 views

NSS/Linux permissions. rsync issue.

I have a small business customer (3 users). Essentially, everyone has full access to all files. When I setup a new OES2-SP3 server (S1) I created a DATA folder on NSS and copied over the user files using rsync. I am now using rsync on another remote OES2-SP3 server (S2), running as root, to backup all the changed files onto the NSS volume on S2. It's working pretty much as expected but there are two files that always appear in the list of files that rsync finds and I know they haven't changed.

rsync -X  -avhuz -e ssh root@S1:/media/nss/DATA/ /media/nss/BACKUP 2>&1 | ssh root@S1  "cat >> /media/nss/DATA/remote_backup_server/rsynclog.txt"


After the DATA folder was setup, things looked like this for each file:
Linux owner: root
Linux group: root
Linux octal permissions: 777
eDirectory creator: S1

The two files in question show:
Linux owner: admin
Linux group: root
octal permissions: 777
eDirectory creator: admin

"admin" is the OES admin account which is LUM enabled.

I also have no problems with files created by other users who are not LUM enabled. Those files show:
Linux owner: nobody
Linux group: root
octal permissions: 666
eDirectory creator: <user>

So, the question is: Why does rsync think the files created by my LUM enabled admin user have changed each time it runs?
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
Labels (2)
Tags (5)
0 Likes
11 Replies
Anonymous_User Absent Member.
Absent Member.

Re: NSS/Linux permissions. rsync issue.

* KBOYLE (Wed, 16 Mar 2011 20:36:02 GMT)
> So, the question is: Why does rsync think the files created by my LUM
> enabled admin user have changed each time it runs?


Rsync does not actually have a concept of "changed" since it does not
keep state. Rsync transfers files if local and remote file differ. By
default this is determined by size and mtime. I suggest you check your
"-avhuz" to see what you're asking rsync to check and see if you have
the same problem without the "-avhuz".

Thorsten
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

Thorsten Kampe;2086638 wrote:
* KBOYLE (Wed, 16 Mar 2011 20:36:02 GMT)
> So, the question is: Why does rsync think the files created by my LUM
> enabled admin user have changed each time it runs?


Rsync does not actually have a concept of "changed" since it does not
keep state. Rsync transfers files if local and remote file differ. By
default this is determined by size and mtime. I suggest you check your
"-avhuz" to see what you're asking rsync to check and see if you have
the same problem without the "-avhuz".

Thorsten

Hi Thorsten,

Thank you for the suggestion. Since I knew the files hadn't changed, and since thousands of other files sync correctly, it never occurred to me to compare the files on the two servers to see what was different. I did compare them and what I learned is interesting.

First, some additional information not provided in the OP. This anomaly affects a folder and the file it contains. The file, in this case, is a download of Groupwise. It is identical in all respects (visual inspection) on the source server and the backup server. The folder is also identical in all respects *except* for the owner.

On the source server the owner of the folder is "admin". On the backup server the owner is "nobody".

So, this new information answers a few questions and raises a few more.

Q1. Normally, is a different owner enough cause rsync to think the file/folder is different? It would appear the answer is "yes".

Q2. Why would the backup server have an owner of "nobody" and why can't rsync synchronize this owner? Does it have something to do with the user being LUM enabled?

Q3. Why would rsync think the file was different?

Presumably, after rsync does its analysis very little data is actually transferred between the two systems. Still, I would like to get to the bottom of this mystery.
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: NSS/Linux permissions. rsync issue.

* KBOYLE (Thu, 17 Mar 2011 16:36:03 GMT)
> Thank you for the suggestion. Since I knew the files hadn't changed,
> and since thousands of other files sync correctly, it never occurred to
> me to compare the files on the two servers to see what was different. I
> did compare them and what I learned is interesting.
>
> First, some additional information not provided in the OP. This anomaly
> affects a folder and the file it contains. The file, in this case, is a
> download of Groupwise. It is identical in all respects (visual
> inspection) on the source server and the backup server. The folder is
> also identical in all respects *except* for the owner.
>
> On the source server the owner of the folder is "admin". On the backup
> server the owner is "nobody".
>
> So, this new information answers a few questions and raises a few
> more.
>
> Q1. Normally, is a different owner enough cause rsync to think the
> file/folder is different? It would appear the answer is "yes".


Please read my previous posting: "By default this [if files differ] is
determined by size and mtime." This comes directly from the man page so
I cannot tell you if this is /really/ the case but I would believe it
is.

> Q2. Why would the backup server have an owner of "nobody" and why can't
> rsync synchronize this owner? Does it have something to do with the user
> being LUM enabled?


That would be one possibility. The other one is that rsync is not trying
to sync the owner (maybe because you haven't asked it to). There are
passages about ownership in the man page but I never needed that, so I
suggest you have a look at it.

> Q3. Why would rsync think the file was different?
>
> Presumably, after rsync does its analysis very little data is actually
> transferred between the two systems. Still, I would like to get to the
> bottom of this mystery.


The solution to your mystery is buried in the man page of rsync and in
the universal KISS principle. As I already stated, you would have to
simplify by getting rid of all non-standard options.

You are using "-avhuz" and "-a" is equivalent to "-rlptgoD" so you're
actually running "-rlptgoDvhuz" (I'm not kidding). Also using ssh to
sync remotely is not useful for troubleshooting; simply sync local to
local.

Thorsten
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

Thorsten Kampe;2087178 wrote:

The solution to your mystery is buried in the man page of rsync and in
the universal KISS principle. As I already stated, you would have to
simplify by getting rid of all non-standard options.

You are using "-avhuz" and "-a" is equivalent to "-rlptgoD" so you're
actually running "-rlptgoDvhuz" (I'm not kidding). Also using ssh to
sync remotely is not useful for troubleshooting; simply sync local to
local.

Thorsten

Hi Thorsten,

This is the first time I have tried to use rsync and, as you point out, I first need to understand what effect the options have. I read the man pages (~1800 lines) before I initially selected the options. I re-read them to gain a better understanding of what is happening. This is what I learned:

As you point out...
You are using "-avhuz" and "-a" is equivalent to "-rlptgoD" so you're
actually running "-rlptgoDvhuz" (I'm not kidding)

But there is more: "D" is the same as "--devices --specials"; I also have "-X" and "-e". So, for everyone's benefit this is what I am asking for:

  • -r, --recursive recurse into directories
  • -l, --links copy symlinks as symlinks
  • -p, --perms preserve permissions
  • -t, --times preserve times
  • -g, --group preserve group
  • -o, --owner preserve owner (super-user only)
  • --devices preserve device files (super-user only)
  • --specials preserve special files
  • -v, --verbose increase verbosity
  • -h, --human-readable output numbers in a human-readable format
  • -u, --update skip files that are newer on the receiver
  • -z, --compress compress file data during the transfer
  • -X, --xattrs preserve extended attrs (implies -p) [n.s.]
  • -e, --rsh=COMMAND specify the remote shell to use


Next, I ran a series of tests to pinpoint the problem:


  1. I copied only the problematic folder/file locally, using rsync with all the options except "-e". It worked... the "admin" owner is preserved.

  2. Next I deleted the copied file and repeated the test locally using ssh. It worked... the owner is preserved.

  3. Finally, I copied only the problematic folder/file from the remote system using rsync with all the same options from the previous test. The owner name was not preserved.

  4. I added the "-i" option to output a change-summary for all updates then re-ran the test. This is what I got:
    receiving file list ... done
    .d....o.. gw801HP_client_win_en/
    .f....o.. gw801HP_client_win_en/gw801HP_client_win_en.exe

    What this says is that rsync recognizes the owner is different and will attempt to sync the owner which, of course, doesn't happen.
  5. Just to be complete, I ran test #2 on the remote server. It worked... the "admin" owner is preserved.

    That pretty much answers the question in the OP and verifies that the owner name is not preserved when copying from a remote system. There may be other factors that affect this e.g. NSS or LUM enabled user but I think it is time to open a Service Request.
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: NSS/Linux permissions. rsync issue.

* KBOYLE (Sat, 19 Mar 2011 00:06:05 GMT)
> What this says is that rsync recognizes the owner is different and
> will attempt to sync the owner which, of course, doesn't happen.
> - Just to be complete, I ran test #2 on the remote server. It
> worked... the "admin" owner is preserved.
>
> That pretty much answers the question in the OP and verifies that the
> owner name is not preserved when copying from a remote system.


So as far as I understand the admin owner cannot be set while other
owners are correct? Try "id admin" and "id WORKING_OWNER" on both
machines and see if you find something...

Thorsten
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

I believe that the user would have to be LUM-enabled for the "owner" to be set since rsync is "linux" and doesn't access the NSS API or SMS/TSA stuff.

Even NRM in OES2 will show "owner" of nobody on an NSS volume unless all your users are LUM-enabled (or the user that actually wrote the file is LUN-Enabled)

I'm just guessing at this part, mind you.
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

kjhurni;2087909 wrote:
I believe that the user would have to be LUM-enabled for the "owner" to be set since rsync is "linux" and doesn't access the NSS API or SMS/TSA stuff.

Even NRM in OES2 will show "owner" of nobody on an NSS volume unless all your users are LUM-enabled (or the user that actually wrote the file is LUN-Enabled)

I'm just guessing at this part, mind you.

Hi Kevin... Thomas,

Admin is LUM enabled on both systems.
server:~ # id admin
uid=601(admin) gid=601(admingroup) groups=601(admingroup)
server:~ #

When an eDirectory user who is not LUM enabled creates a file on an NSS volume, eDirectory shows the correct user while Linux shows the owner as nobody. When a LUM enabled user creates a file on an NSS volume, Linux correctly shows the owner as the LUM enabled user. I believe that is how it is supposed to work and I am happy with it.

The issue appears to be with rsync. rsync is able to synchronize the "nobody" owner but doesn't sync the LUM enabled owner when accessing a remote server via ssh. My tests show that it works correctly when the source and destination are on the same server.

I have opened an SR and will let everyone know what I find.

Thanks for the suggestions.
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

Thanks for the update Kevin

I sync files via rsync on NSS but I don't do it via SSH. However, the data we sync to the remote servers is "read only" if you will so we don't care who the owner is. In most cases "us" are the owners and "we" are LUM-enabled so perhaps that's why I've never noticed.

Still intriguing to find out, yes?
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

kjhurni;2088009 wrote:
Thanks for the update Kevin

I sync files via rsync on NSS but I don't do it via SSH. However, the data we sync to the remote servers is "read only" if you will so we don't care who the owner is. In most cases "us" are the owners and "we" are LUM-enabled so perhaps that's why I've never noticed.

Still intriguing to find out, yes?

Hi Kevin,

I only want to backup at the end of the day (a one-way sync) and I want to encrypt the data transfers so rsync over SSH works well for me. I'm surprised you aren't concerned about ownership and extended attributes. Do you not need that information if the files ever need to be restored?
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

In our case, the only thing we replicate is "software" that is pushed out (or available) via ZENworks

Since the user needs RF rights to get to the data, we just gave "everyone" RF rights to the data and don't need to worry about syncing the owners or the file rights.

We use NetBackup for "real" backups as it's faster/better for us than rsync.

So for example, we rsync the GroupWise 32-bit client. no need to worry about who wrote the files onto the server (coincidentally it's only one of us admins as we're the only ones who have WRITE access, and we're also LUM-enabled so it just works).
0 Likes
Knowledge Partner
Knowledge Partner

Re: NSS/Linux permissions. rsync issue.

KBOYLE;2087959 wrote:
Hi Kevin... Thomas,

Admin is LUM enabled on both systems.
server:~ # id admin
uid=601(admin) gid=601(admingroup) groups=601(admingroup)
server:~ #

I have opened an SR and will let everyone know what I find.

Issue resolved and I learned something in the process...
source-server:~ # id admin
uid=601(admin) gid=601(admingroup) groups=601(admingroup)

dest-server:~ # id admin
uid=600(admin) gid=600(admingroup) groups=600(admingroup),601(linux)

My "admin" user is LUM enabled on both servers however the Linux "uid" is different. My destination server does not have a user with uid=601 and therefore the owners could not be synchronized.
_____
Kevin Boyle - Knowledge Partner - Calgary, Alberta, Canada
Who are the Knowledge Partners?
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.