Big news! The community will be moving to a new platform April 21. Read more.
Big news! The community will be moving to a new platform April 21. Read more.
Admiral
Admiral
1223 views

Migrating from Netware to Linux - faster "sync" ?

Hey all,

Planning on doing a TransferID migration from netware 6.5 to linux this weekend and have done some trial data copies.. Main copy is already done, but the "sync" is still taking a large number of hours that I need to try and fit into a smaller window. Any tips on getting sync to go faster? I'm currently using the miggui but wondering if you do the commands manually, leaving things out like displaying progress bar, etc.. if it'll get to it quicker.

#novell on efnet
Labels (2)
0 Likes
8 Replies
Knowledge Partner Knowledge Partner
Knowledge Partner

joebrug;2268232 wrote:
Hey all,

Planning on doing a TransferID migration from netware 6.5 to linux this weekend and have done some trial data copies.. Main copy is already done, but the "sync" is still taking a large number of hours that I need to try and fit into a smaller window. Any tips on getting sync to go faster? I'm currently using the miggui but wondering if you do the commands manually, leaving things out like displaying progress bar, etc.. if it'll get to it quicker.


So, we have 6-year old HP DL-385 (no G, that's old they are) that are slower than you know what.
We're migrating to HP DL385 G6 or G7, but we're using VMware ESXi, so the target OES11 server is a Guest VM.

Both servers are on a gigabit switch.
We average approx. 15-20 GB/hour (it's not the AMOUNT of data, it's how many small files that affects things).

We're able to get the sync to fit in a small window (even our large ones, maybe 2-4 hours).

Is that not fast enough (although you may have LOTS more small files that change daily).

Basically we cart the server down, run the sync during the day until it finishes. Obviously it skips things that were in use. Then we kick everyone off the server at 3:00 and start the sync.
Since you're on NetWare to OES, I would NOT recommend rsync. The rsync on NetWare is very touchy/finicky and I don't think you can migrate trustees either.

From OES2 to OES11, you could use rsync, although I'm having mixed results with the trustees at the moment.

One thing we DO have is a "static" set of read-only data on the server, that we do NOT migrate over. We have a central office server that contains the "master" data and the clients run rsync to pull the data. So we don't even back it up either (other than the main office).

I can't remember the EXACT numbers we got when we did NetWare to OES2, but I still think we were averaging somewhere around 12-17GB/hour depending on. The key point was to make sure you were gigabit and didn't have a mis-match because someone in your datacom unit set the damn switch to 100-full or something other than auto-auto.

🙂

Oh, and if you have disk quotas, make sure that you either clear them or keep compression enabled. THAT will slow things down because if it tries to copy something, and there's not enough space, it'll retry like X times and then continue on and you find out like 10 hours later it was slow because you didn't have enough directory quota space.

🙂
0 Likes
Admiral
Admiral

I think the main amount of "migration time" is spent comparing the source and destination directories to see what is no longer on source and can be deleted from destination. I did a sync yesterday after doing the initial "consolidation" file copy and the sync was already over 6 hours before the nightly backup started running on netware and really slowed things to a crawl by eating up the CPU.

In looking at the log file.. it says:
***Syncing files and/or trustees************
executing command: migfiles --source-server (IP) --source-path "SYS:random" --destination-full-path "/usr/novell/sys/random" --list-extensions

What is --list-extensions used for??

Then 5 seconds later:
Syncing data from source path(s) to target path(s) using migfiles
executing command:migfiles --source-server (IP) --verbose --source-path "@sourcepaths.in" --destination-full-path "@targetpaths.in" --progress --progress-interval 1 --source-ldap-port 636 --delete --sync "11-06-2013 16:55" --delete-file-on-restore-error

^ that was taking over 6 hours before I had to stop it.. Nothing else useful in log file

#novell on efnet
0 Likes
Absent Member.
Absent Member.

Am 13.06.2013 23:26, schrieb joebrug:
>
> I think the main amount of "migration time" is spent comparing the
> source and destination directories to see what is no longer on source
> and can be deleted from destination. I did a sync yesterday after doing
> the initial "consolidation" file copy and the sync was already over 6
> hours before the nightly backup started running on netware and really
> slowed things to a crawl by eating up the CPU.



That's my expirience too. Synching 2,5 GB with a lot of small files
could'nt be completed in one night. I've rsynced (outside of migration
process) files during business hours, hoping this will spedd up the
process, but most time was spent comparing, writing the diffs was done
in less the 5 hours. We decided to do this final step over a long weekend.


But you may try to devide such an large volume in several migration jobs
(per folders). When a job (a migrated folder) is finished, point the
users to the new, already migrated folder and remove rights to the old
one (the source). Never tried this, but it should work (in therory).
Problem is, the ID is'nt transfered at this moment, you have to
'workaround' that fact until the ID is transfered.

A workaround (*if pathes are not hardcoded in your environment*) could
be to use directory map objects.

E.g. home dircetorys

Instead of using

map h:=Server\User:home\%cn

create a directory map object 'DO_Home' which points to the above path,
then map

map h:=.Do_Home.context_of_do_object\%cn.

This way you have a 'symlink' to the data, which by editing it can
easily switch between old and new server


Tom
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

joebrug;2268248 wrote:
I think the main amount of "migration time" is spent comparing the source and destination directories to see what is no longer on source and can be deleted from destination. I did a sync yesterday after doing the initial "consolidation" file copy and the sync was already over 6 hours before the nightly backup started running on netware and really slowed things to a crawl by eating up the CPU.

In looking at the log file.. it says:
***Syncing files and/or trustees************
executing command: migfiles --source-server (IP) --source-path "SYS:random" --destination-full-path "/usr/novell/sys/random" --list-extensions

What is --list-extensions used for??

Then 5 seconds later:
Syncing data from source path(s) to target path(s) using migfiles
executing command:migfiles --source-server (IP) --verbose --source-path "@sourcepaths.in" --destination-full-path "@targetpaths.in" --progress --progress-interval 1 --source-ldap-port 636 --delete --sync "11-06-2013 16:55" --delete-file-on-restore-error

^ that was taking over 6 hours before I had to stop it.. Nothing else useful in log file


Wow, we've never gotten that slow/bad (that I remember)

For example, we migrated about 140 GB of data, (about 970,000 files) with miggui (initial sync) took:
About 7 hours

Only about 2 GB of data (approx. 4,000 files) changed by the next day, so the final sync finished in about:
2 hours

Granted, our hardware may be different.


We do break up the miggui into multiple projects (ie, we don't create one project and drag the entire volume over).

Mainly because:
a) If there's a problem and it bombs out, we don't have to start ALL over again for the entire data set
b) We exclude some directories that we know have static content (ie, our rsync setup)

I'm not sure if migfiles is much quicker than miggui.
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Oh, and you may wish to re-ask this over in the migration sub-forum in case there's additional light that can be shed on this. Sometimes Ramesh wanders by that forum (he does the miggui).
--Kevin
0 Likes
Admiral
Admiral

I ran a couple of tests over last day or so.. using migfiles instead of the gui, just so I could watch it as it goes. It takes a long time to "Scanning source path" and "scanning destination path" for any files that have been updated since last sync. Not much you can do about that I guess.. 😕 the actual file copy of the changes isnt that bad.. maybe I'll just kick everyone out and try to do a full file copy while its down 🙂

#novell on efnet
0 Likes
Absent Member.
Absent Member.

oops, not GB, I meant TB, sorry
0 Likes
Absent Member.
Absent Member.

Try this increases Netware read performance

nss /CacheBalance=85 /fileflushtimer=75 /bufferflushtimer=75 /closedfilecachesize=50000
nss /ReadAheadBlks=data:128
nss /AllocAheadBlks=63
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.