jimdh Absent Member.
Absent Member.
3536 views

Webclient Questions

I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

Couple questions:

1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

Any guidance would be appreciate.

0 Likes
11 Replies
Micro Focus Expert
Micro Focus Expert

RE: Webclient Questions

Can you tell me what WebClient .NET is?

Are yo working with Silk ... Silk > Silk Performer > Silk Performer 16.5 > Silk Performer Workbench > Load Testing Specific Application Types > .NET Support > Silk Performer .NET Explorer > Using the .NET Message Sample > Working With the .NET Message Sample > Exploring the .NET Message Sample

or something else?

0 Likes
Micro Focus Expert
Micro Focus Expert

RE: Webclient Questions

(407) Proxy Authentication Required error is the access to the URL resource requires the prior use of a proxy server ... see www.checkupdown.com/.../E407.html ... somehow your login /  authentication is getting lost or perhaps was cached to begin with .. I would think that the US Treasury Dept is very concerned about authentication.

0 Likes
rstoll Absent Member.
Absent Member.

RE: Webclient Questions

Throwing this out there based on the feedback - could load balancing be happening?   So you're going out one server ten times in a row and suddenly going out a different server where the previous access is not established??  Dunno.  We use IE to obtain the SDN files from Treasury manually.

0 Likes
jimdh Absent Member.
Absent Member.

RE: Webclient Questions

The files we are downloading from the US Treasury website are generally available public files, they are available to anyone to download (they contain lists of suspected 'bad guys' from around the world).  The entire program is copied below - it is quite simple except that it is using WebClient functions to do the actual file downloads.  Our biggest struggle is that when the WebClient function has a problem it crashes the entire AcuGT runtime instead of letting us capture the error and handle it gracefully.

      IDENTIFICATION DIVISION.

      PROGRAM-ID. WEBCLIENT-CBL.

      DATE-WRITTEN. 10/09/2015.

      AUTHOR.

      REMARKS.

     *

     * .NET Web Client research program

     *

      ENVIRONMENT DIVISION.

      CONFIGURATION SECTION.

      SPECIAL-NAMES.

           COPY "WebClient".

           .

     *INPUT-OUTPUT SECTION.

      FILE-CONTROL.

     *

     *

      DATA DIVISION.

      FILE SECTION.

     *

     *

       WORKING-STORAGE SECTION.

      01  PROGRAM-NAME  pic x(20) value "WEBCLIENT.CBLV.23.00".

     *

     * Handles:

     *    Window handles:

      77  WStatus-Handle             handle of window.

      77  WCall-Handle               handle.

      77  WCancel-Handle             handle of thread.

      77  WWebClient-Handle

             usage is handle of "@System.System.Net.WebClient".

      77  WS-ADDR-SDN         pic x(89) value                          

          "www.treasury.gov/.../sdn.csv".          

      77  WS-FileName-SDN     pic x(18) value                        

          "C:\TMP\SDN".                                        

      77  WS-ADDR-ALT         pic x(89) value                          

          "www.treasury.gov/.../alt.csv".          

      77  WS-FileName-ALT     pic x(18) value                        

          "C:\TMP\ALT".                                        

      77  WS-ADDR-SDNCMTS     pic x(89) value                          

          "www.treasury.gov/.../sdn_comments.csv".

      77  WS-FileName-SDNCMTS pic x(18) value                        

          "C:\TMP\SDNCMTS".                                        

      77  WS-ADDR-CON         pic x(89) value                          

          "www.treasury.gov/.../cons_pr

     -    "im.csv".          

      77  WS-FileName-CON     pic x(18) value                        

          "C:\TMP\CONPRM".                                        

      77  WS-ADDR-CONALT         pic x(89) value                          

          "www.treasury.gov/.../cons_al

     -    "t.csv".          

      77  WS-FileName-CONALT     pic x(18) value                        

          "C:\TMP\CONAKA".                                        

      77  WS-ADDR-CONCMTS     pic x(89) value                          

          "www.treasury.gov/.../cons_co

     -    "mments.csv".

      77  WS-FileName-CONCMTS pic x(18) value                        

          "C:\TMP\CONCMTS".                                        

      PROCEDURE DIVISION.

     *

     * CREATE  - instantiate a NON-GUI CLASS *

          CREATE "@System"

          NAMESPACE is "System.Net"

          CLASS-NAME is "WebClient"

          HANDLE is WWebClient-Handle.

     *    * Set the Default Credentials *

          Modify WWebClient-Handle

             UseDefaultCredentials is 1.

     *    * Invoke the Download file method *

          PERFORM 10 TIMES

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-SDN, WS-FILENAME-SDN)            

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-ALT, WS-FILENAME-ALT)            

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-SDNCMTS, WS-FILENAME-SDNCMTS)            

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-CON, WS-FILENAME-CON)            

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-CONALT, WS-FILENAME-CONALT)            

            Modify WWebClient-Handle

             "@DownloadFile" (WS-Addr-CONCMTS, WS-FILENAME-CONCMTS)

          END-PERFORM.

          Exit Program.

0 Likes
Knowledge Partner
Knowledge Partner

RE: Webclient Questions

Since the HTTPS protocol is being used, I would think this could be done quite simply using RMnet (which I think was available in extend 9).


Tom Morrison
Consultant

0 Likes
Chuck Edgin Absent Member.
Absent Member.

RE: Webclient Questions

Tom beat me to the punch in suggesting RMNet. However, we'd need to be sure the files are pure text files, as RMNet has a limitation that it doesn't handle binary payloads.

0 Likes
Knowledge Partner
Knowledge Partner

RE: Webclient Questions

The files referenced in the example code are text - CSV.


Tom Morrison
Consultant

0 Likes
neidingd Honored Contributor.
Honored Contributor.

RE: Webclient Questions

you can use BITSAdmin.exe  which is part of Windows 7.

But for later Windows Versions the Windows PowerShell cmdlets should be better.

technet.microsoft.com/.../ff382721.aspx

0 Likes
Knowledge Partner
Knowledge Partner

RE: Webclient Questions

The following program, in RM/COBOL dialect (because that's what I have installed), uses RMNet to do an HTTP GET on one of the files, and store the result in "results.txt".  The data that are stored are identical to that fetched by the browser(!).  The example uses XML Extensions for the convenience of storing the resulting document (returned in memory) in a single instruction.  No other XML Extensions methods are used.

RMNet will return the HTTP status and even the result HTTP headers (optional) so that you can have a lot of control over the behavior.

       identification division.
       program-id.  BadGuy.
       data division.
       working-storage section.
       01  response-payload            usage pointer.
       01  response-error              usage pointer.
       01  response-status             pic 9(3) value zero.

       78  Get-Address value
             "https://www.treasury.gov/ofac/downloads/alt.csv".
           Copy "lixmlall.cpy".
       linkage section.
       01 http-response pic x(500).
       01 http-error pic x.

       procedure division.
       a.
           XML INITIALIZE.
           if not xml-ok go to z.

           call "NetInit" giving response-status.
           call "NetSetSSLCA" using "ca-bundle.crt".
           call "HttpGet"
             using
               Get-Address
               response-payload
             giving
               response-status.

           if response-status not = 0
               call "NetGetError" using response-error
               set address of http-response to response-error
               display http-response(1:)
               set address of http-response to null
               call "NetFree" using response-error
           else
               if response-payload not equal null
                   XML PUT TEXT
                       response-payload
                       "response.txt"
                   if not xml-ok go z end-if
               end-if
           end-if.

           call "NetCleanup".

           stop run.

       z.
           Perform Display-Error-Status
           XML TERMINATE.
           Stop Run.

       Display-Error-Status.
           If Not XML-OK
               Display "!!!!!!!!XML Error Occurred!!!!!!!!"
               Display "XML Status: " XML-Status
               Perform With Test After Until XML-NoMore
                   XML GET STATUS-TEXT
                   Display XML-StatusText
               End-Perform
           End-If.


Tom Morrison
Consultant

0 Likes
Micro Focus Expert
Micro Focus Expert

RE: Webclient Questions

You might consider using the DECLARATIVES to catch the WebClient .Net error ... USE AFTER STANDARD {EXCEPTION} PROCEDURE ON OBJECT

{ERROR    }

0 Likes
jimdh Absent Member.
Absent Member.

RE: Webclient Questions

Thanks so much for the direction on capturing the errors through declaratives, now we can at least capture the error cleanly without crashing the runtime.  I appreciate the help.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.