Webclient Questions

I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

Couple questions:

1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

Any guidance would be appreciate.

  • Can you tell me what WebClient .NET is?

    Are yo working with Silk ... Silk > Silk Performer > Silk Performer 16.5 > Silk Performer Workbench > Load Testing Specific Application Types > .NET Support > Silk Performer .NET Explorer > Using the .NET Message Sample > Working With the .NET Message Sample > Exploring the .NET Message Sample

    or something else?

  • (407) Proxy Authentication Required error is the access to the URL resource requires the prior use of a proxy server ... see www.checkupdown.com/.../E407.html ... somehow your login /  authentication is getting lost or perhaps was cached to begin with .. I would think that the US Treasury Dept is very concerned about authentication.

  • Throwing this out there based on the feedback - could load balancing be happening?   So you're going out one server ten times in a row and suddenly going out a different server where the previous access is not established??  Dunno.  We use IE to obtain the SDN files from Treasury manually.

  • The files we are downloading from the US Treasury website are generally available public files, they are available to anyone to download (they contain lists of suspected 'bad guys' from around the world).  The entire program is copied below - it is quite simple except that it is using WebClient functions to do the actual file downloads.  Our biggest struggle is that when the WebClient function has a problem it crashes the entire AcuGT runtime instead of letting us capture the error and handle it gracefully.

          IDENTIFICATION DIVISION.

          PROGRAM-ID. WEBCLIENT-CBL.

          DATE-WRITTEN. 10/09/2015.

          AUTHOR.

          REMARKS.

         *

         * .NET Web Client research program

         *

          ENVIRONMENT DIVISION.

          CONFIGURATION SECTION.

          SPECIAL-NAMES.

               COPY "WebClient".

               .

         *INPUT-OUTPUT SECTION.

          FILE-CONTROL.

         *

         *

          DATA DIVISION.

          FILE SECTION.

         *

         *

           WORKING-STORAGE SECTION.

          01  PROGRAM-NAME  pic x(20) value "WEBCLIENT.CBLV.23.00".

         *

         * Handles:

         *    Window handles:

          77  WStatus-Handle             handle of window.

          77  WCall-Handle               handle.

          77  WCancel-Handle             handle of thread.

          77  WWebClient-Handle

                 usage is handle of "@System.System.Net.WebClient".

          77  WS-ADDR-SDN         pic x(89) value                          

              "www.treasury.gov/.../sdn.csv".          

          77  WS-FileName-SDN     pic x(18) value                        

              "C:\TMP\SDN".                                        

          77  WS-ADDR-ALT         pic x(89) value                          

              "www.treasury.gov/.../alt.csv".          

          77  WS-FileName-ALT     pic x(18) value                        

              "C:\TMP\ALT".                                        

          77  WS-ADDR-SDNCMTS     pic x(89) value                          

              "www.treasury.gov/.../sdn_comments.csv".

          77  WS-FileName-SDNCMTS pic x(18) value                        

              "C:\TMP\SDNCMTS".                                        

          77  WS-ADDR-CON         pic x(89) value                          

              "www.treasury.gov/.../cons_pr

         -    "im.csv".          

          77  WS-FileName-CON     pic x(18) value                        

              "C:\TMP\CONPRM".                                        

          77  WS-ADDR-CONALT         pic x(89) value                          

              "www.treasury.gov/.../cons_al

         -    "t.csv".          

          77  WS-FileName-CONALT     pic x(18) value                        

              "C:\TMP\CONAKA".                                        

          77  WS-ADDR-CONCMTS     pic x(89) value                          

              "www.treasury.gov/.../cons_co

         -    "mments.csv".

          77  WS-FileName-CONCMTS pic x(18) value                        

              "C:\TMP\CONCMTS".                                        

          PROCEDURE DIVISION.

         *

         * CREATE  - instantiate a NON-GUI CLASS *

              CREATE "@System"

              NAMESPACE is "System.Net"

              CLASS-NAME is "WebClient"

              HANDLE is WWebClient-Handle.

         *    * Set the Default Credentials *

              Modify WWebClient-Handle

                 UseDefaultCredentials is 1.

         *    * Invoke the Download file method *

              PERFORM 10 TIMES

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDN, WS-FILENAME-SDN)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-ALT, WS-FILENAME-ALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDNCMTS, WS-FILENAME-SDNCMTS)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CON, WS-FILENAME-CON)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONALT, WS-FILENAME-CONALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONCMTS, WS-FILENAME-CONCMTS)

              END-PERFORM.

              Exit Program.

  • Since the HTTPS protocol is being used, I would think this could be done quite simply using RMnet (which I think was available in extend 9).

  • Tom beat me to the punch in suggesting RMNet. However, we'd need to be sure the files are pure text files, as RMNet has a limitation that it doesn't handle binary payloads.

  • The files referenced in the example code are text - CSV.

  • you can use BITSAdmin.exe  which is part of Windows 7.

    But for later Windows Versions the Windows PowerShell cmdlets should be better.

    technet.microsoft.com/.../ff382721.aspx

  • The following program, in RM/COBOL dialect (because that's what I have installed), uses RMNet to do an HTTP GET on one of the files, and store the result in "results.txt".  The data that are stored are identical to that fetched by the browser(!).  The example uses XML Extensions for the convenience of storing the resulting document (returned in memory) in a single instruction.  No other XML Extensions methods are used.

    RMNet will return the HTTP status and even the result HTTP headers (optional) so that you can have a lot of control over the behavior.

           identification division.
           program-id.  BadGuy.
           data division.
           working-storage section.
           01  response-payload            usage pointer.
           01  response-error              usage pointer.
           01  response-status             pic 9(3) value zero.

           78  Get-Address value
                 "www.treasury.gov/.../alt.csv".
               Copy "lixmlall.cpy".
           linkage section.
           01 http-response pic x(500).
           01 http-error pic x.

           procedure division.
           a.
               XML INITIALIZE.
               if not xml-ok go to z.

               call "NetInit" giving response-status.
               call "NetSetSSLCA" using "ca-bundle.crt".
               call "HttpGet"
                 using
                   Get-Address
                   response-payload
                 giving
                   response-status.

               if response-status not = 0
                   call "NetGetError" using response-error
                   set address of http-response to response-error
                   display http-response(1:)
                   set address of http-response to null
                   call "NetFree" using response-error
               else
                   if response-payload not equal null
                       XML PUT TEXT
                           response-payload
                           "response.txt"
                       if not xml-ok go z end-if
                   end-if
               end-if.

               call "NetCleanup".

               stop run.

           z.
               Perform Display-Error-Status
               XML TERMINATE.
               Stop Run.

           Display-Error-Status.
               If Not XML-OK
                   Display "!!!!!!!!XML Error Occurred!!!!!!!!"
                   Display "XML Status: " XML-Status
                   Perform With Test After Until XML-NoMore
                       XML GET STATUS-TEXT
                       Display XML-StatusText
                   End-Perform
               End-If.

  • You might consider using the DECLARATIVES to catch the WebClient .Net error ... USE AFTER STANDARD {EXCEPTION} PROCEDURE ON OBJECT

    {ERROR    }