Webclient Questions

I am running AcuGT V9.2.3 on a Windows 7 PC and need a AcuGT program to download multiple data files from a US Treasury Dept website for processing.  I wrote a sample program using WebClient .NET features to pull the files from the website, most of the time this program works just fine but other times it is failing with a Webclient error that reads:  The remote server returned an error: (407) Proxy Authentication Required" and then the program simply crashes.  What is odd is the random nature of this failure, I can run the program 10 times in a row and it will run just fine, then I can run it 5 times and it will fail with this error.  Five minutes later I can run the program again and it runs just fine.  The results appear to be completely random at this point.

Couple questions:

1) Is there a better way to have my AcuGT program download files from a website or is Webclient the best approach?

2) If Webclient is the best approach is there a way to capture errors like this so that my program doesn't simply crash when these errors occur?

Any guidance would be appreciate.

  • Can you tell me what WebClient .NET is?

    Are yo working with Silk ... Silk > Silk Performer > Silk Performer 16.5 > Silk Performer Workbench > Load Testing Specific Application Types > .NET Support > Silk Performer .NET Explorer > Using the .NET Message Sample > Working With the .NET Message Sample > Exploring the .NET Message Sample

    or something else?

  • (407) Proxy Authentication Required error is the access to the URL resource requires the prior use of a proxy server ... see www.checkupdown.com/.../E407.html ... somehow your login /  authentication is getting lost or perhaps was cached to begin with .. I would think that the US Treasury Dept is very concerned about authentication.

  • Throwing this out there based on the feedback - could load balancing be happening?   So you're going out one server ten times in a row and suddenly going out a different server where the previous access is not established??  Dunno.  We use IE to obtain the SDN files from Treasury manually.

  • The files we are downloading from the US Treasury website are generally available public files, they are available to anyone to download (they contain lists of suspected 'bad guys' from around the world).  The entire program is copied below - it is quite simple except that it is using WebClient functions to do the actual file downloads.  Our biggest struggle is that when the WebClient function has a problem it crashes the entire AcuGT runtime instead of letting us capture the error and handle it gracefully.



          DATE-WRITTEN. 10/09/2015.




         * .NET Web Client research program





               COPY "WebClient".






          DATA DIVISION.

          FILE SECTION.




          01  PROGRAM-NAME  pic x(20) value "WEBCLIENT.CBLV.23.00".


         * Handles:

         *    Window handles:

          77  WStatus-Handle             handle of window.

          77  WCall-Handle               handle.

          77  WCancel-Handle             handle of thread.

          77  WWebClient-Handle

                 usage is handle of "@System.System.Net.WebClient".

          77  WS-ADDR-SDN         pic x(89) value                          


          77  WS-FileName-SDN     pic x(18) value                        


          77  WS-ADDR-ALT         pic x(89) value                          


          77  WS-FileName-ALT     pic x(18) value                        


          77  WS-ADDR-SDNCMTS     pic x(89) value                          


          77  WS-FileName-SDNCMTS pic x(18) value                        


          77  WS-ADDR-CON         pic x(89) value                          


         -    "im.csv".          

          77  WS-FileName-CON     pic x(18) value                        


          77  WS-ADDR-CONALT         pic x(89) value                          


         -    "t.csv".          

          77  WS-FileName-CONALT     pic x(18) value                        


          77  WS-ADDR-CONCMTS     pic x(89) value                          


         -    "mments.csv".

          77  WS-FileName-CONCMTS pic x(18) value                        




         * CREATE  - instantiate a NON-GUI CLASS *

              CREATE "@System"

              NAMESPACE is "System.Net"

              CLASS-NAME is "WebClient"

              HANDLE is WWebClient-Handle.

         *    * Set the Default Credentials *

              Modify WWebClient-Handle

                 UseDefaultCredentials is 1.

         *    * Invoke the Download file method *

              PERFORM 10 TIMES

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDN, WS-FILENAME-SDN)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-ALT, WS-FILENAME-ALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-SDNCMTS, WS-FILENAME-SDNCMTS)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CON, WS-FILENAME-CON)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONALT, WS-FILENAME-CONALT)            

                Modify WWebClient-Handle

                 "@DownloadFile" (WS-Addr-CONCMTS, WS-FILENAME-CONCMTS)


              Exit Program.

  • Since the HTTPS protocol is being used, I would think this could be done quite simply using RMnet (which I think was available in extend 9).

  • Tom beat me to the punch in suggesting RMNet. However, we'd need to be sure the files are pure text files, as RMNet has a limitation that it doesn't handle binary payloads.

  • The files referenced in the example code are text - CSV.

  • you can use BITSAdmin.exe  which is part of Windows 7.

    But for later Windows Versions the Windows PowerShell cmdlets should be better.


  • The following program, in RM/COBOL dialect (because that's what I have installed), uses RMNet to do an HTTP GET on one of the files, and store the result in "results.txt".  The data that are stored are identical to that fetched by the browser(!).  The example uses XML Extensions for the convenience of storing the resulting document (returned in memory) in a single instruction.  No other XML Extensions methods are used.

    RMNet will return the HTTP status and even the result HTTP headers (optional) so that you can have a lot of control over the behavior.

           identification division.
           program-id.  BadGuy.
           data division.
           working-storage section.
           01  response-payload            usage pointer.
           01  response-error              usage pointer.
           01  response-status             pic 9(3) value zero.

           78  Get-Address value
               Copy "lixmlall.cpy".
           linkage section.
           01 http-response pic x(500).
           01 http-error pic x.

           procedure division.
               XML INITIALIZE.
               if not xml-ok go to z.

               call "NetInit" giving response-status.
               call "NetSetSSLCA" using "ca-bundle.crt".
               call "HttpGet"

               if response-status not = 0
                   call "NetGetError" using response-error
                   set address of http-response to response-error
                   display http-response(1:)
                   set address of http-response to null
                   call "NetFree" using response-error
                   if response-payload not equal null
                       XML PUT TEXT
                       if not xml-ok go z end-if

               call "NetCleanup".

               stop run.

               Perform Display-Error-Status
               XML TERMINATE.
               Stop Run.

               If Not XML-OK
                   Display "!!!!!!!!XML Error Occurred!!!!!!!!"
                   Display "XML Status: " XML-Status
                   Perform With Test After Until XML-NoMore
                       XML GET STATUS-TEXT
                       Display XML-StatusText

  • You might consider using the DECLARATIVES to catch the WebClient .Net error ... USE AFTER STANDARD {EXCEPTION} PROCEDURE ON OBJECT

    {ERROR    }