Home » Uncategorized » Java URLConnection and HttpURLConnection Examples

Java URLConnection and HttpURLConnection Examples

https://www.codejava.net/java-se/networking/java-urlconnection-and-httpurlconnection-examples

In this article, you will learn how to use the URLConnectionand HttpURLConnection classes for developing Java network applications through various examples.

For understanding the API for URLConnection and HttpURLConnection, please refer to this tutorial.

Here is the list of examples in this article:

Example #1: Download a web page’s HTML source code

The following code snippet downloads HTML source code of a webpage and save it to a file:

1234567891011121314151617181920String url = "https://google.com";String filePath = "Google.html"; URL urlObj = new URL(url);URLConnection urlCon = urlObj.openConnection(); InputStream inputStream = urlCon.getInputStream();BufferedInputStream reader = new BufferedInputStream(inputStream); BufferedOutputStream writer = new BufferedOutputStream(new FileOutputStream(filePath)); byte[] buffer = new byte[4096];int bytesRead = -1; while ((bytesRead = reader.read(buffer)) != -1) {writer.write(buffer, 0, bytesRead);} writer.close();reader.close();

As you can see, this code opens a connection from the specified URL, gets an input stream and an output stream. Then it reads data from the input stream and writes the data to a specified file.

Adding exception handling code and parameterize the URL and file path, we have a complete program as follows:

1234567891011121314151617181920212223242526272829303132333435363738394041424344import java.net.*;import java.io.*;import java.util.*; public class DownloadWebPage { public static void main(String[] args) {if (args.length < 2) {System.out.println("Syntax: <url> <file>");return;} String url = args[0];String filePath = args[1]; try { URL urlObj = new URL(url);URLConnection urlCon = urlObj.openConnection(); InputStream inputStream = urlCon.getInputStream();BufferedInputStream reader = new BufferedInputStream(inputStream); BufferedOutputStream writer = new BufferedOutputStream(new FileOutputStream(filePath)); byte[] buffer = new byte[4096];int bytesRead = -1; while ((bytesRead = reader.read(buffer)) != -1) {writer.write(buffer, 0, bytesRead);} writer.close();reader.close(); System.out.println("Web page saved"); } catch (MalformedURLException e) {System.out.println("The specified URL is malformed: " + e.getMessage());} catch (IOException e) {System.out.println("An I/O error occurs: " + e.getMessage());}}}

You can run this program from command line using this syntax:

1java DownloadWebPage <url> <file>

For example, download the Google’s homepage:

1java DownloadWebPage https://google.com Google.html

The page’s HTML code is saved to the file Google.html.

Example #2: Check HTTP Response Code

The above program will fail silently if the server returns a response code other than 200 – the HTTP response code indicates the server returns the document without any problem. For example, if you run the above program using this command:

1java DownloadWebPage http://facebook.com Facebook.html

The program terminates normally but the file Facebook.html is empty (0 byte). So let use the following code to check the server’s response code before reading and writing data:

123456789URL urlObj = new URL(url);HttpURLConnection httpCon = (HttpURLConnection) urlObj.openConnection(); int responseCode = httpCon.getResponseCode(); if (responseCode != HttpURLConnection.HTTP_OK) {System.out.println("Server returned response code " + responseCode + ". Download failed.");System.exit(0);}

This code will terminate the program if the response code is not 200 (HTTP_OK). Update, recompile and run the program again:

1java DownloadWebPage http://facebook.com Facebook.html

And you should see the following output:

1Server returned response code 301. Download failed.

The HTTP status code 301 indicates that the requested document is moved permanently. That means we should use https:// instead of http://. Run the program again with this command:

1java DownloadWebPage https://facebook.com Facebook.html

And you see the file Facebook.html is saved with HTML code.

Example #3: Set Client’s HTTP Request Header Fields

Use the setRequestProperty(String key, String value) method of the URLConnection class to set header fields for the request. The client’s header fields provide additional information about the client and how the client expects response from the server.

Here’s an example:

1234567URL urlObj = new URL(url);URLConnection urlCon = urlObj.openConnection(); urlCon.setRequestProperty("User-Agent", "Java Client; Mac OS");urlCon.setRequestProperty("Accept", "text/html");urlCon.setRequestProperty("Accept-Language", "en-US");urlCon.setRequestProperty("Connection", "close");

As you can see, this code snippet specifies 4 header fields for the request:

– User-Agent: information about the client such as browser type, operating system, architecture, etc.

– Accept: the content type understood by the client

– Accept-Language: the language understood by the client

– Connection: type of the connection. In this case, the connection is closed after a request-response roundtrip finished.

Example #4: Read all Server’s Header Fields

We can use the getHeaderFields() method of the URLConnection class to read all header fields sent from the server. Here’s an example:

12345678910111213141516String url = "https://google.com";URL urlObj = new URL(url);URLConnection urlCon = urlObj.openConnection(); Map<String, List<String>> map = urlCon.getHeaderFields();  for (String key : map.keySet()) {System.out.println(key + ":"); List<String> values = map.get(key); for (String aValue : values) {System.out.println("\t" + aValue);}}

Run this code and you should see the output looks like this (from google.com):

123456789101112131415Transfer-Encoding:chunkednull:HTTP/1.1 200 OKServer:gwsDate:Sat, 09 Dec 2017 15:38:16 GMTAccept-Ranges:noneCache-Control:private, max-age=0 Content-Type:text/html; charset=ISO-8859-1

This information varies, depending on each server.

Example #5: Read Common Header Fields

Using descriptive get header fields methods, the following code snippet reads and prints the values of common header fields such as response code, response message, content type, content encoding, content length, and so on:

1234567891011121314151617181920212223String url = "https://facebook.com"; URL urlObj = new URL(url);HttpURLConnection httpCon = (HttpURLConnection) urlObj.openConnection(); int responseCode = httpCon.getResponseCode();String responseMessage = httpCon.getResponseMessage();String contentType = httpCon.getContentType();String contentEncoding = httpCon.getContentEncoding();int contentLength = httpCon.getContentLength(); long date = httpCon.getDate();long expiration = httpCon.getExpiration();long lastModified = httpCon.getLastModified(); System.out.println("Response Code: " + responseCode);System.out.println("Response Message: " + responseMessage);System.out.println("Content Type: " + contentType);System.out.println("Content Encoding: " + contentEncoding);System.out.println("Content Length: " + contentLength);System.out.println("Date: " + new Date(date));System.out.println("Expiration: " + new Date(expiration));System.out.println("Last Modified: " + new Date(lastModified));

The output would be:

12345678Response Code: 200Response Message: OKContent Type: text/html; charset=UTF-8Content Encoding: nullContent Length: -1Date: Sun Dec 10 10:52:31 ICT 2017Expiration: Sat Jan 01 07:00:00 ICT 2000Last Modified: Thu Jan 01 07:00:00 ICT 1970

Example #6: Set HTTP Request Method 

By default the HTTP request method is GET. You can call the setRequestMethod(String method) to set the request method which is one of GET, POST, HEAD, PUT, DELETE, TRACE, and OPTIONS.

For example, the following code sets the request method to HEAD:

123URL urlObj = new URL(url);HttpURLConnection httpCon = (HttpURLConnection) urlObj.openConnection();httpCon.setRequestMethod("HEAD");

When processing a HEAD request, the server returns a response without the body content. Only the header fields are returned. Hence the method name “HEAD”.

Also note that when you set the request method to POST, you must also enable output for the connection, as POST method is about send data to the server. For example:

12httpCon.setDoOutput(true);httpCon.setRequestMethod("POST");

See how to send a POST request in the next example.

Example #7: Send HTTP POST Request

To send an HTTP POST request along with parameters, you need to constructor the parameters in the following form:

1param1=value1&param2=value2&param3=value3

The following code snippet demonstrates how to send a login request to Twitter via HTTP POST:

12345678910111213141516171819String url = "https://twitter.com/sessions";String email = "yourname@gmail.com";String password = "yourpass";  URL urlObj = new URL(url);HttpURLConnection httpCon = (HttpURLConnection) urlObj.openConnection(); httpCon.setDoOutput(true);httpCon.setRequestMethod("POST"); String parameters = "username=" + email;parameters += "password=" + password;  OutputStreamWriter writer = new OutputStreamWriter(httpCon.getOutputStream());writer.write(parameters);writer.flush();

Of course this won’t work because Twitter requires more complex parameters for the login process (i.e. authentication key). Anyway, let try to combine this code with the example of reading all header fields and save to file to see what will happen.

And for more practical URLConnection/HttpURLConnection examples, refer to the following articles:

Uncategorized