Getting started with Amazon S3 sdk with java

Put the sdk in your pom.xml of your maven project:

1. <dependency>
2.     <groupId>com.amazonaws</groupId>
3.     <artifactId>aws-java-sdk</artifactId>
4.     <version>1.9.13</version>
5. </dependency>

Check out the simplest sample code https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonS3/S3Sample.java

Before running, make sure you setup API credentials properly.Follow the instruction here to do this :

http://mohiplanet.blogspot.com/2014/12/setting-up-aws-s3-api-java-client.html

Creating a directory:-

http://voidweb.com/2013/03/how-to-create-a-folder-programmatically-in-s3-using-amazon-aws-java-sdk/ has good examples for creating a directory

Uploading a file:-
Here is an for uploading a file which will be publicly accessible:
http://mohiplanet.blogspot.com/2014/11/aws-s3-multipart-file-upload-with.html
About content type:-
Note:- You have to select appropriate content types for each of your files. S3 SDK is not gonna do it automatically  for you.
Here is a sample that you can use during uploading with correct content type:-

1. public ObjectMetadata generateCorrectContentTypeObjMetaFromKeyName(String key) throws InvalidContentTypeException {
2. ObjectMetadata metadata = new ObjectMetadata();
3. String contentType = getFileContentTypeFromFileName(key);
4. if (contentType.length() == 0) {
5. throw new InvalidContentTypeException();
6. }
7. LOG.log(Level.INFO, "Setting content type : " + contentType);
8. metadata.setContentType(contentType);
9. return metadata;
10. }

Set content type according to your key name with:

1. initRequest.setObjectMetadata(generateCorrectContentTypeObjMetaFromKeyName(keyName));

 Details and definition of getFileContentTypeFromFileName(key) are here:-
 http://mohiplanet.blogspot.com/2014/12/amazon-web-services-s3-api-java-client.html

More examples:-
https://github.com/aws/aws-sdk-java/tree/master/src/samples/AmazonS3

Command line tools for working fast:-
http://s3tools.org/s3cmd

Amazon Web Services S3 API Java Client : update file object key with correct content type

By default the uploaded file content type is :
application/x-www-form-urlencoded; charset=utf-8

To change this you can do something like this :

1. static AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
3. public void validateKeyContentType(String key) {
4.         try {
5.             ObjectMetadata metadata = generateCorrectContentTypeObjMetaFromKeyName(key);
6.             
7.             final CopyObjectRequest request = new CopyObjectRequest(bucket, key, bucket, key).withSourceBucketName(bucket)
8.                 .withSourceKey(key)
9.                 .withNewObjectMetadata(metadata).withCannedAccessControlList(CannedAccessControlList.PublicRead);
10.             s3Client.copyObject(request);
11.         } catch (Exception exception) {
12.             LOG.info(exception.getMessage());
13.         }
14.     }
15. public ObjectMetadata generateCorrectContentTypeObjMetaFromKeyName(String key){
16.         ObjectMetadata metadata = new ObjectMetadata();
17.         String contentType = getFileContentTypeFromFileName(key);
18.         
19.         LOG.log(Level.INFO, "Setting content type : " + contentType);
20.         metadata.setContentType(contentType);
21.         return metadata;
22.     }
23. /*
24. These method takes some argument like "bucket1uploads/file100.mp4" and returns it's content type "video/mp4" by parsing the text (javax.activation)
25. */
26.     public static String getFileContentTypeFromFileName(String filePath) {
27. //javax.activation is too old and misses these content types
28.         MimetypesFileTypeMap ftmp = new MimetypesFileTypeMap();
29.         ftmp.addMimeTypes("audio/mp3 mp3 MP3");
30.         ftmp.addMimeTypes("video/mp4 mp4 MP4");
31.         ftmp.addMimeTypes("video/mkv mkv MKV");
32.         ftmp.addMimeTypes("video/webm webm WEBM");
33.         ftmp.addMimeTypes("video/flv flv FLV");
34.         ftmp.addMimeTypes("video/x-flv x-flv X-FLV");
35.         return ftmp.getContentType(filePath);
36.     }

Setting up AWS S3 API Java Client credentials

You may get this during Amazon s3 java API client first run:
Exception in thread "main" java.lang.IllegalArgumentException: AWS credential profiles file not found in the given path: /root/.aws/credentials
        at com.amazonaws.auth.profile.internal.ProfilesConfigFileLoader.loadProfiles(ProfilesConfigFileLoader.java:45)
        at com.amazonaws.auth.profile.ProfilesConfigFile.loadProfiles(ProfilesConfigFile.java:173)
        at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:109)
        at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:89)
        at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:117)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3691)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3647)
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:626)
        at com.vantage.photowall.toolset.s3client.PhotowallAWSS3Client.listKeys(PhotowallAWSS3Client.java:179)
        at com.vantage.photowall.toolset.s3client.AWSS3BucketAPIClientCLI.main(AWSS3BucketAPIClientCLI.java:55)

To solve this, simply :

Download crendentials file from https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonS3/credentials

1. wget --no-check-certificate https://github.com/aws/aws-sdk-java/raw/master/src/samples/AmazonS3/credentials

put it in ~/.aws/ directory:

1. mkdir ~/.aws/
2. mv credentials ~/.aws/credentials

open credentials file and fill access key id and secret access key :

like this:

1. # Move this credentials file to (~/.aws/credentials)
2. # after you fill in your access and secret keys in the default profile
4. # WARNING: To avoid accidental leakage of your credentials,
5. # DO NOT keep this file in your source directory.
6. [default]
7. aws_access_key_id=YOUR_ACCESS_KEY_ID
8. aws_secret_access_key=YOUR_SECRET_ACCESS_KEY

Save and close the file.

Run API client program again :)

Related:-
Getting started with Amazon S3 sdk with java
AWS S3 multipart file upload with public access

Amazon Web Services S3 API Java Client : update file object key with correct content type

AWS S3 multipart file upload with public access

1. import java.io.File;
2. import java.io.IOException;
3. import java.util.ArrayList;
4. import java.util.List;
5. 
   
6. import com.amazonaws.auth.profile.ProfileCredentialsProvider;
7. import com.amazonaws.services.s3.AmazonS3;
8. import com.amazonaws.services.s3.AmazonS3Client;
9. import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
10. import com.amazonaws.services.s3.model.CannedAccessControlList;
11. import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
12. import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
13. import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
14. import com.amazonaws.services.s3.model.PartETag;
15. import com.amazonaws.services.s3.model.UploadPartRequest;
17. public class MultipartPublicFileUpload {
19.     static AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
21.     static public boolean multiPartFileUploadWithPublicAccess(String existingBucketName, String keyName, String filePath) {
22.         // Create a list of UploadPartResponse objects. You get one of these
23.         // for each part upload.
24.         List<PartETag> partETags = new ArrayList<PartETag>();
26.         // Step 1: Initialize.
27.         InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(existingBucketName, keyName);
28.         initRequest.setCannedACL(CannedAccessControlList.PublicRead);//This is where we are enabling this file for public access
29.         InitiateMultipartUploadResult initResponse
30.             = s3Client.initiateMultipartUpload(initRequest);
32.         File file = new File(filePath);
33.         long contentLength = file.length();
34.         long partSize = 5242880; // Set part size to 5 MB.
36.         try {
37.             // Step 2: Upload parts.
38.             long filePosition = 0;
39.             for (int i = 1; filePosition < contentLength; i++) {
40.                 // Last part can be less than 5 MB. Adjust part size.
41.                 partSize = Math.min(partSize, (contentLength - filePosition));
43.                 // Create request to upload a part.
44.                 UploadPartRequest uploadRequest = new UploadPartRequest()
45.                     .withBucketName(existingBucketName).withKey(keyName)
46.                     .withUploadId(initResponse.getUploadId()).withPartNumber(i)
47.                     .withFileOffset(filePosition)
48.                     .withFile(file)
49.                     .withPartSize(partSize);
51.                 // Upload part and add response to our list.
52.                 partETags.add(
53.                     s3Client.uploadPart(uploadRequest).getPartETag());
55.                 filePosition += partSize;
56.             }
58.             // Step 3: Complete.
59.             CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(
60.                 existingBucketName,
61.                 keyName,
62.                 initResponse.getUploadId(),
63.                 partETags);
65.             s3Client.completeMultipartUpload(compRequest);
66.             return true;
67.         } catch (Exception e) {
68.             s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
69.                 existingBucketName, keyName, initResponse.getUploadId()));
70.             return false;
71.         }
72.     }
74.     public static void main(String[] args) throws IOException {
75.         String existingBucketName = "myphotowall";
76.         String keyName = "photo/myuploadedtestfile3.txt";
77.         String filePath = "/tmp/mytestfile.txt";
78.         multiPartFileUploadWithPublicAccess(existingBucketName, keyName, filePath);
80.     }
81. }

Before you run make sure you have setup the API clients crendentials properly.See http://mohiplanet.blogspot.com/2014/12/setting-up-aws-s3-api-java-client.html

Note * : This will upload file with content type application/x-www-form-urlencoded; charset=utf-8

Please refer to http://mohiplanet.blogspot.com/2014/12/amazon-web-services-s3-api-java-client.html for updating keys with correct content type or setting correct content type with object metadata while uploading.

References :
http://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html